[dancer-users] High CPU usage on DB

[Zahir Lalani]

Thx All

No transactions - its mainly selects but I will add in Racke's suggestion of finish and see what that does.

As for Starman - so far not had any issues - but we have set it to limit the max requests so that it gets cleaned every so often. I do want to try out uwsgi - have read very good things - thx for the heads up

Z

> -----Original Message-----
> From: dancer-users [mailto:dancer-users-bounces at dancer.pm] On Behalf
> Of Hermann Calabria
> Sent: 10 December 2016 21:21
> To: Perl Dancer users mailing list <dancer-users at dancer.pm>
> Subject: Re: [dancer-users] High CPU usage on DB
> 
> If you're using Dancer::Plugin::Database (and I suspect D2 has the same
> issue), you're using transactions, and a crash occurs mid-transaction, note
> that there is no automatic rollback() on die().  I documented this issue here,
> and bigpresh (the plugin author) was kind enough to offer some
> suggestions:
> 
> https://github.com/bigpresh/Dancer-Plugin-Database/issues/76
> 
> With enough of these, you end up with a bunch of unfinished transactions,
> and the obvious problems eventually start occurring -- memory leaks, high
> cpu usage, hung processes, etc.
> 
> Unrelated, but may be helpful: We gave up on Starman. Just could not get it
> stable in a high-traffic production environment. We switched to uwsgi (with
> nginx) and haven't looked back since. uwsgi is fast, compact, has tons of
> useful options.  It is used by Python stacks, so it is well-supported and
> updated, including commercial support if you need it.
> 
> Good luck on your project!
> 
> -----Original Message-----
> From: Zahir Lalani
> Sent: Saturday, December 10, 2016 1:52 AM
> To: Perl Dancer users mailing list
> Subject: Re: [dancer-users] High CPU usage on DB
> 
> Hi Warren
> 
> We are looking into the processes more - we are also looking at all the fine
> tuning of PG parameters to make sure none of those are having an impact
> 
> On the Dancer side - maybe I need to check that we are doing things
> correctly. We use Dancer2::Plugin::Database in the following way.
> 
> In each module that performs DB requests, we 'use' the above package
> 
> In a typical DB layer request we will do:
> 
> Package mmm;
> use Dancer2 appname => 'APP';
> Use Dancer2::Plugin::Database
> 
> .........
> $sth = database->prepare("select xxxx from yyy(?)"); $sth->execute($id);
> 
> $result = $sth->fetchrow_hashref();
> 
> return $result;
> ............
> 
> There is no explicit closing of connections.
> 
> We use starman to run our app, and it looks like the number of starman
> workers matches exactly the number of open connections to the DB - so it
> seems that the plugin holds one open connection per instance.
> 
> Does the above all sound the correct use of the plugin?
> 
> Z
> 
> > -----Original Message-----
> > From: dancer-users [mailto:dancer-users-bounces at dancer.pm] On Behalf
> > Of Warren Young
> > Sent: 08 December 2016 17:33
> > To: Perl Dancer users mailing list <dancer-users at dancer.pm>
> > Subject: Re: [dancer-users] High CPU usage on DB
> >
> > On Dec 8, 2016, at 10:27 AM, Zahir Lalani <ZahirLalani at oliver.agency>
> > wrote:
> > >
> > > We use the Dancer database module to create a connection
> >
> > Are you caching those connections or re-opening a fresh one for every
> > request?
> >
> > And if the latter, are you *closing* the connection before returning
> > the queried data to the client?  Are you sure, evidenced by monitoring
> > the OS’s open file handles counter?
> >
> > You don’t say which OS this is, but if it’s Linux:
> >
> >     ls /proc/$(pidof my-dancer-program)/fd | wc -l
> >
> > > so in this case we have 20 connections to the DB
> >
> > So you believe.  Double-check.
> >
> > > a restart of Starman cures the problem
> >
> > That sure sounds like a leaked resource of some kind.  I have you
> > chasing DB file handles.  You say it isn’t leaked memory.  If not
> > either of those, then it could be web sockets or any similar sort of
> > thing.
> >
> > _______________________________________________
> > dancer-users mailing list
> > dancer-users at dancer.pm
> > http://lists.preshweb.co.uk/mailman/listinfo/dancer-users
> _______________________________________________
> dancer-users mailing list
> dancer-users at dancer.pm
> http://lists.preshweb.co.uk/mailman/listinfo/dancer-users
> 
> _______________________________________________
> dancer-users mailing list
> dancer-users at dancer.pm
> http://lists.preshweb.co.uk/mailman/listinfo/dancer-users