[dancer-users] High CPU usage on DB

[Hermann Calabria]

If you're using Dancer::Plugin::Database (and I suspect D2 has the same 
issue), you're using transactions, and a crash occurs mid-transaction, note 
that there is no automatic rollback() on die().  I documented this issue 
here, and bigpresh (the plugin author) was kind enough to offer some 
suggestions:

https://github.com/bigpresh/Dancer-Plugin-Database/issues/76

With enough of these, you end up with a bunch of unfinished transactions, 
and the obvious problems eventually start occurring -- memory leaks, high 
cpu usage, hung processes, etc.

Unrelated, but may be helpful: We gave up on Starman. Just could not get it 
stable in a high-traffic production environment. We switched to uwsgi (with 
nginx) and haven't looked back since. uwsgi is fast, compact, has tons of 
useful options.  It is used by Python stacks, so it is well-supported and 
updated, including commercial support if you need it.

Good luck on your project!

-----Original Message----- 
From: Zahir Lalani
Sent: Saturday, December 10, 2016 1:52 AM
To: Perl Dancer users mailing list
Subject: Re: [dancer-users] High CPU usage on DB

Hi Warren

We are looking into the processes more - we are also looking at all the fine 
tuning of PG parameters to make sure none of those are having an impact

On the Dancer side - maybe I need to check that we are doing things 
correctly. We use Dancer2::Plugin::Database in the following way.

In each module that performs DB requests, we 'use' the above package

In a typical DB layer request we will do:

Package mmm;
use Dancer2 appname => 'APP';
Use Dancer2::Plugin::Database

.........
$sth = database->prepare("select xxxx from yyy(?)");
$sth->execute($id);

$result = $sth->fetchrow_hashref();

return $result;
............

There is no explicit closing of connections.

We use starman to run our app, and it looks like the number of starman 
workers matches exactly the number of open connections to the DB - so it 
seems that the plugin holds one open connection per instance.

Does the above all sound the correct use of the plugin?

Z

> -----Original Message-----
> From: dancer-users [mailto:dancer-users-bounces at dancer.pm] On Behalf
> Of Warren Young
> Sent: 08 December 2016 17:33
> To: Perl Dancer users mailing list <dancer-users at dancer.pm>
> Subject: Re: [dancer-users] High CPU usage on DB
>
> On Dec 8, 2016, at 10:27 AM, Zahir Lalani <ZahirLalani at oliver.agency>
> wrote:
> >
> > We use the Dancer database module to create a connection
>
> Are you caching those connections or re-opening a fresh one for every
> request?
>
> And if the latter, are you *closing* the connection before returning the
> queried data to the client?  Are you sure, evidenced by monitoring the OS’s
> open file handles counter?
>
> You don’t say which OS this is, but if it’s Linux:
>
>     ls /proc/$(pidof my-dancer-program)/fd | wc -l
>
> > so in this case we have 20 connections to the DB
>
> So you believe.  Double-check.
>
> > a restart of Starman cures the problem
>
> That sure sounds like a leaked resource of some kind.  I have you chasing 
> DB
> file handles.  You say it isn’t leaked memory.  If not either of those, 
> then it
> could be web sockets or any similar sort of thing.
>
> _______________________________________________
> dancer-users mailing list
> dancer-users at dancer.pm
> http://lists.preshweb.co.uk/mailman/listinfo/dancer-users
_______________________________________________
dancer-users mailing list
dancer-users at dancer.pm
http://lists.preshweb.co.uk/mailman/listinfo/dancer-users