[dancer-users] Dancer::Plugin::Database Hangs Randomly on check_connection

[James Baer]

Hello - I'm having an issue with Dancer::Plugin::Database that has been
hard to reproduce and track down.

I have a new app that I just moved to its production environment. It
connects to a database that is in a remote data center over a vpn tunnel.
Everything functions normally as expected 98% of the time, however
occasionally I will hit a worker thread (deployed via Starman) that just
hangs trying to get a database handle. It hangs in the _check_connection
subroutine until it reaches some timeout. Then will reconnect just fine
with a new connection.

I'm using plugin version 1.82 and I have traced it down to this line in the
plugin:
if ($dbh->{Active} && (my $result = $dbh->ping)) {

Right at this if statement is where it hangs, i assume while calling the
ping function. It seems there is some confusion as to whether the dbh is
still active, it then calls ping which hangs for a number of minutes (not
sure how many exactly yet, b/t 3-5). This renders this worker thread
unresponsive until it times out. It will eventuall fall into the else block
and just return false, which then a new connection is established just
fine.

I have successfully deployed a number of apps using this same plugin with
no problems this only difference here is the physical location so there
maybe something network related, just not sure where.

I have dug into DBI, etc and the ping functions works fine on all my test
scripts. Again, this only happens once in a while so I cannot reproduce on
command, which makes it difficult to debug.


Here is the output of my debug statements, showing the one thread that
hangs:

[27037] debug @0.005053> [hit #4]Database connection_check_threshold [30]
in /usr/local/share/perl5/Dancer/Plugin/Database.pm l. 69
[27037] debug @0.005189> [hit #4]Database handle last check [1398093938] in
/usr/local/share/perl5/Dancer/Plugin/Database.pm l. 70
[27037] debug @0.005345> [hit #4]Database calling check_connection in
/usr/local/share/perl5/Dancer/Plugin/Database.pm l. 83
[27037] debug @0.005481> [hit #4]Database.pm in check_connection before if
active and ping in /usr/local/share/perl5/Dancer/Plugin/Database.pm l. 265
... other threads working ...
[27037] debug @962.134467> [hit #4]Database in check_connectiion false
return  in /usr/local/share/perl5/Dancer/Plugin/Database.pm l. 285
[27037] debug @962.134667> [hit #4]Database connection went away,
reconnecting in /usr/local/share/perl5/Dancer/Plugin/Database.pm l. 90
[27037] debug @962.134894> [hit #4]Database calling get_connection in
/usr/local/share/perl5/Dancer/Plugin/Database.pm l. 97
[27037] debug @962.135059> [hit #4]Adding mysql_enable_utf8 to DBI
connection params to enable UTF-8 support in /usr/local/share/perl5/Dance



Any help or suggestions on where to look next is appreciated.

Thank you.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.preshweb.co.uk/pipermail/dancer-users/attachments/20140421/7a994a2a/attachment.html>