Get backtrace for stuck Dancer process
I’ve got a long-running Dancer app which froze up today for no obvious reason, with no useful logs either at the Apache front-end proxy level or at the Dancer level. If this were a program written in a language that compiles to native code, I’d force a core dump and do a backtrace on the process in a debugger to find out where it got hung up, which would usually tell me how to avoid that problem. Is there a way to do that with Perl, and if not, can anyone suggest an alternative? My best alternative is to set up some kind of watchdog, but I’d rather avoid the problem in the first place, if possible.
On 9/27/18 11:48 PM, Warren Young wrote:
I’ve got a long-running Dancer app which froze up today for no obvious reason, with no useful logs either at the Apache front-end proxy level or at the Dancer level.
If this were a program written in a language that compiles to native code, I’d force a core dump and do a backtrace on the process in a debugger to find out where it got hung up, which would usually tell me how to avoid that problem.
Is there a way to do that with Perl, and if not, can anyone suggest an alternative?
My best alternative is to set up some kind of watchdog, but I’d rather avoid the problem in the first place, if possible.
At any rate, you can strace the Dancer process(es) and see what system calls it does. Regards Racke
_______________________________________________ dancer-users mailing list dancer-users@dancer.pm http://lists.preshweb.co.uk/mailman/listinfo/dancer-users
-- Ecommerce and Linux consulting + Perl and web application programming. Debian and Sympa administration. Provisioning with Ansible.
On Sep 28, 2018, at 1:13 AM, Stefan Hornburg (Racke) <racke@linuxia.de> wrote:
At any rate, you can strace the Dancer process(es) and see what system calls it does.
I’m quite familiar with strace, but I rejected that out of hand at the time, since most Perl code is running way above the OS’s syscall interface. There are only a couple of cases I can think of where what it says might help. E.g. stuck on accept(2). No, I really do want to be able to produce a Perl-level stack trace, on demand, without killing the program, as via Carp::confess(). It looks like “perl -d” can’t attach to a running program, either, which would let me use some of these fine Perl modules to produce a backtrace. I found a partial solution, though: https://metacpan.org/pod/App%3a%3aStacktrace Unfortunately for me, it requires Perl 5.10, and this problem happened on a box that’s still running 5.8. (Don’t ask, can’t avoid that.)
On Sep 28, 2018, at 1:11 PM, Clive Eisen <clive@hildebrand.co.uk> wrote:
On 28 Sep 2018, at 19:39, Warren Young <warren@etr-usa.com> wrote: Unfortunately for me, it requires Perl 5.10, and this problem happened on a box that’s still running 5.8. (Don’t ask, can’t avoid that.)
perlbrew?
Perhaps as a last resort. We can’t even use CPAN at this site, due to firewall restrictions, so upgrades of Dancer and such have had to happen via Carton. I suppose perlbrew is just one more step down that same path…
On 28 Sep 2018, at 20:40, Warren Young <warren@etr-usa.com> wrote:
Perhaps as a last resort. We can’t even use CPAN at this site, due to firewall restrictions, so upgrades of Dancer and such have had to happen via Carton.
I suppose perlbrew is just one more step down that same path…
https://metacpan.org/pod/Dancer2 <https://metacpan.org/pod/Dancer2> 'Dancer2 can optionally use XS modules for speed, but at its core remains fatpackable (packable by App::FatPacker <https://metacpan.org/pod/App::FatPacker>) so you could easily deploy Dancer2 applications on hosts that do not support custom CPAN modules.’ — C
Can you modify the code in the program or at least add a statement to the invocation? If so, here are some hints from an older perlmonks post: https://www.perlmonks.org/?node_id=640319 Maybe you could catch a signal and invoke one of the methods described in the post. There is Devel::STrace if you can handle the volume of information. Since it doesn't die, I'm not sure Carp::Always would be of much help except in a signal handler you set up. Does that help any? -john On 9/28/2018 12:39 PM, Warren Young wrote:
On Sep 28, 2018, at 1:13 AM, Stefan Hornburg (Racke) <racke@linuxia.de> wrote:
At any rate, you can strace the Dancer process(es) and see what system calls it does. I’m quite familiar with strace, but I rejected that out of hand at the time, since most Perl code is running way above the OS’s syscall interface.
There are only a couple of cases I can think of where what it says might help. E.g. stuck on accept(2).
No, I really do want to be able to produce a Perl-level stack trace, on demand, without killing the program, as via Carp::confess().
It looks like “perl -d” can’t attach to a running program, either, which would let me use some of these fine Perl modules to produce a backtrace.
I found a partial solution, though:
https://metacpan.org/pod/App%3a%3aStacktrace
Unfortunately for me, it requires Perl 5.10, and this problem happened on a box that’s still running 5.8. (Don’t ask, can’t avoid that.) _______________________________________________ dancer-users mailing list dancer-users@dancer.pm http://lists.preshweb.co.uk/mailman/listinfo/dancer-users
-- John J. McDermott, CPLP Learning and Performance Consultant jjm at jkintl.com 575/737-8556 Check out my blog posts at blog.learningtree.com Add an A for the Arts To STEM and get STEAM and a strong engine to move forward.
In addition to my earlier suggestion, you might consider use of AOP: https://metacpan.org/pod/Aspect I have not used it for the need you have, but there may be a way you can get what you need. On 9/28/2018 12:39 PM, Warren Young wrote:
On Sep 28, 2018, at 1:13 AM, Stefan Hornburg (Racke) <racke@linuxia.de> wrote:
At any rate, you can strace the Dancer process(es) and see what system calls it does. I’m quite familiar with strace, but I rejected that out of hand at the time, since most Perl code is running way above the OS’s syscall interface.
There are only a couple of cases I can think of where what it says might help. E.g. stuck on accept(2).
No, I really do want to be able to produce a Perl-level stack trace, on demand, without killing the program, as via Carp::confess().
It looks like “perl -d” can’t attach to a running program, either, which would let me use some of these fine Perl modules to produce a backtrace.
I found a partial solution, though:
https://metacpan.org/pod/App%3a%3aStacktrace
Unfortunately for me, it requires Perl 5.10, and this problem happened on a box that’s still running 5.8. (Don’t ask, can’t avoid that.)
participants (4)
-
Clive Eisen -
John McDermott, CPLP -
Stefan Hornburg (Racke) -
Warren Young