How can a fork+system() Dancer app not create zombie processes?
Hi all! When running a simple and reduced Dancer 1 application which forks and runs perldoc -f system , I am getting many zombie processes after making requests. This does not happen with https://github.com/shlomif/shlomif-perl-snippets/blob/master/tcp-socket-serv... which does the same thing using IO::Socket and plain TCP. To reproduce, do: hg clone https://shlomif@bitbucket.org/shlomif/app-notifier cd app-notifier hg checkout zombies-processes-bug cd App-Notifier-Service/ And then run “bash run-server.bash” and in a different shell session “bash run-client.bash.” . You can run 'pstree -p' to see the zombies processes. This happens on Mageia v6 x86-64 and Fedora 25 x86-64 with perl-Dancer-1.320.200-2.mga6 . I am quoting a discussion I had on #dancer on irc.perl.org about it. Any further insights about a solution will be welcome. Regards, Shlomi Fish ``` Apr 02 16:47:21 <rindolf> Hi all! My dancer service here (run run-server.bash and then run-client.bash in the App-Notifier-Service directory and note the branch) generates many zombie processes - how can I prevent it from happening? https://bitbucket.org/shlomif/app-notifier/branch/zombies-processes-bug Apr 02 16:54:58 <racke> how do you start the dancer service? Apr 02 16:55:57 <rindolf> racke: perl -I"`pwd`/lib" bin/app.pl --port=6300 Apr 02 16:56:25 <racke> so that is Dancer 1 ? Apr 02 16:56:40 <rindolf> racke: yes, it is Apr 02 16:58:36 <racke> you start only one instance so i might not be able to handle the incoming request Apr 02 16:59:32 <rindolf> racke: it handles most requests fine Apr 02 16:59:36 <racke> better use something like plackup -s Starman --workers=5 Apr 02 17:01:13 <rindolf> racke: what do I need it for? Apr 02 17:01:38 <racke> as I explained before Apr 02 17:01:44 <racke> you need more than one instance Apr 02 17:02:33 <rindolf> racke: see https://metacpan.org/pod/App::Notifier::Service for my use case Apr 02 17:02:38 <rindolf> racke: why? Apr 02 17:03:08 <rindolf> racke: i don't need to handle moer than one request at the time Apr 02 17:03:23 <racke> hm Apr 02 17:03:30 <racke> if you think so Apr 02 17:05:01 <racke> you probably don't handle fork correctly, that is likely to cause your zombies Apr 02 17:05:27 <rindolf> racke: I followed perldoc perlipc Apr 02 17:10:42 <racke> well why not use something like IPC::RUn? Apr 02 17:16:33 <rindolf> racke: 1. I used fork+system - it is enough for my needs, 2. I provided a link to the code - did you take a look? 3. I found IPC::Run buggy when i used it in a different project Apr 02 17:17:22 <rindolf> racke: and also - this is the reduced-to-a-minimum code Apr 02 17:19:23 <racke> yeah I did take a look Apr 02 17:19:49 <rindolf> racke: ok, thanks! Apr 02 17:20:09 <racke> I am not familar with using fork so ... Apr 02 17:20:33 <racke> but it looks like it produces your army of zombies :-) Apr 02 18:01:22 <rindolf> racke: the IO-Socket-INET equivalent does not spawn any zombies - https://github.com/shlomif/shlomif-perl-snippets/blob/master/tcp-socket-serv... Apr 02 18:07:07 <rindolf> http://stackoverflow.com/questions/13614498/forking-to-run-code-in-a-child-p... - there is this too ``` -- ----------------------------------------------------------------- Shlomi Fish http://www.shlomifish.org/ My Favourite FOSS - http://www.shlomifish.org/open-source/favourite/ Chuck Norris can end world hunger, but he thinks that hungry people make humanity a more challenging adversary. — http://www.shlomifish.org/humour/bits/facts/Chuck-Norris/ Please reply to list if it's a mailing list post
On Mon, 3 Apr 2017 11:01:23 +0300 Shlomi Fish wrote:
When running a simple and reduced Dancer 1 application which forks and runs perldoc -f system , I am getting many zombie processes after making requests.
Forking a web app is not as simple as it first seems. You need to take an approach like the following (hopefully the comments will explain without me regurgitating here, but feel free to ask for clarification): https://github.com/ctrlo/GADS/blob/cb8c9478dee8eac1f40a44486634173bed8d666d/... The key point being that you need to fork both a child and a grand-child in order to correctly reap all the forked processes. Andy
Hi Andy! On Mon, 3 Apr 2017 09:15:37 +0100 Andrew Beverley <andy@andybev.com> wrote:
On Mon, 3 Apr 2017 11:01:23 +0300 Shlomi Fish wrote:
When running a simple and reduced Dancer 1 application which forks and runs perldoc -f system , I am getting many zombie processes after making requests.
Forking a web app is not as simple as it first seems.
Do you mean forking any web app or just Dancer apps?
You need to take an approach like the following (hopefully the comments will explain without me regurgitating here, but feel free to ask for clarification):
https://github.com/ctrlo/GADS/blob/cb8c9478dee8eac1f40a44486634173bed8d666d/...
I don't understand the code there in part because it uses too many custom APIs like "guard"s and "try". Furthermore, the licence is the AGPLv3 which cannot be used inside my MIT Expat code. If someone can provide a simple interface to spawn a background child process in Dancer, I will appreciate it. Something like: use Dancer::Plugin::SpawnProc; # Inside the handler: dancer_spawn_proc (sub { # I am the child process. system("mpv", 'no-exist.webm'); exit(0); }); ========== Regards, Shlomi Fish
The key point being that you need to fork both a child and a grand-child in order to correctly reap all the forked processes.
Andy
-- ----------------------------------------------------------------- Shlomi Fish http://www.shlomifish.org/ List of Text Editors and IDEs - http://shlom.in/IDEs You gotta go out there, believe in the ball, and throw yourself. — The Wise Janitor in http://en.wikipedia.org/wiki/Not_another_teen_movie Please reply to list if it's a mailing list post - http://shlom.in/reply .
On 04/03/2017 12:22 PM, Shlomi Fish wrote:
Hi Andy!
On Mon, 3 Apr 2017 09:15:37 +0100 Andrew Beverley <andy@andybev.com> wrote:
On Mon, 3 Apr 2017 11:01:23 +0300 Shlomi Fish wrote:
When running a simple and reduced Dancer 1 application which forks and runs perldoc -f system , I am getting many zombie processes after making requests.
Forking a web app is not as simple as it first seems.
Do you mean forking any web app or just Dancer apps?
You need to take an approach like the following (hopefully the comments will explain without me regurgitating here, but feel free to ask for clarification):
https://github.com/ctrlo/GADS/blob/cb8c9478dee8eac1f40a44486634173bed8d666d/...
I don't understand the code there in part because it uses too many custom APIs like "guard"s and "try". Furthermore, the licence is the AGPLv3 which cannot be used inside my MIT Expat code.
Shlomi, the point is the double fork - I suggest to disregard the code in the second else condition and put your code there :-). Regards Racke
If someone can provide a simple interface to spawn a background child process in Dancer, I will appreciate it. Something like:
use Dancer::Plugin::SpawnProc;
# Inside the handler: dancer_spawn_proc (sub { # I am the child process. system("mpv", 'no-exist.webm'); exit(0); });
==========
Regards,
Shlomi Fish
The key point being that you need to fork both a child and a grand-child in order to correctly reap all the forked processes.
Andy
-- Ecommerce and Linux consulting + Perl and web application programming. Debian and Sympa administration.
On Mon, 3 Apr 2017 13:22:16 +0300 Shlomi Fish wrote:
Forking a web app is not as simple as it first seems.
Do you mean forking any web app or just Dancer apps?
Yes, any web application, and probably any application that runs continuously. The parent process needs to reap the child processes.
You need to take an approach like the following (hopefully the comments will explain without me regurgitating here, but feel free to ask for clarification):
https://github.com/ctrlo/GADS/blob/cb8c9478dee8eac1f40a44486634173bed8d666d/...
I don't understand the code there in part because it uses too many custom APIs
As Racke says, you just need to use that concept. I.e. the parent process forks a child; the child process terminates immediately after it has spawned a grandchild, which allows the parent to reap the child. The grandchild is then left hanging and becomes init's responsibility.
like "guard"s
You should consider a guard, to ensure that nothing in the grandchild causes it to hang around. For example, if there is an exception which is then caught before the _exit(), you can end up with the grandchild hanging around serving web requests to nobody ;-)
and "try".
You can ignore that.
Furthermore, the licence is the AGPLv3 which cannot be used inside my MIT Expat code.
IANAL, but I wouldn't say that small snippet of code is copywritable. Nonetheless, I can speak with authority for Ctrl O, and you are welcome to freely copy without condition that whole forking section if you want. Andy
Hi Andy, On Mon, 3 Apr 2017 12:21:36 +0100 Andrew Beverley <andy@andybev.com> wrote:
On Mon, 3 Apr 2017 13:22:16 +0300 Shlomi Fish wrote:
Forking a web app is not as simple as it first seems.
Do you mean forking any web app or just Dancer apps?
Yes, any web application, and probably any application that runs continuously. The parent process needs to reap the child processes.
I see.
You need to take an approach like the following (hopefully the comments will explain without me regurgitating here, but feel free to ask for clarification):
https://github.com/ctrlo/GADS/blob/cb8c9478dee8eac1f40a44486634173bed8d666d/...
I don't understand the code there in part because it uses too many custom APIs
As Racke says, you just need to use that concept. I.e. the parent process forks a child; the child process terminates immediately after it has spawned a grandchild, which allows the parent to reap the child. The grandchild is then left hanging and becomes init's responsibility.
I see - thanks!
like "guard"s
You should consider a guard, to ensure that nothing in the grandchild causes it to hang around. For example, if there is an exception which is then caught before the _exit(), you can end up with the grandchild hanging around serving web requests to nobody ;-)
OK, I will.
and "try".
You can ignore that.
OK.
Furthermore, the licence is the AGPLv3 which cannot be used inside my MIT Expat code.
IANAL, but I wouldn't say that small snippet of code is copywritable. Nonetheless, I can speak with authority for Ctrl O, and you are welcome to freely copy without condition that whole forking section if you want.
Thanks! Regards, Shlomi Fish
Andy
-- ----------------------------------------------------------------- Shlomi Fish http://www.shlomifish.org/ Freecell Solver - http://fc-solve.shlomifish.org/ Doing linear scans over an associative array is like trying to club someone to death with a loaded Uzi. — Larry Wall Please reply to list if it's a mailing list post - http://shlom.in/reply .
Hi Andy, On Mon, 3 Apr 2017 12:21:36 +0100 Andrew Beverley <andy@andybev.com> wrote:
On Mon, 3 Apr 2017 13:22:16 +0300 Shlomi Fish wrote:
Forking a web app is not as simple as it first seems.
Do you mean forking any web app or just Dancer apps?
Yes, any web application, and probably any application that runs continuously. The parent process needs to reap the child processes.
sorry it took me so long to try that. I've now tried your suggestion of double-forking and I'm still getting zombies. See: * https://bitbucket.org/shlomif/app-notifier/branch/zombies-processes-bug How can it be fixed? Regards, Shlomi Fish
You need to take an approach like the following (hopefully the comments will explain without me regurgitating here, but feel free to ask for clarification):
https://github.com/ctrlo/GADS/blob/cb8c9478dee8eac1f40a44486634173bed8d666d/...
I don't understand the code there in part because it uses too many custom APIs
As Racke says, you just need to use that concept. I.e. the parent process forks a child; the child process terminates immediately after it has spawned a grandchild, which allows the parent to reap the child. The grandchild is then left hanging and becomes init's responsibility.
like "guard"s
You should consider a guard, to ensure that nothing in the grandchild causes it to hang around. For example, if there is an exception which is then caught before the _exit(), you can end up with the grandchild hanging around serving web requests to nobody ;-)
and "try".
You can ignore that.
Furthermore, the licence is the AGPLv3 which cannot be used inside my MIT Expat code.
IANAL, but I wouldn't say that small snippet of code is copywritable. Nonetheless, I can speak with authority for Ctrl O, and you are welcome to freely copy without condition that whole forking section if you want.
Andy
-- ----------------------------------------------------------------- Shlomi Fish http://www.shlomifish.org/ UNIX Fortune Cookies - http://www.shlomifish.org/humour/fortunes/ Reality to be conquered, must be obeyed. — Francis Bacon
On Thu, 13 Apr 2017 13:11:38 +0300 Shlomi Fish wrote:
sorry it took me so long to try that. I've now tried your suggestion of double-forking and I'm still getting zombies. See:
* https://bitbucket.org/shlomif/app-notifier/branch/zombies-processes-bug
How can it be fixed?
You need to also copy the code for the exit: POSIX::_exit(0); Reasons here: stackoverflow.com/questions/5422831/ Incidentally I noticed this: if (fork() eq 0) eq is for comparing strings, and you could just write: if (!fork) Andy
On Fri, 14 Apr 2017 11:58:46 +0100 Andrew Beverley <andy@andybev.com> wrote:
On Thu, 13 Apr 2017 13:11:38 +0300 Shlomi Fish wrote:
sorry it took me so long to try that. I've now tried your suggestion of double-forking and I'm still getting zombies. See:
* https://bitbucket.org/shlomif/app-notifier/branch/zombies-processes-bug
How can it be fixed?
You need to also copy the code for the exit:
POSIX::_exit(0);
Reasons here: stackoverflow.com/questions/5422831/
Thanks!
Incidentally I noticed this:
if (fork() eq 0)
eq is for comparing strings, and you could just write:
if (!fork)
But I want this check to fail if fork() returns undef(). Regards, Shlomi Fish
Andy
-- ----------------------------------------------------------------- Shlomi Fish http://www.shlomifish.org/ Rethinking CPAN - http://shlom.in/rethinking-cpan <mst> I find it’s usually safe to assume that whatever shlomif’s doing, there isn’t a good reason for it. Please reply to list if it's a mailing list post - http://shlom.in/reply .
Hi all, I want to send my Dancer2 TT page as an email *and* continue to generate a normal web response. This forking discussion seems like an excellent approach. I have: 1. Created a fork in 'hook before_layout_render' (based on a 'var want_email = 1;'). 2. The parent continues to serve the webpage (as normal, after undef'ing want_email), the grandchild sends the email (in 'hook after_layout_render') *(I'm doing this in before_layout_render so I can change the layout (app->template_engine->layout('email');) to a lighter, more email-friendly layout)* An additional benefit of this approach is that the email is sent asynchronously, since Gmail-dispatched emails seem to take 2-3s to send. This all works well and I am grateful for this discussion and Andy's example code, but I am unclear about the guard, per below. I understand what it is intended to do, but if I put: my $guard = guard { info "I'm dying"; POSIX::_exit(0); }; in my code, the code immediately prints the "I'm dying" and exits. Am I missing something in how to use Guard properly? BTW, if someone has a better pattern for emailing a page, I'd welcome it. cheers, Nathan On 3 April 2017 at 21:21, Andrew Beverley <andy@andybev.com> wrote:
On Mon, 3 Apr 2017 13:22:16 +0300 Shlomi Fish wrote:
like "guard"s
You should consider a guard, to ensure that nothing in the grandchild causes it to hang around. For example, if there is an exception which is then caught before the _exit(), you can end up with the grandchild hanging around serving web requests to nobody ;-)
On Thu, 29 Jun 2017 15:55:59 +1000 Nathan Bailey wrote:
Hi all, I want to send my Dancer2 TT page as an email *and* continue to generate a normal web response.
This forking discussion seems like an excellent approach.
I'm not convinced - it seems like overkill to me.
An additional benefit of this approach is that the email is sent asynchronously, since Gmail-dispatched emails seem to take 2-3s to send.
I would initially investigate on why emails are taking so long to send. Can you deliver the email locally instead? That should happen with very little noticeable delay. It's then effectively asynchronous, as the local mailer will be attempting to send the email whilst the web application continues to run.
This all works well and I am grateful for this discussion and Andy's example code, but I am unclear about the guard, per below. I understand what it is intended to do, but if I put: my $guard = guard { info "I'm dying"; POSIX::_exit(0); }; in my code, the code immediately prints the "I'm dying" and exits.
Have you loaded a suitable guard module (e.g. Guard or Scope::Guard)? Otherwise (from memory) the application will exit before it raises an exception of the missing guard function. Andy
Hi Nathan It might be helpful to address this problem through a separation of concerns. You've got the Dancer app's job which is to serve web pages. Then there's another job - to send emails - and from this viewpoint it makes sense to implement that as a separate piece of software. On a large scale this can be done using a queuing service where the Dancer app puts a message onto the message queue (managed by a broker like RabbitMQ or ActiveMQ....) and your email sending software pulls the message off the queue and sends the email. On a smaller scale I've done this without a message broker by treating a table in a database as a message queue - the Dancer app writes a line to the table with details of the message to send. A separate script - a daemon based on this https://metacpan.org/pod/Daemon::Control checks for any new records in the table, sends the appropriate message and marks the row as 'sent'. There's more than one way to skin a cat so that's just another approach to consider. Good luck! Andrew On 29 Jun 2017 06:56, "Nathan Bailey" <web@polynate.net> wrote:
Hi all, I want to send my Dancer2 TT page as an email *and* continue to generate a normal web response.
This forking discussion seems like an excellent approach.
I have: 1. Created a fork in 'hook before_layout_render' (based on a 'var want_email = 1;'). 2. The parent continues to serve the webpage (as normal, after undef'ing want_email), the grandchild sends the email (in 'hook after_layout_render')
*(I'm doing this in before_layout_render so I can change the layout (app->template_engine->layout('email');) to a lighter, more email-friendly layout)*
An additional benefit of this approach is that the email is sent asynchronously, since Gmail-dispatched emails seem to take 2-3s to send.
This all works well and I am grateful for this discussion and Andy's example code, but I am unclear about the guard, per below. I understand what it is intended to do, but if I put: my $guard = guard { info "I'm dying"; POSIX::_exit(0); }; in my code, the code immediately prints the "I'm dying" and exits.
Am I missing something in how to use Guard properly?
BTW, if someone has a better pattern for emailing a page, I'd welcome it.
cheers, Nathan
On 3 April 2017 at 21:21, Andrew Beverley <andy@andybev.com> wrote:
On Mon, 3 Apr 2017 13:22:16 +0300 Shlomi Fish wrote:
like "guard"s
You should consider a guard, to ensure that nothing in the grandchild causes it to hang around. For example, if there is an exception which is then caught before the _exit(), you can end up with the grandchild hanging around serving web requests to nobody ;-)
_______________________________________________ dancer-users mailing list dancer-users@dancer.pm http://lists.preshweb.co.uk/mailman/listinfo/dancer-users
Agree with Andrew We use Gearman and it works very well. The advantage is that you scale your asynch backend independently of your main application, and it allows you main app to be far more responsive rather waste time doing things that can be done asynch Z From: dancer-users [mailto:dancer-users-bounces@dancer.pm] On Behalf Of Andrew Solomon Sent: 29 June 2017 11:27 To: Perl Dancer users mailing list <dancer-users@dancer.pm> Subject: Re: [dancer-users] How can a fork+system() Dancer app not create zombie processes? Hi Nathan It might be helpful to address this problem through a separation of concerns. You've got the Dancer app's job which is to serve web pages. Then there's another job - to send emails - and from this viewpoint it makes sense to implement that as a separate piece of software. On a large scale this can be done using a queuing service where the Dancer app puts a message onto the message queue (managed by a broker like RabbitMQ or ActiveMQ....) and your email sending software pulls the message off the queue and sends the email. On a smaller scale I've done this without a message broker by treating a table in a database as a message queue - the Dancer app writes a line to the table with details of the message to send. A separate script - a daemon based on this https://metacpan.org/pod/Daemon::Control checks for any new records in the table, sends the appropriate message and marks the row as 'sent'. There's more than one way to skin a cat so that's just another approach to consider. Good luck! Andrew On 29 Jun 2017 06:56, "Nathan Bailey" <web@polynate.net<mailto:web@polynate.net>> wrote: Hi all, I want to send my Dancer2 TT page as an email and continue to generate a normal web response. This forking discussion seems like an excellent approach. I have: 1. Created a fork in 'hook before_layout_render' (based on a 'var want_email = 1;'). 2. The parent continues to serve the webpage (as normal, after undef'ing want_email), the grandchild sends the email (in 'hook after_layout_render') (I'm doing this in before_layout_render so I can change the layout (app->template_engine->layout('email');) to a lighter, more email-friendly layout) An additional benefit of this approach is that the email is sent asynchronously, since Gmail-dispatched emails seem to take 2-3s to send. This all works well and I am grateful for this discussion and Andy's example code, but I am unclear about the guard, per below. I understand what it is intended to do, but if I put: my $guard = guard { info "I'm dying"; POSIX::_exit(0); }; in my code, the code immediately prints the "I'm dying" and exits. Am I missing something in how to use Guard properly? BTW, if someone has a better pattern for emailing a page, I'd welcome it. cheers, Nathan On 3 April 2017 at 21:21, Andrew Beverley <andy@andybev.com<mailto:andy@andybev.com>> wrote: On Mon, 3 Apr 2017 13:22:16 +0300 Shlomi Fish wrote:
like "guard"s
You should consider a guard, to ensure that nothing in the grandchild causes it to hang around. For example, if there is an exception which is then caught before the _exit(), you can end up with the grandchild hanging around serving web requests to nobody ;-) _______________________________________________ dancer-users mailing list dancer-users@dancer.pm<mailto:dancer-users@dancer.pm> http://lists.preshweb.co.uk/mailman/listinfo/dancer-users
Am 29.06.2017 um 12:27 schrieb Andrew Solomon:
Hi Nathan
It might be helpful to address this problem through a separation of concerns. You've got the Dancer app's job which is to serve web pages. Then there's another job - to send emails - and from this viewpoint it makes sense to implement that as a separate piece of software. On a large scale this can be done using a queuing service where the Dancer app puts a message onto the message queue (managed by a broker like RabbitMQ or ActiveMQ....) and your email sending software pulls the message off the queue and sends the email.
The software to queue e-mails is called a MTA. Installing a local postfix (or exim or any other MTA the OP is comfort- able with) configured to accept local e-mail without authentication should get the submission time down in the ms range. The further pro- cessing is then done by a software that is built to handle e-mail: the MTA. Jochen
participants (7)
-
Andrew Beverley -
Andrew Solomon -
Jochen Lutz -
Nathan Bailey -
Shlomi Fish -
Stefan Hornburg (Racke) -
Zahir Lalani