Web-front end for long-running processes
Hello, I'm planning a web site (w/ Dancer, of course :) ) that will execute shell scripts (to compute some results) and eventually return the results to the user. The web-side is simple enough, but the shell scripts might take anywhere between 10 seconds to 10 minutes to execute. Is there a recommended way to manage external jobs for this type of scenario ? One extreme is to use SGE/PBS and build a whole database-backed-up queuing system. The other is perhaps to execute the shell scripts (serialized, one after the other) and just send the results to the users by email. But if anyone has experience with something similar, any advice will be appreciated. (This is supposed to be a short-term project, just a front-end to some unix scripts - so I prefer to keep it simple, not build a full-fledged infrastructure from scratch). Thanks, -gordon
WebSockets ! http://www.slideshare.net/dkrotkine/dancing-with-websocket On 20 September 2012 17:46, Assaf Gordon <gordon@cshl.edu> wrote:
Hello,
I'm planning a web site (w/ Dancer, of course :) ) that will execute shell scripts (to compute some results) and eventually return the results to the user.
The web-side is simple enough, but the shell scripts might take anywhere between 10 seconds to 10 minutes to execute.
Is there a recommended way to manage external jobs for this type of scenario ? One extreme is to use SGE/PBS and build a whole database-backed-up queuing system. The other is perhaps to execute the shell scripts (serialized, one after the other) and just send the results to the users by email.
But if anyone has experience with something similar, any advice will be appreciated. (This is supposed to be a short-term project, just a front-end to some unix scripts - so I prefer to keep it simple, not build a full-fledged infrastructure from scratch).
Thanks, -gordon _______________________________________________ Dancer-users mailing list Dancer-users@perldancer.org http://www.backup-manager.org/cgi-bin/listinfo/dancer-users
damien krotkine wrote, On 09/20/2012 11:58 AM:
WebSockets !
Sounds interesting, I'll have to brush-up on HTML5-ism. But this only takes care of the server-client side communication, not the unix process handling (I neglected to mention that users might close their browser window and come back to it later, expecting to see their jobs still running). Sadly, It seems Apache doesn't support WebSockets proxied to perl (at least I couldn't find any reference to that), and "sudo cpan Dancer::Plugin::WebSocket" fails with too many errors (on Perl 5.14.2) - so I'm not going to pursue it ATM.
On 20 September 2012 17:46, Assaf Gordon <gordon@cshl.edu> wrote:
Hello,
I'm planning a web site (w/ Dancer, of course :) ) that will execute shell scripts (to compute some results) and eventually return the results to the user.
The web-side is simple enough, but the shell scripts might take anywhere between 10 seconds to 10 minutes to execute.
Is there a recommended way to manage external jobs for this type of scenario ? One extreme is to use SGE/PBS and build a whole database-backed-up queuing system. The other is perhaps to execute the shell scripts (serialized, one after the other) and just send the results to the users by email.
But if anyone has experience with something similar, any advice will be appreciated. (This is supposed to be a short-term project, just a front-end to some unix scripts - so I prefer to keep it simple, not build a full-fledged infrastructure from scratch).
Thanks, -gordon _______________________________________________ Dancer-users mailing list Dancer-users@perldancer.org http://www.backup-manager.org/cgi-bin/listinfo/dancer-users
On 20.09.2012 22:27, Assaf Gordon wrote:
damien krotkine wrote, On 09/20/2012 11:58 AM:
WebSockets !
Sounds interesting, I'll have to brush-up on HTML5-ism. But this only takes care of the server-client side communication, not the unix process handling (I neglected to mention that users might close their browser window and come back to it later, expecting to see their jobs still running).
Sadly, It seems Apache doesn't support WebSockets proxied to perl (at least I couldn't find any reference to that), and "sudo cpan Dancer::Plugin::WebSocket" fails with too many errors (on Perl 5.14.2) - so I'm not going to pursue it ATM.
Try Server Side Events, it's way easier. You'll need an event-driven back-end though, like Twiggy. To mix it up place nginx in front of apache and twiggy to have requests to uri like /event to be proxied to twiggy, and others to apache.
A follow-up to set the record straight: Assaf Gordon wrote, On 09/20/2012 12:27 PM:
[...] "sudo cpan Dancer::Plugin::WebSocket" fails with too many errors (on Perl 5.14.2) - so I'm not going to pursue it ATM.
The problem wasn't in "Dancer::Plugin::WebSocket", but in its dependencies (AnyMQ + Twiggy). With Naveed's help I was able to install the websocket plugin: AnyMQ required me to explicitly install the "AE" (AnyEvent) and "EV" (libEV) modules, with their most recent versions. For some reason, those weren't automatically pulled by cpan. -gordon
For the server side, there is Directory::Queue on CPAN maybe? Niels L On Thu, 2012-09-20 at 14:24 -0400, Assaf Gordon wrote:
A follow-up to set the record straight:
Assaf Gordon wrote, On 09/20/2012 12:27 PM:
[...] "sudo cpan Dancer::Plugin::WebSocket" fails with too many errors (on Perl 5.14.2) - so I'm not going to pursue it ATM.
The problem wasn't in "Dancer::Plugin::WebSocket", but in its dependencies (AnyMQ + Twiggy).
With Naveed's help I was able to install the websocket plugin: AnyMQ required me to explicitly install the "AE" (AnyEvent) and "EV" (libEV) modules, with their most recent versions. For some reason, those weren't automatically pulled by cpan.
-gordon _______________________________________________ Dancer-users mailing list Dancer-users@perldancer.org http://www.backup-manager.org/cgi-bin/listinfo/dancer-users
Directory::Queue looks awesome. Thanks for sharing. It's probably the simplest and most robust queueing system in the world :) -Naveed On Thu, Sep 20, 2012 at 2:26 PM, Niels Larsen <niels@genomics.dk> wrote:
For the server side, there is Directory::Queue on CPAN maybe?
Niels L
On Thu, 2012-09-20 at 14:24 -0400, Assaf Gordon wrote:
A follow-up to set the record straight:
Assaf Gordon wrote, On 09/20/2012 12:27 PM:
[...] "sudo cpan Dancer::Plugin::WebSocket" fails with too many errors (on Perl 5.14.2) - so I'm not going to pursue it ATM.
The problem wasn't in "Dancer::Plugin::WebSocket", but in its dependencies (AnyMQ + Twiggy).
With Naveed's help I was able to install the websocket plugin: AnyMQ required me to explicitly install the "AE" (AnyEvent) and "EV" (libEV) modules, with their most recent versions. For some reason, those weren't automatically pulled by cpan.
-gordon _______________________________________________ Dancer-users mailing list Dancer-users@perldancer.org http://www.backup-manager.org/cgi-bin/listinfo/dancer-users
_______________________________________________ Dancer-users mailing list Dancer-users@perldancer.org http://www.backup-manager.org/cgi-bin/listinfo/dancer-users
On Thu, Sep 20, 2012 at 11:46 AM, Assaf Gordon <gordon@cshl.edu> wrote:
Hello,
I'm planning a web site (w/ Dancer, of course :) ) that will execute shell scripts (to compute some results) and eventually return the results to the user.
The web-side is simple enough, but the shell scripts might take anywhere between 10 seconds to 10 minutes to execute.
Is there a recommended way to manage external jobs for this type of scenario ? One extreme is to use SGE/PBS and build a whole database-backed-up queuing system.
If you want to go this route, I like to use https://metacpan.org/module/Dancer::Plugin::Stomp. Installing and deploying the whole queuing system is simply a matter of running: sudo cpan POE::Component::MessageQueue sudo mq.pl -Naveed
The other is perhaps to execute the shell scripts (serialized, one after the other) and just send the results to the users by email.
But if anyone has experience with something similar, any advice will be appreciated. (This is supposed to be a short-term project, just a front-end to some unix scripts - so I prefer to keep it simple, not build a full-fledged infrastructure from scratch).
Thanks, -gordon _______________________________________________ Dancer-users mailing list Dancer-users@perldancer.org http://www.backup-manager.org/cgi-bin/listinfo/dancer-users
Hi, On Thu, Sep 20, 2012 at 4:46 PM, Assaf Gordon <gordon@cshl.edu> wrote:
I'm planning a web site (w/ Dancer, of course :) ) that will execute shell scripts (to compute some results) and eventually return the results to the user.
The web-side is simple enough, but the shell scripts might take anywhere between 10 seconds to 10 minutes to execute.
Is there a recommended way to manage external jobs for this type of scenario ? One extreme is to use SGE/PBS and build a whole database-backed-up queuing system. The other is perhaps to execute the shell scripts (serialized, one after the other) and just send the results to the users by email.
But if anyone has experience with something similar, any advice will be appreciated. (This is supposed to be a short-term project, just a front-end to some unix scripts - so I prefer to keep it simple, not build a full-fledged infrastructure from scratch).
Use a server side queue, assign a secure random ID to each job (Session::Token is good for this). A simple DBI table would be enough for the most simple cases, just make sure you archive completed jobs on a regular basis. (look at pt-archiver from Percona if you use mysql and if this archive process becomes a bottleneck or causes too much lock contention). After accepting the job and storing it (reliably) in your queue, redirect the user to /job_status/ID. This is the page that the user can leave open (or bookmark, to come back later). This page should periodically check what is the status of the job. I would suggest a meta refresh tag, that is progressively enhanced via JS to a AJAX call or websocket or pusher.com or whatever you feel most comfortable with. This way even the most primitive no-JS browser will be able to use your system. You can also use that page to display a message to your user "Your job is currently being processed. You can leave this page open to track the progress/await completion, or you can bookmark it and come back later". If you have the AJAX bit up and running and depending how what kind of feedback the shell script produces, you could even show a ETA to your uses here... When the script finishes its part, mark the job as complete. Your page/AJAX status will notice this on the next refresh/pool, or if you use websockets/pusher it will notice it immediately. Just redirect the user again to a success/complete page where you can show the status of the job to your user. That's how I would do it. Bye, -- Pedro Melo @pedromelo http://www.simplicidade.org/ http://about.me/melo xmpp:melo@simplicidade.org mailto:melo@simplicidade.org
On 2012-09-20 16:46, Assaf Gordon wrote:
I'm planning a web site (w/ Dancer, of course :) ) that will execute shell scripts (to compute some results) and eventually return the results to the user.
The web-side is simple enough, but the shell scripts might take anywhere between 10 seconds to 10 minutes to execute.
Is there a recommended way to manage external jobs for this type of scenario ?
Just another idea to add to the other more concrete suggestions... If you're happy with javascript being involved, there's a browser notification system similar to Growl, you could try, called jsGrowl. I've seen it used in the Cobbler web UI where you start a job, then later on the notification pops up in the corner of the browser, with a link to an 'events log' page. This is independent of the queue/job-control. It simply makes a background AJAX request to the server every two seconds asking for new 'events', and pops up if there is one. https://github.com/cobbler/cobbler/blob/master/web/content/cobbler.js https://github.com/cobbler/cobbler/tree/master/web/content regards, oliver.
participants (7)
-
Assaf Gordon -
damien krotkine -
Naveed Massjouni -
Niels Larsen -
Oliver Gorwits -
Pedro Melo -
Роман Галеев