Hi, On Thu, Sep 20, 2012 at 4:46 PM, Assaf Gordon <gordon@cshl.edu> wrote:
I'm planning a web site (w/ Dancer, of course :) ) that will execute shell scripts (to compute some results) and eventually return the results to the user.
The web-side is simple enough, but the shell scripts might take anywhere between 10 seconds to 10 minutes to execute.
Is there a recommended way to manage external jobs for this type of scenario ? One extreme is to use SGE/PBS and build a whole database-backed-up queuing system. The other is perhaps to execute the shell scripts (serialized, one after the other) and just send the results to the users by email.
But if anyone has experience with something similar, any advice will be appreciated. (This is supposed to be a short-term project, just a front-end to some unix scripts - so I prefer to keep it simple, not build a full-fledged infrastructure from scratch).
Use a server side queue, assign a secure random ID to each job (Session::Token is good for this). A simple DBI table would be enough for the most simple cases, just make sure you archive completed jobs on a regular basis. (look at pt-archiver from Percona if you use mysql and if this archive process becomes a bottleneck or causes too much lock contention). After accepting the job and storing it (reliably) in your queue, redirect the user to /job_status/ID. This is the page that the user can leave open (or bookmark, to come back later). This page should periodically check what is the status of the job. I would suggest a meta refresh tag, that is progressively enhanced via JS to a AJAX call or websocket or pusher.com or whatever you feel most comfortable with. This way even the most primitive no-JS browser will be able to use your system. You can also use that page to display a message to your user "Your job is currently being processed. You can leave this page open to track the progress/await completion, or you can bookmark it and come back later". If you have the AJAX bit up and running and depending how what kind of feedback the shell script produces, you could even show a ETA to your uses here... When the script finishes its part, mark the job as complete. Your page/AJAX status will notice this on the next refresh/pool, or if you use websockets/pusher it will notice it immediately. Just redirect the user again to a success/complete page where you can show the status of the job to your user. That's how I would do it. Bye, -- Pedro Melo @pedromelo http://www.simplicidade.org/ http://about.me/melo xmpp:melo@simplicidade.org mailto:melo@simplicidade.org