[Dancer-users] Web-front end for long-running processes

[Pedro Melo]

Hi,

On Thu, Sep 20, 2012 at 4:46 PM, Assaf Gordon <gordon at cshl.edu> wrote:
> I'm planning a web site (w/ Dancer, of course :) ) that will execute shell scripts (to compute some results) and eventually return the results to the user.
>
> The web-side is simple enough, but the shell scripts might take anywhere between 10 seconds to 10 minutes to execute.
>
> Is there a recommended way to manage external jobs for this type of scenario ?
> One extreme is to use SGE/PBS and build a whole database-backed-up queuing system.
> The other is perhaps to execute the shell scripts (serialized, one after the other) and just send the results to the users by email.
>
> But if anyone has experience with something similar, any advice will be appreciated.
> (This is supposed to be a short-term project, just a front-end to some unix scripts - so I prefer to keep it simple, not build a full-fledged infrastructure from scratch).

Use a server side queue, assign a secure random ID to each job
(Session::Token is good for this). A simple DBI table would be enough
for the most simple cases, just make sure you archive completed jobs
on a regular basis. (look at pt-archiver from Percona if you use mysql
and if this archive process becomes a bottleneck or causes too much
lock contention).

After accepting the job and storing it (reliably) in your queue,
redirect the user to /job_status/ID. This is the page that the user
can leave open (or bookmark, to come back later). This page should
periodically check what is the status of the job. I would suggest a
meta refresh tag, that is progressively enhanced via JS to a AJAX call
or websocket or pusher.com or whatever you feel most comfortable with.
This way even the most primitive no-JS browser will be able to use
your system.

You can also use that page to display a message to your user "Your job
is currently being processed. You can leave this page open to track
the progress/await completion, or you can bookmark it and come back
later". If you have the AJAX bit up and running and depending how what
kind of feedback the shell script produces, you could even show a ETA
to your uses here...

When the script finishes its part, mark the job as complete. Your
page/AJAX status will notice this on the next refresh/pool, or if you
use websockets/pusher it will notice it immediately. Just redirect the
user again to a success/complete page where you can show the status of
the job to your user.

That's how I would do it.

Bye,
-- 
Pedro Melo
@pedromelo
http://www.simplicidade.org/
http://about.me/melo
xmpp:melo at simplicidade.org
mailto:melo at simplicidade.org