Detecting disconnected HTTP clients during streamed output
I have a handler that transcodes video files using ffmpeg which all works fine. However, if the web client disconnects or otherwise stops reading from the socket, my handler will continue reading from the ffmpeg filehandle and writing out to the writer object until ffmpeg has completed the transcode. This wouldn't be a problem is we lived in a perfect world where every web client would send a single request to a URL and not make additional requests until the first one had been completed, but of course we don't live in an ideal world,.. and I would prefer my webapp not kill the server it is running on my spawning off more CPU intensive processes than necessary. Is there any way for me to determine how much data is sat waiting to be flushed to the web client so that if it exceeded a predetermined number of bytes I can assume that I should kill my ffmpeg file handle and exit the handler prematurely,.. or any other way to determine if the web client has gone away? Apologies if I haven't explained myself especially articulately. This is the handler in question: *get '/stream/:file/:format' => sub {* * my $file = params->{file};* * my $format = params->{format};* * return send_file(* * 'favicon.ico',* * streaming => 1,* * callbacks => {* * override => sub {* * my ($respond, $response) = @_;* * my $mime = 'video/mp4';* * my @ffmpeg_args = qw(-threads 0 -i $file);* * if (exists $ffmpeg_formats{$format}) {* * $mime = 'video/' . $format;* * push @ffmpeg_args, @{$ffmpeg_formats{$format}};* * } else {* * push @ffmpeg_args, @{$ffmpeg_formats{mp4}};* * }* * push @ffmpeg_args, '-';* * my $writer = $respond->([200, ['Content-Type' => $mime]]);* * open(my $fh, '-|', join(' ', '/usr/bin/ffmpeg', @ffmpeg_args));* * my $buffer;* * while (read($fh, $buffer, 1024) ) {* * $writer->write($buffer);* * }* * close($fh);* * },* * },* * );* *};* -- Nicola Worthington nicolaw@tfb.net
Just an idea. Why don't you use ajax and only allow on conversion per user at a time. You can create an ajax query to check on progress, plus the user can cancel. Basically detach the conversion from the web client, but allow client to check on progress and have access to the file when it has finished. On 21/12/2013 5:40 am, Nicola Worthington wrote:
I have a handler that transcodes video files using ffmpeg which all works fine. However, if the web client disconnects or otherwise stops reading from the socket, my handler will continue reading from the ffmpeg filehandle and writing out to the writer object until ffmpeg has completed the transcode.
This wouldn't be a problem is we lived in a perfect world where every web client would send a single request to a URL and not make additional requests until the first one had been completed, but of course we don't live in an ideal world,.. and I would prefer my webapp not kill the server it is running on my spawning off more CPU intensive processes than necessary.
Is there any way for me to determine how much data is sat waiting to be flushed to the web client so that if it exceeded a predetermined number of bytes I can assume that I should kill my ffmpeg file handle and exit the handler prematurely,.. or any other way to determine if the web client has gone away?
Apologies if I haven't explained myself especially articulately.
This is the handler in question:
*get '/stream/:file/:format' => sub {* * my $file = params->{file};* * my $format = params->{format};* * * * return send_file(* *'favicon.ico',* * streaming => 1,* * callbacks => {* *override => sub {* * my ($respond, $response) = @_;* * * * my $mime = 'video/mp4';* * my @ffmpeg_args = qw(-threads 0 -i $file);* * * * if (exists $ffmpeg_formats{$format}) {* * $mime = 'video/' . $format;* * push @ffmpeg_args, @{$ffmpeg_formats{$format}};* * } else {* * push @ffmpeg_args, @{$ffmpeg_formats{mp4}};* * }* *push @ffmpeg_args, '-';* * * * my $writer = $respond->([200, ['Content-Type' => $mime]]);* *open(my $fh, '-|', join(' ', '/usr/bin/ffmpeg', @ffmpeg_args));* * my $buffer;* *while (read($fh, $buffer, 1024) ) {* * $writer->write($buffer);* * }* *close($fh);* * },* * },* * );* *};*
-- Nicola Worthington nicolaw@tfb.net <mailto:nicolaw@tfb.net>
_______________________________________________ dancer-users mailing list dancer-users@dancer.pm http://lists.preshweb.co.uk/mailman/listinfo/dancer-users
That's certainly a possibility, but since this app is primarily aimed at mobile devices that will be landing directly on stream URLs (for the most part not launched from their browser but through a handful of other apps that will simply make a call to the OS to open a URL for them to stream media directly) it would make this approach less than idea and not as robust as I was hoping for. :( On 21 December 2013 03:52, Matthew Mallard <mqm@q-technologies.com.au>wrote:
Just an idea.
Why don't you use ajax and only allow on conversion per user at a time. You can create an ajax query to check on progress, plus the user can cancel. Basically detach the conversion from the web client, but allow client to check on progress and have access to the file when it has finished.
On 21/12/2013 5:40 am, Nicola Worthington wrote:
I have a handler that transcodes video files using ffmpeg which all works fine. However, if the web client disconnects or otherwise stops reading from the socket, my handler will continue reading from the ffmpeg filehandle and writing out to the writer object until ffmpeg has completed the transcode.
This wouldn't be a problem is we lived in a perfect world where every web client would send a single request to a URL and not make additional requests until the first one had been completed, but of course we don't live in an ideal world,.. and I would prefer my webapp not kill the server it is running on my spawning off more CPU intensive processes than necessary.
Is there any way for me to determine how much data is sat waiting to be flushed to the web client so that if it exceeded a predetermined number of bytes I can assume that I should kill my ffmpeg file handle and exit the handler prematurely,.. or any other way to determine if the web client has gone away?
Apologies if I haven't explained myself especially articulately.
This is the handler in question:
*get '/stream/:file/:format' => sub {* * my $file = params->{file};* * my $format = params->{format};*
* return send_file(* * 'favicon.ico',* * streaming => 1,* * callbacks => {* * override => sub {* * my ($respond, $response) = @_;*
* my $mime = 'video/mp4';* * my @ffmpeg_args = qw(-threads 0 -i $file);*
* if (exists $ffmpeg_formats{$format}) {* * $mime = 'video/' . $format;* * push @ffmpeg_args, @{$ffmpeg_formats{$format}};* * } else {* * push @ffmpeg_args, @{$ffmpeg_formats{mp4}};* * }* * push @ffmpeg_args, '-';*
* my $writer = $respond->([200, ['Content-Type' => $mime]]);* * open(my $fh, '-|', join(' ', '/usr/bin/ffmpeg', @ffmpeg_args));* * my $buffer;* * while (read($fh, $buffer, 1024) ) {* * $writer->write($buffer);* * }* * close($fh);* * },* * },* * );* *};*
-- Nicola Worthington nicolaw@tfb.net
_______________________________________________ dancer-users mailing listdancer-users@dancer.pmhttp://lists.preshweb.co.uk/mailman/listinfo/dancer-users
_______________________________________________ dancer-users mailing list dancer-users@dancer.pm http://lists.preshweb.co.uk/mailman/listinfo/dancer-users
-- Nicola Worthington +44 (0)3333 406 333 nicolaw@tfb.net
I'm not sure I understand what you're saying here. I think we might have our wires crossed or I've not explained myself very well. Web servers communicate over TCP. Therefore until the point that either the servers established TCP connection detects that the client has connected or timed out, it will be able to tell how much data has been successfully send and acknowledged as received by the client browser. This is a function of the underlying TCP network connection. I'm simply looking for a way to get at this socket level information as exposed to the PSGI application through the web server. On Dec 22, 2013 3:57 PM, "Daniel Perrett" <dperrett@cambridge.org> wrote:
As far as I understand it, this is impossible with any HTTP server, because once a client makes an HTTP request, the server processes it and responds - and there is no communication between them in the mean time. A server has no way of knowing if the client is still interested, and the server cannot initiate contact with the client. You could pull some tricks to demand multiple HTTP requests (generate 3xx responses on the fly to play for time), but this is likely to be a headache, and from the sounds of it won't fit your use case either. WebSockets might help you, but I don't know much about them, and I don't think it will work if you don't have any control over the client.
To me it sounds like the root of the problem is you are offering to do resource-intensive tasks for unidentifiable users where the only thing you know about them is that they have the URL.
Daniel
-----dancer-users-bounces@dancer.pm wrote: ----- To: Perl Dancer users mailing list <dancer-users@dancer.pm> From: Nicola Worthington Sent by: dancer-users-bounces@dancer.pm Date: 12/21/2013 12:57PM Subject: Re: [dancer-users] Detecting disconnected HTTP clients during streamed output
That's certainly a possibility, but since this app is primarily aimed at mobile devices that will be landing directly on stream URLs (for the most part not launched from their browser but through a handful of other apps that will simply make a call to the OS to open a URL for them to stream media directly) it would make this approach less than idea and not as robust as I was hoping for. :(
On 21 December 2013 03:52, Matthew Mallard <mqm@q-technologies.com.au>wrote:
Just an idea.
Why don't you use ajax and only allow on conversion per user at a time. You can create an ajax query to check on progress, plus the user can cancel. Basically detach the conversion from the web client, but allow client to check on progress and have access to the file when it has finished.
On 21/12/2013 5:40 am, Nicola Worthington wrote:
I have a handler that transcodes video files using ffmpeg which all works fine. However, if the web client disconnects or otherwise stops reading from the socket, my handler will continue reading from the ffmpeg filehandle and writing out to the writer object until ffmpeg has completed the transcode.
This wouldn't be a problem is we lived in a perfect world where every web client would send a single request to a URL and not make additional requests until the first one had been completed, but of course we don't live in an ideal world,.. and I would prefer my webapp not kill the server it is running on my spawning off more CPU intensive processes than necessary.
Is there any way for me to determine how much data is sat waiting to be flushed to the web client so that if it exceeded a predetermined number of bytes I can assume that I should kill my ffmpeg file handle and exit the handler prematurely,.. or any other way to determine if the web client has gone away?
Apologies if I haven't explained myself especially articulately.
This is the handler in question:
*get '/stream/:file/:format' => sub {* * my $file = params->{file};* * my $format = params->{format};*
* return send_file(* * 'favicon.ico',* * streaming => 1,* * callbacks => {* * override => sub {* * my ($respond, $response) = @_;*
* my $mime = 'video/mp4';* * my @ffmpeg_args = qw(-threads 0 -i $file);*
* if (exists $ffmpeg_formats{$format}) {* * $mime = 'video/' . $format;* * push @ffmpeg_args, @{$ffmpeg_formats{$format}};* * } else {* * push @ffmpeg_args, @{$ffmpeg_formats{mp4}};* * }* * push @ffmpeg_args, '-';*
* my $writer = $respond->([200, ['Content-Type' => $mime]]);* * open(my $fh, '-|', join(' ', '/usr/bin/ffmpeg', @ffmpeg_args));* * my $buffer;* * while (read($fh, $buffer, 1024) ) {* * $writer->write($buffer);* * }* * close($fh);* * },* * },* * );* *};*
-- Nicola Worthington nicolaw@tfb.net
_______________________________________________ dancer-users mailing list dancer-users@dancer.pm http://lists.preshweb.co.uk/mailman/listinfo/dancer-users
_______________________________________________ dancer-users mailing list dancer-users@dancer.pm http://lists.preshweb.co.uk/mailman/listinfo/dancer-users
-- Nicola Worthington +44 (0)3333 406 333 nicolaw@tfb.net
_______________________________________________ dancer-users mailing list dancer-users@dancer.pm http://lists.preshweb.co.uk/mailman/listinfo/dancer-users
_______________________________________________ dancer-users mailing list dancer-users@dancer.pm http://lists.preshweb.co.uk/mailman/listinfo/dancer-users
Typo: client has DISconnected or timed out On Dec 22, 2013 4:11 PM, "Nicola Worthington" <nicolaw@tfb.net> wrote:
I'm not sure I understand what you're saying here. I think we might have our wires crossed or I've not explained myself very well.
Web servers communicate over TCP. Therefore until the point that either the servers established TCP connection detects that the client has connected or timed out, it will be able to tell how much data has been successfully send and acknowledged as received by the client browser. This is a function of the underlying TCP network connection. I'm simply looking for a way to get at this socket level information as exposed to the PSGI application through the web server. On Dec 22, 2013 3:57 PM, "Daniel Perrett" <dperrett@cambridge.org> wrote:
As far as I understand it, this is impossible with any HTTP server, because once a client makes an HTTP request, the server processes it and responds - and there is no communication between them in the mean time. A server has no way of knowing if the client is still interested, and the server cannot initiate contact with the client. You could pull some tricks to demand multiple HTTP requests (generate 3xx responses on the fly to play for time), but this is likely to be a headache, and from the sounds of it won't fit your use case either. WebSockets might help you, but I don't know much about them, and I don't think it will work if you don't have any control over the client.
To me it sounds like the root of the problem is you are offering to do resource-intensive tasks for unidentifiable users where the only thing you know about them is that they have the URL.
Daniel
-----dancer-users-bounces@dancer.pm wrote: ----- To: Perl Dancer users mailing list <dancer-users@dancer.pm> From: Nicola Worthington Sent by: dancer-users-bounces@dancer.pm Date: 12/21/2013 12:57PM Subject: Re: [dancer-users] Detecting disconnected HTTP clients during streamed output
That's certainly a possibility, but since this app is primarily aimed at mobile devices that will be landing directly on stream URLs (for the most part not launched from their browser but through a handful of other apps that will simply make a call to the OS to open a URL for them to stream media directly) it would make this approach less than idea and not as robust as I was hoping for. :(
On 21 December 2013 03:52, Matthew Mallard <mqm@q-technologies.com.au>wrote:
Just an idea.
Why don't you use ajax and only allow on conversion per user at a time. You can create an ajax query to check on progress, plus the user can cancel. Basically detach the conversion from the web client, but allow client to check on progress and have access to the file when it has finished.
On 21/12/2013 5:40 am, Nicola Worthington wrote:
I have a handler that transcodes video files using ffmpeg which all works fine. However, if the web client disconnects or otherwise stops reading from the socket, my handler will continue reading from the ffmpeg filehandle and writing out to the writer object until ffmpeg has completed the transcode.
This wouldn't be a problem is we lived in a perfect world where every web client would send a single request to a URL and not make additional requests until the first one had been completed, but of course we don't live in an ideal world,.. and I would prefer my webapp not kill the server it is running on my spawning off more CPU intensive processes than necessary.
Is there any way for me to determine how much data is sat waiting to be flushed to the web client so that if it exceeded a predetermined number of bytes I can assume that I should kill my ffmpeg file handle and exit the handler prematurely,.. or any other way to determine if the web client has gone away?
Apologies if I haven't explained myself especially articulately.
This is the handler in question:
*get '/stream/:file/:format' => sub {* * my $file = params->{file};* * my $format = params->{format};*
* return send_file(* * 'favicon.ico',* * streaming => 1,* * callbacks => {* * override => sub {* * my ($respond, $response) = @_;*
* my $mime = 'video/mp4';* * my @ffmpeg_args = qw(-threads 0 -i $file);*
* if (exists $ffmpeg_formats{$format}) {* * $mime = 'video/' . $format;* * push @ffmpeg_args, @{$ffmpeg_formats{$format}};* * } else {* * push @ffmpeg_args, @{$ffmpeg_formats{mp4}};* * }* * push @ffmpeg_args, '-';*
* my $writer = $respond->([200, ['Content-Type' => $mime]]);* * open(my $fh, '-|', join(' ', '/usr/bin/ffmpeg', @ffmpeg_args));* * my $buffer;* * while (read($fh, $buffer, 1024) ) {* * $writer->write($buffer);* * }* * close($fh);* * },* * },* * );* *};*
-- Nicola Worthington nicolaw@tfb.net
_______________________________________________ dancer-users mailing list dancer-users@dancer.pm http://lists.preshweb.co.uk/mailman/listinfo/dancer-users
_______________________________________________ dancer-users mailing list dancer-users@dancer.pm http://lists.preshweb.co.uk/mailman/listinfo/dancer-users
-- Nicola Worthington +44 (0)3333 406 333 nicolaw@tfb.net
_______________________________________________ dancer-users mailing list dancer-users@dancer.pm http://lists.preshweb.co.uk/mailman/listinfo/dancer-users
_______________________________________________ dancer-users mailing list dancer-users@dancer.pm http://lists.preshweb.co.uk/mailman/listinfo/dancer-users
participants (3)
-
Daniel Perrett -
Matthew Mallard -
Nicola Worthington