Streaming binary data without known size? Content-length problem.
While $working on a RESTFul application, I came across a situation where I needed to pack up a directory of files (already gzipped) and send them to the user. Rather than pulling everything into memory with Archive::Tar or creating a temporary file with File::Temp that would meet an uncertain fate after file_send depending on the deployment, I decided I would try and forkexec using a pipe and just stream the tar to the user. This was easy enough to accomplish with open and '-|' and then exec ('/path/to/tar', '-c', '-f', '-' ...); in the child. After that, it was a matter of duplicating parts of send_file, and I think this is where I got it wrong. Testing with raw Telnet saw the binary data come through, so I know that at least the piping worked, but when I tested the route as users would use it, the browser downloaded an empty file and curl -v, mentioned something about "* Excess found in a non pipelined read: excess = 7989". I looked in the printed headers, and saw that it defined Content-Length: 0, which I assume is part of the problem. I tried NOT sending it using the code below using: $response->{headers}->remove_header('Content-Length'); but was unable to remove it, and I'm not sure that's what I want to do anyway. Hijacking the stream and just catting binary data doesn't seem to be part of the spec, though I'd swear I've seen it done elswhere before, but I thought I would pick the collective brains of the mailing list and see. I eventually ended up just using File::Temp::tmpnam and having tar output to that file, which I then added to a var called 'cleanup' that an after hook deletes. Then I sent with file_send as normal. So, I suppose my main question is: would this work if I were somehow able to remove the Content-Length header? I'm thinking not and, if not, should I be using chunked transfer instead? Does Dancer support this? CODE: sub send_data { Dancer::Continuation::Route::FileSent->new( return_value => _send_data(@_) )->throw } sub _send_data { my ($fh, %options) = @_; my $env = Dancer::SharedData->request->env; die "Must have streaming" if ! $env->{'psgi.streaming'}; die "Must provide content_type\n" unless $options{content_type}; die "Must provide filename\n" unless $options{filename}; binmode $fh; my $response = Dancer::SharedData->response() || Dancer::Response->new(); $response->header('Content-Type' => $options{content_type}); $response->push_header('Content-Disposition' => "attachment; filename=\"$options{filename}\"" ); $response->streamed( sub { my ( $status, $headers ) = @_; return sub { my $respond = shift; my $writer = $respond->( [ $status, $headers, ] ); my $content = $response->content; my $bytes = $options{'bytes'} || '43008'; my $buf; while ( ( my $read = sysread $fh, $buf, $bytes ) != 0 ) { $writer->write($buf); } }; } ); return $response if $response; }
On 13-05-09 12:26 PM, Andy Walker wrote:
I'm thinking not and, if not, should I be using chunked transfer instead? Does Dancer support this?
Take my word with a fairly big grain of salt, but I don't think Dancer support chunked transfer right now. But since it's all sitting on psgi, it shouldn't be too hard to do something about that. I've opened a new issue[1] for that. I'll try to go and experiment with it soonishly. Joy, `/anick [1] https://github.com/PerlDancer/Dancer2/issues/263
participants (2)
-
Andy Walker -
Yanick Champoux