[dancer-users] Streaming binary data without known size? Content-length problem.

[Andy Walker]

While $working on a RESTFul application, I came across a situation where I
needed to pack up a directory of files (already gzipped) and send them to
the user.  Rather than pulling everything into memory with Archive::Tar or
creating a temporary file with File::Temp that would meet an uncertain fate
after file_send depending on the deployment, I decided I would try and
forkexec using a pipe and just stream the tar to the user.

This was easy enough to accomplish with open and '-|' and then

exec ('/path/to/tar', '-c', '-f', '-' ...);

in the child.

After that, it was a matter of duplicating parts of send_file, and I think
this is where I got it wrong.  Testing with raw Telnet saw the binary data
come through, so I know that at least the piping worked, but  when I tested
the route as users would use it, the browser downloaded an empty file and
curl -v, mentioned something about "* Excess found in a non pipelined read:
excess = 7989".

I looked in the printed headers, and saw that it defined Content-Length: 0,
which I assume is part of the problem.  I tried NOT sending it using the
code below using:

$response->{headers}->remove_header('Content-Length');

but was unable to remove it, and I'm not sure that's what I want to do
anyway.  Hijacking the stream and just catting binary data doesn't seem to
be part of the spec, though I'd swear I've seen it done elswhere before,
but I thought I would pick the collective brains of the mailing list and
see.

I eventually ended up just using File::Temp::tmpnam and having tar output
to that file, which I then added to a var called 'cleanup' that an after
hook deletes.  Then I sent with file_send as normal.

So, I suppose my main question is: would this work if I were somehow able
to remove the Content-Length header?  I'm thinking not and, if not, should
I be using chunked transfer instead?  Does Dancer support this?

CODE:

sub send_data {
    Dancer::Continuation::Route::FileSent->new(
        return_value => _send_data(@_)
    )->throw
}

sub _send_data {
    my ($fh, %options) = @_;
    my $env = Dancer::SharedData->request->env;

    die "Must have streaming" if ! $env->{'psgi.streaming'};
    die "Must provide content_type\n" unless $options{content_type};
    die "Must provide filename\n" unless $options{filename};
    binmode $fh;

    my $response = Dancer::SharedData->response() ||
Dancer::Response->new();
    $response->header('Content-Type' => $options{content_type});
    $response->push_header('Content-Disposition' =>
        "attachment; filename=\"$options{filename}\""
    );

    $response->streamed( sub {
            my ( $status, $headers ) = @_;

            return sub {
                my $respond = shift;

                my $writer = $respond->( [
                    $status,
                    $headers,
                ] );

                my $content = $response->content;

                my $bytes = $options{'bytes'} || '43008';
                my $buf;

                while ( ( my $read = sysread $fh, $buf, $bytes ) != 0 ) {
                    $writer->write($buf);
                }
            };
        }
    );
    return $response if $response;
}
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.preshweb.co.uk/pipermail/dancer-users/attachments/20130509/b2e4251e/attachment.htm>