[Dancer-users] Feedback requested: streaming support

[sawyer x]

Hey everyone,

I've been working on a nice feature and I wanted to get some feedback from
the community before it's merged. We want to merge this by the beginning of
next week and release it, so *early feedback is important*. Don't let this
linger for long. In short: skim though the email, check the examples, let me
know if it makes sense. That is all.

We've been asked several times for streaming support. While we're built on
PSGI and that allows you to separate that part into another layer (outside
of Dancer, but within the same server instance), people wanted to be able to
do streaming from inside Dancer - which makes sense, obviously.

Recently I've hacked on this and set up a new branch on the Dancer github
repo: "feature/send_data_streaming". Right now it adds support for
send_file() but I intend to refactor this out to another keyword.

I have a few bells and whistles here but before I show it, I want to explain
the type of feedback I'm interested in:
1. Is it useful for you?
2. Are the examples here helpful? Is the syntax understandable?
3. If not, what would be a more suitable syntax, in your opinion?
4. Are there enough features? Do you need more power, more options?
5. How are you doing today? Feel free to share. :)

*The basics:*
Basically I've added the ability to ask send_file() to send the file (or
straight content, as send_file() supports a scalar ref as pure data to send
to the user) streamingly. It uses the PSGI spec for this, so it's 100%
compatible, no magic of any kind. You can ask it to be streamed using a
simple flag:
    send_file( 'log.txt', streaming => 1 );
This will send the file one line at a time.

This also works on scalar refs:
    send_file( \'my string to stream back', streaming => 1, ... );

*Method callbacks:*
I wanted users to be able to control what happens before each line and after
each line streamed. You can do that using method callbacks. These callbacks
also get the line as a parameter so you can do stuff with it.

    send_file(
        'log.txt',
        streaming => 1,
        before_cb => sub { my $line = shift; say "About to send a line:
$line" },
        after_cb  => sub { my $line = shift; say "Just sent the line: $line"
  },
    );

However, this wasn't enough for me. What if I don't want to stream one line
at a time. What if it's a binary and I want to stream it 512K at a time?
You're able to do that too. You can override the method that gets the
content's file handle and you can control what happens:

    send_file(
        $binfile,
        streaming => 1,
        around_cb => sub {
            my $content_fh = shift;
            # now I can do whatever I want with it
        },
    );

*Complete control:*

Here's another thought. While we're at it, how about we allow people to take
over the ENTIRE operation? We can do that, using a method override. Take
into account you should understand how PSGI streaming works, but this means
you can really go all out.

    send_file(
        $binfile,
        streaming   => 1,
        override_cb => sub {
            my ( $respond, $response ) = @_;

            my $writer = $respond->( [ $newstatus, $newheaders ] );
            $writer->write('some line');
        },
    );

This might seem scary. It is, a bit. Like I said, you should know what
you're doing. And if you do - hell - you get ALL the control right there.

Things to note:
1. I want to refactor out the logic so it will be available in another
keyword, like "send_streaming(...)". Will that be useful for anyone?
2. This might mean that you won't need to use "override_cb".
3. Do the callback names make sense?
4. Would you prefer getting the original method inside the "around_cb"?
Should an around_cb with the content be named differently? What do you
suggest?
5. I intend to add nonblocking streaming which will allow to run a request,
answer it and run other stuff at the same request call.
6. "Blocking streaming" doesn't block the server, just the specific request.
Other requests aren't blocked.

Apologies for a long mail and thanks in advanced to anyone answering. :)

Have a great weekend,
Sawyer.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.backup-manager.org/pipermail/dancer-users/attachments/20110818/8322a726/attachment.htm>