Hi sawyer

We've been asked several times for streaming support. While we're built on PSGI and that allows you to separate that part into another layer (outside of Dancer, but within the same server instance), people wanted to be able to do streaming from inside Dancer - which makes sense, obviously.

Recently I've hacked on this and set up a new branch on the Dancer github repo: "feature/send_data_streaming". Right now it adds support for send_file() but I intend to refactor this out to another keyword.

I have a few bells and whistles here but before I show it, I want to explain the type of feedback I'm interested in: 1. Is it useful for you? 2. Are the examples here helpful? Is the syntax understandable? 3. If not, what would be a more suitable syntax, in your opinion? 4. Are there enough features? Do you need more power, more options? 5. How are you doing today? Feel free to share. :)

I'm fine but very busy with $work, how are you ? :) OK here is the first thought : I'm not sure it's a good idea to add more options and stuff to send_file ? If you want to stream stuff, it opens the door to a lot of options, and modes, as you demonstrates in your email. So what about send_stream instead of send_file ? My other comments assume you decide to stick with send_file. But I'd rather see send_stream() :)

The basics: Basically I've added the ability to ask send_file() to send the file (or straight content, as send_file() supports a scalar ref as pure data to send to the user) streamingly. It uses the PSGI spec for this, so it's 100% compatible, no magic of any kind. You can ask it to be streamed using a simple flag:     send_file( 'log.txt', streaming => 1 ); This will send the file one line at a time.

This syntax is fine for me, except that I don't think sending the file by default line by line is a good idea. Most of the time when you want to stream something, it's because you're sending a big binary data, where the concept of line doesn't exist. So I'd say by default, when specifying nothing, send the data chunk by chunk, with chunk size something reasonable and constant, like 42 KB ?

This also works on scalar refs:     send_file( \'my string to stream back', streaming => 1, ... );

agreed

Method callbacks: I wanted users to be able to control what happens before each line and after each line streamed. You can do that using method callbacks. These callbacks also get the line as a parameter so you can do stuff with it.

As Matthew Vickers said, callbacks => { before => sub { my $line = shift; say "About to send a line: $line" }, after => sub { my $line = shift; say "Just sent the line: $line" }, someother_callback => sub { ... } } is a nicer syntax imho.

However, this wasn't enough for me. What if I don't want to stream one line at a time. What if it's a binary and I want to stream it 512K at a time? You're able to do that too. You can override the method that gets the content's file handle and you can control what happens:

    send_file(         $binfile,         streaming => 1,         around_cb => sub {             my $content_fh = shift;             # now I can do whatever I want with it         },     );

Don't you need an additional argument, a ref on the sub to call, like in Moose 'around' keyword ? If not, then it's not an 'around', it's a wrapper. The $content_fh is pointing to the file you are trying to send, or the FH you should write to ? So instead of around_cb, something like data_send_callback, or a less crappy name actually :) Anyway do you see my point ? I assume this method gets called multiple time (once per chunk ?). If not, then it's not a callback. But I think it is :)

Complete control:

Here's another thought. While we're at it, how about we allow people to take over the ENTIRE operation? We can do that, using a method override. Take into account you should understand how PSGI streaming works, but this means you can really go all out.

    send_file(         $binfile,         streaming   => 1,         override_cb => sub {             my ( $respond, $response ) = @_;

            my $writer = $respond->( [ $newstatus, $newheaders ] );             $writer->write('some line');         },     );

Sounds good I guess. I'd rename it to something containing psgi (so the user knows it has to follow PSGI streaming )

This might seem scary. It is, a bit. Like I said, you should know what you're doing. And if you do - hell - you get ALL the control right there.

Things to note: 1. I want to refactor out the logic so it will be available in another keyword, like "send_streaming(...)". Will that be useful for anyone?

Ah yes definitely, I'd vote for this, so that send_file stays the simple reassuring method we all know. It's already stressful for a bunch of users to send a file instead of rendering a page. Don't make it scarrier :)

2. This might mean that you won't need to use "override_cb". 3. Do the callback names make sense? 4. Would you prefer getting the original method inside the "around_cb"? Should an around_cb with the content be named differently? What do you suggest? 5. I intend to add nonblocking streaming which will allow to run a request, answer it and run other stuff at the same request call.

I'm interested to see how you'd do that.

6. "Blocking streaming" doesn't block the server, just the specific request. Other requests aren't blocked.

Apologies for a long mail and thanks in advanced to anyone answering. :)

Have a great weekend, Sawyer.

_______________________________________________ Dancer-users mailing list Dancer-users@perldancer.org http://www.backup-manager.org/cgi-bin/listinfo/dancer-users