Re: [Dancer-users] recommended way to return large dynamic data?
I guess you don't want to split it your response into pages (50 hits or so per page), you seem to want to stream a file to your users (without writing it to server hard disk). I looked into that a long while ago, got it to work in a test, but never used it in production. It's mentioned http://search.cpan.org/~xsawyerx/Dancer-1.3110/lib/Dancer.pm#send_file.
As you can see, streaming with send_file is possible for a while now. On Wed, Oct 17, 2012 at 4:24 AM, Maurice Mengel <mauricemengel@gmail.com>wrote:
I guess you don't want to split it your response into pages (50 hits or so per page), you seem to want to stream a file to your users (without writing it to server hard disk). I looked into that a long while ago, got it to work in a test, but never used it in production.
It's mentioned http://search.cpan.org/~xsawyerx/Dancer-1.3110/lib/Dancer.pm#send_file. _______________________________________________ Dancer-users mailing list Dancer-users@perldancer.org http://www.backup-manager.org/cgi-bin/listinfo/dancer-users
Hello Sawyer, sawyer x wrote, On 10/17/2012 03:51 PM:
As you can see, streaming with send_file is possible for a while now.
I really want to see, but I failed :) Several people suggested "send_file" (on and off the mailing list), but from what I gathered, "send_file" requires either a physical file (which I don't have), or a scalar-ref containing the entire buffer to be sent (which I don't have). I couldn't find a working example of using "send_file" while dynamically generating the output. Also, I'm using apache as a front-end, and I'm not sure about the "streaming" capabilities of the apache/psgi/plack/dancer stack in this situation - would it work ? Thanks, -gordon
On Wed, Oct 17, 2012 at 10:02 PM, Assaf Gordon <gordon@cshl.edu> wrote:
Several people suggested "send_file" (on and off the mailing list), but from what I gathered, "send_file" requires either a physical file (which I don't have), or a scalar-ref containing the entire buffer to be sent (which I don't have).
I couldn't find a working example of using "send_file" while dynamically generating the output.
True. This is not really supported well in Dancer. The stream support was added for streaming an actual file. The fact that we can fake files from in-memory scalars is just an added benefit. :) I intend to fully support streaming upload and download in Dancer 2.
Also, I'm using apache as a front-end, and I'm not sure about the "streaming" capabilities of the apache/psgi/plack/dancer stack in this situation - would it work ?
I'm not sure. I know that almost no servers (including Apache) support non-buffered file uploads, for example, so even if Dancer supports it (and it will), the servers will buffer it completely for you. I'm not sure how it is on downloads though. Having pure streamed download support might just be a feature strong enough for you to move to Dancer 2.
sawyer x wrote, On 10/17/2012 04:10 PM:
On Wed, Oct 17, 2012 at 10:02 PM, Assaf Gordon <gordon@cshl.edu <mailto:gordon@cshl.edu>> wrote:
Several people suggested "send_file" (on and off the mailing list), but from what I gathered, "send_file" requires either a physical file (which I don't have), or a scalar-ref containing the entire buffer to be sent (which I don't have).
I couldn't find a working example of using "send_file" while dynamically generating the output.
True. This is not really supported well in Dancer. The stream support was added for streaming an actual file. The fact that we can fake files from in-memory scalars is just an added benefit. :)
Got it to work. Thanks to everyone who helped. See a working demo here: http://cowbell.cancan.cshl.edu/dispatch.cgi/dynamic_content and the code here: https://github.com/agordon/dancer_bootstrap_fontawesome_template/blob/master... Hope this will save some time for other people.
Also, I'm using apache as a front-end, and I'm not sure about the "streaming" capabilities of the apache/psgi/plack/dancer stack in this situation - would it work ?
I'm not sure. I know that almost no servers (including Apache) support non-buffered file uploads, for example, so even if Dancer supports it (and it will), the servers will buffer it completely for you. I'm not sure how it is on downloads though.
From preliminary testing, it seems Apache doesn't buffer the data before sending it to the client. I was able to generate and transmit 1GB of data without either HTTPD or Perl consuming more memory than usual.
-gordon
Nice! :) On Wed, Oct 17, 2012 at 11:48 PM, Assaf Gordon <gordon@cshl.edu> wrote:
sawyer x wrote, On 10/17/2012 04:10 PM:
On Wed, Oct 17, 2012 at 10:02 PM, Assaf Gordon <gordon@cshl.edu <mailto: gordon@cshl.edu>> wrote:
Several people suggested "send_file" (on and off the mailing list), but from what I gathered, "send_file" requires either a physical file (which I don't have), or a scalar-ref containing the entire buffer to be sent (which I don't have).
I couldn't find a working example of using "send_file" while dynamically generating the output.
True. This is not really supported well in Dancer. The stream support was added for streaming an actual file. The fact that we can fake files from in-memory scalars is just an added benefit. :)
Got it to work. Thanks to everyone who helped.
See a working demo here: http://cowbell.cancan.cshl.edu/dispatch.cgi/dynamic_content
and the code here:
https://github.com/agordon/dancer_bootstrap_fontawesome_template/blob/master...
Hope this will save some time for other people.
Also, I'm using apache as a front-end, and I'm not sure about the
"streaming" capabilities of the apache/psgi/plack/dancer stack in this situation - would it work ?
I'm not sure. I know that almost no servers (including Apache) support
non-buffered file uploads, for example, so even if Dancer supports it (and it will), the servers will buffer it completely for you. I'm not sure how it is on downloads though.
From preliminary testing, it seems Apache doesn't buffer the data before sending it to the client. I was able to generate and transmit 1GB of data without either HTTPD or Perl consuming more memory than usual.
-gordon
You might be interested in this usage case: https://gist.github.com/3987355 It allows you to run async code in a fork, wait for it in a condition variable and send it asynchronously to the user. It's based on Gordon's (ab)use of my insane additions to the send_file() function in Dancer. s.
Beautiful! Now there's just one more thing I'm trying to do: have an automagic way to capture STDOUT and send it asynchronously back to the client (using async_send() ). This will allow using dancer to send output just like old cgi programs (and also execute external programs to generate big output). Something like Tie::STDOUT - but it only captures STDOUT inside perl code (no external programs) Or Capture::Tiny - but it seems to do buffering. So I'm not a sure how to implement it yet. -Gordon On Oct 31, 2012, at 10:34 AM, "sawyer x" <xsawyerx@gmail.com<mailto:xsawyerx@gmail.com>> wrote: You might be interested in this usage case: https://gist.github.com/3987355 It allows you to run async code in a fork, wait for it in a condition variable and send it asynchronously to the user. It's based on Gordon's (ab)use of my insane additions to the send_file() function in Dancer. s. _______________________________________________ dancer-users mailing list dancer-users@dancer.pm<mailto:dancer-users@dancer.pm> http://lists.preshweb.co.uk/mailman/listinfo/dancer-users
Output buffering is handled by the Perl process, so you can use the $| variable to control that. On Wed, Oct 31, 2012 at 4:50 PM, Gordon, Assaf <gordon@cshl.edu> wrote:
Beautiful!
Now there's just one more thing I'm trying to do: have an automagic way to capture STDOUT and send it asynchronously back to the client (using async_send() ). This will allow using dancer to send output just like old cgi programs (and also execute external programs to generate big output).
Something like Tie::STDOUT - but it only captures STDOUT inside perl code (no external programs) Or Capture::Tiny - but it seems to do buffering.
So I'm not a sure how to implement it yet.
-Gordon
On Oct 31, 2012, at 10:34 AM, "sawyer x" <xsawyerx@gmail.com<mailto: xsawyerx@gmail.com>> wrote:
You might be interested in this usage case: https://gist.github.com/3987355
It allows you to run async code in a fork, wait for it in a condition variable and send it asynchronously to the user. It's based on Gordon's (ab)use of my insane additions to the send_file() function in Dancer.
s. _______________________________________________ dancer-users mailing list dancer-users@dancer.pm<mailto:dancer-users@dancer.pm> http://lists.preshweb.co.uk/mailman/listinfo/dancer-users _______________________________________________ dancer-users mailing list dancer-users@dancer.pm http://lists.preshweb.co.uk/mailman/listinfo/dancer-users
participants (4)
-
Assaf Gordon -
Gordon, Assaf
-
Maurice Mengel -
sawyer x