[dancer-users] Starman large file upload

[Zahir Lalani]

Hi John

As I said, our back-end code does not get called for literally 1-2 mins. It seems the server goes into a massive swap space issue.

The post is using a standard jquery library (jQuery-File-Upload) and this uses multi-part file upload in an ajax post

This works really well with <300MB files - our 600MB test fails badly

Our setup is Apache as the front end proxy, which then forwards data to Starman on port 5000. Starman of course runs the Dancer code. As I said, the temp file is being created but at the end of the transfer Starman eats up a lot of resource but we don't see our code being called for ages. So not sure what code I should send you. If you need to know the dancer launch configs or apache - I can certainly send those.

We are also looking into chunked uploads, but are working out how we handle this at the dancer end - the chunks upload fine, but each one creates a new upload at the back end - we should be looking at the content range header or similar to prevent this - Anyone implemented this in Dancer before?

Thx again

Z

> -----Original Message-----
> From: dancer-users [mailto:dancer-users-bounces at dancer.pm] On Behalf
> Of John Stoffel
> Sent: 13 April 2017 21:40
> To: Perl Dancer users mailing list <dancer-users at dancer.pm>
> Subject: Re: [dancer-users] Starman large file upload
> 
> 
> Zahir> Hit a bit of a wall
> 
> Zahir> We are using a standard jquery uploader connected to Dancer2 and
> Zahir> that in general works great
> 
> Can you give us an example of your code so we can help debug this?
> But from my googling around, it's almost certainly a problem with how the
> file is written from the client, through starman, to dancer and then to disk.
> 
> Are you trying to parse forms or parts in the upload by any chance?
> 
> Zahir> We are testing uploading large files (600M) and having a lot of
> issues:
> 
> Zahir> We see the file uploading to /tmp as expected. At the end of the
> Zahir> upload, Starman has a lot of CPU and Mem usage and the defined
> Zahir> upload route (our code) does not get hit for several minutes. It
> Zahir> looks like Starman is loading this into memory and hitting swap
> Zahir> issues
> 
> So you need to show us how you're using starman and what it's config
> is before we can help here.
> 
> Just increasing memory won't solve the problem.  You need to figure out
> why starman is buffering the upload and how to get it out of the middle.
> 
> John
> _______________________________________________
> dancer-users mailing list
> dancer-users at dancer.pm
> http://lists.preshweb.co.uk/mailman/listinfo/dancer-users