Hello guys! I have found that dancer's memory consumption is indeterminant. I am writing RESTful statistics service. Have discovered that dancer can consume lots more memory with one request than another with essentially same POST body size. Some tests: http://pastebin.com/yuy1ZKfn The output: http://pastebin.com/YDHqLzQh You can see that when I send ~12MB json string (~20MB hash in memory) dancer can consume ~ 24MB when data is sent as nested hash or ~81MB (4x!) when sent as-is. Tranfering dozens of MBs of statistics is normal in my service, so it can hit swap easily by clients mistake. What can I do to make dancer behave predictably in memory consumption? May be I am doing something wrong?
"Konstantin A. Pustovalov" <lamoz@adriver.ru> writes:
Hello guys!
I have found that dancer's memory consumption is indeterminant. I am writing RESTful statistics service. Have discovered that dancer can consume lots more memory with one request than another with essentially same POST body size. Some tests: http://pastebin.com/yuy1ZKfn The output: http://pastebin.com/YDHqLzQh
I, too, experience large memory usage (which doesn't seem to decrease) with non-trivial POST requests. When I investigated, I couldn't reproduce it, so I blamed my own code for some leakage and/or the perl lazy GC. Sorry, I know this doesn't help much.... -- Marco
On Tue, Oct 16, 2012 at 10:42:53AM +0200, Marco wrote:
I, too, experience large memory usage (which doesn't seem to decrease)
It doesn't decrease because perl doesn't release memory back to the OS. It will, however, re-use memory that it is no longer using. -- David Cantrell | A machine for turning tea into grumpiness
I have found that dancer's memory consumption is indeterminant. I believe I have found the source of my problem here - https://github.com/sukria/Dancer/blob/devel/lib/Dancer/Request.pm#L381 _build_params copies all first-level keys of huge POSTed hashes into params (there must be some more places where data is copied as memory consumption is more than 2x). Being newbie dancer I am not sure about further investigation. Is it a valid behavior? I think that only application/x-www-form-urlencoded// should be copied or this behavior should be switchable perhaps. Any ideas on how to live with that?
While this seems like a serious issue for you, and I'm sorry you're experiencing this, I still prefer that we focus on getting Dancer 2 out, and it will be easier to fix it on 2 than on 1. Making sure this doesn't happen to you on Dancer 2 will be a good point to start. :) On Tue, Oct 16, 2012 at 1:50 PM, Konstantin A. Pustovalov <lamoz@adriver.ru>wrote:
I have found that dancer's memory consumption is indeterminant.
I believe I have found the source of my problem here - https://github.com/sukria/Dancer/blob/devel/lib/Dancer/Request.pm#L381 _build_params copies all first-level keys of huge POSTed hashes into params (there must be some more places where data is copied as memory consumption is more than 2x). Being newbie dancer I am not sure about further investigation. Is it a valid behavior? I think that only application/x-www-form-urlencoded ** should be copied or this behavior should be switchable perhaps. Any ideas on how to live with that?
_______________________________________________ Dancer-users mailing list Dancer-users@perldancer.org http://www.backup-manager.org/cgi-bin/listinfo/dancer-users
participants (4)
-
David Cantrell -
Konstantin A. Pustovalov -
Marco -
sawyer x