Confidential Hi all Wondering if I can get some guidance. Our application is growing in use and have recently found that UWSGI runs out of worker resources - but when we use uwsgitop, its more that memory has shot up. The only way to resolve is a restart of uwsgi. That sounds like a memory leak, but I am not sure what tools can help me in that regard. I assumed that Perl was quite good at cleanup, but something is not right here. Any guidance appreciated Z
We have experienced the same issue -- memory leaks -- with our Dancer-based app. Not sure if it's Dancer, underlying Perl, or UWSGI itself, but haven't been able to debug and would certainly appreciate any tips! As a workaround, we use the reload-on-rss option on UWSGI to reload bloated workers. Here's a recipe, with sample settings: processes = 6 # have 6 workers loaded simultaneously reload-on-rss = 800 # if a worker reaches 800MB RSS allocated, reload it after it finishes. harakiri = 120 # (for good measure) hung workers die after 120 seconds The reload-on-rss setting obviously depends on the size of your app, with a bit of headroom for it to bloat until it gets restarted. Make sure (# of processes) * (reload-on-rss) < (available memory). On 3/29/2021 9:39:58 AM, Zahir Lalani <zahirlalani@oliver.agency> wrote: Confidential Hi all Wondering if I can get some guidance. Our application is growing in use and have recently found that UWSGI runs out of worker resources - but when we use uwsgitop, its more that memory has shot up. The only way to resolve is a restart of uwsgi. That sounds like a memory leak, but I am not sure what tools can help me in that regard. I assumed that Perl was quite good at cleanup, but something is not right here. Any guidance appreciated Z _______________________________________________ dancer-users mailing list dancer-users@lists.preshweb.co.uk https://lists.preshweb.co.uk/mailman/listinfo/dancer-users
Confidential That’s a good tip – had not seen that particular setting, we were getting ready to do a full scheduled restart across the servers! Will give your suggestion a go in the meantime Z From: dancer-users <dancer-users-bounces@lists.preshweb.co.uk> On Behalf Of Hermann Calabria Sent: 29 March 2021 18:52 To: dancer-users@lists.preshweb.co.uk; dancer-users@dancer.pm Subject: Re: [dancer-users] Debugging memory leaks We have experienced the same issue -- memory leaks -- with our Dancer-based app. Not sure if it's Dancer, underlying Perl, or UWSGI itself, but haven't been able to debug and would certainly appreciate any tips! As a workaround, we use the reload-on-rss option on UWSGI to reload bloated workers. Here's a recipe, with sample settings: processes = 6 # have 6 workers loaded simultaneously reload-on-rss = 800 # if a worker reaches 800MB RSS allocated, reload it after it finishes. harakiri = 120 # (for good measure) hung workers die after 120 seconds The reload-on-rss setting obviously depends on the size of your app, with a bit of headroom for it to bloat until it gets restarted. Make sure (# of processes) * (reload-on-rss) < (available memory). On 3/29/2021 9:39:58 AM, Zahir Lalani <zahirlalani@oliver.agency<mailto:zahirlalani@oliver.agency>> wrote: Confidential Hi all Wondering if I can get some guidance. Our application is growing in use and have recently found that UWSGI runs out of worker resources - but when we use uwsgitop, its more that memory has shot up. The only way to resolve is a restart of uwsgi. That sounds like a memory leak, but I am not sure what tools can help me in that regard. I assumed that Perl was quite good at cleanup, but something is not right here. Any guidance appreciated Z _______________________________________________ dancer-users mailing list dancer-users@lists.preshweb.co.uk<mailto:dancer-users@lists.preshweb.co.uk> https://lists.preshweb.co.uk/mailman/listinfo/dancer-users
On Mon, 29 Mar 2021 16:39:53 +0000 Zahir Lalani wrote:
That sounds like a memory leak, but I am not sure what tools can help me in that regard. I assumed that Perl was quite good at cleanup, but something is not right here.
Have you tried to investigate whether it's a memory leak in the application itself? I've had a few of these over the years caused by circular references. They're tricky to debug, but generally I use Devel::Gladiator to dump the arena on each page request, and then compare them over time. If it's an application memory leak then you will see the counts consistently increasing for a particular object, which then helps to pin down the cause. If it's a Perl internal memory leak (which is unlikely) then the counts will remain consistent. The other thing to bear in mind, as I understand it, is that Perl will not release memory back to the operating system once used. It will, however, use it itself in the future. So if you have a process that uses a lot of memory and then releases it, the Perl process itself will not release that memory and instead it will be used for the application's other needs as it runs. Finally, you can dump the contents of the running process which may provide an insight into what is using the memory. You'll need to dump the contents with gdb: gdb --pid=1234 (gdb) gcore (gdb) detach (gdb) exit Then see if something appears lots of times: strings core.1234 | sort | uniq -c | sort -nr | less Andy
Confidential Nice - thank you Andy Z
-----Original Message----- From: Andrew Beverley <andy@andybev.com> Sent: 30 March 2021 09:44 To: Perl Dancer users mailing list <dancer-users@lists.preshweb.co.uk> Cc: Zahir Lalani <ZahirLalani@oliver.agency>; Perl Dancer users mailing list <dancer-users@dancer.pm> Subject: Re: [dancer-users] Debugging memory leaks
On Mon, 29 Mar 2021 16:39:53 +0000 Zahir Lalani wrote:
That sounds like a memory leak, but I am not sure what tools can help me in that regard. I assumed that Perl was quite good at cleanup, but something is not right here.
Have you tried to investigate whether it's a memory leak in the application itself? I've had a few of these over the years caused by circular references.
They're tricky to debug, but generally I use Devel::Gladiator to dump the arena on each page request, and then compare them over time. If it's an application memory leak then you will see the counts consistently increasing for a particular object, which then helps to pin down the cause. If it's a Perl internal memory leak (which is unlikely) then the counts will remain consistent.
The other thing to bear in mind, as I understand it, is that Perl will not release memory back to the operating system once used. It will, however, use it itself in the future. So if you have a process that uses a lot of memory and then releases it, the Perl process itself will not release that memory and instead it will be used for the application's other needs as it runs.
Finally, you can dump the contents of the running process which may provide an insight into what is using the memory. You'll need to dump the contents with gdb:
gdb --pid=1234 (gdb) gcore (gdb) detach (gdb) exit
Then see if something appears lots of times:
strings core.1234 | sort | uniq -c | sort -nr | less
Andy
participants (3)
-
Andrew Beverley -
Hermann Calabria -
Zahir Lalani