Hi all, I have an annoyingly intermittent problem where on successful authentication the session info disappears and the user is endlessly redirected to the login page. It happens about once a month on a site with 10-20 logins daily. Once it happens to a user, it keeps happening for that user only. A restart of the app usually fixes the issue, and I am yet to see someone who had this problem to report it again at a later stage. Had about 10 cases so far in total. The cookie is generated on accessing the login page, stored in the browser and sent with the login POST request so I can verify cookies are enabled. The user is then authenticated, data added to the session including an 'authenticated' flag and they are redirected to their dashboard. Right before the redirection the session data is present. They make a request for the dashboard following the redirect, this triggers a session authenticated check which fails as no data is available for the session. They are then redirected for authentication and so on in a loop. If I restart the app the issue vanishes. hook before => sub { # <snip> if (!session('authenticated')) { forward '/login'; } } post '/login' => sub { # user authentication and session storage here, then the debug code and the redirecton: my $authenticated = session('authenticated'); $log->info("User authenticated ($authenticated), redirecting to $target_route"); return redirect $target_route; }; I don't have an explicit check on cookie size - not sure how to check it doesn't exceed 4K - but find it unlikely this is the issue. If that was the problem it should be happening persistently for a user regardless of restarts etc? Hope all this makes sense, I am looking for general advice on how to troubleshoot this but any ideas at all will be very much appreciated. Nik
Nik We have the same issue - we have multiple servers and hundreds of logins per day. We believe something is killing a starman worker and then that user gets bounced to another server and so loses the session. We are trying to check logs to see what errors would lead us to the issue but no luck yet - we have to schedule a restart when we begin to run low on active workers Z
-----Original Message----- From: dancer-users <dancer-users-bounces@dancer.pm> On Behalf Of Nikola Mitev Sent: 21 November 2018 17:17 To: dancer-users@dancer.pm Subject: [dancer-users] Debugging sessions
Hi all,
I have an annoyingly intermittent problem where on successful authentication the session info disappears and the user is endlessly redirected to the login page. It happens about once a month on a site with 10-20 logins daily. Once it happens to a user, it keeps happening for that user only. A restart of the app usually fixes the issue, and I am yet to see someone who had this problem to report it again at a later stage. Had about 10 cases so far in total.
The cookie is generated on accessing the login page, stored in the browser and sent with the login POST request so I can verify cookies are enabled. The user is then authenticated, data added to the session including an 'authenticated' flag and they are redirected to their dashboard. Right before the redirection the session data is present. They make a request for the dashboard following the redirect, this triggers a session authenticated check which fails as no data is available for the session. They are then redirected for authentication and so on in a loop. If I restart the app the issue vanishes.
hook before => sub { # <snip> if (!session('authenticated')) { forward '/login'; } }
post '/login' => sub { # user authentication and session storage here, then the debug code and the redirecton: my $authenticated = session('authenticated'); $log->info("User authenticated ($authenticated), redirecting to $target_route"); return redirect $target_route; };
I don't have an explicit check on cookie size - not sure how to check it doesn't exceed 4K - but find it unlikely this is the issue. If that was the problem it should be happening persistently for a user regardless of restarts etc?
Hope all this makes sense, I am looking for general advice on how to troubleshoot this but any ideas at all will be very much appreciated.
Nik _______________________________________________ dancer-users mailing list dancer-users@dancer.pm http://lists.preshweb.co.uk/mailman/listinfo/dancer-users
Zahir Wouldn't that cause the user to lose the session but be fine on re- authentication (on the new worker)? In my case the user fails to login repeatedly every time they try until I restart the app. I am running with 100 workers, if anything I should probably reduce that as I doubt simultaneous users ever peak beyond 50. Nik On Wed, 2018-11-21 at 18:16 +0000, Zahir Lalani wrote:
Nik
We have the same issue - we have multiple servers and hundreds of logins per day. We believe something is killing a starman worker and then that user gets bounced to another server and so loses the session. We are trying to check logs to see what errors would lead us to the issue but no luck yet - we have to schedule a restart when we begin to run low on active workers
Z
-----Original Message----- From: dancer-users <dancer-users-bounces@dancer.pm> On Behalf Of Nikola Mitev Sent: 21 November 2018 17:17 To: dancer-users@dancer.pm Subject: [dancer-users] Debugging sessions
Hi all,
I have an annoyingly intermittent problem where on successful authentication the session info disappears and the user is endlessly redirected to the login page. It happens about once a month on a site with 10-20 logins daily. Once it happens to a user, it keeps happening for that user only. A restart of the app usually fixes the issue, and I am yet to see someone who had this problem to report it again at a later stage. Had about 10 cases so far in total.
The cookie is generated on accessing the login page, stored in the browser and sent with the login POST request so I can verify cookies are enabled. The user is then authenticated, data added to the session including an 'authenticated' flag and they are redirected to their dashboard. Right before the redirection the session data is present. They make a request for the dashboard following the redirect, this triggers a session authenticated check which fails as no data is available for the session. They are then redirected for authentication and so on in a loop. If I restart the app the issue vanishes.
hook before => sub { # <snip> if (!session('authenticated')) { forward '/login'; } }
post '/login' => sub { # user authentication and session storage here, then the debug code and the redirecton: my $authenticated = session('authenticated'); $log->info("User authenticated ($authenticated), redirecting to $target_route"); return redirect $target_route; };
I don't have an explicit check on cookie size - not sure how to check it doesn't exceed 4K - but find it unlikely this is the issue. If that was the problem it should be happening persistently for a user regardless of restarts etc?
Hope all this makes sense, I am looking for general advice on how to troubleshoot this but any ideas at all will be very much appreciated.
Nik _______________________________________________ dancer-users mailing list dancer-users@dancer.pm http://lists.preshweb.co.uk/mailman/listinfo/dancer-users
_______________________________________________ dancer-users mailing list dancer-users@dancer.pm http://lists.preshweb.co.uk/mailman/listinfo/dancer-users
On Nov 21, 2018, at 10:17 AM, Nikola Mitev <nik@mitev.eu> wrote:
It happens about once a month on a site with 10-20 logins daily. Once it happens to a user, it keeps happening for that user only.
You haven’t said which engines you’re using. It might be simplest to just post the config.yml file and the $environment.yml file it loads, if any. It might also help to know the HTTP server configuration. Apache vs Starman, FastCGI vs HTTP proxying, etc. I suspect you’ve got a parallel access locking problem that only shows up when two users happen to hit the app at the same time, but that’s just speculation on the information you’ve given so far.
I don't have an explicit check on cookie size - not sure how to check it doesn't exceed 4K
Unless you’re using Dancer2::Session::Cookie, I don’t see how that can be the case. That’s the whole point of separating the notion of cookie and session: all of the real per-user data goes into the Session object, and the cookie is used only to look that Session object up. But you don’t need to guess: any modern browser’s developer toolset will give you the cookie value. It should be just a small-ish blob of hex noise.
I am looking for general advice on how to troubleshoot this but any ideas at all will be very much appreciated.
What happens if you switch to Dancer2::Session::YAML? It’s slower, but for volumes as low as yours, it shouldn’t matter. The benefit is that you can then inspect the session objects directly when this happens.
On Wed, 2018-11-21 at 11:22 -0700, Warren Young wrote:
On Nov 21, 2018, at 10:17 AM, Nikola Mitev <nik@mitev.eu> wrote:
It happens about once a month on a site with 10-20 logins daily. Once it happens to a user, it keeps happening for that user only. You haven’t said which engines you’re using. It might be simplest to just post the config.yml file and the $environment.yml file it loads, if any.
It might also help to know the HTTP server configuration. Apache vs Starman, FastCGI vs HTTP proxying, etc.
I am using Starman with Nginx in front of it, session engine is Cookie: session: 'Cookie' engines: session: Cookie: secret_key: 'removed' cookie_name: "removed" is_secure: 1 is_http_only: 1 with_request_address: 1 default_duration: 604800
I suspect you’ve got a parallel access locking problem that only shows up when two users happen to hit the app at the same time, but that’s just speculation on the information you’ve given so far.
Would that lock persist for the unlucky user until an app restart, or would it only cause an issue for the request which collided with that of another user?
I don't have an explicit check on cookie size - not sure how to check it doesn't exceed 4K Unless you’re using Dancer2::Session::Cookie, I don’t see how that can be the case. That’s the whole point of separating the notion of cookie and session: all of the real per-user data goes into the Session object, and the cookie is used only to look that Session object up.
That's what I am using and the docs mention size over 4K could cause issues.
But you don’t need to guess: any modern browser’s developer toolset will give you the cookie value. It should be just a small-ish blob of hex noise.
I was thinking more about checking it server-side. Checked it in Chrome, it is less than 1K.
I am looking for general advice on how to troubleshoot this but any ideas at all will be very much appreciated. What happens if you switch to Dancer2::Session::YAML? It’s slower, but for volumes as low as yours, it shouldn’t matter. The benefit is that you can then inspect the session objects directly when this happens.
Haven't tried it yet, but will give it a go. Thanks for all the input! Nik
On Nov 22, 2018, at 1:23 AM, Nikola Mitev <nik@mitev.eu> wrote:
I am using Starman with Nginx in front of it
That allows parallel access, which means every code path in your app that touches global state needs to be protected against simultaneous modification, else you risk data corruption, which could explain your symptom. “Global state” means global variables in the Perl app, files on the server, database records, shared memory, etc. Basically, anything that persists beyond the running of each Dancer route handler call. For such a low usage volume app, I’d recommend switching to a simple single-threaded HTTP server such as HTTP::Server::Simple. HTTP servers that can handle multiple simultaneous connections only help you when you often have multiple users making dynamic requests at the same time. When that happens to a single-threaded HTTP server, the second user just waits for the first to finish their request, so problems like this cannot happen. That does mean you have a potential scalability problem, but I think too many people think they’re going to be the next Google and therefore need to architect their apps accordingly. I recommend keeping the Nginx front end, because it lets you offload static content hits to it. Without that, even a single user can hit the Dancer app multiple times in parallel, since then the Dancer app also has to serve all of the static assets, too. With a low-volume web app, each app hit is usually one dynamic pull plus multiple static pulls, so it doesn’t matter that you’re using a single-threaded HTTP server if you’ve got a front-end proxy like Nginx handling all of the static pulls.
On Nov 22, 2018, at 12:54 PM, Warren Young <warren@etr-usa.com> wrote:
On Nov 22, 2018, at 1:23 AM, Nikola Mitev <nik@mitev.eu> wrote:
I am using Starman with Nginx in front of it
...every code path in your app that touches global state needs to be protected…
To be clear, by “touch” I mean “modify”. If your app is fully read-only, then none of what I wrote matters.
Thank you for the suggestion, it makes a lot of sense for me as the app does write to a database and the risk of data corruption is certainly not worth the efficiency gains at the current load (which is very unlikely to increase drastically). I will definitely look into a single threaded server - whether it will resolve the session issues we will wait and see :) Nik On Thu, 2018-11-22 at 12:55 -0700, Warren Young wrote:
On Nov 22, 2018, at 12:54 PM, Warren Young <warren@etr-usa.com> wrote:
On Nov 22, 2018, at 1:23 AM, Nikola Mitev <nik@mitev.eu> wrote:
I am using Starman with Nginx in front of it
...every code path in your app that touches global state needs to be protected…
To be clear, by “touch” I mean “modify”. If your app is fully read- only, then none of what I wrote matters. _______________________________________________ dancer-users mailing list dancer-users@dancer.pm http://lists.preshweb.co.uk/mailman/listinfo/dancer-users
I doubt this is related, but DPAE::Database had/has a DB handle caching issue which creates a MySQL connection issue, see https://github.com/PerlDancer/Dancer2-Plugin-Auth-Extensible-Provider-Databa... It's possible that something in your app (at the "system" module level) is caching something badly (although it wouldn't explain why it only affects one user...) -N On Fri, 23 Nov 2018 at 07:38, Nikola Mitev <nik@mitev.eu> wrote:
Thank you for the suggestion, it makes a lot of sense for me as the app does write to a database and the risk of data corruption is certainly not worth the efficiency gains at the current load (which is very unlikely to increase drastically).
I will definitely look into a single threaded server - whether it will resolve the session issues we will wait and see :)
Nik
On Thu, 2018-11-22 at 12:55 -0700, Warren Young wrote:
On Nov 22, 2018, at 12:54 PM, Warren Young <warren@etr-usa.com> wrote:
On Nov 22, 2018, at 1:23 AM, Nikola Mitev <nik@mitev.eu> wrote:
I am using Starman with Nginx in front of it
...every code path in your app that touches global state needs to be protected…
To be clear, by “touch” I mean “modify”. If your app is fully read- only, then none of what I wrote matters. _______________________________________________ dancer-users mailing list dancer-users@dancer.pm http://lists.preshweb.co.uk/mailman/listinfo/dancer-users
dancer-users mailing list dancer-users@dancer.pm http://lists.preshweb.co.uk/mailman/listinfo/dancer-users
On Fri, 2018-11-23 at 12:16 +1100, Nathan Bailey wrote:
I doubt this is related, but DPAE::Database had/has a DB handle caching issue which creates a MySQL connection issue, see https://github.com/PerlDancer/Dancer2-Plugin-Auth-Extensible- Provider-Database/issues/3
I do use cached connections to the database, but the database is on localhost so it should only go away if Postgres crashes? If id did crash I will know about it :)
It's possible that something in your app (at the "system" module level) is caching something badly (although it wouldn't explain why it only affects one user...)
Exactly - for everyone else all works as usual. Nik
-N
On Fri, 23 Nov 2018 at 07:38, Nikola Mitev <nik@mitev.eu> wrote:
Thank you for the suggestion, it makes a lot of sense for me as the app does write to a database and the risk of data corruption is certainly not worth the efficiency gains at the current load (which is very unlikely to increase drastically).
I will definitely look into a single threaded server - whether it will resolve the session issues we will wait and see :)
Nik
On Thu, 2018-11-22 at 12:55 -0700, Warren Young wrote:
On Nov 22, 2018, at 12:54 PM, Warren Young <warren@etr-usa.com> wrote:
On Nov 22, 2018, at 1:23 AM, Nikola Mitev <nik@mitev.eu> wrote:
I am using Starman with Nginx in front of it ...every code path in your app that touches global state needs to be protected… To be clear, by “touch” I mean “modify”. If your app is fully read- only, then none of what I wrote matters.
dancer-users mailing list dancer-users@dancer.pm http://lists.preshweb.co.uk/mailman/listinfo/dancer-users
dancer-users mailing list dancer-users@dancer.pm http://lists.preshweb.co.uk/mailman/listinfo/dancer-users
_______________________________________________ dancer-users mailing list dancer-users@dancer.pm http://lists.preshweb.co.uk/mailman/listinfo/dancer-users
Hello all I hope someone can help. We are finally making the move from Centos to Ubuntu and are finding we are needing to make quite a few adjustments to the code base due to the version of perl being a lot newer. However, one issue I cannot seem to resolve is the following Test File: #!/usr/bin/perl use lib '.'; use Dancer2 appname => 'archive_assets'; use strict; use warnings; use Data::Dumper; use TryCatch; try { say "1234"; } catch { print "Something went wrong \n"; } Perl -c output [archive_assets:1475837] core @2023-06-29 17:29:54> Built config from files: /var/www/OMG/tools/archive_assets/environments/development.yml in (eval 274) l. 910 Couldn't find declarator 'try' at /usr/lib/x86_64-linux-gnu/perl5/5.34/Devel/Declare/Context/Simple.pm line 47. Devel::Declare::Context::Simple::skip_declarator(TryCatch=HASH(0x55fcf9da2b50)) called at /usr/lib/x86_64-linux-gnu/perl5/5.34/TryCatch.pm line 83 TryCatch::_parse_try("TryCatch", "main", "try", 0) called at /usr/lib/x86_64-linux-gnu/perl5/5.34/TryCatch.pm line 44 TryCatch::__ANON__("try", 0) called at /usr/lib/x86_64-linux-gnu/perl5/5.34/Devel/Declare.pm line 277 Devel::Declare::linestr_callback("const", "try", 0) called at z.pl line 11 I have tried hunting for an answer and it seems its an issue with Devel::Declare? Any ideas? Z
On Nov 22, 2018, at 1:37 PM, Nikola Mitev <nik@mitev.eu> wrote:
it makes a lot of sense for me as the app does write to a database
The fact that it does is not a guarantee of corruption in parallel execution, just a risk. If you’re using an ACID-compliant database and are doing transactions properly, then you’ve removed the risk of multiple parallel writes stomping on each other. The thing is, you have to do that in *every* such path. All queries, all file I/O, all network I/O… If there’s a missing lock in any of those paths, there’s a possible data race. So, you have two choices: 1. Find and fix all the data races, or 2. Construct your app in such a way that data races are impossible. Choking everything down to a single thread at the HTTP layer is one of the entry points to path #2.
participants (4)
-
Nathan Bailey -
Nikola Mitev -
Warren Young -
Zahir Lalani