Problem with uwsgi CPU running at 100%

1

Hi Community,

Setup:

  • [Palo Alto] —> (RADIUS protocol) —> [Freeradius + PrivacyIdea vm]
  • PrivacyIdea: 3.11, running on AWS t3a.small (2GB RAM with 2vCPUs)

Problem:
When trying to auth, the /usr/bin/uwsgi service goes up to 100% and then times out after 10sec. Palo Alto errors ‘Invalid username or password’.

process:

/usr/bin/uwsgi --ini /usr/share/uwsgi/conf/default.ini --xmlconfig /etc/uwsgi/apps-enabled/privacyidea.xml

(Temporary) Solution:
Increase resources for AWS vm.

Logs:

/var/log/privacyidea/privacyidea.log

[2025-02-27 22:13:12,038][10752][127881809324992][INFO][privacyidea.lib.crypto:839] initializing HSM class: <class 'privacyidea.lib.security.default.DefaultSecurityModule'>
[2025-02-27 22:13:12,039][10752][127881809324992][INFO][privacyidea.lib.crypto:271] Initialized HSM object {'obj': <privacyidea.lib.security.default.DefaultSecurityModule object at 0x744ec5175960>}
[2025-02-27 22:13:12,210][10752][127881809324992][INFO][privacyidea.lib.resolvers.LDAPIdResolver:99] Could not import gssapi package. Kerberos authentication not available
[2025-02-27 22:13:12,419][10752][127881809324992][INFO][privacyidea.lib.user:271] user 'brad.pitt' found in resolver 'aws-eu-MyLDAP.org'
[2025-02-27 22:13:12,419][10752][127881809324992][INFO][privacyidea.lib.user:273] userid resolved to '21ab8339-e8c8-4868-982d-6e4b5ec5bab6' 
[2025-02-27 22:13:12,500][10752][127881809324992][INFO][privacyidea.lib.pooling:117] Created a new engine registry: <privacyidea.lib.pooling.SharedEngineRegistry object at 0x744ec4de33d0>
[2025-02-27 22:13:12,500][10752][127881809324992][INFO][privacyidea.lib.pooling:82] Creating a new engine and connection pool for key sqlaudit
[2025-02-27 22:13:23,626][10752][127881809324992][INFO][privacyidea.lib.user:447] User brad.pitt from realm MyLDAP.org tries to authenticate

/var/log/uwsgi/app/privacyidea.log

Thu Feb 27 21:59:46 2025 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /validate/check (ip 127.0.0.1) !!!
Thu Feb 27 21:59:46 2025 - uwsgi_response_writev_headers_and_body_do(): Broken pipe [core/writer.c line 306] during POST /validate/check (127.0.0.1)
OSError: write error
Thu Feb 27 21:59:47 2025 - ... monitored exception detected, respawning worker 2 (pid: 790)...
Thu Feb 27 21:59:47 2025 - Respawned uWSGI worker 2 (new pid: 5741)
Thu Feb 27 21:59:47 2025 - mapping worker 2 to CPUs: 1
The configuration name is: production
Additional configuration will be read from the file /etc/privacyidea/pi.cfg
Using PI_LOGLEVEL and PI_LOGFILE.
Using PI_LOGLEVEL 20.
Using PI_LOGFILE /var/log/privacyidea/privacyidea.log.
Thu Feb 27 21:59:49 2025 - WSGI app 0 (mountpoint='') ready in 2 seconds on interpreter 0x5fe6103f58d0 pid: 5741 (default app)

Question

Has anyone faced such a problem? The PrivacyIdea server is serving just a handful of users. At the time of debugging only single user was tested. In my opinion, fixing it by throwing more resources at it is just a workaround.

2

This can depend on your actual privacyIDEA configuration and used tokentypes.
Also, there are a lot of parameters to fiddle around with uwsgi and nginx.
Have you tried using apache with mod_wsgi just to analyse and identify a difference?

3

Confirming that after changing from nginx to apache, resource use ‘calmed’ down. I also changed the AWS PrivacyIdea instance to t2.small (1vCPU, 2GB) and so far so good. I use TOTP with SHA1.

However, initially it behaved fine under nginx but after some time, it got ‘hungry’. Let’s see what happens now after a few weeks.

I’m also observing that specifically one user (my username!) takes more time to process and then others. I tried to recreate it in the AD and also remove it from all the groups it’s in … but no difference :thinking: