No PI daemons after server reboot

thomascapote · November 12, 2020, 6:05pm

After running apt get update && apt get upgrade and rebooting the server, I’m getting 502 Bad Gateway error instead of the privacyIDEA login page. Also, there are zero PI daemons running now where, before the restart, there were several.

Did the installation script not install a systemd configuration for daemon auto-start?
How do I correctly do that now, without “breaking” anything?

Details

I followed the installation instructions here for Ubuntu 18.04 and Nginx. The installation worked reliably for two weeks, until this reboot.

Prior to the reboot, I thought to check whether and how many daemons existed.

thomas@mfa_proxy:~$ ps aux | grep 'priva[c]y'
privacy+  58503  0.0  0.2 237532  2188 ?        S    Oct30   1:05 /usr/bin/uwsgi --ini /usr/share/uwsgi/conf/default.ini --xmlconfig /etc/uwsgi/apps-enabled/privacyidea.xml --daemonize /var/log/uwsgi/app/privacyidea.log
privacy+  58699  0.0  3.6 290460 32720 ?        S    Oct30   0:13 /usr/bin/uwsgi --ini /usr/share/uwsgi/conf/default.ini --xmlconfig /etc/uwsgi/apps-enabled/privacyidea.xml --daemonize /var/log/uwsgi/app/privacyidea.log
privacy+  85360  0.0  2.0 370348 18456 ?        S    Nov06   0:14 /usr/bin/uwsgi --ini /usr/share/uwsgi/conf/default.ini --xmlconfig /etc/uwsgi/apps-enabled/privacyidea.xml --daemonize /var/log/uwsgi/app/privacyidea.log
privacy+ 132526  0.0  3.4 244548 30680 ?        S    00:58   0:01 /usr/bin/uwsgi --ini /usr/share/uwsgi/conf/default.ini --xmlconfig /etc/uwsgi/apps-enabled/privacyidea.xml --daemonize /var/log/uwsgi/app/privacyidea.log
privacy+ 133455  0.0  8.3 264552 74872 ?        S    01:55   0:02 /usr/bin/uwsgi --ini /usr/share/uwsgi/conf/default.ini --xmlconfig /etc/uwsgi/apps-enabled/privacyidea.xml --daemonize /var/log/uwsgi/app/privacyidea.log

After rebooting as follows…

thomas@mfa_proxy:~$ sudo shutdown -r now
Connection to 172.16.1.56 closed by remote host.
Connection to 172.16.1.56 closed.

… there are zero daemons running and attempts to access the privacyIDEA instances URL via browser return 502 Bad Gateway. This tells me that the WSGI socket doesn’t exist and/or has nothing “running on the other side”.

A search for daemons returns nothing:

thomas@mfa_proxy:~$ ps aux | grep 'privac[y]'
thomas@mfa_proxy:~$

Every time I refresh the browser, the following two lines are written to the Nginx error log (at /var/log/privacyidea/error.log):

2020/11/12 13:02:22 [crit] 1227#1227: *75 connect() to unix:/run/uwsgi/app/privacyidea/privacyidea.socket failed (2: No such file or directory) while connecting to upstream, client: 172.16.1.1, server: mfa_proxy, request: "GET / HTTP/1.1", upstream: "uwsgi://unix:/run/uwsgi/app/privacyidea/privacyidea.socket:", host: "privacyidea.ncsglobalinc.com"
2020/11/12 13:02:23 [crit] 1227#1227: *75 connect() to unix:/run/uwsgi/app/privacyidea/privacyidea.socket failed (2: No such file or directory) while connecting to upstream, client: 172.16.1.1, server: mfa_proxy, request: "GET /favicon.ico HTTP/1.1", upstream: "uwsgi://unix:/run/uwsgi/app/privacyidea/privacyidea.socket:", host: "privacyidea.ncsglobalinc.com", referrer: "https://privacyidea.ncsglobalinc.com/"

Clearly, the calls to the UWSGI socket are failing.

I’ll deeply appreciate any help getting the daemons back and configuring them to stay back after subsequent reboots.

thomascapote · November 12, 2020, 9:26pm

First, for anyone who has put any time into thinking about my issue… thank you!

I’ve found a solution, as follows. Hope this helps someone!

It turns out that the uwsgi package was not installed. Clearly, it was installed at some point, because my system had a directory /etc/uwsgi (which included a PI configuration file /etc/uwsgi/apps-available/privacyidea.xml; more on this below) and a logging directory at /var/log/uwsgi, but mysteriously was uninstalled (?!) prior to the last reboot. (It seems that processes run under that lost installation continued to run until the reboot, because the app was running fine until reboot.)

In any case, I (re-)installed the package with sudo apt install uwsgi. Repeated attempts to launch it continued to fail, without leaving any traces in any log files. Then I noticed a line in the XML config file at /etc/uwsgi/apps-available/privacyidea.xml that looks like this:

    <plugin>python</plugin>

A quick Google search showed I needed to install a uwsgi plugin specific to this directive: sudo apt install uwsgi-plugin-python3. NOTE the “3” here, as it is important. I installed uwsgi-plugin-python — without the 3 — and efforts to start the uwsgi service continued to fail silently! So, be sure to install the python3 version of the plugin. (This obviously depends on the OS version and privacyIDEA version…)

Attempts to start the uwsgi service with sudo service uwsgi start or sudo /etc/init.d/uwsgi start continued to fail silently until I realized that the XML configuration file at /etc/uwsgi/apps-available needed to be linked into the /etc/uwsgi/apps-enabled.

ln -s /etc/uwsgi/apps-available/privacyidea.xml /etc/uwsgi/apps-enabled

Absent this step, the uwsgi subsystem was failing silently because it was not actually trying to start any app at all!

At this point failures were logged to /var/log/uwsgi/privacyidea, but log entries simply signaled failure without giving useful information. Desperately random Google searching provided the final piece of the puzzle… App initialization was failing because the PID file couldn’t be written to /run/app/privacyidea/ because the parent directory, /run/uwsgi, did not exist.

These commands fixed this final issue.

sudo mkdir /run/uwsgi
sudo chown www-data:www-data /run/uwsgi

Importantly, the user and group in the chown command are taken from the privacyidea.xml configuration file in /etc/uwsgi/sites-enabled.

Do not forget (as I did) to restart both nginx and uwsgi, in that order.

sudo service nginx restart
sudo service uwsgi restart

I hope I haven’t missed anything. If you notice I have forgotten something, please advise and I’ll edit this post (if I’m allowed to).

cornelinux · November 12, 2020, 10:12pm

Hi @thomascapote
thank you for your detailed analysis.

Are you running on Ubuntu 18.04?

Do you remember, which version of privacyIDEA you were running before the update?

Thanks a lot
Cornelius

thomascapote · November 12, 2020, 10:33pm

I am indeed running Ubuntu 18.04:

thomas@mfa_proxy:~$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.5 LTS"

Unfortunately, I do not remember the version of PI from before the latest update. So sorry. If you can think of how to find it (logs of startup?), I’ll help however I can.

Thank you,
Thomas

cornelinux · November 13, 2020, 5:41am

Maybe you can find it in the apt history:

grep privacyidea /var/log/apt/history.log

thomascapote · November 13, 2020, 8:09pm

Of course. Thank you for the guidance. The current version installed is privacyidea:amd64 (3.4.1-1bionic, automatic), according to apt-show.

thomas@mfa_proxy:~$ sudo apt show privacyidea
Package: privacyidea
Version: 3.4.1-1bionic
...

Every mention in /var/log/apt/history.log shows this same version. The history shows that I installed and uninstalled a couple of times.

I first installed privacyidea-nginx but found that certain useful modules were missing in this installation of Nginx.
I then uninstalled that package and instead separately installed nginx-extras, mysql-server, mysql-server, and privacyidea. All the prior configurations and data were left in place, so this was painless (though anxiety-inducing!).

Maybe this somehow created the problem. Sorry to not have thought to mention these steps before.

Here is a list of other packages I installed explicitly.

# To replace what was uninstalled along with privacyidea-nginx.
privacyidea/stable,now 3.4.1-1bionic amd64 [installed]
mysql-client/bionic-updates,bionic-security,now 5.7.32-0ubuntu0.18.04.1 all [installed]
mysql-server/bionic-updates,bionic-security,now 5.7.32-0ubuntu0.18.04.1 all [installed]
nginx-extras/bionic-updates,bionic-security,now 1.14.0-0ubuntu1.7 amd64 [installed]  # includes key modules
openssh-server/bionic-updates,bionic-security,now 1:7.6p1-4ubuntu0.3 amd64 [installed]
uwsgi/bionic-updates,bionic-security,now 2.0.15-10.2ubuntu2.1 amd64 [installed]
uwsgi-plugin-python3/bionic-updates,bionic-security,now 2.0.15-10.2ubuntu2.1 amd64 [installed]

# Additional module(s) for Nginx.
lua-cjson/bionic,now 2.1.0+dfsg-2.1 amd64 [installed]

# For access to Microsoft SQL Server within user-resolver(s).
msodbcsql17/bionic,now 17.6.1.1-1 amd64 [installed]
mssql-tools/bionic,now 17.6.1.1-1 amd64 [installed]
unixodbc-dev/bionic,now 2.3.7 amd64 [installed]  # implies all other needed pkgs!

# Utilities for building pyodbc (inside PI's virtual environment).
build-essential/bionic,now 12.4ubuntu1 amd64 [installed]
python3-dev/bionic-updates,now 3.6.7-1~18.04 amd64 [installed]
python3-distutils/bionic-updates,now 3.6.9-1~18.04 all [installed]

# General command-line tools.
jq/bionic,now 1.5+dfsg-2 amd64 [installed]
tree/bionic,now 1.7.0-5 amd64 [installed]

Thankfully, all is working now, but I thank you very much for your attention to this matter.

cornelinux · November 13, 2020, 9:12pm

Probably s.th. strange has happened.

Given, the privacyidea-apache package is probably a bit more robust than privacyidea-nginx.
I was not sure, if the dependencies were set right,
but I checked again, it is totally fine, we depend on uwsgi-plugin-python3. But it should be basically fine.