PI 3.1dev2 + uwsgi + nginx : connect to PI's unix socket FAILs (111:Connection refused)?

I’ve installed

uwsgi --version
	2.0.18
python3 --version
	Python 3.6.5
nginx -v
	nginx version: nginx/1.17.3

on linux.

my UWSGI config works well here with CGI apps.

I’m working on setting it up with python/privacyIDEA.

PI 3.1dev2 has been pip installed,

virtualenv /opt/privacyidea
cd /opt/privacyidea
source bin/activate
pip install privacyidea==3.1dev2
...

pip show privacyidea | grep Version
	Version: 3.1.dev2

I set up UWSGI systemd templates, for one service per app usage (https://uwsgi-docs.readthedocs.io/en/latest/Systemd.html),

/etc/systemd/system/uwsgi-app@.service
	[Unit]
	Description=%i uWSGI app
	After=syslog.target

	[Service]
	ExecStart=/usr/sbin/uwsgi \
	 --ini /etc/uwsgi/apps-available/%i.ini \
	 --socket /run/uwsgi/%i.sock
	User=wwwrun-%i
	Group=www
	Restart=on-failure
	KillSignal=SIGQUIT
	Type=notify
	StandardError=syslog
	NotifyAccess=all

/etc/systemd/system/uwsgi-app@.socket
	[Unit]
	Description=Socket for uWSGI app %i

	[Socket]
	ListenStream=/run/uwsgi/%i.sock
	SocketUser=wwwrun-%i
	SocketGroup=www
	SocketMode=0660

	[Install]
	WantedBy=sockets.target

The uwsgi PI-config,

/etc/uwsgi/apps-available/privacyidea.ini
	[uwsgi]
	master = true
	cheap = true
	idle = 600
	die-on-idle = true
	processes = 8
	harakiri = 3600
	reload-mercy = 8
	cpu-affinity = 1
	max-requests = 2000
	limit-as = 512
	reload-on-as = 256
	reload-on-rss = 192
	no-orphans = true
	vacuum = true
	logger     = file:/var/log/uwsgi/error.log
	req-logger = file:/var/log/uwsgi/request.log
	stats = /run/uwsgi/stats.socket

	manage-script-name = true
	plugins = python3,logfile
	wsgi-file = /etc/privacyidea/privacyideaapp.wsgi

the referenced wsgi-file

/etc/privacyidea/privacyideaapp.wsgi
	import sys
	sys.stdout = sys.stderr
	from privacyidea.app import create_app
	application = create_app(config_name="production", config_file="/etc/privacyidea/pi.cfg")

and the PI config

/etc/privacyidea/pi.cfg
	import logging
	SUPERUSER_REALM = ['super']
	SQLALCHEMY_DATABASE_URI = 'sqlite:////etc/privacyidea/data.sqlite'
	SECRET_KEY = 't0p s3cr3t'
	PI_PEPPER = "Never know..."
	PI_ENCFILE              = '/etc/privacyidea/enckey'
	PI_AUDIT_KEY_PRIVATE    = '/etc/privacyidea/private.pem'
	PI_AUDIT_KEY_PUBLIC     = '/etc/privacyidea/public.pem'
	PI_AUDIT_SQL_TRUNCATE = True
	PI_ENGINE_REGISTRY_CLASS = "shared"
	PI_AUDIT_POOL_SIZE = 20
	PI_LOGLEVEL = logging.DEBUG
	PI_LOGFILE = "/var/log/privacyidea/privacyidea.log"

nginx config includes

upstream uwsgi_privacyidea {
        server unix:/run/uwsgi/privacyidea.sock;
}

server {
	listen 10.0.0.1:8888 ssl http2;
	server_name pi.pgnd.loc;
	...
	location / {
		try_files $uri @privacyidea;
	}
	location @privacyidea {
		internal;
		gzip off;
		include uwsgi_params;
		uwsgi_param UWSGI_SCHEME $scheme;
		uwsgi_param SERVER_SOFTWARE nginx/$nginx_version;
		uwsgi_pass uwsgi_privacyidea;
	}
}

The app socket’s active

systemctl start uwsgi-app@privacyidea.socket
systemctl status  uwsgi-app@privacyidea.socket -l
	● uwsgi-app@privacyidea.socket - Socket for uWSGI app privacyidea
	   Loaded: loaded (/etc/systemd/system/uwsgi-app@.socket; enabled; vendor preset: disabled)
	   Active: active (listening) since Mon 2019-08-19 10:11:44 PDT; 103ms ago
	   Listen: /run/uwsgi/privacyidea.sock (Stream)

ls -al /run/uwsgi/
	total 0
	drwxr-xr-x  2 root               root   80 Aug 19 10:11 ./
	drwxr-xr-x 47 root               root 1.4K Aug 19 08:52 ../
	srw-rw----  1 wwwrun-privacyidea www     0 Aug 19 10:11 privacyidea.sock=

after restarting the webserver

systemctl restart nginx

access to

https://pi.pgnd.loc:8888

returns

502 Bad Gateway

& in nginx logs

2019/08/19 10:05:18 [error] 2812#2812: *46 connect() to unix:/run/uwsgi/privacyidea.sock failed (111: Connection refused) while connecting to upstream, client: 10.0.0.9, server: pi.pgnd.loc, request: "GET / HTTP/2.0", upstream: "uwsgi://unix:/run/uwsgi/privacyidea.sock:", host: "pi.pgnd.loc:8888"

I’m unclear as to why the connetion’s refused.

So far, haven’t found the right way to debug/log the problem …

Have I missed, or misconfigured, some req’d config?
Any hints as to how/where to troubleshoot this?

Welcome to the privacyIDEA community.
I am sorry, I did not read all your details. I am also not the nginx expert.
But you might look at the configs for our Ubuntu nginx setup here


which might give you a clue.

Check which distribution you are running on if it could be an selinux issue.

Welcome to the community! You should probably have a look at the uwsgi logfile, which will tell you why uwsgi is unable to load the privacyIDEA WSGI app.

1 Like

@cornelinux

Alas, that link’s where I started with this process.

There’s something unique/privacyidea-specific to my setup – atm PI is the only uwsgi app with which I’m having any issues.

No, there’s no selinux in place …

@fredreichbier

I was getting nothing of any use in my logfile until I DISabled

	PI_LOGLEVEL  = 10
	PI_LOGFILE   = '/var/log/privacyidea/privacyidea.log'
-	PI_LOGCONFIG = '/etc/privacyidea/logging.cfg'

Not clear what my problem there is; I’d cribbed it from,

https://privacyidea.readthedocs.io/en/latest/installation/system/logging.html

… I’ll deal with that later …

NOW, my logs show

	/var/log/uwsgi/error.log

		*** Starting uWSGI 2.0.18 (64bit) on [Tue Aug 20 08:45:27 2019] ***
		...
		 *** WARNING: you have enabled harakiri without post buffering. Slow upload could be rejected on post-unbuffered webservers *** 

(don’t know what that’s about … as in nginx, “uwsgi_request_buffering on;” is the default, and it’s not DISabled here)

	...
	- SystemD socket activation detected -
	uwsgi socket 1 attached to UNIX address /run/uwsgi/privacyidea.sock fd 3
	Python version: 3.6.5 (default, Mar 31 2018, 19:45:04) [GCC]
	Python main interpreter initialized at 0x55b16b657b00
	python threads support enabled
	your server socket listen backlog is limited to 100 connections
	your mercy for graceful operations on workers is 60 seconds
	mapped 364600 bytes (356 KB) for 4 cores
	*** Operational MODE: preforking ***
	Traceback (most recent call last):
	  File "/etc/privacyidea/privacyideaapp.wsgi", line 4, in <module>
	    from privacyidea.app import create_app
	ModuleNotFoundError: No module named 'privacyidea'
	unable to load app 0 (mountpoint='') (callable not found or import error)
	*** no app loaded. going in full dynamic mode ***
	*** uWSGI is running in multiple interpreter mode ***
	spawned uWSGI master process (pid: 38993)
	bind(): Permission denied [core/socket.c line 230]
	...brutally killing workers...
	VACUUM WARNING: unix socket /run/uwsgi/privacyidea.sock changed inode. Skip removal
	unlink(): No such file or directory [core/uwsgi.c line 1673]
	...

re-reading @cornelinux 's link, I needed to add a missing

	/etc/uwsgi/apps-available/privacyidea.ini
		...
+		venv = /opt/privacyidea
		...

for a fix.

now, on URL access,

	tail -f /var/log/uwsgi/error.log
		...
		*** Operational MODE: preforking ***
		The configuration name is: production
		Additional configuration can be read from the file /etc/privacyidea/pi.cfg
		The config file specified in PI_LOGCONFIG does not exist.
		Could not use PI_LOGCONFIG. Using PI_LOGLEVEL and PI_LOGFILE.
		Using PI_LOGLEVEL 10.
		Using PI_LOGFILE /var/log/privacyidea/privacyidea.log.
		WSGI app 0 (mountpoint='') ready in 3 seconds on interpreter 0x55dceb26a8d0 pid: 41996 (default app)
		*** uWSGI is running in multiple interpreter mode ***
		spawned uWSGI master process (pid: 41996)
		bind(): Permission denied [core/socket.c line 230]
		...brutally killing workers...
		VACUUM WARNING: unix socket /run/uwsgi/privacyidea.sock changed inode. Skip removal
		unlink(): No such file or directory [core/uwsgi.c line 1673]

so, a permission problem.

now to figure out why …

pebkac!

	ls -al /run/uwsgi/
		total 0
!!		drwxr-xr-x  2 root               root   80 Aug 20 09:19 ./
		drwxr-xr-x 46 root               root 1.4K Aug 20 09:19 ../
		...
		srw-rw----  1 wwwrun-privacyidea www     0 Aug 20 09:19 privacyidea.sock=

that’s been inadvertently changed …

fixing that

chmod 770 /run/uwsgi
chown wwwrun:www /run/uwsgi

does the trick

1 Like