Deploying Python Web Applications with nginx and uWSGI Emperor

pyweb

A guide about Python, web apps for Ubuntu, Debian, Fedora, CentOS, AlmaLinux, Rocky Linux, Arch Linux

You’ve just written a great Python web application. Now, you want to share it with the world. In order to do that, you need a server, and some software to do that for you.

The following is a comprehensive guide on how to accomplish that, on multiple Linux-based operating systems, using nginx and uWSGI Emperor. It doesn’t force you to use any specific web framework — Flask, Django, Pyramid, Bottle will all work. Written for Ubuntu, Debian, Fedora, CentOS 7, Alma Linux, Rocky Linux and Arch Linux (should be helpful for other systems, too). Now with an Ansible Playbook.

Revision 8 (2022-02-20): works with Fedora 35, AlmaLinux 8, RockyLinux 8

CI status for the associated Ansible Playbook: ci-status

For easy linking, I set up some aliases: https://go.chriswarrick.com/pyweb and https://go.chriswarrick.com/uwsgi-tut (powered by a Django web application, deployed with nginx and uWSGI!).

Prerequisites

In order to deploy your web application, you need a server that gives you root and ssh access — in other words, a VPS (or a dedicated server, or a datacenter lease…). If you’re looking for a great VPS service for a low price, I recommend Hetzner Cloud (reflink [1]), which offers a pretty good entry-level VPS for €3.49 + VAT / month (with higher plans available for equally good prices). If you want to play along at home, without buying a VPS, you can create a virtual machine on your own, or use Vagrant with a Vagrant box for Fedora 35 (fedora/35-cloud-base).

Your server should also run a modern Linux-based operating system. This guide was written and tested on:

  • Ubuntu 18.04 LTS, 20.04 LTS or newer

  • Debian 10 (buster), 11 (bullseye) or newer

  • Fedora 33 or newer (with SELinux enabled and disabled)

  • CentOS 7 (with SELinux enabled and disabled) — manual guide should also work on RHEL 7.

  • AlmaLinux 8 (with SELinux enabled and disabled) — manual guide should also work on RHEL 8. Referred to as “EL8” collectively with Rocky Linux.

  • Rocky Linux 8 (with SELinux enabled and disabled) — manual guide should also work on RHEL 8. Referred to as “EL8” collectively with AlmaLinux.

  • Arch Linux

Debian 8 (jessie) 9 (stretch), Ubuntu 16.04 LTS, and Fedora 24 through 32 are not officially supported, even though they still probably work.

What if you’re using Docker? The story is a bit complicated, and this guide does not apply, but do check the Can I use Docker? at the end of this post for some hints on how to approach it.

Users of other Linux distributions (and perhaps other Unix flavors) can also follow this tutorial. This guide assumes systemd as your init system; if you are not using systemd, you will have to get your own daemon files somewhere else. In places where the instructions are split three-way, try coming up with your own, reading documentation and config files; the Arch Linux instructions are probably the closest to upstream (but not always). Unfortunately, all Linux distributions have their own ideas when it comes to running and managing nginx and uWSGI.

nginx and uWSGI are considered best practices by most people. nginx is a fast, modern web server, with uWSGI support built in (without resorting to reverse proxying). uWSGI is similarly aimed at speed. The Emperor mode of uWSGI is recommended for init system integration by the uWSGI team, and it’s especially useful for multi-app deployments. (This guide is opinionated.)

Automate everything: Ansible Playbook

A Playbook that automates everything in this tutorial is available. ci-status

How to use

  1. Install Ansible on your control computer (not necessarily the destination server).

  2. Clone the Playbook from GitHub.

  3. Read README.md. You should also understand how Ansible works.

  4. Configure (change three files: hosts, group_vars/all, and group_vars/os_<destination OS>

  5. Make sure all the dependencies are installed on your destination server

  6. Run ansible-playbook -v nginx-uwsgi.yml -i hosts and watch magic happen.

  7. Skip over to End result and test your site.

The manual guide

Even though I personally recommend the Playbook as a much less error-prone way to set up your app, it might not be compatible with everyone’s system, or otherwise be the wrong solution. The original manual configuration guide is still maintained.

Even if you are using the Playbook, you should still read this to find out what happens under the hood, and to find out about other caveats/required configuration changes.

Getting started

Start by installing Python 3 (with venv), nginx and uWSGI. I recommend using your operating system’s packages. Make sure you are downloading the latest versions available for your OS (update the package cache first). For uWSGI, we need the logfile and python3 plugins. (Arch Linux names the python3 plugin python; the logfile plugin may be built-in — check with your system repositories!). I’ll also install Git to clone the tutorial app, but it’s optional if your workflow does not involve Git.

Ubuntu, Debian:

apt install python3 python3-venv uwsgi uwsgi-emperor uwsgi-plugin-python3 nginx-full git

Fedora:

dnf install python3 uwsgi uwsgi-plugin-python3 uwsgi-logger-file nginx git

CentOS 7:

yum install epel-release
yum install python36 uwsgi uwsgi-plugin-python36 uwsgi-logger-file nginx git wget

EL8 (AlmaLinux 8, Rocky Linux 8):

dnf install epel-release
dnf install python36 uwsgi uwsgi-plugin-python3 uwsgi-logger-file nginx git wget

Arch Linux:

pacman -S python uwsgi uwsgi-plugin-python nginx git

Preparing your application

This tutorial will work for any web framework. I will use a really basic Flask app that has just one route (/), a static hello.png file and a favicon.ico for demonstration purposes. The app is pretty basic, but all the usual advanced features (templates, user logins, database access, etc.) would work without any other web server-related config. Note that the app does not use app.run(). While you could add it, it would be used for local development and debugging only, and would have to be prepended by if __name__ == '__main__': (if it wasn’t, that server would run instead of uWSGI, which is bad)

The app will be installed somewhere under the /srv directory, which is a great place to store things like this. I’ll choose /srv/myapp for this tutorial, but for real deployments, you should use something more distinguishable — the domain name is a great idea.

If you don’t use Flask, this tutorial also has instructions for other web frameworks (Django, Pyramid, Bottle) in the configuration files; it should be adjustable to any other WSGI-compliant framework/script nevertheless.

We’ll start by creating a virtual environment, which is very easy with Python 3:

mkdir /srv/myapp
python3 -m venv --prompt myapp /srv/myapp/venv

(The --prompt option is not supported on some old versions of Python, but you can just skip it if that’s the case, it’s just to make the prompt after source bin/activate more informative.)

Now, we need to put our app there and install requirements. An example for the tutorial demo app:

cd /srv/myapp
git clone https://github.com/Kwpolska/flask-demo-app appdata
venv/bin/pip install -r appdata/requirements.txt

I’m storing my application data in the appdata subdirectory so that it doesn’t clutter the virtual environment (or vice versa). You may also install the uwsgi package in the virtual environment, but it’s optional.

What this directory should be depends on your web framework. For example, for a Django app, you should have an appdata/manage.py file (in other words, appdata is where your app structure starts). I also assumed that the appdata folder should have a static subdirectory with all static files, including favicon.ico if you have one (we will add support for both in nginx).

At this point, you should chown this directory to the user and group your server is going to run as. This is especially important if uwsgi and nginx run as different users (as they do on Fedora). Run one of the following commands:

Ubuntu, Debian:

chown -R www-data:www-data /srv/myapp

Fedora, CentOS, EL8 (AlmaLinux, Rocky Linux):

chown -R uwsgi:nginx /srv/myapp

Arch Linux:

chown -R http:http /srv/myapp

Configuring uWSGI and nginx

We need to write a configuration file for uWSGI and nginx.

uWSGI configuration

Start with this, but read the notes below and change the values accordingly:

[uwsgi]
socket = /srv/myapp/uwsgi.sock
chmod-socket = 775
chdir = /srv/myapp/appdata
master = true
binary-path = /srv/myapp/venv/bin/uwsgi
virtualenv = /srv/myapp/venv
module = flaskapp:app
uid = www-data
gid = www-data
processes = 1
threads = 1
plugins = python3,logfile
logger = file:/srv/myapp/uwsgi.log

Save this file as:

  • Ubuntu, Debian: /etc/uwsgi-emperor/vassals/myapp.ini

  • Fedora, CentOS, EL8 (AlmaLinux, Rocky Linux): /etc/uwsgi.d/myapp.ini

  • Arch Linux: /etc/uwsgi/vassals/myapp.ini (create the directory first and chown it to http: mkdir -p /etc/uwsgi/vassals; chown -R http:http /etc/uwsgi/vassals)

The options are:

  • socket — the socket file that will be used by your application. It’s usually a file path (Unix domain socket). You could use a local TCP socket, but it’s not recommended.

  • chdir — the app directory.

  • binary-path — the uWSGI executable to use. Remove if you didn’t install the (optional) uwsgi package in your virtual environment.

  • virtualenv — the virtual environment for your application.

  • module — the name of the module that houses your application, and the object that speaks the WSGI interface, separated by colons. This depends on your web framework:

    Framework Flask, Bottle Django Pyramid
    Package module where app is defined project.wsgi
    (project is the package with settings.py)     		module where app is defined
    Callable Flask: app instance
    Bottle: app = bottle.default_app()     		application app = config.make_wsgi_app()
    Module package:app project.wsgi:application package:app
    Caveats Make sure app is not in an if __name__ == '__main__': block Add environment variable for settings:
    env = DJANGO_SETTINGS_MODULE=project.settings     		Make sure app is not in an if __name__ == '__main__': block (the demo quickstart does that!)

  • uid and gid — the names of the user account to use for your server. Use the same values as in the chown command above.

  • processes and threads — control the resources devoted to this application. Because this is a simple hello app, I used one process with one thread, but for a real app, you will probably need more (you need to see what works the best; there is no algorithm to decide). Also, remember that if you use multiple processes, they don’t share memory (you need a database to share data between them).

  • plugins — the list of uWSGI plugins to use. For Arch Linux, use plugins = python (the logfile plugin is always active). For CentOS 7 only (i.e. not for EL8), use plugins = python36.

  • logger — the path to your app-specific logfile. (Other logging facilities are available, but this one is the easiest, especially for multiple applications on the same server)

  • env — environment variables to pass to your app. Useful for configuration, may be specified multiple times. Example for Django: env = DJANGO_SETTINGS_MODULE=project.settings

You can test your configuration by running uwsgi --ini /path/to/myapp.ini (disable the logger for stderr output or run tail -f /srv/myapp/uwsgi.log in another window).

If you’re using Fedora, CentOS, or EL8, there are two configuration changes you need to make globally: in /etc/uwsgi.ini, disable the emperor-tyrant option (which we don’t need, as it sets uid/gid for every process based on the owner of the related .ini config file — we use one global setup) and set gid = nginx. We’ll need this so that nginx can talk to your socket.

nginx configuration

We need to configure our web server. Here’s a basic configuration that will get us started:

Save this file as:

  • Ubuntu, Debian: /etc/nginx/sites-enabled/myapp.conf

  • Fedora, CentOS, EL8 (AlmaLinux, Rocky Linux): /etc/nginx/conf.d/myapp.conf

  • Arch Linux: add include /etc/nginx/conf.d/*.conf; to your http directive in /etc/nginx/nginx.conf and use /etc/nginx/conf.d/myapp.conf

server {
    # for a public HTTP server:
    listen 80;
    # for a public HTTPS server:
    # listen 443 ssl;
    server_name localhost myapp.local;
    location / {
        include uwsgi_params;
        uwsgi_pass unix:/srv/myapp/uwsgi.sock;
    }
    location /static {
        alias /srv/myapp/appdata/static;
    }
    location /favicon.ico {
        alias /srv/myapp/appdata/static/favicon.ico;
    }
}

Note that this file is a very basic and rudimentary configuration. This configuration is fine for local testing, but for a real deployment, you will need to adjust it:

  • set listen to 443 ssl and create a http→https redirect on port 80 (you can get a free SSL certificate from Let’s Encrypt; make sure to configure SSL properly).

  • set server_name to your real domain name

  • you might also want to add custom error pages, log files, or change anything else that relates to your web server — consult other nginx guides for details

  • nginx usually has some server already enabled by default — edit /etc/nginx/nginx.conf or remove their configuration files from your sites directory to disable it

Service setup

After you’ve configured uWSGI and nginx, you need to enable and start the system services.

For Arch Linux

All you need is:

systemctl enable nginx emperor.uwsgi
systemctl start nginx emperor.uwsgi

Verify the service is running with systemctl status emperor.uwsgi

For Fedora, CentOS, EL8

Make sure you followed the extra note about editing /etc/uwsgi.ini earlier and run:

systemctl enable nginx uwsgi
systemctl start nginx uwsgi

Verify the service is running with systemctl status uwsgi

If you disabled SELinux, this is enough to get an app working and you can skip over to the next section.

If you want to use SELinux, you need to do the following to allow nginx to read static files:

setenforce 0
chcon -R system_u:system_r:httpd_t:s0 /srv/myapp/appdata/static
setenforce 1

We now need to install a SELinux policy (that I created for this project; updated 2020-05-02) to allow nginx and uWSGI to communicate. Download nginx-uwsgi.pp and run:

semodule -i nginx-uwsgi.pp

Hopefully, this is enough (you can delete the file). In case it isn’t, please read SELinux documentation, check audit logs, and look into audit2allow.

For Ubuntu and Debian

Ubuntu and Debian (still!) use LSB services for uWSGI. Because LSB services are awful, we’re going to set up our own systemd-based (native) service.

Start by disabling the LSB service that comes with Ubuntu and Debian:

systemctl stop uwsgi-emperor
systemctl disable uwsgi-emperor

Copy the .service file from the uWSGI systemd documentation to /etc/systemd/system/emperor.uwsgi.service. Change the ExecStart line to:

ExecStart=/usr/bin/uwsgi --ini /etc/uwsgi-emperor/emperor.ini

You can now reload systemd daemons and enable the services:

systemctl daemon-reload
systemctl enable nginx emperor.uwsgi
systemctl reload nginx
systemctl start emperor.uwsgi

Verify the service is running with systemctl status emperor.uwsgi. (Ignore the warning about no request plugin)

End result

Your web service should now be running at http://localhost/ (or wherever you set up server to listen).

If you used the demo application, you should see something like this (complete with the favicon and image greeting):

/images/nginx-uwsgi-demo.png

If you want to test with cURL:

curl -v http://localhost/
curl -I http://localhost/favicon.ico
curl -I http://localhost/static/hello.png

Troubleshooting

Hopefully, everything works. If it doesn’t:

  • Check your nginx, system (journalctl, systemctl status SERVICE) and uwsgi (/srv/myapp/uwsgi.log) logs.

  • Make sure you followed all instructions.

  • If you get a default site, disable that site in nginx config (/etc/nginx/nginx.conf or your sites directory).

  • If you have a firewall installed, make sure to open the ports your web server runs on (typically 80/443). For firewalld (Fedora, CentOS, EL8):

firewall-cmd --add-service http
firewall-cmd --add-service https

  • If it still does not work, feel free to ask in the comments, mentioning your distribution, installation method, and what doesn’t work.

Can I use Docker?

This blog post is written for systems running standalone. But Docker is a bit special, in that it offers a limited subset of OS features this workflow expects. The main issue is with user accounts, which generally work weird in Docker, and I had issues with setuid/setgid as used by uWSGI. Another issue is the lack of systemd, which means that another part of the tutorial fails to apply.

This tutorial uses uWSGI Emperor, which can run multiple sites at once, and offers other management features (such as seamless code restarts with touch /etc/uwsgi/vassals/myapp.ini) that may not be useful or easy to use in a Docker environment. You’d probably also run uWSGI and nginx in separate containers in a typical Docker deployment.

Regardless, many parts of this tutorial can be used with Docker, although with the aforementioned adjustments. I have done some work on this topic. This tutorial has an Ansible Playbook attached, and the tutorial/playbook are compatible with five Linux distros in multiple versions. How do I know that there were no unexpected bugs in an older version? I could grab a Vagrant image or set up a VM. I do that when I need specific testing, but doing it for each of the distros on each update would take at least half an hour, probably even more. Yeah, that needs automating. I decided to use GitHub Actions for the CI, which can run anything, as long as you provide a Dockerfile.

The Docker images were designed to support running the Playbook and testing it. But the changes, setups and patches could be a good starting point if you wanted to make your own Docker containers that could run in production. You can take a look at the Docker files for CI The images support all 5 distros using their base images, but you could probably use Alpine images, or the python docker images; be careful not to mix Python versions in the latter case.

That said, I still prefer to run without Docker, directly on the system. Less resources wasted and less indirection. Which is why this guide does it the traditional way.