Uploaded files returned 403 from nginx because the app's umask made them owner-only

We serve user-uploaded images directly from disk with nginx, while a Python app handles the upload and writes the file to /srv/media/uploads/. After a maintenance change that moved the app from a supervisor process to a systemd unit, users reported that new avatars showed as broken images. Old avatars still worked.

nginx's error log:

2026/10/27 10:14:22 [error] 1288#1288: *5310 open() "/srv/media/uploads/2026/10/u_44120.jpg" failed
(13: Permission denied), client: 203.0.113.18, server: media.example.com,
request: "GET /uploads/2026/10/u_44120.jpg HTTP/2.0"

Permission denied for nginx's worker processes, which run as the www-data user. Comparing an old file and a new one:

ls -l /srv/media/uploads/2026/10/ | head -4

-rw-r--r-- 1 app app  48211 Oct 25 16:02 u_40981.jpg
-rw-r--r-- 1 app app  51874 Oct 25 17:40 u_41377.jpg
-rw------- 1 app app  49302 Oct 27 10:13 u_44120.jpg
-rw------- 1 app app  50117 Oct 27 10:14 u_44123.jpg

Old files: rw-r--r-- (644), readable by everyone. New files: rw------- (600), readable only by the app user. nginx, running as www-data, could no longer read them.

The application code had not changed. It wrote files with Python's ordinary open(path, "wb"), which does not specify a mode.

How a new file's permissions are decided

When a program creates a file, it asks the kernel for a mode, and the kernel removes the bits set in the process's umask:

final mode = requested mode AND NOT umask

Python's open() requests 0666 (read and write for everyone) for new files, and os.mkdir requests 0777. The umask then takes bits away:

The umask is a per-process attribute, inherited from the parent process. Nothing in the file system or the application decides it.

What changed

Under the old setup, the app was started by a supervisor launched from a login shell, which inherited the system default umask of 0022, so files came out 0644.

The new systemd unit had been copied from a hardened template:

[Service]
User=app
Group=app
UMask=0077
ExecStart=/srv/app/venv/bin/gunicorn app.wsgi --bind 127.0.0.1:8000

UMask=0077 gives owner-only files. That is a sensible default for services that handle secrets, and it is why the template had it. For a service whose files are meant to be read by another user, it breaks things. Note that without a UMask= line, systemd's default for services is 0022; it was the template, not systemd itself, that changed it.

This is also why the bug only affected new files. Permissions are fixed when a file is created; changing a process's umask later does nothing to files that already exist. Everything uploaded before the migration kept its 0644 mode, and everything after it got 0600, which is exactly the split the directory listing showed.

The process's current umask is visible in /proc:

grep Umask /proc/$(systemctl show -p MainPID --value app)/status

Umask:	0077

Environment

We wanted nginx to read uploads without making them world-readable, which 0644 effectively does.

1. A shared group, with setgid directories

Create a group for "can read media", add both users, and make the upload tree belong to it:

sudo groupadd media
sudo usermod -aG media app
sudo usermod -aG media www-data
sudo chgrp -R media /srv/media/uploads
sudo find /srv/media/uploads -type d -exec chmod 2750 {} +
sudo find /srv/media/uploads -type f -exec chmod 0640 {} +

The 2 in 2750 is the setgid bit on directories. New files and subdirectories created inside a setgid directory inherit the directory's group (media) instead of the creating process's primary group (app). Without it, new files would be app:app and nginx would still be locked out.

2. A umask that grants group read

[Service]
User=app
Group=app
UMask=0027

With 0027, new files are 0640 (owner read/write, group read, others nothing) and directories 0750. Combined with the setgid directories, every new upload is app:media 0640: the app can write it, nginx can read it, other users on the host cannot.

nginx's workers needed a restart to pick up the new group membership, because supplementary groups are read when a process starts.

3. Make the app explicit about modes

Relying on the umask alone makes behaviour depend on how the process was started. The upload code now sets the mode explicitly, using os.open with a mode and then os.fchmod to be sure the result does not depend on the umask:

def write_upload(path: Path, data: bytes) -> None:
    fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o640)
    try:
        os.fchmod(fd, 0o640)          # umask may have removed bits; set exactly what we want
        os.write(fd, data)
    finally:
        os.close(fd)

O_EXCL also guarantees we never overwrite an existing file by accident, since upload names are generated and should be unique.

4. Do not fix it with chmod 777

One suggestion during the incident was chmod -R 777 /srv/media/uploads. That makes every upload writable by every user on the machine, including any compromised process, which could then replace images or plant files nginx would serve. The group approach gives nginx exactly the read access it needs.

1. New files get the right owner, group and mode.

curl -s -F "file=@test.jpg" https://app.example.com/api/avatar -H "Authorization: Bearer $T" >/dev/null
ls -l /srv/media/uploads/2026/10/ | tail -1

-rw-r----- 1 app media 50433 Oct 27 11:02 u_44190.jpg

2. nginx can read them, others cannot.

sudo -u www-data head -c 4 /srv/media/uploads/2026/10/u_44190.jpg | xxd
sudo -u nobody  head -c 4 /srv/media/uploads/2026/10/u_44190.jpg

00000000: ffd8 ffe0                                ....
head: cannot open '/srv/media/uploads/2026/10/u_44190.jpg' for reading: Permission denied

The JPEG header bytes for www-data, permission denied for nobody.

3. End to end. curl -I https://media.example.com/uploads/2026/10/u_44190.jpg returns 200. We also fixed the 312 files uploaded during the incident with the chgrp and chmod commands above.

4. Configuration test. Our deployment smoke test now uploads a file and fetches it through nginx, so a permissions problem fails the deploy instead of reaching users.

5. Security implications. The final state is more restrictive than before the incident: uploads used to be world-readable (0644), and now only the owner and the media group can read them. The 0077 umask remains in our template for services that do not share files, and the template now has a comment explaining when to change it.

Takeaways

• New file permissions are the requested mode minus the process's umask, inherited from whatever started it.
• Changing how a service is started (shell, supervisor, systemd) can change its umask.
• Share files between services with a common group, setgid directories and a 0027 umask, not world-readable modes.
• Set file modes explicitly in code when another process must read the file.

References: umask(2) Linux manual page, systemd.exec, UMask=, nginx core module, Python os.open.