Nginx 502 Bad Gateway with PHP-FPM: socket permission denied and pm.max_children

We moved a customer portal (a Laravel app) from an aging Ubuntu 20.04 VM to a fresh Ubuntu 24.04 server, provisioned with our usual Ansible playbook. DNS switched at 22:00. By 22:01 every request returned Nginx's plain "502 Bad Gateway" page. The Nginx error log had one line per request:

2024/10/15 22:01:07 [crit] 1842#1842: *3 connect() to unix:/run/php/php8.3-fpm.sock failed (13: Permission denied) while connecting to upstream, client: 203.0.113.24, server: portal.example.com, request: "GET / HTTP/2.0", upstream: "fastcgi://unix:/run/php/php8.3-fpm.sock:", host: "portal.example.com"

We fixed that within 20 minutes, which is the first half of this story. Two days later, during the Monday morning login peak, 502s came back for about 8 percent of requests, with a different message:

[error] 1843#1843: *48211 connect() to unix:/run/php/php8.3-fpm.sock failed (11: Resource temporarily unavailable) while connecting to upstream

And in the PHP-FPM log:

[15-Oct-2024 09:02:14] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it

Reproducing it

The permission error reproduced on every request. For the load-related one, we pointed a load generator at a slow endpoint on a staging copy:

# 60 concurrent clients against a page that takes ~400 ms of PHP time
hey -z 60s -c 60 https://staging-portal.example.com/dashboard

With the default pool, 502s started within seconds.

What we ruled out

• PHP-FPM not running. systemctl status php8.3-fpm was active, and the socket file existed. A stopped FPM gives (2: No such file or directory) or (111: Connection refused), not error 13.
• AppArmor or SELinux. Ubuntu ships no AppArmor profile for Nginx by default, and dmesg had no apparmor="DENIED" entries.
• Wrong socket path. The path in the Nginx config matched listen in the pool file exactly.
• PHP fatal errors. Those return 500 or an empty response from FPM, not a failed connect().

Root cause

The first problem was ownership. Our playbook changed the pool user and group to deploy so the app could write to its storage directory, and also set listen.owner and listen.group to deploy. Nginx runs its workers as www-data, and the socket was created as deploy:deploy with mode 0660. www-data was neither owner nor group member, so the kernel rejected the connect().

The second problem was capacity. The Ubuntu package ships the www pool with pm = dynamic and pm.max_children = 5. With five workers and requests taking 300 to 500 ms, the pool could serve about 12 requests per second. Above that, connections queued in the socket's listen backlog. When the backlog filled, new connect() calls failed immediately with EAGAIN, which Nginx logs as "Resource temporarily unavailable" and turns into a 502.

What is happening under the hood

Nginx talks to PHP-FPM over FastCGI, here through a Unix domain socket. Connecting to a Unix socket needs write permission on the socket file, checked like any other file. PHP-FPM creates the socket at startup and applies listen.owner, listen.group and listen.mode (default 0660). If those do not include the Nginx worker user, every connection fails.

Each FPM child process handles exactly one request at a time. pm.max_children is therefore the maximum concurrency of PHP on that server. Extra requests wait in the kernel listen queue, whose length is listen.backlog capped by net.core.somaxconn. Over TCP, a full queue usually shows up as slow connects or timeouts (504). Over Unix sockets on Linux, the connect fails right away, so you see 502.

Environment

Minimal reproduction

; /etc/php/8.3/fpm/pool.d/www.conf (broken excerpt)
[www]
user = deploy
group = deploy
listen = /run/php/php8.3-fpm.sock
listen.owner = deploy
listen.group = deploy
listen.mode = 0660
pm = dynamic
pm.max_children = 5

$ ls -l /run/php/php8.3-fpm.sock
srw-rw---- 1 deploy deploy 0 Oct 15 21:58 /run/php/php8.3-fpm.sock
$ sudo -u www-data php -r 'var_dump(stream_socket_client("unix:///run/php/php8.3-fpm.sock"));'
PHP Warning:  stream_socket_client(): Unable to connect to unix:///run/php/php8.3-fpm.sock (Permission denied)

Who is affected

Any Nginx and PHP-FPM setup where the pool's socket owner or group does not include the Nginx user, which commonly happens after changing pool users, using a custom Nginx user, or copying configs between distributions. The capacity issue affects anyone running the packaged default pool in production.

The fix

1. Socket ownership. The pool can run as deploy while its socket stays owned by the Nginx user. Process user and socket owner are independent settings:

 [www]
 user = deploy
 group = deploy
 listen = /run/php/php8.3-fpm.sock
-listen.owner = deploy
-listen.group = deploy
+listen.owner = www-data
+listen.group = www-data
 listen.mode = 0660

We did not use listen.mode = 0666. It works, but lets every local user talk to PHP-FPM.

2. Pool sizing from memory. We measured the average resident memory of FPM children under real traffic:

ps --no-headers -o rss -C php-fpm8.3 | awk '{ sum += $1; n++ } END { printf "%d workers, avg %.0f MB\n", n, sum/n/1024 }'
# 5 workers, avg 74 MB

The server has 8 GB. After reserving about 2 GB for the OS, Nginx, Redis and OPcache shared memory, we had roughly 6 GB for PHP: 6,000 / 74 is about 80. We picked 60 to leave headroom for memory spikes on report pages.

pm = dynamic
pm.max_children = 60
pm.start_servers = 12
pm.min_spare_servers = 8
pm.max_spare_servers = 20
pm.max_requests = 1000
pm.status_path = /fpm-status
request_slowlog_timeout = 5s
slowlog = /var/log/php8.3-fpm-slow.log

pm.max_requests recycles children to contain slow memory leaks in extensions. The slow log tells us which code paths hold workers longest.

3. Expose the status page to localhost only:

location = /fpm-status {
    allow 127.0.0.1;
    deny all;
    include fastcgi_params;
    fastcgi_param SCRIPT_FILENAME $fastcgi_script_name;
    fastcgi_pass unix:/run/php/php8.3-fpm.sock;
}

Then sudo systemctl reload php8.3-fpm nginx.

Alternatives and trade-offs

• Add www-data to the deploy group. Also works, but grants Nginx read access to everything the deploy group can read, including .env.
• Use TCP (127.0.0.1:9000). Avoids file permissions, slightly slower, and a full queue shows up as timeouts instead of instant 502s.
• pm = static. Predictable memory and no fork latency on spikes, at the cost of holding all workers in memory all the time. Good on dedicated app servers.
• Raise pm.max_children without measuring. The classic mistake: the server swaps or the OOM killer arrives, and you trade 502s for a dead box.

How to test the fix

sudo php-fpm8.3 -t                       # config syntax check
ls -l /run/php/php8.3-fpm.sock           # expect www-data www-data
curl -s http://127.0.0.1/fpm-status      # pool status
hey -z 60s -c 60 https://staging-portal.example.com/dashboard

The status page showed max children reached: 0 and listen queue: 0 after the load test, and hey reported zero non-2xx responses.

Confirming in production

During the next Monday peak:

• Nginx error log had zero connect() ... failed lines; the previous Monday had 4,312.
• /fpm-status peaked at 41 active processes out of 60, max listen queue: 0.
• Memory stayed under 5.9 GB used with no swap activity.
• p95 response time for /dashboard went from 2.4 s (including queueing) to 460 ms.

Regression guard

• A Prometheus php-fpm exporter scrapes /fpm-status; we alert when max_children_reached increases or listen_queue is above zero for five minutes.
• The Ansible role now templates listen.owner from the nginx_user variable and has a Molecule test that runs sudo -u www-data against the socket after converge.
• pm.max_children is computed in the role from ansible_memtotal_mb and a measured per-worker figure, instead of a hard-coded value.

Security implications

The fix keeps the socket restricted to the web server user, which matters: anyone who can write to an FPM socket can ask PHP to execute any PHP file on disk with arbitrary environment parameters. Avoid 0666, keep the pool user separate from the Nginx user, and never expose /fpm-status or FPM's TCP port publicly. Running out of workers is also a cheap denial of service vector, so rate limiting slow endpoints in Nginx with limit_req is a sensible companion.

Prevention checklist

• Keep FPM listen.owner/listen.group aligned with the Nginx worker user whenever you change pool users.
• Never ship the packaged pm.max_children = 5 to production.
• Size the pool from measured worker RSS and available RAM.
• Enable the FPM status page and slow log on day one.
• Treat "server reached pm.max_children" warnings as alerts, not noise.

References

• PHP manual: FPM configuration — https://www.php.net/manual/en/install.fpm.configuration.php
• Nginx: ngx_http_fastcgi_module — https://nginx.org/en/docs/http/ngx_http_fastcgi_module.html
• Nginx: ngx_http_limit_req_module — https://nginx.org/en/docs/http/ngx_http_limit_req_module.html
• Linux man page: unix(7) — https://man7.org/linux/man-pages/man7/unix.7.html