Too many open files under load while ulimit -n says 65535: systemd LimitNOFILE

Our image thumbnail service handles uploads for a marketplace app. During a promotion on a Thursday evening, error rates on /thumbs jumped from near zero to 18 percent within ten minutes. The load balancer reported connection resets, and the service journal filled with this:

Jan 23 19:42:07 thumbs-02 uvicorn[1187]: socket.accept() out of system resource
Jan 23 19:42:07 thumbs-02 uvicorn[1187]: socket: <asyncio.TransportSocket fd=9, family=2, type=1, proto=6, laddr=('0.0.0.0', 8000)>
Jan 23 19:42:07 thumbs-02 uvicorn[1187]: OSError: [Errno 24] Too many open files

Requests that did get in sometimes failed later while writing a temporary file:

OSError: [Errno 24] Too many open files: '/var/tmp/thumbs/7f3c2a.webp'

The first person on call SSHed in, ran ulimit -n, saw 65535, and concluded the limit could not be the problem. That cost us about forty minutes.

Reproducing it

On a staging VM with the same image we drove concurrent keep-alive connections with a load tool and watched the descriptor count of the process:

PID=$(systemctl show -p MainPID --value thumbs.service)
watch -n1 "ls /proc/$PID/fd | wc -l"

The count climbed steadily to 1,024 and errors started on the next connection, every time. The number 1,024 was the giveaway: it is not a number anyone on the team had configured.

What we ruled out

• The kernel-wide limit. cat /proc/sys/fs/file-max returned a value in the millions and /proc/sys/fs/file-nr showed the system using a few thousand handles in total.
• The shell limit. ulimit -n in our SSH session really was 65535, set in /etc/security/limits.d/90-nofile.conf by the base image. That limit applies to login sessions, not to our service.
• Ephemeral port exhaustion. ss -s showed a normal number of TIME-WAIT sockets and the error was EMFILE (per-process), not EADDRNOTAVAIL.

Root cause

The service's own limits told the real story:

grep 'open files' /proc/$PID/limits

Max open files            1024                 524288               files

The process ran with a soft limit of 1,024 and a hard limit of 524,288, which are systemd's defaults for services. systemd starts services directly from PID 1, so the PAM limits.conf files that set 65535 for SSH sessions were never applied. Under the promotion, each client connection used one descriptor, each in-flight request opened a temp file, and a leak in our code (below) held another socket per request. We crossed 1,024 well before CPU or memory were stressed.

What is happening under the hood

Every open file, socket, pipe and epoll instance consumes a file descriptor, and each process has an RLIMIT_NOFILE with a soft and a hard value. A process can raise its own soft limit up to the hard limit, but most runtimes, including CPython, do not do that automatically.

Where that limit comes from depends on how the process was started:

Since systemd 240 the built-in default for services is a soft limit of 1024 and a hard limit of 524288. The soft limit stays low on purpose: programs that use select() cannot handle descriptor numbers above 1023, so raising it for everyone would break them. Programs that need more are expected to ask, either in the unit file or by raising their own soft limit.

Environment

Minimal reproduction

# Start any command as a transient service and print its limit.
sudo systemd-run --wait --pipe sh -c 'ulimit -n'
# 1024

# Same command in your SSH shell.
ulimit -n
# 65535

Who is affected

Any long-running service under systemd that holds many sockets or files: Python and Node.js servers, Ruby app servers, Elasticsearch-style data stores, proxies. Some runtimes raise their own soft limit at startup (Go has done so since 1.19, for example), which hides the problem until a different language is deployed with the same unit template. Containers are a separate case and follow the container runtime's settings.

The fix

We fixed both halves: the limit, and the leak that made us reach it.

1. Set the limit in a drop-in, so package upgrades never overwrite it:

sudo systemctl edit thumbs.service

# /etc/systemd/system/thumbs.service.d/override.conf
[Service]
LimitNOFILE=65536

systemctl edit reloads the unit files when you save. A restart applies the new limit:

sudo systemctl restart thumbs.service
systemctl show thumbs.service -p LimitNOFILE -p LimitNOFILESoft

LimitNOFILE=65536
LimitNOFILESoft=65536

A single value sets both soft and hard. LimitNOFILE=65536:524288 would set them separately.

2. Fix the descriptor leak. lsof grouped by type showed thousands of established TCP connections to our object storage endpoint, far more than in-flight requests:

sudo lsof -p "$PID" -a -i TCP | awk 'NR>1 {print $9}' | cut -d'>' -f2 | sort | uniq -c | sort -rn | head

The handler created an httpx.AsyncClient per request and never closed it, so each one kept its connection pool open until garbage collection got around to it:

-@app.post("/thumbs")
-async def make_thumb(file: UploadFile):
-    client = httpx.AsyncClient(timeout=10)
-    await client.put(storage_url(file.filename), content=await render(file))
+@asynccontextmanager
+async def lifespan(app: FastAPI):
+    app.state.http = httpx.AsyncClient(timeout=10, limits=httpx.Limits(max_connections=200))
+    yield
+    await app.state.http.aclose()
+
+app = FastAPI(lifespan=lifespan)
+
+@app.post("/thumbs")
+async def make_thumb(file: UploadFile, request: Request):
+    await request.app.state.http.put(storage_url(file.filename), content=await render(file))

One shared client with a bounded pool caps storage connections at 200 regardless of traffic.

Alternatives and trade-offs

1. Raise DefaultLimitNOFILE= in /etc/systemd/system.conf. Affects every service on the host. Convenient for single-purpose machines, but it hides which services really need it.
2. Raise the soft limit from the application with resource.setrlimit(resource.RLIMIT_NOFILE, (hard, hard)). Portable across init systems, but the setting lives in code instead of in the deployment config ops engineers look at.
3. LimitNOFILE=infinity. Tempting and rarely a good idea. The value maps to the kernel's fs.nr_open ceiling, and some programs loop over every possible descriptor at startup or in child processes, which then becomes very slow.

How to test the fix

PID=$(systemctl show -p MainPID --value thumbs.service)
grep 'open files' /proc/$PID/limits
# Max open files            65536                65536                files

# Drive 3,000 concurrent keep-alive connections from a load host.
hey -z 2m -c 3000 https://thumbs-staging.example.com/healthz

ls /proc/$PID/fd | wc -l

During the run the descriptor count settled around 3,300 and returned to about 260 afterwards. Before the leak fix it kept growing after the load stopped.

Confirming in production

After rolling out both changes we replayed the promotion traffic profile in staging at twice the peak, then watched production through the next promotion:

• Zero Errno 24 lines in journalctl -u thumbs.service across four hosts for the following week.
• Open descriptors per process peaked at 4,100 during the next sale, with a ceiling of 65,536.
• Established connections to object storage stayed at or below 200 per process, the pool limit.
• The 5xx rate on /thumbs stayed under 0.05 percent.

Regression guard

• Our node exporter textfile collector publishes open fds / soft limit per service, and we alert when it stays above 70 percent for ten minutes. This catches both a lower limit after a unit change and a new leak.
• The unit template in our configuration management now requires an explicit LimitNOFILE= for any network-facing service, and a CI check fails if it is missing.
• A load test in the release pipeline asserts the descriptor count returns to baseline five minutes after load stops.

Security implications

Raising LimitNOFILE increases how many descriptors one compromised or misbehaving process can hold. It does not open new access, but it moves the denial-of-service ceiling. Keep the value sized to need (we picked 65,536, not unlimited), keep per-client connection limits at the load balancer, and keep the bounded HTTP pool so a slow storage endpoint cannot make the service open connections without end.

Prevention checklist

• Check /proc/<pid>/limits of the actual process, never ulimit -n in your shell.
• Set LimitNOFILE= explicitly in a drop-in for every busy service.
• Reuse HTTP clients and database pools; never create one per request.
• Alert on descriptor usage as a percentage of the soft limit.
• Include a soak test that verifies descriptors return to baseline.

References

• systemd.exec manual, LimitNOFILE= and process properties — https://www.freedesktop.org/software/systemd/man/latest/systemd.exec.html
• systemd-system.conf manual, DefaultLimitNOFILE= — https://www.freedesktop.org/software/systemd/man/latest/systemd-system.conf.html
• Linux man-pages, getrlimit(2) and RLIMIT_NOFILE — https://man7.org/linux/man-pages/man2/getrlimit.2.html
• Linux man-pages, proc_pid_limits(5) — https://man7.org/linux/man-pages/man5/proc_pid_limits.5.html
• HTTPX documentation, clients and connection pooling — https://www.python-httpx.org/advanced/clients/