Linux "no space left on device" from Docker overlay2, logs and build cache

We run a small SaaS on a single 80 GB cloud VM with Docker Compose: an API, a background worker, Postgres and Redis. At 03:12 the uptime monitor reported the API returning 500s. The API logs said the database could not commit, and the Postgres container logs were blunt:

PANIC:  could not write to file "pg_wal/xlogtemp.31": No space left on device
LOG:  server process (PID 31) was terminated by signal 6: Aborted

When we tried to deploy a quick fix, the image pull failed too:

failed to register layer: write /usr/lib/x86_64-linux-gnu/libLLVM-15.so.1: no space left on device

Reproducing it

The condition was simply a full root filesystem:

$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        78G   78G     0 100% /

Docker keeps everything under /var/lib/docker on the root filesystem by default, so the question was which part of Docker had grown.

What we ruled out

• Inode exhaustion. "No space left on device" is also what you get when a filesystem runs out of inodes, even with free blocks. df -i / showed 9 percent inode usage, so this was real space.
• Postgres data growth. The database volume was 6.1 GB, close to what it had been for months.
• Deleted files still held open. A process holding a deleted file keeps its blocks allocated. lsof +L1 showed nothing large, so df and du should agree.

Root cause

docker system df and a du pass told the story:

$ docker system df
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          47        5         24.8GB    21.3GB (85%)
Containers      6         5         31.6GB    0B (0%)
Local Volumes   4         2         6.4GB     212MB (3%)
Build Cache     318       0         11.9GB    11.9GB

$ sudo du -xh --max-depth=1 /var/lib/docker | sort -h | tail -4
6.4G    /var/lib/docker/volumes
12G     /var/lib/docker/buildkit
31G     /var/lib/docker/containers
25G     /var/lib/docker/overlay2

Three separate leaks added up to roughly 65 GB. The worker container had a 31 GB JSON log file in /var/lib/docker/containers/<id>/<id>-json.log, because a retry loop logged a full payload at info level every second and the default json-file log driver never rotates. Our deploy script built images on the host with docker compose build on every release, leaving 40-plus old image versions and about 12 GB of BuildKit cache. Nothing ever cleaned up either.

What is happening under the hood

The overlay2 storage driver stores each image layer as a directory under /var/lib/docker/overlay2, plus a writable upper layer for each container. When you build or pull a new version of an image with the same tag, the old one loses its tag and becomes a dangling image, but its layers stay on disk until something prunes them. BuildKit keeps its own cache of intermediate results for fast rebuilds, which also grows without bound unless garbage collection limits are set.

Container stdout and stderr are captured by the logging driver. The default json-file driver writes one JSON line per log line into a file under /var/lib/docker/containers, and by default has no max-size, so a chatty container grows its log forever. docker logs reads from that file.

Postgres is the first to complain because it pre-allocates WAL segments and panics when a write fails, which is the safe behaviour for a database.

Environment

Minimal reproduction

# A container that logs fast with the default json-file driver
docker run -d --name noisy alpine sh -c 'while true; do echo "$(date) retrying job payload=..."; done'

# Watch the log file grow without limit
watch -n 5 'sudo du -h $(docker inspect --format "{{.LogPath}}" noisy)'

Who is affected

Any Docker host that uses the default logging configuration and builds or pulls images regularly without pruning. Single-host deployments and self-hosted CI runners are the usual victims. Kubernetes nodes rotate container logs through the kubelet and evict pods under disk pressure, so they fail differently, but image garbage collection thresholds still matter there.

The fix

1. Recover space safely, in this order. We stopped the worker first so it would stop writing, then truncated its log instead of deleting it. Deleting a log file that the daemon still holds open does not free the space until the file handle closes.

docker compose stop worker
sudo truncate -s 0 "$(docker inspect --format '{{.LogPath}}' app-worker-1)"

docker builder prune --all --force --filter until=72h
docker image prune --all --force --filter until=168h
docker system df

We avoided docker system prune --volumes, because removing unused volumes can delete data that belongs to a stopped service. After pruning, the disk was at 31 percent and Postgres recovered on restart from its WAL with no data loss.

2. Rotate container logs by default. We set daemon-wide defaults in /etc/docker/daemon.json:

{
  "log-driver": "local",
  "log-opts": {
    "max-size": "20m",
    "max-file": "5"
  }
}

The local driver stores logs in a compressed, more compact format and rotates by default, while docker logs still works. Daemon logging defaults only apply to containers created after the daemon restart, so we ran sudo systemctl restart docker and then docker compose up -d --force-recreate.

We also pinned it per service in compose.yaml so the setting travels with the project:

services:
  worker:
    image: registry.example.com/acme/worker:${TAG}
    logging:
      driver: local
      options:
        max-size: "20m"
        max-file: "5"

3. Stop building on the production host. Images are now built in CI and pulled by tag. A systemd timer runs a weekly prune that keeps anything from the last week:

# /etc/systemd/system/docker-prune.service
[Service]
Type=oneshot
ExecStart=/usr/bin/docker image prune --all --force --filter until=168h
ExecStart=/usr/bin/docker builder prune --all --force --filter until=168h

4. Fix the noisy worker. The retry loop now uses exponential backoff and logs the payload ID, not the payload.

Alternatives and trade-offs

• Move /var/lib/docker to a separate disk with data-root in daemon.json. A full Docker disk then cannot take down the OS, but it still takes down your containers.
• Ship logs elsewhere with the journald, fluentd or awslogs driver. Fine for central logging, but remember journald has its own size limits to configure.
• Keep json-file with max-size. It works and is the most widely supported, but takes more disk per line than local.

How to test the fix

docker inspect --format '{{.HostConfig.LogConfig}}' app-worker-1
# {local map[max-file:5 max-size:20m]}

# Generate noise and confirm the log directory stays bounded
docker run -d --name noisy alpine sh -c 'while true; do echo spam; done'
sleep 120
sudo du -sh /var/lib/docker/containers/$(docker inspect --format '{{.Id}}' noisy)
docker rm -f noisy

The container directory stayed under 100 MB no matter how long it ran.

Confirming in production

Two weeks after the fix, root filesystem usage had stayed between 28 and 34 percent. docker system df showed 9 images (5 active) and build cache at 0 B since nothing builds on the host anymore. The worker's log directory sat at 61 MB, which is five rotated files at their compressed size.

Regression guard

• A node_exporter alert on node_filesystem_avail_bytes{mountpoint="/"} below 20 percent for 15 minutes, and a second, paging alert below 10 percent.
• A CI check that fails if any service in compose.yaml lacks a logging block.
• A dashboard panel for /var/lib/docker size, collected by a tiny textfile exporter script.

Security implications

Unbounded logs are a denial of service vector: anything that makes the app log per request, such as an attacker sending malformed input, can fill the disk. Rotation closes that path. The original bug also logged full job payloads, which contained customer email addresses; we purged the old logs and changed the logging to IDs only. Be careful with docker system prune on shared hosts, since it removes stopped containers that other teams may still need for forensics.

Prevention checklist

• Set log-driver and log-opts in daemon.json on every Docker host from day one.
• Build images in CI, not on production hosts.
• Schedule image and build cache pruning with a time filter.
• Alert on disk usage at 80 percent, not 100.
• Never log full payloads at info level inside retry loops.

References

• Docker: configure logging drivers — https://docs.docker.com/engine/logging/configure/
• Docker: local file logging driver — https://docs.docker.com/engine/logging/drivers/local/
• Docker: prune unused objects — https://docs.docker.com/engine/manage-resources/pruning/
• Docker: docker system df — https://docs.docker.com/reference/cli/docker/system/df/
• Docker: OverlayFS storage driver — https://docs.docker.com/engine/storage/drivers/overlayfs-driver/