PyTorch DataLoader: Unexpected bus error encountered in worker in Docker and Kubernetes

We moved image classification training from a workstation to Kubernetes GPU nodes. The first epoch started, and within minutes every run died:

ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm).
Traceback (most recent call last):
  ...
RuntimeError: DataLoader worker (pid 214) is killed by signal: Bus error. It is possible that dataloader's workers are out of shared memory. Please try to raise your shared memory limit.

The same code, same data and same Python environment ran for days on the workstation.

Reproducing it

Inside the training container:

$ df -h /dev/shm
Filesystem      Size  Used Avail Use% Mounted on
shm              64M   63M  1.0M  99% /dev/shm

On the workstation, /dev/shm was 32 GB (half of RAM, the Linux default for tmpfs). The crash only happened with num_workers > 0; with num_workers=0 training ran, very slowly.

What we ruled out

• Container memory limit. The pod had 64 GiB and was using about 20; no OOM kill.
• Corrupt images. A bus error is not a decoding exception, and the same files worked with num_workers=0.
• A PyTorch bug. The error message itself names the likely cause.

Root cause

PyTorch DataLoader workers are separate processes. They send batches back to the main process through shared memory, which on Linux lives in /dev/shm. Containers get a private /dev/shm of 64 MB by default in Docker, and Kubernetes pods inherit the container runtime default unless you mount something else. Our batches were 256 images at 224x224 in float32, about 150 MB per batch, with 8 workers each prefetching 2 batches. Writing to a full tmpfs-backed mapping triggers SIGBUS.

What is happening under the hood

With num_workers > 0, each worker loads and transforms samples, collates a batch into tensors, and moves those tensors into shared memory so the main process can use them without copying. The amount of shared memory needed is roughly:

batch_size x bytes_per_sample x num_workers x prefetch_factor
256 x (3 x 224 x 224 x 4 bytes) x 8 x 2  ~= 2.4 GB

Shared memory segments are files in /dev/shm. When the filesystem is full, a page fault on a mapped page that cannot be backed results in a bus error, not a clean Python exception.

Environment

Minimal reproduction

docker run --rm --gpus all pytorch/pytorch:2.5.1-cuda12.4-cudnn9-runtime \
  python -c "
import torch
from torch.utils.data import DataLoader, TensorDataset
ds = TensorDataset(torch.randn(10000, 3, 224, 224))
for _ in DataLoader(ds, batch_size=256, num_workers=8): pass
"

Fails with the bus error; adding --shm-size=8g makes it pass.

The fix

Kubernetes: mount a memory-backed emptyDir at /dev/shm.

spec:
  containers:
    - name: trainer
      resources:
        limits:
          memory: 64Gi
          nvidia.com/gpu: 1
      volumeMounts:
        - name: dshm
          mountPath: /dev/shm
  volumes:
    - name: dshm
      emptyDir:
        medium: Memory
        sizeLimit: 8Gi

Memory used in a memory-backed emptyDir counts against the container's memory limit, so size the limit to include it. We set 8 GiB, more than three times our calculated peak.

Docker and local runs: --shm-size or IPC host.

docker run --gpus all --shm-size=8g trainer:latest
# or, for single-tenant machines only:
docker run --gpus all --ipc=host trainer:latest

In Compose, shm_size: "8gb".

Right-size the DataLoader too. Eight workers was more than the node's CPU share could feed. Profiling showed 6 workers saturated the GPU, so we reduced it, which also lowered shared memory use.

Alternatives and trade-offs

• num_workers=0. No shared memory needed, and data loading becomes the bottleneck; our epoch time tripled.
• torch.multiprocessing.set_sharing_strategy('file_system'). Changes how tensors are shared and can avoid file descriptor limits, but does not remove the need for shared memory space.
• --ipc=host. Gives the container the host's shared memory; acceptable on dedicated machines, not on shared nodes, because it removes IPC isolation.

How to test the fix

The minimal reproduction above passes with the new settings. In the cluster, a smoke job runs one epoch on a 5,000-image subset with production DataLoader settings before each long training run.

Confirming in production

• Training runs complete; no bus errors in a month of daily runs.
• /dev/shm usage peaks at about 1.9 GB per pod, visible through a node exporter textfile metric we added.
• GPU utilisation during training averages 92 percent with 6 workers.

Regression guard

• The training job Helm chart always mounts /dev/shm with a size derived from batch size and worker count values.

• The smoke epoch runs in the training pipeline before the full run.

• A short section in the training platform docs explains the shared memory formula with a worked example, because the same question came up from three teams in a month.

Security implications

Avoid --ipc=host and host-path mounts of /dev/shm on shared nodes: they let a container read and write shared memory segments of other processes on the host. A memory-backed emptyDir gives each pod its own isolated shared memory with a size limit.

Prevention checklist

• Containers default to 64 MB of /dev/shm; plan for more with multi-worker data loading.
• Mount a memory-backed emptyDir in Kubernetes and count it in memory limits.
• Estimate shared memory from batch size, workers and prefetch factor.
• Tune num_workers with profiling instead of guessing.
• Do not use host IPC on shared machines.

References

• PyTorch DataLoader documentation (multi-process data loading) — https://pytorch.org/docs/stable/data.html#multi-process-data-loading
• PyTorch multiprocessing best practices — https://pytorch.org/docs/stable/notes/multiprocessing.html
• Docker run, --shm-size — https://docs.docker.com/reference/cli/docker/container/run/
• Kubernetes emptyDir volumes — https://kubernetes.io/docs/concepts/storage/volumes/#emptydir