GPU at 30 percent utilisation during training: the data loader was the bottleneck

We moved a product image classifier to an 8-GPU node to speed up experiments. Training was barely faster than on the old 4-GPU machine. nvidia-smi dmon told us why:

# gpu   sm   mem
    0   31    12
    1   28    11
    2   34    13
    ...

Streaming multiprocessor utilisation around 30 percent on every GPU. Meanwhile, top on the host showed the 16 DataLoader worker processes at 100 percent CPU each.

Reproducing it

The PyTorch profiler over 50 steps:

with torch.profiler.profile(
    activities=[torch.profiler.ProfilerActivity.CPU, torch.profiler.ProfilerActivity.CUDA],
    schedule=torch.profiler.schedule(wait=5, warmup=5, active=40),
    on_trace_ready=torch.profiler.tensorboard_trace_handler("./prof"),
) as prof:
    for step, batch in enumerate(loader):
        train_step(batch); prof.step()
        if step >= 50: break

The trace showed long gaps on the GPU timeline between steps, during which the main process waited in DataLoader.__next__. Per step: about 90 ms of GPU compute and about 210 ms waiting for data.

What we ruled out

• Small batch size. Batch size 256 per GPU already saturated memory reasonably.
• Synchronisation in training code. No .item() calls or prints in the hot loop.
• Storage throughput. Local NVMe read at over 2 GB/s; data volume per second was far below that.

Root cause

The input pipeline did all work on CPU per step: decode full-size JPEGs (average 2.5 MB, 3000x2000 pixels), random resized crop, colour jitter, normalisation. With 16 workers for 8 GPUs (2 per GPU), CPU could produce about 1,200 images per second, while the GPUs could consume about 3,500. The GPUs were waiting for data most of the time. The worker count was a leftover from the old 4-GPU machine.

What is happening under the hood

Training throughput is limited by the slowest stage: reading, decoding, augmenting, transferring to GPU, computing. DataLoader parallelises the CPU stages across worker processes and prefetches batches, but it cannot exceed what the CPU cores can do. JPEG decoding of large images and per-pixel augmentation are CPU-heavy. When the GPU gets faster (or more GPUs share one CPU), the input pipeline often becomes the bottleneck.

Signs: GPU utilisation well below 90 percent, CPU workers at full load, and profiler gaps between GPU kernels.

Environment

Minimal reproduction

Iterate the DataLoader alone, without training, and measure images per second; compare with training throughput:

t = time.time(); n = 0
for x, y in loader:
    n += x.shape[0]
    if n > 50_000: break
print(n / (time.time() - t), "images/s")    # ~1,200 vs ~3,500 the GPUs can consume

The fix

1. More workers, pinned memory, persistent workers.

loader = DataLoader(
    ds, batch_size=256, sampler=sampler,
    num_workers=7,                 # per GPU process; 56 of 64 cores in total
    pin_memory=True,               # faster host-to-device copies
    persistent_workers=True,       # no worker restart each epoch
    prefetch_factor=4,
)

And .to(device, non_blocking=True) for batches, which overlaps copies with compute when memory is pinned.

2. Decode smaller. Training uses 224x224 crops, so decoding 3000x2000 images was wasteful. We used torchvision's JPEG decoding with reduced-size decoding where supported, and, more importantly, step 3.

3. Preprocess offline once. A one-time job resized all images so the shorter side is 320 pixels and stored them in sharded WebDataset tar files. Random crops and flips still happen during training, now on much smaller images. This reduced average file size from 2.5 MB to 45 KB.

4. Move augmentation to the GPU. Normalisation and colour jitter now run on GPU tensors in the training step, batched, after transfer. They cost almost nothing on the GPU compared with the model.

images = images.to(device, non_blocking=True).float().div_(255)
images = gpu_color_jitter(images)            # batched ops on GPU
images = (images - mean) / std

Alternatives and trade-offs

• NVIDIA DALI moves decoding and augmentation to the GPU with its own pipeline. Very fast; another framework to learn.
• Bigger CPU instances for the same GPUs. Sometimes the simplest fix, if the cloud offers the right shape.
• Mixed approach: offline preprocessing removes most of the cost and keeps the pipeline simple, which is why we led with it.

How to test the fix

Loader-only throughput rose from 1,200 to 9,800 images per second. With training, GPU utilisation went to 93 to 97 percent and the profiler showed no gaps between steps.

Confirming in production

• Epoch time dropped from 58 to 19 minutes; full training from 2.5 days to under a day.
• GPU cost per experiment dropped by about two thirds.
• Model accuracy was unchanged within seed noise, confirming the offline resize did not hurt.

Regression guard

• Training jobs log GPU utilisation and data-wait time per step; a warning is emitted if data wait exceeds 10 percent of step time.

• New datasets go through the preprocessing job before training is allowed on the multi-GPU pool.

• A quarterly review of GPU utilisation across all training jobs, sorted by GPU hours, picks the next pipelines to optimise.

Security implications

None directly. The preprocessed shards are derived from customer-uploaded product images and inherit the source data's access controls and retention.

Prevention checklist

• Measure input pipeline throughput separately from training throughput.
• Profile and look for gaps on the GPU timeline.
• Size DataLoader workers to the CPU cores available per GPU.
• Preprocess expensive, deterministic steps offline.
• Move batched augmentation to the GPU.
• Re-check the input pipeline whenever you move to faster or more GPUs.

References

• PyTorch profiler — https://pytorch.org/tutorials/recipes/recipes/profiler_recipe.html
• PyTorch performance tuning guide — https://pytorch.org/tutorials/recipes/recipes/tuning_guide.html
• DataLoader documentation — https://pytorch.org/docs/stable/data.html
• NVIDIA DALI — https://docs.nvidia.com/deeplearning/dali/user-guide/docs/
• WebDataset — https://github.com/webdataset/webdataset