Our image model trained on an expensive GPU that sat idle 70 percent of the time. Profiling showed the GPU waiting on CPU-side JPEG decoding and augmentation in too few DataLoader workers. More workers, pinned memory, GPU-side augmentation and preprocessed shards tripled throughput.
We enabled fp16 autocast to speed up training and the loss turned NaN within 300 steps. A custom attention score and a loss computed in half precision overflowed fp16's range, and we had not used a gradient scaler. bf16 on our GPUs, fp32 for the sensitive ops and a NaN guard fixed it.
Two training runs with the same seed and data ended with validation accuracy 0.3 points apart, enough to flip a model comparison. Enabling deterministic algorithms then crashed with a CuBLAS error. Seeding every RNG, DataLoader workers, cuDNN flags and CUBLAS_WORKSPACE_CONFIG made runs repeatable.
After upgrading to PyTorch 2.6, every checkpoint load in our evaluation pipeline failed with Weights only load failed. The default for weights_only in torch.load flipped to True, and our checkpoints contained a NumPy scalar. We fixed the checkpoints instead of turning the safety off.
After rebuilding our inference image, models silently ran on CPU on half the GPU fleet. The new PyTorch wheel targets CUDA 12, and older nodes still had a driver that only supports CUDA 11. We aligned drivers, added a startup check that fails without a GPU, and pinned images.
Training jobs that ran fine on a workstation crashed in containers with a bus error in DataLoader workers. Workers pass batches through shared memory, and containers get only 64 MB of /dev/shm by default. A memory-backed emptyDir in Kubernetes and --shm-size in Docker fixed it.
Our text classification service crashed with CUDA out of memory every few hours, always on a big batch. Logits kept for drift monitoring were holding autograd graphs alive, batches were sized by count not tokens, and the allocator was fragmented. How we measured each one and fixed it.