Our image model trained on an expensive GPU that sat idle 70 percent of the time. Profiling showed the GPU waiting on CPU-side JPEG decoding and augmentation in too few DataLoader workers. More workers, pinned memory, GPU-side augmentation and preprocessed shards tripled throughput.
We enabled fp16 autocast to speed up training and the loss turned NaN within 300 steps. A custom attention score and a loss computed in half precision overflowed fp16's range, and we had not used a gradient scaler. bf16 on our GPUs, fp32 for the sensitive ops and a NaN guard fixed it.
Two training runs with the same seed and data ended with validation accuracy 0.3 points apart, enough to flip a model comparison. Enabling deterministic algorithms then crashed with a CuBLAS error. Seeding every RNG, DataLoader workers, cuDNN flags and CUBLAS_WORKSPACE_CONFIG made runs repeatable.