#cuda

AI & Machine Learning Bug Report

Identical PyTorch training runs give different results: seeds, cuDNN and CUBLAS_WORKSPACE_CONFIG

Ryan Pham · Dec 23, 2025 · 6 min read

Two training runs with the same seed and data ended with validation accuracy 0.3 points apart, enough to flip a model comparison. Enabling deterministic algorithms then crashed with a CuBLAS error. Seeding every RNG, DataLoader workers, cuDNN flags and CUBLAS_WORKSPACE_CONFIG made runs repeatable.

AI & Machine Learning Bug Report

torch.cuda.is_available() is False after an image rebuild: driver too old for CUDA 12

Ryan Pham · Feb 3, 2025 · 6 min read

After rebuilding our inference image, models silently ran on CPU on half the GPU fleet. The new PyTorch wheel targets CUDA 12, and older nodes still had a driver that only supports CUDA 11. We aligned drivers, added a startup check that fails without a GPU, and pinned images.

Performance & Scaling Bug Report

PyTorch CUDA out of memory in an inference service: autograd, batch size and fragmentation

Daniel Tran · Dec 26, 2024 · 9 min read

Our text classification service crashed with CUDA out of memory every few hours, always on a big batch. Logits kept for drift monitoring were holding autograd graphs alive, batches were sized by count not tokens, and the allocator was fragmented. How we measured each one and fixed it.