#reproducibility

DevOps & CI/CD

CI/CD for machine learning: testing data, code and models before they ship

Ryan Pham · Jun 24, 2026 · 5 min read

Software CI checks code. ML systems also change when data changes, when a model is retrained, or when a dependency shifts numerics. This guide lays out a CI/CD pipeline for ML: versioning, tests for data and models, reproducible training, promotion gates and deployment.

AI & Machine Learning Bug Report

Identical PyTorch training runs give different results: seeds, cuDNN and CUBLAS_WORKSPACE_CONFIG

Ryan Pham · Dec 23, 2025 · 6 min read

Two training runs with the same seed and data ended with validation accuracy 0.3 points apart, enough to flip a model comparison. Enabling deterministic algorithms then crashed with a CuBLAS error. Seeding every RNG, DataLoader workers, cuDNN flags and CUBLAS_WORKSPACE_CONFIG made runs repeatable.