Software CI checks code. ML systems also change when data changes, when a model is retrained, or when a dependency shifts numerics. This guide lays out a CI/CD pipeline for ML: versioning, tests for data and models, reproducible training, promotion gates and deployment.
A new model that wins offline can still lose in production, through skew, latency, or effects on users that offline data cannot show. This guide covers a staged rollout for models: offline gates, shadow traffic, canaries on business metrics, A/B tests when needed, and fast rollback.
After we redeployed our MLflow tracking server on Kubernetes, every run older than the deploy showed metrics but no artifacts, and model loading failed. The server wrote artifacts to a local directory in the pod. We moved to proxied artifact storage on S3 and recovered what we could.
Accuracy is the metric everyone wants, and for many models the labels arrive weeks or months after the prediction. This guide covers what to monitor in the meantime: data quality, input drift with PSI and KS tests, prediction drift, and alert design that does not cry wolf.
Our fraud model scored 0.91 AUC on holdout data and close to random in production. The serving service re-implemented preprocessing in another language: the scaler was refit on each batch, one-hot categories came in a different order and text was not lowercased. Shipping the pipeline with the model fixed it.
Data scientists promoted a new churn model by setting the champion alias, as the newer MLflow docs recommend. Our deploy script still loaded models:/churn/Production, which pointed at the old version. Stages are deprecated; we moved the whole pipeline to aliases and added a promotion check.