CI/CD for machine learning: testing data, code and models before they ship

Our early ML deployments followed a familiar pattern: a data scientist trained a model in a notebook, saved a file, and an engineer wrapped it in a service. It worked until someone asked which data the production model was trained on, or why the retrained model behaved differently, or whether a library upgrade changed predictions. Nobody could answer quickly.

A software CI/CD pipeline tests one thing that changes: code. An ML system has three: code, data and the trained model. Each can break production on its own. This guide describes the pipeline we built for our ranking and classification models. It is tool-agnostic; we use GitHub Actions, MLflow, and a workflow orchestrator for training, but the stages map to any stack.

Version all three inputs

Every production model must be traceable to:

• Code: a Git commit for training and serving code.
• Data: an immutable snapshot of the training data, identified by a version (a dated, read-only table partition, a DVC or lakeFS commit, or a manifest of file hashes).
• Environment: a locked dependency file and a container image digest.

The training run records all three in the model registry entry. If you cannot rebuild a model from these, you cannot debug it.

Stage 1: code CI on every pull request

Fast checks, minutes, no GPUs:

• Linting, type checks, unit tests for feature code and utilities.
• Feature logic tests with small fixed inputs and expected outputs, including edge cases (nulls, unseen categories, extreme values).
• Training smoke test: train on a tiny sample for a few steps and assert the loss decreases and the pipeline runs end to end. This catches broken imports, shape mismatches and config errors before an expensive run.
• Serving contract test: the serving code loads a small model artifact and answers a request with the expected schema.

Stage 2: data validation before training

Training on bad data produces a bad model that passes code tests. Before each training run:

• Schema checks: columns, types, non-null constraints.
• Distribution checks against the previous training snapshot: row counts, null rates, category sets, numeric ranges.
• Label checks: label distribution and label delay (are recent rows labelled yet?).
• Leakage checks: no features computed after the prediction time (see our post on point-in-time joins).

Failures stop the pipeline and notify the owner. Warnings are attached to the run.

Stage 3: reproducible training

Training runs in a pipeline, not a notebook: fixed container image, pinned data version, recorded config and seed, logged to the experiment tracker. For models where comparisons matter, run several seeds or use deterministic settings (see our post on reproducible PyTorch training). The output is a candidate model version in the registry, tagged with its inputs.

Stage 4: model evaluation gates

A candidate must pass gates before promotion:

Gate results are written as tags on the model version. Humans review a short report for significant models, but the gates themselves are automated.

results = evaluate(candidate, champion, holdout)
for gate in GATES:
    passed, detail = gate.check(results)
    client.set_model_version_tag(MODEL, candidate.version, f"gate.{gate.name}", "passed" if passed else "failed")
    if not passed:
        raise GateFailed(gate.name, detail)

Stage 5: deployment

Promotion moves an alias in the registry; deployment pipelines react to alias changes. Rollout goes through shadow and canary stages with business guardrails (see our post on shadow and canary deployments), and rollback moves the alias back.

Serving and training share feature code as a package, versioned together, to prevent training-serving skew.

Stage 6: continuous training, if you need it

Some models need regular retraining as data changes. Treat scheduled retraining as the same pipeline triggered by time or by drift signals, with the same gates. Automatic retraining without gates is how a broken upstream table quietly becomes a broken production model.

Dependency upgrades

Library upgrades can change model behaviour: numeric defaults, random number generators, serialisation formats. Upgrades to ML libraries run the evaluation suite against the current champion's artifacts and data snapshot, and compare predictions on a fixed sample with a tolerance. Pickled model formats also tie you to library versions; prefer portable formats where possible.

Where to start

You do not need every stage on day one. In our experience the order of value is: version data and environments, add a smoke training test in CI, add evaluation gates against the current champion, then automate deployment through the registry. Each step pays off on its own, and each makes the next one easier.

Common mistakes

Key takeaways

• Code, data and model each need versioning and tests.
• Validate data before training, not after deployment.
• Gate promotion on evaluation, segments, behaviour and operational limits.
• Deploy through aliases with shadow, canary and fast rollback.
• Treat library upgrades as model changes.

References

• Google Cloud, MLOps: continuous delivery and automation pipelines in machine learning — https://cloud.google.com/architecture/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning
• The ML Test Score rubric (Breck et al.) — https://research.google/pubs/the-ml-test-score-a-rubric-for-ml-production-readiness-and-technical-debt-reduction/
• Beyond Accuracy: Behavioral Testing of NLP Models with CheckList — https://arxiv.org/abs/2005.04118
• MLflow model registry — https://mlflow.org/docs/latest/model-registry.html