A model can win every offline metric and still hurt the business when it goes live. We have seen all the usual reasons: preprocessing that differs in production, latency that is fine in a notebook and too slow at peak, a recommendation model that improves click rate and reduces revenue, a fraud model whose new thresholds overwhelm the review team. Offline evaluation answers "is this model better on historical data?" Production answers different questions.

This guide is the rollout process we use for every model that makes decisions affecting users or money. It is deliberately boring: each stage reduces a specific risk before the next one exposes more users.

Stage 0: offline gates

Before a model leaves the training pipeline:

  • Beats the current model on a time-based holdout, by a margin larger than run-to-run noise.
  • No segment regresses beyond a tolerance: by country, customer type, device, or whatever segments matter.
  • Calibration checks for models whose scores drive thresholds.
  • Fairness and policy checks where required.
  • Latency and size within serving budgets, measured on the target hardware.
  • Parity test between the training pipeline and the serving artifact (see our post on training-serving skew).

The result is recorded as tags on the model version in the registry; promotion jobs refuse versions without passing tags.

Stage 1: shadow deployment

The new model receives a copy of real production traffic, computes predictions, and logs them, but its predictions are not used. The current model still makes every decision.

What shadow mode tells you:

  • Operational health: errors, latency at real load, memory, cold starts.
  • Prediction agreement with the current model: how often decisions would differ, and in which segments.
  • Feature health: whether production features look like training features for this model.
  • Score distributions: whether thresholds will trigger the expected volume of actions.

Implementation choices: mirror traffic at the gateway or service mesh level, or have the serving service call both models and log the challenger asynchronously. We use the second, because it lets us log both predictions with the same request id:

async def predict(request):
    features = build_features(request)
    champion = await champion_model.predict(features)
    asyncio.create_task(shadow(features, request.id))      # never blocks the response
    return champion

async def shadow(features, request_id):
    try:
        challenger = await asyncio.wait_for(challenger_model.predict(features), timeout=0.5)
        log_prediction(request_id, "challenger", challenger)
    except Exception as exc:
        metrics.increment("shadow.errors", tags={"type": type(exc).__name__})

The shadow path has its own timeout and error handling so it cannot hurt the real response. Shadow traffic costs compute; for expensive models we shadow a sample.

Shadow mode cannot tell you how users react, because they never see the new model's output.

Stage 2: canary

A small share of real decisions, typically 1 to 5 percent, use the new model. Monitor:

  • The same operational metrics as in shadow mode.
  • Business guardrail metrics for the canary group versus the rest: conversion, revenue per session, approval rate, manual review volume, complaints.
  • Downstream effects: a fraud model that flags more transactions increases review queue length; a pricing model changes refund requests.

Define in advance what aborts the canary, for example "review queue volume more than 20 percent above control" or "error rate above 0.5 percent". Automate the abort where you can.

Canary groups should be assigned by a stable key (user id, account id), not per request, so a user does not bounce between models and you can measure per-user outcomes.

Stage 3: A/B test, when the question is business impact

If the new model is meant to improve a business metric, a canary is too small and too short to measure it. Run a proper experiment: decide the primary metric and minimum detectable effect, compute sample size and duration, and do not stop early when the curve looks good. Many model changes, such as retrains on fresh data with the same objective, do not need an A/B test; a canary with guardrails is enough.

Stage 4: full rollout and rollback

Ramp in steps (5, 25, 50, 100 percent) with the same guardrails. Keep the previous model deployed and warm for a while, and make rollback a configuration change: in our case, moving the registry alias back to previous_champion and restarting the serving pods. Time a rollback drill once a quarter.

Logging for comparison

Every stage depends on logging the right data:

  • Request id, model name and version, timestamp.
  • Input features or a hash plus a sample of full features.
  • Prediction, score, and the decision taken.
  • Later, joined: outcomes and labels.

Without these, you cannot answer "would the new model have done better on last week's traffic?" or "which model made this decision?", which is also a frequent audit question.

How long each stage takes

For a routine retrain of an existing model, our stages take about a week: a day of offline evaluation, three days of shadow traffic to cover weekday and weekend patterns, two days of canary at 5 percent, then a ramp over a day. For a new model type or a new decision, we extend shadow and canary to two weeks each and usually add an A/B test. The schedule is written in the model's rollout plan before training starts, so nobody shortens it under deadline pressure.

Common mistakes

Mistake Why it hurts Do this instead
Going from offline straight to 100 percent Production-only problems hit everyone Shadow, then canary, then ramp
Shadow path in the request's critical path Challenger slowness hurts users Asynchronous with its own timeout
Per-request random assignment Users see both models, metrics are muddy Assign by stable user or account key
No pre-defined abort criteria Debates during incidents Write guardrails and thresholds first
Not logging model version per decision Cannot audit or compare Log version, score and decision per request

Key takeaways

  • Offline gates, shadow, canary, and ramp each remove a different risk.
  • Shadow mode tests operations and agreement without user impact.
  • Canaries need business guardrails and stable assignment.
  • Use A/B tests when the goal is business impact, with fixed duration.
  • Keep rollback a configuration change and practise it.

References