Monitoring ML models in production when labels arrive weeks later

Our credit risk model predicts whether a loan will default within 12 months. Its real accuracy is known a year after each prediction. Our churn model's labels take 60 days. A fraud model waits for chargebacks, which take weeks. If we only monitored accuracy, we would learn about a broken model long after it had done damage.

The good news: most model failures in production are visible before labels arrive, because they start with the inputs. A schema change upstream, a new customer segment, a broken feature pipeline or a changed default value all show up in the data the model sees. This guide is how we monitor models in the label gap.

Layer 1: data quality, before anything statistical

The most common production incident for our models is not drift. It is broken data: a column suddenly all null, a unit change from cents to dollars, a category renamed upstream. Cheap, deterministic checks catch these:

• Null rate per feature compared with a baseline.
• Range checks for numeric features (min, max, and known valid bounds).
• Unknown category rate for categorical features.
• Row counts per hour or per batch compared with expectations.
• Schema checks: types and presence of fields.

def quality_checks(df, baseline):
    issues = []
    for col in baseline.columns:
        null_rate = df[col].isna().mean()
        if null_rate > baseline.null_rate[col] + 0.05:
            issues.append(f"{col}: null rate {null_rate:.1%} vs baseline {baseline.null_rate[col]:.1%}")
        if col in baseline.categorical:
            unknown = (~df[col].isin(baseline.categories[col])).mean()
            if unknown > 0.02:
                issues.append(f"{col}: {unknown:.1%} unseen categories")
    return issues

These checks run on every scoring batch and page someone when they fail, because a broken input feature is almost always a real incident.

Layer 2: input drift

Drift means the distribution of inputs in production differs from the training data, or from a recent reference window. Two common measures:

Population Stability Index (PSI) for binned features. Bin the reference distribution (deciles for numeric features, categories for categorical ones), compute the share of production data per bin, and sum:

import numpy as np

def psi(ref_counts: np.ndarray, cur_counts: np.ndarray, eps: float = 1e-6) -> float:
    ref = ref_counts / ref_counts.sum() + eps
    cur = cur_counts / cur_counts.sum() + eps
    return float(np.sum((cur - ref) * np.log(cur / ref)))

A widely used rule of thumb: below 0.1 little change, 0.1 to 0.25 moderate, above 0.25 significant. Treat these as starting points and calibrate per feature.

Kolmogorov-Smirnov test for continuous features compares the full distributions. With large samples, KS p-values become tiny for differences nobody cares about, so use the KS statistic (the maximum distance between distributions) with a threshold rather than the p-value.

Choose the reference carefully. Training data is the natural reference, but if training data is a year old, everything drifts a little. We compare against both training data and the trailing four weeks, and alert mostly on sudden changes against the recent window.

Layer 3: prediction drift

The model's output distribution is the single most informative signal without labels. If the share of transactions scored as high risk jumps from 2 to 7 percent overnight, either the world changed or something broke. Monitor:

• Mean and percentiles of scores.
• Share above each decision threshold (what actually drives actions).
• For classifiers, predicted class distribution.

Prediction drift catches cases where many features drift a little and none crosses its individual threshold.

Layer 4: proxy and delayed labels

Some signals arrive faster than true labels: early payment behaviour for credit, first-week engagement for churn, manual review outcomes for fraud. They are imperfect but early. When true labels arrive, compute accuracy by prediction cohort (the week the prediction was made), not by label arrival date, so you can see when a problem started.

Segment, do not just average

Overall drift numbers hide segment problems. A new partner sending traffic from one country can shift that country's inputs dramatically while the global distribution barely moves. We compute drift and prediction metrics per major segment (country, channel, product) and look at the worst segments first.

Alert design

Drift monitoring fails most often through noise. Many features, many segments and daily windows produce false alarms, and teams start ignoring them. What worked for us:

• Page only on data quality failures and large prediction drift. Those are usually real and urgent.
• Report input drift in a daily summary, ranked by feature importance multiplied by drift, not alphabetically.
• Require persistence: drift above threshold for several consecutive windows before alerting.
• Weight by importance: a large shift in a feature the model barely uses matters less than a small shift in its top feature.
• Record decisions: when a drift alert is investigated and judged harmless, note why, so the next person does not repeat the work.

What to do when drift appears

Drift is a question, not an answer. The usual sequence: check data quality and pipelines first, then check whether the change is real in the world (a marketing campaign, a new market, seasonality), then estimate impact using proxy labels or slices with fast labels, and only then decide to retrain, adjust thresholds or do nothing. Retraining on broken data makes things worse.

Common mistakes

Key takeaways

• Most model failures show up in inputs before labels arrive.
• Run cheap data quality checks on every batch and page on failures.
• Track input drift with PSI or KS statistics and prediction drift on decision thresholds.
• Segment metrics and weight by feature importance.
• Treat drift as a prompt to investigate, not an automatic retrain trigger.

References

• Google, ML Test Score: monitoring tests — https://research.google/pubs/the-ml-test-score-a-rubric-for-ml-production-readiness-and-technical-debt-reduction/
• SciPy ks_2samp — https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.ks_2samp.html
• Evidently AI open source documentation — https://docs.evidentlyai.com/
• Great Expectations documentation — https://docs.greatexpectations.io/