Model with 0.91 AUC offline performs near random online: training-serving skew

A new fraud detection model for card-not-present payments passed every offline check: 0.91 AUC on a time-based holdout, well calibrated, stable across segments. Two weeks after launch in shadow mode, comparing its scores with confirmed fraud labels gave an AUC of 0.56. Barely better than a coin.

Reproducing it

We logged the exact feature vectors the serving service sent to the model and compared them with what the training pipeline would compute for the same transactions, recomputed offline from raw data:

feature                 training_mean   serving_mean   max_abs_diff
amount_scaled                0.00            0.03          4.12
merchant_category_5411       0.08            0.00          1.00
merchant_category_5812       0.05            0.13          1.00
email_domain_is_free         0.41            0.07          1.00

The same transaction produced different features in training and in serving.

What we ruled out

• Concept drift. Fraud patterns change, but not this fast, and offline evaluation on the most recent month before launch was still 0.90.
• Label delay. We waited for chargebacks to mature before computing online AUC.
• The model artifact. Loading the same artifact in a notebook and feeding it training-style features reproduced offline scores exactly.

Root cause

Training used a scikit-learn pipeline in Python. The serving service, written in Go for latency, re-implemented preprocessing by hand from a design document. Three differences:

1. Scaling: training fit a StandardScaler on the training set. Serving computed mean and standard deviation from each incoming micro-batch, so the same amount got different scaled values depending on what else was in the batch.
2. One-hot order: training's OneHotEncoder sorted categories; serving used the order categories appeared in a config file. Columns were shifted, so the model read "restaurant" where it expected "grocery".
3. Text normalisation: training lowercased email domains before comparing against a list of free providers; serving did not, so Gmail.com was not recognised.

What is happening under the hood

A model learns a function of its input features. If production computes those features differently, the model is evaluated on a distribution it never saw, and offline metrics stop meaning anything. This is training-serving skew. Common sources:

• Preprocessing re-implemented in a different language or framework.
• Statistics (means, vocabularies, category lists) recomputed instead of loaded from training.
• Different data sources for the same feature (warehouse in training, live service in production) with different freshness or definitions.
• Time travel in training data, where training features include information not yet available at prediction time.

Skew is silent: the model returns plausible numbers either way.

Environment

Minimal reproduction

from sklearn.preprocessing import StandardScaler
import numpy as np

train = np.array([[10.0], [20.0], [30.0], [1000.0]])
scaler = StandardScaler().fit(train)
print(scaler.transform([[50.0]]))                  # training-consistent value

batch = np.array([[50.0], [55.0], [60.0]])
print((batch - batch.mean()) / batch.std())        # serving's per-batch scaling: very different

The fix

1. Ship the preprocessing with the model. We export the whole scikit-learn pipeline, preprocessing and model, to ONNX with skl2onnx, so the Go service only passes raw fields and the exported graph does every transformation exactly as in training:

from skl2onnx import to_onnx

pipeline.fit(X_train, y_train)
onx = to_onnx(pipeline, X_train[:1], options={id(pipeline.steps[-1][1]): {"zipmap": False}})
with open("fraud_pipeline.onnx", "wb") as f:
    f.write(onx.SerializeToString())

Custom transformers needed small changes to be convertible: the email domain feature became a lowercase plus lookup step implemented with supported operators, and the free-provider list is embedded in the graph.

2. A parity test before every release. For 10,000 recent transactions, compute predictions with the Python pipeline and with the exported ONNX model inside the Go service's test harness, and require them to match:

np.testing.assert_allclose(python_scores, onnx_scores, rtol=1e-4, atol=1e-5)

3. Log features at serving time. The service logs a sample of raw inputs and transformed features with the prediction. A nightly job recomputes features for the same transactions with the training code and reports differences per feature.

Alternatives and trade-offs

• Serve in Python with the original pipeline. Simplest way to avoid skew; we needed lower latency, and ONNX gave both.
• Feature store with one definition used for training and serving. The right long-term answer for shared features; more infrastructure.
• Re-implement carefully with tests. Possible, and every future change has to be made twice.

How to test the fix

Parity test above in CI, plus a shadow run for one week: online AUC on matured labels reached 0.89, close to offline.

Confirming in production

• Shadow online AUC 0.89 over two weeks, then the model went live.
• The nightly skew report has shown differences only below 1e-4 since, except once when a raw field changed format upstream, which the report caught within a day.

Regression guard

• Parity test as a release gate for model and service changes.
• Nightly skew report with alerts on any feature whose difference exceeds a tolerance.
• Feature distributions in serving compared with training distributions (see our post on drift monitoring).

Security implications

For a fraud model, skew is a security problem: a model that cannot detect fraud lets it through. During shadow mode no decisions depended on the new model, which is exactly why shadow deployments matter for risk models. Feature logs contain payment-related data, so they are stored with the same access controls and retention as transaction data, with card numbers never logged.

Prevention checklist

• Never re-implement preprocessing by hand for serving; export it with the model.
• Load fitted statistics and vocabularies from training artifacts.
• Run a parity test between training and serving code paths.
• Log serving features and compare with training computations.
• Launch risky models in shadow mode first.

References

• Rules of Machine Learning, training-serving skew (Google) — https://developers.google.com/machine-learning/guides/rules-of-ml#training-serving_skew
• scikit-learn pipelines and composite estimators — https://scikit-learn.org/stable/modules/compose.html
• sklearn-onnx — https://onnx.ai/sklearn-onnx/
• ONNX Runtime — https://onnxruntime.ai/docs/