scikit-learn InconsistentVersionWarning: unpickling a model from another version

Our scoring service hosts about 30 small scikit-learn models for pricing and routing decisions. A Dependabot pull request bumped scikit-learn from 1.4.2 to 1.7.2 along with NumPy and SciPy. Tests passed and the change was merged. After deploy, the logs filled with:

/usr/local/lib/python3.12/site-packages/sklearn/base.py: InconsistentVersionWarning: Trying to unpickle
estimator DecisionTreeRegressor from version 1.4.2 when using version 1.7.2. This might lead to breaking
code or invalid results. Use at your own risk. For more info please refer to:
https://scikit-learn.org/stable/model_persistence.html#security-maintainability-limitations

A week later, the pricing team noticed that the average discount suggested by one model had changed by 0.4 percent without any model update.

Reproducing it

We loaded that model in two environments, one with 1.4.2 and one with 1.7.2, and scored the same 100,000 historical rows:

max_abs_diff:   0.0213
mean_abs_diff:  0.0009
rows_changed:   3.1%

Most models gave identical predictions; this one, a pipeline with a HistGradientBoostingRegressor and a custom preprocessing step depending on library defaults, did not.

What we ruled out

• Input data changes. We scored the same stored rows in both environments.
• Randomness. Prediction is deterministic for these models.

Root cause

Pickled scikit-learn models store the internal state of Python objects, not a version-independent description of the model. Loading them with a different version can work, fail, or work with subtly different behaviour, because internal attributes, default parameters or numerical code paths change between versions. Our service had no rule tying the library version to the version used in training, and our tests only checked that models loaded and returned numbers of the right shape.

What is happening under the hood

pickle and joblib serialise the object graph of an estimator: its class path and attributes. When unpickling, the current library version's class is used with the stored attributes. scikit-learn records the version an estimator was pickled with and emits InconsistentVersionWarning when it differs. The documentation is explicit that loading across versions is not supported.

Sources of drift include changes in default parameters for components that are re-initialised, changes in tree traversal or numeric precision, deprecations that change behaviour of preprocessing steps, and differences in NumPy or SciPy underneath.

There is also a security aspect: unpickling can execute arbitrary code, so pickles must only come from trusted sources.

Environment

Minimal reproduction

python -m venv old && old/bin/pip install scikit-learn==1.4.2 joblib
old/bin/python -c "import joblib; from sklearn.tree import DecisionTreeRegressor as D; import numpy as np; \
  joblib.dump(D().fit(np.random.rand(100,3), np.random.rand(100)), 'm.joblib')"
python -m venv new && new/bin/pip install scikit-learn==1.7.2 joblib
new/bin/python -W error::UserWarning -c "import joblib; joblib.load('m.joblib')"   # warning becomes an error

The fix

1. Pin the environment with each model. The training pipeline writes a small manifest next to every model:

{
  "model": "pricing-discount",
  "version": "27",
  "python": "3.12.6",
  "requirements": {"scikit-learn": "1.4.2", "numpy": "1.26.4", "scipy": "1.13.1"},
  "sha256": "9f1c...e2"
}

2. Check at load time and fail loudly. The service compares the manifest with installed versions and refuses to load a mismatched model:

import warnings
from importlib.metadata import version
from sklearn.exceptions import InconsistentVersionWarning

def load_model(path: str, manifest: dict):
    for pkg, required in manifest["requirements"].items():
        if version(pkg) != required:
            raise RuntimeError(f"{manifest['model']}: needs {pkg}=={required}, found {version(pkg)}")
    with warnings.catch_warnings():
        warnings.simplefilter("error", InconsistentVersionWarning)
        return joblib.load(path)

3. Upgrade by retraining, not by loading. Library upgrades now go through retraining all models with the new versions, comparing predictions on a fixed evaluation set, and deploying models and service together.

4. Portable formats where possible. We exported 24 of the 30 models to ONNX with skl2onnx, which serving runs with ONNX Runtime, independent of scikit-learn's version. For models with components ONNX cannot represent, we use skops, which stores models in a format that avoids arbitrary code execution and lets us review the types it contains before loading.

Alternatives and trade-offs

• Pin scikit-learn forever. Stops the drift and blocks security and performance updates; dependencies like NumPy eventually force an upgrade anyway.
• One service image per model version. Perfect isolation, more images to maintain.
• Ignore the warning when tests pass. That is what caused this incident.

How to test the fix

A CI job loads every model with the service's environment and compares predictions with reference outputs stored at training time, with a tolerance of 1e-9 for pickles and 1e-5 for ONNX exports. Loading a model with a mismatched manifest must raise.

Confirming in production

• We rolled back the scikit-learn upgrade, retrained all 30 models with 1.7.2, verified predictions against business expectations, and redeployed models and service together.
• The pricing model's outputs returned to expected behaviour; the 0.4 percent discount shift was traced entirely to the version mismatch.
• No InconsistentVersionWarning in logs since.

Regression guard

• Dependabot pull requests for ML libraries are labelled and routed through the retraining pipeline instead of auto-merge.
• The load-time check and the prediction comparison test in CI.

Security implications

Pickled models are executable code in practice. We restricted write access to the model bucket to the training pipeline, verify the SHA-256 in the manifest before loading, and moved most models to ONNX, which does not execute arbitrary code on load. For the remaining skops files, the loader only accepts an explicit list of trusted types.

Prevention checklist

• Record library versions with every saved model.
• Refuse to load models with mismatched library versions.
• Treat ML library upgrades as model changes: retrain and compare.
• Prefer portable, non-executable formats like ONNX or skops.
• Load pickles only from trusted, access-controlled storage.

References

• scikit-learn model persistence — https://scikit-learn.org/stable/model_persistence.html
• InconsistentVersionWarning — https://scikit-learn.org/stable/modules/generated/sklearn.exceptions.InconsistentVersionWarning.html
• skops persistence — https://skops.readthedocs.io/en/stable/persistence.html
• sklearn-onnx — https://onnx.ai/sklearn-onnx/