All unit tests passed and the release failed in production: our mock had drifted from the real API

We integrate with a parcel carrier's API to create shipping labels. A release that refactored the integration had 96 percent line coverage on the module, 140 passing unit tests and a green pipeline. Within minutes of the deploy, every label request failed:

ERROR shipping.labels - label creation failed for shipment SHP-771203
Traceback (most recent call last):
  File "shipping/carrier.py", line 88, in create_label
    tracking = Tracking(number=resp["tracking"], url=resp["tracking_url"])
  File "shipping/models.py", line 21, in __init__
    raise ValueError(f"invalid tracking number: {number!r}")
ValueError: invalid tracking number: {'number': '1Z999AA10123456784', 'url': 'https://...'}

The carrier's response had tracking as an object:

{
  "label_id": "lbl_8a91",
  "tracking": { "number": "1Z999AA10123456784", "url": "https://track.carrier.example/1Z999..." },
  "label_pdf": "JVBERi0xLjQK..."
}

Our code expected a string in tracking and a separate tracking_url. That was the shape in the carrier's v2 API. We had migrated to v3 in the refactor, and v3 changed it.

The tests

The unit tests mocked the HTTP client:

def test_create_label_returns_tracking(mocker):
    http = mocker.Mock()
    http.post.return_value.json.return_value = {
        "label_id": "lbl_1",
        "tracking": "1Z999AA10123456784",
        "tracking_url": "https://track.example/1Z999",
        "label_pdf": "JVBERi0=",
    }
    client = CarrierClient(http=http, base_url="https://api.carrier.example/v3")
    label = client.create_label(sample_shipment())
    assert label.tracking.number == "1Z999AA10123456784"

The developer had changed the base URL to v3, updated the request body to v3's format (and verified that against the documentation), but left the mocked response in v2's shape. The test passed because it checked our code against our own description of the API, and that description was wrong.

Every one of the 140 tests used a hand-written response dictionary like this one. None of them had ever been compared with a real response.

What a mock proves

A mock replaces a dependency with an object that returns whatever the test says. That is useful for testing our logic in isolation: given this response, does our code build the right label? It proves nothing about whether the response is realistic. When the real API changes, or when we misunderstand it, the mock keeps returning the old shape, and every test keeps passing.

This is mock drift: the test double and the real dependency diverge, and nothing notices. It is especially likely with third-party APIs, which change on someone else's schedule, and with hand-written mocks spread across many tests, where updating all of them is tedious.

Mocks, stubs and fakes

The terms are often used loosely, but the distinction matters here:

Our tests used mocks as stubs, with response shapes duplicated in 140 places. A single fake carrier client, maintained in one place and checked against the real API, would have been one thing to update, not 140, and one thing to verify.

Coverage measured the wrong thing

96 percent coverage meant 96 percent of our lines ran during tests. It said nothing about whether the inputs those lines received resembled reality. High coverage on top of unrealistic doubles gives confidence that is not earned.

Environment

1. Roll back, then fix the parser

We rolled back, fixed the response parsing for v3, and redeployed after the checks below.

2. One fake, built from recorded real responses

We replaced the scattered dictionaries with a single fake HTTP transport that serves recorded responses from the carrier's sandbox. Recording is done by a script that calls the sandbox and stores responses as fixture files:

python -m tools.record_carrier_fixtures --env sandbox --out tests/fixtures/carrier/v3/

recorded create_label_domestic.json        200
recorded create_label_international.json   200
recorded create_label_invalid_postcode.json 422
recorded rate_limited.json                 429

The fake loads them:

class FakeCarrierTransport:
    def __init__(self, fixtures_dir: Path):
        self.fixtures = {p.stem: json.loads(p.read_text()) for p in fixtures_dir.glob("*.json")}
        self.calls: list[dict] = []

    def post(self, path: str, json: dict) -> FakeResponse:
        self.calls.append({"path": path, "json": json})
        name = "create_label_international" if json["to"]["country"] != "GB" else "create_label_domestic"
        return FakeResponse(self.fixtures[name])

Unit tests use the fake. When the API changes, we re-record, and every test that depended on the old shape fails at once, instead of none.

3. Validate fixtures against the provider's schema

The carrier publishes an OpenAPI document for v3. A test validates every recorded fixture against it:

@pytest.mark.parametrize("fixture", sorted(FIXTURES.glob("create_label_*.json")))
def test_fixture_matches_published_schema(fixture, carrier_openapi):
    body = json.loads(fixture.read_text())
    status = body.pop("_status", 200)
    validate_response(carrier_openapi, "POST", "/v3/labels", status, body)

If the carrier publishes a new schema version that changes the response, a scheduled job that downloads it fails this test before our code meets the change in production.

4. A small number of real calls

Some confidence only comes from talking to the real thing. We added a sandbox smoke test that runs on every release candidate (and nightly), making three real calls to the carrier's sandbox: create a domestic label, create an international label, and a request with an invalid postcode. It takes about four seconds. It would have failed on this release immediately.

5. A test mix, not just more unit tests

For integrations with external systems, our guideline is now:

The classic test pyramid still applies: most tests should be fast and isolated. But the base of the pyramid is only trustworthy if something above it checks the assumptions it is built on.

1. The fixtures catch the original bug. We re-recorded fixtures from the v3 sandbox and ran the old parsing code against the new fake. 31 tests failed with the same ValueError seen in production. With the corrected parser, all pass.

2. The schema check catches drift. We edited a fixture to use the v2 shape (string tracking). The schema test failed with 'tracking' is not of type 'object'.

3. The smoke test catches it end to end. Running the sandbox smoke test against the original broken release build failed on the first call.

4. Production. A synthetic check now creates a label against the production API every 15 minutes using a test account the carrier provides, and voids it immediately. It alerts on any failure.

5. Security implications. Recorded fixtures can contain real data. The recording script replaces addresses, names and phone numbers with synthetic values and strips authentication headers before saving, and a CI check rejects fixtures containing strings that look like API keys.

Takeaways

• A mock proves your code works against your assumptions, not against the real dependency.
• Hand-written response mocks spread across many tests drift silently when the API changes.
• Use one fake built from recorded real responses, and validate the recordings against the provider's schema.
• Add a few real sandbox calls to the release pipeline; they catch what no amount of mocked coverage can.

References: Martin Fowler, Mocks Aren't Stubs, Martin Fowler, The Practical Test Pyramid, Python unittest.mock, OpenAPI Specification.