LLM snapshot tests flaky at temperature 0: why outputs still change

We had a pull request check that ran our ticket summariser against 40 fixed support tickets and compared each output to a stored snapshot. The model was called with temperature=0, so we assumed the output would be stable. The check failed on roughly one run in five with diffs like this:

FAILED tests/test_summaries.py::test_snapshot[ticket_017]
AssertionError: snapshot mismatch
- Customer reports that invoices since March show the wrong VAT rate.
+ The customer reports that invoices since March show an incorrect VAT rate.

Same model name, same prompt, same input, different wording. Engineers learned to click "re-run", which is the worst habit a test suite can teach.

Reproducing it

We called the API 50 times in a loop with the same request at temperature 0 and hashed the outputs:

from collections import Counter
outs = Counter(hash_text(summarise(TICKET_017)) for _ in range(50))
print(outs.most_common())
# [('a91f…', 41), ('5c02…', 7), ('e7d4…', 2)]

Three distinct outputs out of 50. Short, highly constrained prompts were stable; longer free-text outputs drifted more, usually after the first sentence.

What we ruled out

• Hidden randomness in our prompt. No timestamps, no shuffled examples. We diffed request bodies byte for byte.
• Model version changes. We pinned a dated model snapshot, and the variation happened within minutes.
• Our own retries. Retries were disabled in the test client.

Root cause

Temperature 0 means greedy decoding: pick the most likely next token. It does not make the computation behind those probabilities exact. On a hosted inference stack, requests are batched with other traffic, kernels pick different execution paths, and floating point addition is not associative, so logits can differ in the last bits between runs. When two candidate tokens are nearly tied, a tiny difference flips the choice, and every token after that follows a different path. Our test asserted exact strings on a system that does not promise exact strings.

What is happening under the hood

A transformer computes logits with large matrix multiplications on GPUs. The sum of many floating point numbers depends on the order in which they are added. Batch size, padding, and kernel choice change that order, and on a shared serving cluster your request is batched together with requests you cannot see. Most of the time the top token wins by a wide margin and nothing changes. Occasionally two tokens are within rounding noise of each other, and the output forks.

Some APIs offer a seed parameter and return a system_fingerprint to make results more repeatable. Providers describe this as best effort: determinism can still break when the backend changes, and some models or endpoints ignore it.

Environment

Minimal reproduction

def test_snapshot(ticket, snapshot):
    out = summarise(ticket, temperature=0)
    assert out == snapshot.read(ticket.id)       # brittle by design

Run it ten times in a row and it will fail at least once on long outputs.

Who is affected

Anyone using string equality, or snapshot tooling, on generated text from hosted models. It also bites caching designs that assume "same input, same output" and evaluation pipelines that run a single sample per case.

The fix

We split the old test into what we actually cared about.

1. Property tests for hard requirements. These are deterministic checks on the output, not on its wording.

import re

def test_summary_properties(ticket):
    out = summarise(ticket, temperature=0)
    assert len(out.split()) <= 60                         # length budget
    assert ticket.order_id in out                         # key fact kept
    assert not re.search(r"\b(I think|As an AI)\b", out)  # style rules
    assert ticket.customer_email not in out               # no PII echo

2. Structured output where possible. Fields like category, priority and product moved into a JSON schema response. Enum fields are stable across runs, and when they are not, that is a real disagreement worth seeing.

result = classify(ticket)          # returns validated pydantic model
assert result.category == ticket.expected_category

3. A scored eval for quality, outside the unit test suite. The 40 tickets became an eval set. Each output is scored by a rubric (facts present, no invented facts, length) and the pull request check compares the average score to the main branch with a tolerance:

score = run_eval(dataset="tickets_v3", samples_per_case=3)
assert score.mean >= BASELINE.mean - 0.02, f"quality dropped: {score.mean:.3f}"

Three samples per case smooths out the occasional fork.

Alternatives and trade-offs

• Record and replay. Store API responses and replay them in unit tests. Great for testing your parsing and control flow, useless for testing the model's behaviour.
• Self-hosted model with deterministic settings. Possible with fixed batch size and deterministic kernels, at a large throughput cost. Worth it only when exact reproducibility is a hard requirement.
• Use seed. It reduced variation in our test but did not remove it, so we did not rely on it.

How to test the fix

We ran the new property tests 30 times in a loop on CI: zero failures. The eval check ran 10 times on an unchanged branch; scores varied by at most 0.008, inside the 0.02 tolerance. Then we deliberately broke the prompt (removed the instruction to include order ids) and the property test failed on every run, so the suite still catches real regressions.

Confirming in production

Over the next month the summariser check had zero flaky failures, and the "re-run" button stopped being muscle memory. Two genuine regressions were caught: one prompt change that made summaries too long and one model upgrade that dropped order ids in 6 percent of cases.

Regression guard

• A pytest plugin that fails if any test in tests/llm/ uses == against a snapshot of generated prose.
• Eval baselines stored per model snapshot and prompt version.
• Flaky test tracking in CI: any test that passes on retry is reported to the owning team.

Security implications

Exact-match tests hid one issue that the property tests found: in two tickets the summary repeated the customer's email address. That is a data minimisation problem for a summary that is shown to other teams. The new PII property check covers it. Nothing in the fix itself changes attack surface.

Prevention checklist

• Do not assert exact strings on generated text.
• Test hard requirements as properties, and push structured fields into schemas.
• Measure quality with an eval set and a tolerance, using several samples.
• Pin model snapshots and record them with eval results.
• Treat a test that passes on retry as a bug in the test.

References

• OpenAI reproducible outputs (seed, system_fingerprint) — https://platform.openai.com/docs/advanced-usage
• OpenAI evals guide — https://platform.openai.com/docs/guides/evals
• What every computer scientist should know about floating point arithmetic — https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html