Four engineers, one invoicing product, no QA team. This is the risk-based testing strategy we settled on: integration tests over mocks, contract tests at service edges, a handful of E2E happy paths and test data that does not rot.
A prompt tweak that looks better on five hand-picked examples can quietly break a tenth of real traffic. This guide shows how to build an eval set from production, choose graders, use LLM-as-judge without fooling yourself and gate changes in CI.
Our CI compared LLM summaries against stored snapshots at temperature 0 and failed about one run in five. Temperature 0 is not a determinism guarantee on hosted models. We replaced string snapshots with property checks and a scored eval.