Flaky tests, test frameworks, E2E, mocking.
Four engineers, one invoicing product, no QA team. This is the risk-based testing strategy we settled on: integration tests over mocks, contract tests at service edges, a handful of E2E happy paths and test data that does not rot.
A prompt tweak that looks better on five hand-picked examples can quietly break a tenth of real traffic. This guide shows how to build an eval set from production, choose graders, use LLM-as-judge without fooling yourself and gate changes in CI.
After we set "type": "module" in package.json, jest.mock stopped replacing our mailer and tests started calling the real SMTP client. Static ESM imports are linked before any mock runs. Here is the mechanism and the fix with unstable_mockModule.
Running the test suite inside our app container dropped every table in the development database. Docker Compose injected DB_DATABASE as a real environment variable, and phpunit.xml's env entries do not override existing variables. Here is the full precedence chain and three layers of defence.
Our CI compared LLM summaries against stored snapshots at temperature 0 and failed about one run in five. Temperature 0 is not a determinism guarantee on hosted models. We replaced string snapshots with property checks and a scored eval.
One CI run in eight failed somewhere in our checkout E2E suite, never in the same test twice. The causes were fixed sleeps, URL checks that raced navigation and text selectors that matched two buttons. Web-first assertions, role locators and traces fixed it.
A good bug report lets someone who has never seen your screen reproduce the problem in minutes. The anatomy we use: environment, exact steps, expected versus actual, evidence, a minimal reproduction, and severity kept separate from priority.
A billing test passed for weeks and failed this morning, 31 January, for every engineer at once. JavaScript's setMonth rolled 31 January into 3 March, and the test read the real clock. We fixed the arithmetic, injected the clock and added property tests over every day.