Building eval sets for LLM features that catch real regressions

Most LLM features are changed by people looking at a few examples and deciding the new output "reads better". That works until a change that improves five demo cases breaks a category of real requests nobody looked at. We learned this when a prompt edit that made support answers friendlier also made the assistant stop asking for order numbers, which doubled the number of follow-up messages for refunds.

An eval set is the test suite for behaviour you cannot assert with equality. This guide covers how we build ours, how we grade them, and how they gate changes. It assumes you already ship at least one LLM feature and want to change it without fear.

Start from real traffic, not imagination

Hand-written examples cluster around what the author thinks users do. Real traffic is messier. Build the first version of the set from production:

1. Sample a few hundred real inputs from the last month, stratified by whatever segments matter: intent, language, customer tier, input length.
2. Add every input from bug reports and complaints. These are the cases that already hurt someone.
3. Add a small set of adversarial or edge cases on purpose: empty input, very long input, mixed languages, requests the feature should refuse.
4. Redact personal data on export (see our post on PII in LLM logs), and give the dataset an owner.

A useful starting size is 200 to 500 cases. Fewer and noise dominates; many more and nobody reviews failures.

Each case stores the input, any context the feature uses (retrieved docs, tool results as fixtures), and what "good" means for that case:

{
  "id": "refund-0142",
  "segment": "refund",
  "input": "I was charged twice for my last order, can you fix it?",
  "context_fixture": "fixtures/refund-0142.json",
  "expect": {
    "must_include_any": ["order number", "order id"],
    "must_not_include": ["refund has been issued"],
    "rubric": "Asks for the order id before promising anything. Does not claim an action was taken."
  }
}

Choose graders from cheapest to most expensive

Use the simplest grader that measures what you care about.

In practice each case combines several: rules for the hard requirements and a rubric score for the soft ones. A case passes only if every hard rule passes.

Use LLM-as-judge carefully

Model graders scale, but they have known biases: they favour longer answers, answers in their own style, and the first option when comparing two. They also miss factual errors they cannot verify. A few rules keep them honest:

• Grade against a specific rubric, not "is this good". Ask for a verdict per criterion, and a short reason first.
• Give the judge the evidence. For faithfulness, pass the retrieved context and ask whether every claim in the answer is supported by it.
• Use a different, preferably stronger model than the one being tested, and pin its version.
• Calibrate against humans. Have two people grade 50 cases, measure agreement with the judge, and fix the rubric where they disagree. Repeat when you change the judge.
• For comparisons, randomise order and run both orders.

JUDGE_PROMPT = """You are grading a support assistant reply.
Context documents:
{context}

User message:
{input}

Assistant reply:
{output}

Rubric:
{rubric}

For each rubric item answer PASS or FAIL with one sentence of evidence.
Then output a final line: SCORE=<number of PASS>/<number of items>"""

Parse the score, store the reasons. The reasons are what you read when a case fails.

Measure noise before trusting differences

Generation is not deterministic even at temperature 0, and judges vary too. Before using the eval as a gate, run it several times on an unchanged system and look at the spread. If the overall score moves by 1.5 points between identical runs, a 1-point improvement means nothing. Two things reduce noise: more cases, and several samples per case (we use three and take the mean).

Gate changes in CI

Every change to a prompt, model, retrieval setting or tool schema runs the eval and compares to the main branch baseline:

- name: LLM evals
  run: python -m evals.run --dataset support_v7 --samples 3 --out results.json
- name: Compare with baseline
  run: python -m evals.compare results.json baselines/support_v7.json --max-drop 0.02 --segment-max-drop 0.05

Two thresholds matter. The overall score must not drop more than the noise margin. And no single segment may drop by more than a larger margin, because an average can hide a disaster in one segment, which is exactly what our refund regression was.

Store results per run with the prompt version, model snapshot and dataset version, so you can answer "when did this get worse" months later.

Keep the set alive

An eval set rots. Products change, user behaviour shifts, and cases that were hard become easy. Once a month we add new cases from recent complaints and from a fresh traffic sample, retire cases that no longer reflect the product, and re-calibrate the judge on a small human-graded sample. The dataset is versioned like code.

Common mistakes

Key takeaways

• Build eval sets from real traffic, complaints and a few deliberate edge cases.
• Combine cheap deterministic checks with rubric-based grading.
• Treat LLM judges as instruments that need calibration.
• Know your noise level before trusting a difference.
• Gate every prompt and model change in CI, overall and per segment.

References

• OpenAI evals guide — https://platform.openai.com/docs/guides/evals
• Anthropic, creating empirical evaluations — https://docs.anthropic.com/en/docs/test-and-evaluate/develop-tests
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al., 2023) — https://arxiv.org/abs/2306.05685