#evals

AI & Machine Learning Bug Report

LLM agent calls the refund tool for order status questions: overlapping tool descriptions

Ryan Pham · Jul 8, 2026 · 7 min read

Our support agent started opening refund requests when customers only asked where their order was. Tool names and descriptions overlapped, and the agent had 23 tools to choose from. Clear tool contracts, fewer tools per step and a tool-choice eval fixed it.

AI & Machine Learning

Building eval sets for LLM features that catch real regressions

Ryan Pham · Nov 19, 2025 · 6 min read

A prompt tweak that looks better on five hand-picked examples can quietly break a tenth of real traffic. This guide shows how to build an eval set from production, choose graders, use LLM-as-judge without fooling yourself and gate changes in CI.

AI & Machine Learning Bug Report

LLM snapshot tests flaky at temperature 0: why outputs still change

Ryan Pham · Jun 26, 2025 · 6 min read

Our CI compared LLM summaries against stored snapshots at temperature 0 and failed about one run in five. Temperature 0 is not a determinism guarantee on hosted models. We replaced string snapshots with property checks and a scored eval.