AI & Machine Learning
Bug Report
LLM snapshot tests flaky at temperature 0: why outputs still change
Our CI compared LLM summaries against stored snapshots at temperature 0 and failed about one run in five. Temperature 0 is not a determinism guarantee on hosted models. We replaced string snapshots with property checks and a scored eval.