#prompt-caching

Performance & Scaling

Keeping LLM API costs predictable in production: accounting, caching, routing and budgets

Daniel Tran · Mar 17, 2026 · 8 min read

Our LLM bill tripled in one month and nobody could say which feature caused it. This guide covers what we built since: per-feature token accounting, prompt and response caching, model routing, output limits, batch processing and budgets with alerts.

AI & Machine Learning Bug Report

Anthropic prompt caching never hits: cache_read_input_tokens always 0

Ryan Pham · Apr 3, 2025 · 7 min read

We added cache_control to a 6,000-token system prompt and the bill went up instead of down. Every request wrote the cache and none read it, because the prompt prefix changed on every call. A stable prefix brought cache reads to 92 percent.