Keeping LLM API costs predictable in production: accounting, caching, routing and budgets

In January our monthly LLM API invoice went from about 6,200 to 18,900 US dollars. Traffic had grown maybe 20%. Finance asked which feature was responsible and we could not answer: every service used the same API key, and the provider's usage page grouped spend by model, not by product feature. It took two engineers four days of log archaeology to find that a new "explain this chart" button was sending the full dashboard JSON (often 40,000 tokens) on every click, and that a retry bug was sending many of those twice.

This guide is the system we built afterwards, for teams running LLM features in production who want spend to be explainable and boring. It is provider-neutral; where a mechanism is provider-specific we say so and point to the docs. By the end you should be able to attribute every token to a feature, cut the obvious waste, and get an alert before the invoice surprises you.

Measure first: per-feature token accounting

You cannot control what you cannot attribute. Every LLM call in our codebase now goes through one small client wrapper that records usage from the response, not from estimates:

@dataclass
class LlmCall:
    feature: str          # "chart_explain", "ticket_summary", ...
    tenant_id: str
    model: str
    input_tokens: int
    cached_input_tokens: int
    output_tokens: int
    latency_ms: int
    cost_usd: float       # computed from a versioned price table

def record(call: LlmCall) -> None:
    metrics.counter("llm_tokens_total", call.input_tokens,
                    tags={"feature": call.feature, "model": call.model, "kind": "input"})
    metrics.counter("llm_tokens_total", call.output_tokens,
                    tags={"feature": call.feature, "model": call.model, "kind": "output"})
    metrics.counter("llm_cost_usd_total", call.cost_usd,
                    tags={"feature": call.feature, "model": call.model})
    usage_log.write(asdict(call))   # warehouse table for monthly reconciliation

Three details matter:

• Feature is mandatory. The wrapper raises if it is missing, so new code cannot bypass attribution.
• Price table is versioned in code and reviewed when providers change prices. Cost is computed at call time so dashboards are live.
• Reconcile monthly against the provider invoice. Our computed number is within 1 to 2% of the invoice; when it drifts further, something is calling the API outside the wrapper.

Separate API keys or projects per service, where your provider supports them, give a second, independent view and let you set limits per service.

Cut waste before optimising

Once spend was attributed, most of the savings came from waste, not clever techniques:

Input tokens dominate cost for most retrieval and analysis features, so look at what goes into the prompt first. Logging token counts per prompt component (system, history, retrieved context, user input) makes the bloat obvious.

Cap output

Output tokens are usually priced several times higher than input tokens. Set max_tokens (or your provider's equivalent) per feature from measured output lengths, with headroom, rather than leaving a large default. Ask for the format you need: a classification feature that returns a label and a one-sentence reason does not need room for three paragraphs. Structured output with a schema also keeps models from padding answers.

Watch for truncation after tightening: track the stop reason (for example finish_reason: "length") per feature and alert if it rises.

Prompt caching

Most providers can now reuse the processed form of a repeated prompt prefix at a reduced price. The details differ, so read your provider's documentation:

• On the OpenAI API, caching applies automatically to prompts above a minimum length, and the response reports cached tokens in its usage details.
• On Anthropic's API, you mark cache breakpoints with cache_control. Its documentation prices cache reads at a fraction of the base input price (0.1x for most models) and 5-minute cache writes at 1.25x, and reports cache_creation_input_tokens and cache_read_input_tokens in usage. Cache reads also do not count toward input-token rate limits on most models.

Caching only works on an identical prefix, so prompt layout matters:

[tools / function definitions]        stable for weeks
[system prompt + policies]            stable for days
[large shared context: docs, schema]  stable for hours
---- cache boundary ----
[conversation history]                changes every turn
[user's latest message]               always new

Things that silently break caching in our experience: a timestamp or request ID at the top of the system prompt, tool definitions serialised in a non-deterministic order, and per-user details mixed into the shared context. After fixing those, our ticket-summary feature went from about 5% to 72% of input tokens served from cache, and its input cost fell by roughly 60%.

Response caching

Prompt caching saves on processing a prefix; response caching skips the call entirely. It fits features where the same input recurs and the answer does not depend on who asks: explaining a static error code, summarising a public document, classifying a product. We key the cache on a hash of model, prompt template version and normalised input, with a TTL per feature. Our knowledge base "explain this article" feature now serves 46% of requests from cache.

Do not cache responses across tenants when the input or the answer contains tenant data, and always include the template version in the key so prompt changes invalidate old entries.

Route by difficulty

Not every request needs the largest model. We route by feature first and by request second:

• By feature: classification, extraction and short summaries use a small model; multi-step reasoning and customer-facing long answers use a larger one.
• By request: for the support assistant, a cheap first pass classifies the question. Simple FAQ-style questions stay on the small model; anything involving account data or multi-step troubleshooting escalates.
• Fallback on validation failure: if the small model's output fails schema or business validation, retry once on the larger model.

Every routing change goes through the same evaluation set as a model change. Routing that saves 40% but quietly drops quality is a cost moved to support.

Batch what can wait

Providers offer asynchronous batch APIs at a discount for work that can complete within hours; both OpenAI's Batch API and Anthropic's Message Batches API document a 50% discount relative to standard pricing. Our nightly re-tagging, weekly report generation and evaluation runs all moved to batch. They now account for about a quarter of our token volume and under 15% of spend.

Budgets and alerts

Measurement and optimisation help on average. Budgets protect you from the bad day:

• Per-feature daily budgets in the wrapper. At 80% of budget we alert the owning team; at 120% non-critical features degrade (smaller model, shorter context or a "temporarily unavailable" message) instead of spending without limit.
• Per-tenant quotas so one customer's bulk job cannot consume everyone's budget. These also limit abuse.
• Anomaly alerts on cost per hour per feature compared with the same hour last week. The "explain this chart" incident would have paged within two hours instead of being found on the invoice.
• Provider-side limits as the last line: monthly spend limits and per-project limits where your provider offers them.

# feature budget config (excerpt)
chart_explain:
  daily_usd: 120
  alert_at: 0.8
  degrade_at: 1.2
  degrade_to: { model: small, max_input_tokens: 4000 }
ticket_summary:
  daily_usd: 300
  alert_at: 0.8
  degrade_at: 1.5

Common mistakes

Key takeaways

• Attribute every call to a feature and tenant through one wrapper, and reconcile against the invoice monthly.
• The biggest savings are usually waste in the prompt: oversized context, resent history, duplicate calls.
• Order prompts for caching and verify the cache hit rate from usage fields.
• Cap output per feature and route easy work to smaller models, backed by evaluation.
• Move non-urgent work to batch APIs.
• Budgets with graceful degradation and hourly anomaly alerts turn surprises into pages.

References

• OpenAI API documentation: Prompt caching — https://platform.openai.com/docs/guides/prompt-caching
• OpenAI API documentation: Batch API — https://platform.openai.com/docs/guides/batch
• Anthropic documentation: Prompt caching — https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching
• Anthropic documentation: Batch processing — https://docs.anthropic.com/en/docs/build-with-claude/batch-processing
• Anthropic documentation: Rate limits — https://docs.anthropic.com/en/api/rate-limits