Anthropic prompt caching never hits: cache_read_input_tokens always 0

Our contract-review feature sends a long system prompt to Claude: about 6,000 tokens of instructions, clause definitions and examples, then the user's document. Prompt caching looked like an easy win, since the instructions are identical for every request. We added a cache_control breakpoint, shipped it, and a week later the LLM line in our cost report was 18 percent higher.

The usage block on every response told the story:

"usage": {
  "input_tokens": 2214,
  "cache_creation_input_tokens": 6188,
  "cache_read_input_tokens": 0,
  "output_tokens": 731
}

Every request created a cache entry. None read one. Cache writes are billed at a premium over normal input tokens, so we were paying more for the same work.

Reproducing it

Two identical requests sent ten seconds apart from a script both showed cache_read_input_tokens: 0. That ruled out traffic patterns: even back-to-back requests missed.

We dumped the exact request body for both and diffed them:

-  "text": "You are a contract review assistant. Today is 2025-03-27T10:14:02Z. ..."
+  "text": "You are a contract review assistant. Today is 2025-03-27T10:14:12Z. ..."

The first line of the system prompt contained the current timestamp, down to the second.

What we ruled out

• The 5-minute cache lifetime. Requests ten seconds apart still missed.
• Minimum cacheable length. Our prefix was over 6,000 tokens, well above the minimum for the model we used.
• Different API keys or workspaces. Both test requests used the same key.

Root cause

A cache hit requires the cached prefix to be byte-for-byte identical up to the breakpoint. Our template put a timestamp at the start of the system prompt and, a few lines later, the customer's organisation name. Both changed per request, so the prefix before the breakpoint was unique every time. A second, smaller issue: our tool list was built from a Python set, so the order of tools could change between processes, and tools come before the system prompt in the cached prefix.

What is happening under the hood

Prompt caching stores the processed prefix of a prompt. The prefix is built in a fixed order: tools, then system, then messages. A cache_control marker on a content block says "cache everything up to and including this block". On the next request, if everything up to that point matches exactly, the API reads it from cache, which is billed at a fraction of the normal input price and is faster. If anything earlier in the prefix differs, it is a miss and a new entry is written.

The default cache entry lives for about five minutes and the lifetime is refreshed every time it is read. There is also a minimum length for a cacheable prefix that depends on the model; shorter prefixes are processed normally without an error.

The response usage reports three numbers: cache_creation_input_tokens (written this time), cache_read_input_tokens (read from cache), and input_tokens (everything after the last breakpoint, billed normally).

Environment

Minimal reproduction

system = [{
    "type": "text",
    "text": f"Today is {datetime.now(UTC).isoformat()}. Customer: {org.name}.\n" + INSTRUCTIONS,
    "cache_control": {"type": "ephemeral"},
}]
tools = [TOOL_DEFS[name] for name in set(enabled_tools)]   # unstable order

client.messages.create(model=MODEL, max_tokens=2048, system=system, tools=tools,
                       messages=[{"role": "user", "content": document}])

The fix

Put everything that is static first, and everything that varies after the breakpoint.

TOOLS = [TOOL_DEFS[name] for name in sorted(TOOL_DEFS)]     # deterministic order

def build_request(org, document: str) -> dict:
    system = [
        {   # static: identical for every customer and every request
            "type": "text",
            "text": INSTRUCTIONS,
            "cache_control": {"type": "ephemeral"},
        },
        {   # dynamic: after the breakpoint, billed as normal input
            "type": "text",
            "text": f"Current date: {date.today().isoformat()}. Customer: {org.name}.",
        },
    ]
    return dict(model=MODEL, max_tokens=2048, tools=TOOLS, system=system,
                messages=[{"role": "user", "content": document}])

Two details mattered:

• We reduced the date to day precision and moved it after the breakpoint. The model only needs the date for "is this contract expired" questions.
• Tool definitions are now built once at import, sorted by name, and serialised the same way everywhere. A unit test compares the JSON of two builds.

For the multi-turn review chat we added a second breakpoint on the last user message, so each turn reuses the previous conversation prefix as well.

Alternatives and trade-offs

• Pass dynamic data in the user message. Equally valid; we kept it in a second system block to keep prompts readable.
• Longer cache lifetime. Useful for traffic that comes in bursts more than five minutes apart; it costs more to write, so measure first.
• Skip caching for low-traffic tenants. If a prefix is used once, the write premium is pure overhead. Caching pays off when the same prefix is read several times within its lifetime.

How to test the fix

def test_prefix_is_stable_across_calls():
    a = build_request(org_a, "doc one")
    b = build_request(org_b, "doc two")
    assert json.dumps(a["tools"]) == json.dumps(b["tools"])
    assert a["system"][0] == b["system"][0]          # cached block identical

Then an integration check against the API: send the same request twice and assert that the second response has cache_read_input_tokens > 0.

Confirming in production

We added the three usage numbers to our request logs and computed a hit ratio:

SELECT date_trunc('hour', created_at) AS hour,
       sum(cache_read_tokens)::float
         / nullif(sum(cache_read_tokens + cache_creation_tokens), 0) AS hit_ratio
FROM llm_requests
WHERE feature = 'contract_review'
GROUP BY 1 ORDER BY 1 DESC LIMIT 24;

During business hours the hit ratio settled at 0.92. Input cost for the feature dropped by about 70 percent compared to before caching, and time to first token improved by roughly a second on long prompts.

Regression guard

• A dashboard panel and alert on the hourly hit ratio below 0.5 during business hours.
• The prefix stability unit test.
• A lint rule in code review: no datetime.now() or request data inside the static prompt constant.

Security implications

Caches are isolated per organisation on the provider side, so caching does not expose one customer's prompt to another tenant of the API. Inside our own app, the change moved customer names out of the shared static block, which is a small privacy improvement: the cached part contains no customer data at all.

Prevention checklist

• Static first, dynamic last: tools, then fixed system text, then per-request data.
• Never put timestamps, ids or random values before a cache breakpoint.
• Build tool lists in a deterministic order.
• Log cache_creation_input_tokens and cache_read_input_tokens from day one.
• Check that traffic actually repeats prefixes within the cache lifetime before counting savings.

References

• Anthropic prompt caching documentation — https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching
• Anthropic pricing — https://www.anthropic.com/pricing
• Messages API reference — https://docs.anthropic.com/en/api/messages