Semantic cache returned the answer to a different question at 0.92 similarity

Our help assistant answers product questions with RAG. Many questions repeat, so we added a semantic cache: embed the question, look for a cached question with cosine similarity above 0.92, and return its answer without calling the model. Cost dropped 38 percent in the first week. Then a support ticket arrived with a screenshot:

User:      how do i cancel my subscription
Assistant: To cancel an order, open Orders, select the order and click "Cancel order" before it ships...

Reproducing it

The cache log showed the hit:

cache_hit query="how do i cancel my subscription" matched="how do i cancel my order" score=0.9237

We ran a sweep over 2,000 recent questions and their nearest cached neighbours, and labelled 300 hits by hand: 11 percent of hits above 0.92 were answers to a different question. Typical pairs:

What we ruled out

• A bug in the vector search. The scores were correct; the questions really are close in embedding space.
• Embedding model quality. A larger embedding model shifted scores but produced the same kinds of confusions.

Root cause

Embedding similarity measures how related two texts are, not whether they ask the same thing. Questions that differ in one decisive word (subscription versus order, after versus within, delete versus deactivate) share almost everything else and land close together. A single global threshold tuned for "high hit rate" let these through. We had not measured the false hit rate before launch, only the hit rate.

What is happening under the hood

Sentence embeddings compress meaning into a vector. Topic, structure and most words dominate the vector; small changes in a key noun or a negation move it only slightly. That makes them great for retrieval, where related is good enough because the model reads the documents, and risky for caching, where the cached answer is returned as-is.

The threshold trade-off is a precision and recall curve. A higher threshold gives fewer wrong hits and fewer hits overall. The right threshold depends on your questions and on how bad a wrong answer is. For billing and account actions, a wrong answer is bad.

Environment

Minimal reproduction

def answer(question: str) -> str:
    vec = embed(question)
    hit = cache.nearest(vec)
    if hit and hit.score >= 0.92:
        return hit.answer               # returned without any further check
    ans = rag_answer(question)
    cache.put(vec, question, ans)
    return ans

The fix

We rebuilt the cache in layers.

1. Exact match first. Normalise the question (lowercase, trim, collapse spaces and punctuation) and check an exact-key cache. A large share of repeats are literally identical, and these hits are always safe.

2. Scope keys by everything that changes the answer. Locale, product, plan tier and help-center version are part of the cache key. A question from a Business plan user never hits a Free plan answer.

3. Only cache safe intents. A lightweight intent classifier runs first. Questions about billing, account deletion, refunds and security are never served from the semantic layer. These intents get exact-match caching only.

4. A measured threshold and a verification step for borderline scores.

def lookup(question: str, scope: Scope) -> str | None:
    key = normalise(question)
    if (hit := exact_cache.get(scope.key(key))):
        return hit

    if classify_intent(question) in UNSAFE_INTENTS:
        return None

    vec = embed(question)
    hit = semantic_cache.nearest(vec, filter=scope.filter())
    if not hit or hit.score < 0.95:
        return None
    if hit.score < 0.975 and not same_question(question, hit.question):
        return None
    return hit.answer

same_question is a small, cheap model call that answers "do these two questions ask for the same information? yes or no". It runs only for scores in the borderline band, and costs a fraction of a full RAG answer.

5. Short TTLs and invalidation on content change. Cache entries carry the ids of the help articles used to produce them. When an article changes, entries that used it are deleted.

Alternatives and trade-offs

• Raise the global threshold. Simple. At 0.97 our wrong-hit rate was 2 percent and hit rate fell by half. Scoping and intent gating did better at the same hit rate.
• Cache retrieval results instead of answers. Reuses the expensive part while still generating a fresh answer. Smaller savings, much lower risk.
• Cross-encoder re-ranker as the verifier. More accurate than embeddings for pair comparison and cheaper than an LLM call; a good next step.

How to test the fix

We turned the 300 hand-labelled pairs plus 500 new ones into a test set, and measure precision of cache hits at each layer:

layer                         hit_rate   wrong_hit_rate
semantic 0.92 (before)         41%        11.0%
exact + scoped + intent gate   27%         0.6%
+ verifier in 0.95-0.975       29%         0.3%

A CI test asserts that wrong-hit rate on the labelled set stays under 1 percent.

Confirming in production

• Cost savings settled at 24 percent, down from 38, with a wrong-hit rate around 0.3 percent in weekly spot checks.
• Support tickets that mention wrong answers returned to their pre-cache baseline.
• A thumbs-down rate comparison between cached and uncached answers shows no significant difference.

Regression guard

• The labelled pair set grows weekly with new borderline pairs from logs.
• Dashboards show hit rate and spot-check precision side by side, so nobody tunes for hit rate alone again.
• Cache responses are tagged in logs with the layer and score that served them.

Security implications

Without scoping, a semantic cache can serve one user an answer generated with another user's context. Our help answers are generic, but we checked: no answer included account-specific data. The rule now is that anything generated with user-specific context is never cached across users, and the cache key always includes tenant and plan. Intent gating also keeps sensitive flows like account deletion away from approximate matches.

Prevention checklist

• Measure wrong-hit rate, not just hit rate, before launch.
• Exact-match first, semantic second.
• Scope cache keys by everything that changes the answer.
• Exclude high-risk intents from semantic caching.
• Invalidate when source content changes.

References

• Redis vector search documentation — https://redis.io/docs/latest/develop/interact/search-and-query/advanced-concepts/vectors/
• Sentence-BERT, cross-encoders vs bi-encoders — https://www.sbert.net/examples/applications/cross-encoder/README.html
• OpenAI embeddings guide — https://platform.openai.com/docs/guides/embeddings