RAG assistant gave confident wrong answers from the wrong product version

Our product is a self-hosted observability agent with a documentation site of about 1,900 pages covering versions 3.x and 4.x. In June we launched a docs assistant in the dashboard: a retrieval-augmented generation (RAG) pipeline that embeds the question, pulls the top chunks from a vector index and asks an LLM to answer with citations. Early feedback was positive. Then the support team flagged a pattern: customers on v4 were being told to set configuration keys that only existed in v3. The answers were fluent, cited real documentation pages and were wrong.

One example from the feedback log:

Q:  How do I limit the agent's memory on 4.2?
A:  Set `buffer.max_mb` in agent.yaml (see "Resource limits"). The default is 256.
    [source: docs/v3/configuration/resource-limits.md]

In 4.x that key was replaced by limits.memory, and the v3 page was cited even though the user's question named the version.

Reproducing it

We pulled 120 thumbs-down conversations from the first eight weeks and re-ran them with retrieval logging on, storing the top 8 chunks for each question with their source path and similarity score. In 47 of the 120, every retrieved chunk came from v3 documentation. In another 31, the right v4 page was retrieved, but the chunk that contained the answer was a table row without its column headers, so the model guessed which column meant what.

What we ruled out

• The LLM hallucinating. In every bad answer we checked, the wrong fact was present in a retrieved chunk. The model was faithfully summarising the wrong context.
• Stale index. The v4 pages were indexed; their chunk IDs existed and matched the current docs commit.
• The embedding model being too weak. Swapping to a larger embedding model on a sample of 200 questions moved recall by only a couple of points. The problem was what we were embedding, not how.

Root cause

Three design gaps, all in the ingestion side:

1. Fixed-size chunking. We split each Markdown file into 800-token windows with 100-token overlap. Tables were cut mid-way, headings ended up in a different chunk from the content beneath them, and a chunk often started in the middle of a sentence. A chunk containing "| limits.memory | 512 | Hard cap... |" with no header row and no heading above it had no idea it was about v4 memory limits.
2. No metadata. Chunks stored only text, embedding and source path. The version lived in the path, and we never filtered on it. v3 and v4 pages describe the same features in almost the same words, so their embeddings were nearly identical, and v3 had more pages, so it won the ties.
3. No retrieval evaluation. We had tested answers by reading them. We had never measured whether the right chunk was retrieved, so none of this was visible before launch.

What is happening under the hood

Vector similarity finds text that is semantically close to the question. Two documentation pages for consecutive versions of the same feature are semantically close to each other and to the question, by design. Nothing in a dense embedding reliably encodes "this is version 4", especially when the version string appears only in a file path that was never embedded. Retrieval therefore returns a mix, and the model, instructed to answer from the context, picks whatever looks most complete.

Chunking decides what a single retrievable unit means. A chunk that does not carry its own context (which page, which section, which version, which table columns) is ambiguous to both the retriever and the model. Overlap does not fix that: it repeats a few sentences, not the heading two screens up.

Environment

Minimal reproduction

The retrieval query had no filter at all:

SELECT id, source_path, content
FROM doc_chunks
ORDER BY embedding <=> $1
LIMIT 8;

Asking "how do I limit memory on 4.2" returned five v3 chunks and three v4 chunks, none of which contained the v4 key next to its description.

Who is affected

Any RAG system over versioned, multi-product, multi-tenant or multi-region content: API docs, policies by country, contracts by customer, runbooks by environment. It is worst when versions are similar enough that embeddings cannot separate them, which is the normal case.

The fix

1. Structure-aware chunking. We parse Markdown into a heading tree and chunk by section. Each chunk is one section (split further only if it exceeds 600 tokens, at paragraph boundaries). Tables are never split mid-table; very large tables are split by row groups with the header row repeated in every piece. Each chunk gets a context prefix that is embedded with it:

def build_chunk(page, section_path: list[str], body: str) -> dict:
    prefix = (
        f"Product: Agent {page.version_major}.x | "
        f"Page: {page.title} | Section: {' > '.join(section_path)}\n\n"
    )
    return {
        "content": prefix + body,
        "version_major": page.version_major,
        "version_minor": page.version_minor,
        "doc_type": page.doc_type,          # reference, guide, release-notes
        "source_path": page.path,
        "anchor": slugify(section_path[-1]),
    }

2. Metadata columns and filters. The version is now a column, and the query filters on it. The user's version comes from the agent telemetry when they are logged in, or is parsed from the question; if neither is available we default to the latest major version and say so in the answer.

SELECT id, source_path, anchor, content
FROM doc_chunks
WHERE version_major = $2
ORDER BY embedding <=> $1
LIMIT 20;

Because pgvector applies the filter after the approximate index scan, a selective filter can return fewer rows than requested. We have only two versions, so we raised hnsw.ef_search to 100 and checked that recall held. With many more versions we would use a partial index per version or pgvector's iterative index scans.

3. Hybrid search. Configuration keys, error codes and CLI flags are exact tokens that dense embeddings handle poorly. We added a PostgreSQL full-text column and merge the two result lists with reciprocal rank fusion:

WITH dense AS (
  SELECT id, row_number() OVER (ORDER BY embedding <=> $1) AS r
  FROM doc_chunks WHERE version_major = $2
  ORDER BY embedding <=> $1 LIMIT 40
), sparse AS (
  SELECT id, row_number() OVER (ORDER BY ts_rank_cd(tsv, q) DESC) AS r
  FROM doc_chunks, websearch_to_tsquery('english', $3) q
  WHERE version_major = $2 AND tsv @@ q
  ORDER BY ts_rank_cd(tsv, q) DESC LIMIT 40
)
SELECT id, sum(1.0 / (60 + r)) AS score
FROM (SELECT * FROM dense UNION ALL SELECT * FROM sparse) u
GROUP BY id ORDER BY score DESC LIMIT 8;

4. Prompt changes that make uncertainty visible. The system prompt now includes the resolved version and tells the model to say when the context does not answer the question rather than borrowing from another version. Citations include the version so users can see it.

Alternatives and trade-offs

• Separate indexes per version. Simple and fast, but cross-version questions ("what changed between 3 and 4") need a different path. We kept one table and filter instead.
• A reranker model over the top 20 to 40 candidates improves ordering further. We added one later; it helped precision but costs latency and does not replace the version filter.
• Generating a short context sentence per chunk with an LLM at ingestion time, rather than a template prefix, can help for unstructured content. Our docs are well structured, so the template was enough.

How to test the fix

We built an evaluation set of 260 questions with the expected source section for each: the 120 bad conversations, 100 questions written by support engineers from real tickets, and 40 cross-version and "not documented" questions. The harness measures retrieval separately from answers:

python -m rag_eval.run --set eval/questions-v1.jsonl --k 8 --report out/retrieval.json

It reports recall@8 (is the expected section in the top 8), the share of retrieved chunks from the wrong version, and MRR. Answer quality is graded separately by support engineers on a sample, so a better-looking answer cannot hide worse retrieval.

Confirming in production

On the evaluation set, before and after:

In production over the following month, thumbs-down rate on assistant answers dropped from 17% to 6%, and support tickets tagged "assistant gave wrong config" went from about 25 a week to 2.

Regression guard

• The retrieval evaluation runs in CI on every change to ingestion code, chunking parameters, embedding model or retrieval SQL, and fails if recall@8 drops more than 2 points or wrong-version share rises above 1%.
• A nightly job re-runs it against the live index to catch drift as docs change.
• New thumbs-down conversations are reviewed weekly and the useful ones are added to the eval set with their expected sources.

Security implications

Metadata filtering is the same mechanism you need for access control in RAG: if documents differ by tenant or permission level, the filter must be enforced in the retrieval query, server side, never left to the prompt. We added a visibility column for internal-only pages at the same time and filter on it based on the user's role. Retrieved documents are also untrusted input to the model; our docs are first-party, but anything user-contributed would need the defences against indirect prompt injection.

Prevention checklist

• Chunk along document structure and keep headings, table headers and page context with each chunk.
• Store version, product, tenant and visibility as filterable columns, and filter in the query.
• Combine dense and keyword retrieval for content with exact identifiers.
• Measure retrieval with a labelled eval set before judging answers.
• Feed real failures back into the eval set continuously.

References

• pgvector README: filtering, HNSW, iterative index scans — https://github.com/pgvector/pgvector
• PostgreSQL documentation: Full Text Search — https://www.postgresql.org/docs/current/textsearch.html
• Anthropic: Introducing Contextual Retrieval — https://www.anthropic.com/news/contextual-retrieval
• Cormack, Clarke and Buettcher, Reciprocal Rank Fusion (SIGIR 2009) — https://doi.org/10.1145/1571941.1572114