RAG bot keeps quoting a policy page we deleted a month ago

A customer forwarded a chat transcript to support: our assistant told them refunds were possible within 60 days. The policy had changed to 30 days five weeks earlier, and the old page had been unpublished. The assistant even cited the old URL, which now returned a 404.

Reproducing it

Asking "what is your refund window" returned the 60-day answer about half the time and the 30-day answer the other half. Looking at the retrieved chunks explained the coin flip:

rank  score  doc_id                       updated_at   status_in_cms
1     0.871  help/refund-policy-2024      2025-03-02   deleted
2     0.866  help/refund-policy           2025-10-27   published
3     0.802  help/returns-overview        2025-09-14   published

Both versions were in the vector store, nearly tied in relevance.

What we ruled out

• Model memory. The model had no reason to know our policy; the 60-day text came from the retrieved chunk.
• Caching. The response cache had a 24-hour TTL and was not involved.
• A failed ingestion run. Runs succeeded every night, and the new page was correctly indexed.

Root cause

The ingestion job listed published pages from the CMS API, chunked them and upserted chunks into the vector store by doc_id#chunk_n. It never removed anything. Unpublished or deleted pages simply stopped appearing in the list, and their chunks stayed in the index forever. Renamed pages had the same problem: the old slug's chunks stayed next to the new ones. There were also orphaned chunks inside living documents: when a page got shorter, chunks with higher indexes from the longer version remained.

What is happening under the hood

Most ingestion pipelines start as "fetch everything, embed, upsert". Upsert handles create and update. It cannot handle delete, because a deleted item is an absence, and absences do not show up in a list of what exists. Three kinds of stale data accumulate:

• Documents deleted or unpublished at the source.
• Documents renamed or moved, whose old ids remain.
• Trailing chunks when a document shrinks.

Over months, the share of stale chunks grows, and retrieval quality drops in ways that are hard to see, because stale chunks are often highly relevant: they are about the same topics as the current ones.

Environment

Minimal reproduction

for page in cms.list_published_pages():
    for i, text in enumerate(chunk(page.body)):
        store.upsert(id=f"{page.id}#{i}", vector=embed(text),
                     payload={"doc_id": page.id, "url": page.url})
# nothing ever removes ids that are no longer produced

When we counted, 18 percent of chunks in the collection belonged to pages that were no longer published or had shrunk.

The fix

1. Replace per document, not per chunk. When a page is ingested, all its existing chunks are deleted and the new set written, in one logical step. Each chunk carries the document version it came from.

def ingest_page(page) -> None:
    version = page.updated_at.isoformat()
    points = [
        Point(id=uuid5(NS, f"{page.id}:{version}:{i}"), vector=embed(text),
              payload={"doc_id": page.id, "doc_version": version, "url": page.url})
        for i, text in enumerate(chunk(page.body))
    ]
    store.upsert(points)
    # remove every chunk for this doc that is not from the new version
    store.delete(filter=And(Match("doc_id", page.id), Not(Match("doc_version", version))))

Writing the new version first and then deleting the old one means queries never see a document with zero chunks during the update.

2. Handle deletes from the source. The CMS sends webhooks for unpublish and delete events. A handler removes all chunks with that doc_id. Webhooks can be missed, so they are a fast path, not the only path.

3. Nightly reconciliation. The job compares the set of doc_ids in the vector store with the set of published pages, and deletes the difference. It also checks that each document's doc_version matches the CMS:

published = {p.id: p.updated_at.isoformat() for p in cms.list_published_pages()}
indexed = store.distinct_doc_versions()          # {doc_id: version}

orphans = indexed.keys() - published.keys()
stale = {d for d, v in indexed.items() if d in published and published[d] != v}

for doc_id in orphans:
    store.delete(filter=Match("doc_id", doc_id))
for doc_id in stale:
    ingest_page(cms.get_page(doc_id))

metrics.gauge("rag.orphan_docs", len(orphans))
metrics.gauge("rag.stale_docs", len(stale))

A safety limit stops the job if it would delete more than 10 percent of documents in one run, which protects against a CMS API returning an empty list during an outage.

4. Filter at query time too. Every chunk payload includes the page status; retrieval filters on status = published. It is a belt-and-braces check for the window between a change and the next sync.

Alternatives and trade-offs

• Full rebuild every night into a new collection, then swap an alias. Simplest correct approach for small corpora. At 21k chunks it would have been fine; we chose incremental sync because the corpus is growing and embeddings cost money.
• Change data capture from the CMS database. Exact, but couples us to the CMS internals.

How to test the fix

Integration tests against a local Qdrant:

def test_unpublished_page_is_removed(cms_fake, store):
    cms_fake.publish("p1", "Refunds within 60 days.")
    run_sync()
    cms_fake.unpublish("p1")
    run_sync()
    assert store.count(filter=Match("doc_id", "p1")) == 0

def test_shrunk_page_has_no_trailing_chunks(cms_fake, store):
    cms_fake.publish("p2", LONG_TEXT)       # 9 chunks
    run_sync()
    cms_fake.update("p2", SHORT_TEXT)       # 2 chunks
    run_sync()
    assert store.count(filter=Match("doc_id", "p2")) == 2

Confirming in production

• The first reconciliation run removed 3,812 chunks from 412 documents.
• Retrieval eval (240 questions with known answers) improved from 0.78 to 0.86 answer accuracy, mostly on questions where old and new versions conflicted.
• Asking about the refund window returned 30 days in 50 out of 50 tries.

Regression guard

• Gauges for orphan and stale documents with an alert above zero for two consecutive nights.
• Every answer logs the doc_version of cited chunks, so support can see what the assistant relied on.
• The deletion safety limit and an alert when it triggers.

Security implications

Deletion sync is also a privacy and legal requirement. Content that was removed because it contained personal data or was published by mistake must also leave the vector store, the embedding backups and any caches. We documented the vector store as a data store in our records of processing, and the delete webhook now feeds our data removal checklist.

Prevention checklist

• Design ingestion for create, update and delete from day one.
• Version chunks by document version and replace per document.
• Reconcile against the source on a schedule; webhooks alone are not enough.
• Filter by status at query time.
• Monitor orphan counts.

References

• Qdrant filtering and delete by filter — https://qdrant.tech/documentation/concepts/filtering/
• Qdrant collection aliases — https://qdrant.tech/documentation/concepts/collections/#collection-aliases
• Airflow DAG documentation — https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/dags.html