Multi-tenant RAG leaked another customer's document through a missing filter

A customer success manager escalated a screenshot on a Tuesday morning. Under an AI answer in our knowledge base product, the new "related questions" panel showed a question and a two-line snippet that clearly came from a different customer's internal documentation, including that company's name.

We treated it as a security incident from the first minute: disabled the related questions feature with its flag, started an incident channel, and began working out scope.

Reproducing it

In staging, with two test tenants, the related panel for tenant A showed tenant B's content within a few queries on overlapping topics like "SSO setup". The main answer path never did.

The related panel's code looked like this:

def related_questions(question: str, tenant_id: str, k: int = 5):
    vec = embed(question)
    hits = vector_search(vec, collection="faq", limit=k)   # tenant_id unused
    return [h.payload["question"] for h in hits]

tenant_id was a parameter and never used.

What we ruled out

• Shared public content. Some FAQ entries are global templates; the leaked snippet was not one of them.
• Main answer path. It used a different function, search_tenant_docs, which always filtered. We verified with logs that no main-path query lacked the filter.
• Embedding or cache sharing. The semantic cache is keyed by tenant; the leak came straight from vector search.

Root cause

Our codebase had two ways to query the vector store: a high-level search_tenant_docs that required a tenant id, and a low-level vector_search helper intended for admin tools and evaluation scripts. The related questions feature was built quickly on top of the low-level helper. Code review missed the unused parameter, and tests used a single tenant, so they could never catch cross-tenant results. Isolation depended on every caller remembering to filter.

What is happening under the hood

Most multi-tenant RAG systems store all tenants in one collection or index and use a metadata filter (tenant_id = X) at query time. This is efficient and perfectly safe if the filter is always applied. The vector store has no idea which tenant is asking; it returns the nearest vectors among whatever it is allowed to search. One query without a filter searches everyone.

Unlike a SQL row leak, a RAG leak can be subtle: the content is paraphrased into an answer or a snippet, so it is easy to miss in testing and hard to find in logs afterwards.

Environment

Minimal reproduction

The pattern, regardless of vector store:

results = index.query(vector=embed(q), top_k=5)                 # all tenants
results = index.query(vector=embed(q), top_k=5,
                      filter={"tenant_id": {"$eq": tenant_id}})   # one tenant

The fix

Immediate containment. The feature flag was off within 20 minutes of the report. We pulled logs for every related-panel request during the six days and joined the returned FAQ ids with their owning tenants. 14 requests from 9 tenants had displayed at least one item from another tenant. We notified the affected customers, both those whose content was shown and those who saw it, following our incident policy.

Structural fix: one door to the vector store. Application code can no longer construct raw vector queries. It goes through a repository that takes a tenant context object and always applies the filter:

from dataclasses import dataclass

@dataclass(frozen=True)
class TenantContext:
    tenant_id: str

class FaqVectorRepository:
    def __init__(self, client):
        self._client = client

    def search(self, ctx: TenantContext, vector: list[float], limit: int = 5):
        if not ctx.tenant_id:
            raise PermissionError("tenant context required")
        return self._client.query_points(
            collection_name="faq",
            query=vector,
            limit=limit,
            query_filter=Filter(must=[
                FieldCondition(key="tenant_id", match=MatchValue(value=ctx.tenant_id)),
            ]),
        ).points

The tenant context is created by authentication middleware from the session, never from request parameters. The low-level client is only importable from the repository module and from an admin package with its own access checks; a lint rule enforces the import boundary.

Defense in depth on results. After retrieval, the repository asserts that every returned point carries the expected tenant_id, and raises (and alerts) if not. It costs nothing and catches mistakes in filter construction.

Alternatives and trade-offs

• Collection or namespace per tenant. Isolation by construction; a query can only see one tenant's data. At 900 tenants that means 900 collections to manage, migrate and monitor. We may move our largest customers to dedicated collections.
• Separate deployments per tenant. Strongest isolation, highest cost; for regulated customers only.
• Database-level row security. If you keep vectors in Postgres with pgvector, row level security policies can enforce the tenant filter in the database itself, which is a strong option.

How to test the fix

Every retrieval test now runs with at least two tenants that have overlapping content:

def test_related_questions_never_cross_tenants(seed_two_tenants, client):
    a, b = seed_two_tenants(shared_topics=["sso", "billing", "api keys"])
    for q in ["how to set up sso", "update billing email", "rotate api key"]:
        items = client.get("/related", params={"q": q}, headers=auth(a)).json()
        assert all(i["tenant_id"] == a.id for i in items)

A property-based test generates random questions and tenants and checks the same invariant against the repository directly.

Confirming in production

• After the fix shipped, we re-enabled the feature for internal tenants for a week, then for everyone.
• The result-tenant assertion has logged zero violations since launch.
• A daily job samples 1,000 retrieval log entries and verifies that every returned document belongs to the requesting tenant.

Regression guard

• Import boundary lint rule for the raw vector client.
• Two-tenant fixtures are the default in the test suite.
• Security review checklist item for any new feature that reads from the vector store.

Security implications

This was a confidentiality breach, small in scope but real. Beyond the fix, we reviewed every other consumer of the low-level helper (two admin tools and an eval script) and confirmed they ran with admin authorisation only. The incident led to a written rule: tenant isolation is enforced in one shared layer and verified on the way out, never left to individual features.

Prevention checklist

• Provide exactly one application-level path to the vector store, which requires tenant context.
• Derive tenant context from authentication, not from request input.
• Assert tenant ownership of results after retrieval.
• Test with multiple tenants that share topics.
• Consider physical separation for high-risk tenants.

References

• OWASP Top 10 for LLM Applications, sensitive information disclosure — https://genai.owasp.org/llmrisk/llm022025-sensitive-information-disclosure/
• Qdrant multitenancy guide — https://qdrant.tech/documentation/guides/multiple-partitions/
• PostgreSQL row security policies — https://www.postgresql.org/docs/current/ddl-rowsecurity.html