Teams often reach for fine-tuning when they need retrieval, or build a retrieval pipeline when the whole corpus fits in the prompt. This guide compares RAG, fine-tuning and long context on freshness, cost, latency, citations and effort, with a decision table and examples.
A customer saw a snippet from another company's document in an AI answer. A new 'related questions' feature queried the vector store through a helper that did not apply the tenant filter. We moved tenant scoping into the repository layer so no code path can skip it.
Our support assistant quoted an old refund policy that had been removed from the help center weeks earlier. The ingestion pipeline only ever upserted chunks and never deleted them. We added source-of-truth sync, document versions and a nightly reconciliation job.
Our docs assistant answered v4 questions with v3 settings, fluently and with citations. Fixed-size chunking split tables from their headings, chunks carried no version metadata and we had no retrieval eval. How we rebuilt chunking, added filters and hybrid search, and measured it.
Small tenants in our RAG product got 2 or 3 chunks back instead of 10, and sometimes none. HNSW finds ef_search candidates first and applies the tenant filter afterwards. Iterative index scans in pgvector 0.8 plus partial indexes for big tenants fixed recall.
Semantic search in our docs assistant took 1.9 seconds and returned odd rankings. The query sorted by L2 distance while the HNSW index was built for cosine, so Postgres scanned every row. Matching the operator to the opclass brought it to 14 ms.
Our nightly embedding backfill died on the same few documents every night. A handful of chunks were far over the embedding model's token limit because the chunker split by characters and some files were minified JSON. Token-aware chunking and batching fixed it.