Every LLM project eventually asks: how does the model learn about our stuff? Our product docs, our support history, our internal policies. There are three main answers: put the right pieces into the prompt at query time (retrieval augmented generation), change the model's weights (fine-tuning), or put everything into the prompt (long context). They solve different problems, and picking the wrong one costs months.

We have shipped all three. This guide is the comparison we wish we had at the start, with the questions that decide it.

What each approach actually changes

  • RAG keeps the model as it is and retrieves relevant documents for each question, then asks the model to answer using them. Knowledge lives in a search index you control.
  • Fine-tuning trains the model on examples so it changes how it behaves: format, tone, task skill, domain vocabulary. Knowledge is absorbed into weights, imperfectly.
  • Long context sends the whole relevant corpus (or a large part of it) with every request, relying on large context windows and prompt caching. Knowledge lives in the prompt.

The most common mistake is using fine-tuning to teach facts. Fine-tuning is good at teaching behaviour. Teaching facts this way is unreliable, hard to update, and gives no citations.

The comparison

Question RAG Fine-tuning Long context
How fresh can knowledge be? Minutes (re-index) Weeks (retrain) Instant (edit prompt)
Corpus size it handles Effectively unlimited Training set size Up to the context window
Citations and traceability Natural None Possible, less precise
Per-request cost Retrieval + moderate prompt Low prompt, possibly cheaper model High prompt; caching helps a lot
Latency Retrieval adds 50-300 ms Lowest Grows with prompt size
Teaches format or style Weakly Strongly Via examples in prompt
Access control per user Filter at retrieval Very hard Build prompt per user
Main failure mode Retrieval misses the right chunk Confident wrong facts, drift Lost-in-the-middle, cost
Engineering effort Ingestion, chunking, eval Data curation, training, eval Lowest to start

When long context is the right answer

If the knowledge fits comfortably in the context window, changes rarely, and the same prefix is reused across many requests, start here. A product with 80 pages of documentation, a policy handbook, or the schema of one database often fits. With prompt caching, the repeated prefix becomes much cheaper after the first request, and you skip building an index.

Watch for two things. Models can miss information placed in the middle of very long prompts, so test recall on facts spread across the document. And costs scale with every request; if traffic is high and the corpus large, RAG becomes cheaper.

When RAG is the right answer

RAG is the default for knowledge that is large, changes often, needs citations, or differs per user. Support assistants, internal search, and anything that must respect document permissions belong here. The cost is the pipeline: ingestion, chunking, deletes (see our post on deleted documents still being answered), tenant filters, and a retrieval eval. Most RAG quality problems are retrieval problems, so measure retrieval separately from generation.

When fine-tuning is the right answer

Fine-tune when the problem is behaviour, not knowledge:

  • Consistent output format that prompting cannot hold, such as a specific JSON layout or a house style.
  • A narrow task done at high volume, where a smaller fine-tuned model can replace a larger general one and cut cost and latency.
  • Domain language the base model handles poorly, like specialised abbreviations, when combined with RAG for the facts.

You need a curated dataset of good examples (hundreds to thousands), an eval to prove the fine-tuned model is better than prompting, and a plan to retrain when the base model is updated.

Combining them

The approaches stack. A common production setup is RAG for facts, a fine-tuned or carefully prompted model for format and tone, and long-context sections for small, stable reference material like a style guide. Another is long context for a single customer's data and RAG across the shared knowledge base.

A worked decision

For our internal engineering assistant:

  1. Corpus: 14,000 runbooks, wiki pages and postmortems, changing daily. Too large for context, changes too fast for training. RAG.
  2. Answers must cite sources. RAG again.
  3. Per-team access rules. Filter at retrieval.
  4. Style: short, command-first answers. Prompting with three examples was enough; no fine-tuning needed.
  5. The on-call policy document (12 pages, stable, needed in many answers) goes into a cached long-context section instead of being chunked.

For a ticket classifier with 40 categories and 200k labelled historic tickets, the answer was different: fine-tune a small model. Knowledge was not the problem; consistent classification at low cost was.

Common mistakes

Mistake Why it hurts Do this instead
Fine-tuning to add facts Unreliable recall, no citations, stale RAG or long context for facts
Building RAG for a 30-page corpus Pipeline cost for little gain Long context with caching
Stuffing huge prompts at high volume Cost and latency scale with every call RAG, or cache stable prefixes
Ignoring permissions Users see documents they should not Filter at retrieval, per-user prompts
No eval before choosing Decision by opinion Small eval set for each candidate approach

Key takeaways

  • RAG adds knowledge, fine-tuning changes behaviour, long context is the simplest when everything fits.
  • Freshness, corpus size, citations and permissions usually decide between RAG and long context.
  • Fine-tune for format, style and narrow high-volume tasks, not for facts.
  • Combining approaches is normal.
  • Build a small eval first and let it decide.

References