Why models need grounding

Large language models learn from vast amounts of public text, but they know nothing about your internal policies, prices or customers, and what they do know stops at their training date. Asked about something they have not seen, they can produce a confident, plausible and wrong answer.

Retrieval-augmented generation, or RAG, addresses this by finding relevant information in your own sources at the moment a question is asked and giving it to the model as evidence. The model's job shifts from remembering to reading.

How it works, step by step

  • Ingest: documents, pages and records are collected from approved sources and converted to clean text.
  • Chunk: content is split into passages small enough to retrieve precisely but large enough to keep their meaning.
  • Index: each passage is stored with an embedding, a numerical representation of its meaning, plus keywords and metadata such as source, date and access rights.
  • Retrieve: the question is matched against the index using both meaning and keywords, and the best candidates are re-ranked.
  • Generate: the model receives the question and the selected passages, with instructions to answer only from them and to cite them.

Where RAG goes wrong

  • Poor source content: outdated, duplicated or contradictory documents produce confused answers.
  • Bad chunking: tables split in half, or headings separated from their text, lose their meaning.
  • Retrieval misses: the right passage exists but is not found, often because people phrase questions differently from how documents are written.
  • Permission leaks: retrieval ignores who is asking and surfaces documents they should not see.
  • Over-answering: the model fills gaps with general knowledge instead of saying it does not know.

Most RAG quality problems are content and retrieval problems, not model problems. Swapping in a bigger model rarely fixes an answer that was never retrieved.

What good looks like in production

  • Hybrid search that combines semantic and keyword matching, followed by re-ranking.
  • Access rules from the source systems applied at retrieval time.
  • Citations on every answer, linked to the exact passage.
  • A clear "I don't know" when evidence is missing, with a route to a person.
  • An evaluation set of real questions and expected answers, run before every change.
  • Content ownership: someone responsible for keeping each source current.

RAG, fine-tuning or both?

RAG is the right default when answers depend on facts that change or must be traceable. Fine-tuning changes how a model behaves, for example following a format or tone consistently, rather than what it knows. Many production systems use RAG for knowledge and careful instructions, or light fine-tuning, for behaviour.

Larger context windows have not made retrieval obsolete. Sending every document with every question is slow and expensive, and models still attend better to a focused set of relevant passages.