How RAG gives a language model access to your own information, each stage of the pipeline, and the design choices that decide answer quality.
What RAG is
Retrieval-augmented generation gives an AI model relevant information from your own sources before it answers. Instead of relying only on what the model learned during training, the system searches your documents, passes the best passages to the model, and asks it to answer from them.
When to use it
RAG suits questions about information that is private, changes often, or must be cited: policies, product documentation, contracts, support history and internal knowledge bases.
The pipeline at a glance
User question → query processing → retrieval → relevant passages → context construction → LLM → grounded response. Each stage can be improved independently, which is why RAG systems are usually tuned step by step rather than all at once.
Ingestion and chunking
Documents are collected, cleaned and split into chunks. Chunk size is a trade-off: small chunks match precise questions but can lose surrounding meaning, while large chunks keep context but dilute relevance. Splitting along natural boundaries such as headings and keeping the section title with each chunk usually works better than cutting at a fixed character count.
Embedding and indexing
Each chunk is converted into an embedding and stored with metadata such as source, date, section and access permissions. Metadata allows filtering, for example restricting results to the current policy version or to documents the user is allowed to see.
Retrieval
At question time, the query is embedded and the closest chunks are retrieved. Many teams combine vector search with keyword search so that exact terms like product codes are not missed. Query rewriting, where the model reformulates an unclear or follow-up question before searching, can also improve results.
Reranking and context construction
A reranker is a second model that scores the retrieved candidates more carefully against the question and keeps the best few. The selected passages are then assembled into the prompt with clear instructions: answer from the provided sources, cite them, and say when the answer is not available.
Generation and citations
The model writes the answer using the supplied context. Showing citations lets users check the source and builds trust. If retrieval returns nothing relevant, the system should say so rather than letting the model guess.
Evaluating a RAG system
Measure retrieval and generation separately. For retrieval, check whether the right passage appears in the results for a set of real questions. For generation, check whether the answer is correct, supported by the retrieved text, and complete. Frameworks such as RAGAS offer automated metrics, but a hand-labelled set of real questions remains the most reliable baseline.
Common mistakes
Indexing outdated or duplicate documents, ignoring access permissions, choosing chunk sizes without testing, skipping evaluation, and assuming a larger model will fix poor retrieval. In practice, most quality problems trace back to what was retrieved rather than to the model.
Key takeaways
- RAG answers from your sources instead of from model memory alone.
- Most quality issues come from ingestion and retrieval, not generation.
- Hybrid search and reranking are common, effective improvements.
- Evaluate retrieval and answers separately, using real questions.
If you want to build a RAG system over your own documents, our RAG Development team designs and ships production pipelines.
RAG Development Services- Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksLewis et al., Meta AI · 2020 · Research paperarxiv.org
- Introducing Contextual RetrievalAnthropic · 2024 · Engineering articleanthropic.com
- RAGAS: Automated Evaluation of Retrieval Augmented GenerationEs et al. · 2023 · Research paperarxiv.org