Remigiusz ZalewskiRemigiusz Zalewski

RAG Explained in 7 Minutes

rag-explained
Sponsor this newsletter →

Introduction

Ask an AI a question it doesn't know the answer to and it won't say "I don't know" - it will confidently make something up: a fact, an API that doesn't exist, a policy your company doesn't have. That's called a hallucination, and one idea fixes it in almost every production AI system you use today: Retrieval-Augmented Generation (RAG).

🎬 Watch the full video here:

The problem: a model frozen in time

Every large language model has a training cutoff. It only knows what it saw in training, up to a certain date. It has never seen your company's internal wiki, the support ticket filed yesterday or a document created five minutes ago. You have a model that is smart, but frozen in time and blind to your own data.

Three ways to deal with it

  1. Retrain or fine-tune the model on your data. Slow, expensive, and out of date again the moment your data changes.
  2. Paste all your documents into the prompt. Fine for a page or two, but real knowledge bases are thousands of documents - you blow past the context limit and pay for every token on every request.
  3. RAG. Combine information retrieval with text generation. Instead of relying only on what the model learned in training, retrieve relevant documents from your data and include them in the prompt, which grounds the answer in your actual content.

That is the whole idea: retrieve first, then generate. Everything else is implementation detail.

Step 1: Ingestion and embeddings

Before anything else you prepare the knowledge base - once, ahead of time, not while a user is waiting for an answer.

  • Take your documents (PDFs, wiki pages, support articles, database rows) and split them into smaller chunks - a paragraph or a section, not a 50-page document.
  • Convert each chunk into an embedding: a list of numbers that represents the meaning of the text. Chunks that mean similar things get similar numbers, even if they share no words.
  • Store the embeddings in a vector database - Pinecone, pgvector, Qdrant, or Azure AI Search in the Microsoft ecosystem. It's a database built to answer one question fast: which of these thousands of chunks is closest in meaning to this one?

Step 2: Retrieval by semantic similarity

When someone asks a question, it is converted into an embedding using the exact same process, and the system runs a similarity search against the vector database for the chunks closest to the question.

This is not keyword matching. Ask "how do I get my money back?" and it correctly matches a chunk titled "refund policy", even though no words overlap - because the meaning is close, and that's what embeddings capture. The output is the top 3, 5 or 10 most relevant chunks.

Step 3: Augmentation

The retrieved chunks aren't shown to the user directly. They're inserted into the prompt alongside the original question, with an instruction like "Using only the following information, answer this question." The model doesn't just see "how do I get a refund" - it sees the actual refund policy text plus the question.

Step 4: Generation

Only now does the language model generate an answer, with real, current, specific context in front of it instead of relying purely on what it memorized. This is why RAG answers can cite their sources ("according to your refund policy") - the model was handed that source text as part of the prompt. Citations are part of the design from day one, together with the second thing you have to design for: what the model says when retrieval comes back empty. A good RAG system answers "I don't have enough information for that" instead of guessing.

The full pipeline

Documents are chunked and embedded into a vector database ahead of time. At question time the question is embedded too, a similarity search pulls the most relevant chunks, those chunks are merged into the prompt, and the model generates an answer grounded in the retrieved context.

Four pitfalls where RAG breaks

  • Retrieval quality is the real bottleneck. Retrieval can pull back the wrong chunks or nothing relevant at all, and the model will still confidently answer either way. RAG doesn't eliminate hallucination; it moves the failure point from "the model made something up" to "the model trusted the wrong document, or answered with no real support". Garbage in, garbage out still applies - which is why the "insufficient context" path isn't optional polish.
  • Chunking strategy matters more than people expect. Chunks that are too large waste context and dilute relevance; chunks that are too small lose the surrounding meaning a sentence needs.
  • RAG is not fine-tuning. It doesn't teach the model a new skill, tone or reasoning ability - it only gives the model new facts at the moment it answers. If the model needs to behave differently, that's a fine-tuning or prompting problem.
  • Your vector database needs the same hygiene as any other database. Stale, duplicate and outdated chunks don't throw errors - they quietly degrade answers until someone notices the bot confidently citing a policy that changed six months ago.

The one thing to remember: RAG doesn't make the model smarter, it makes the model better informed. The four steps are ingest and embed, retrieve by similarity, augment the prompt, generate the answer.

Key Takeaways

  • RAG is "retrieve first, then generate": chunk and embed documents, find the closest chunks by meaning, add them to the prompt, and generate a grounded answer.
  • RAG fails in practice through poor retrieval, bad chunking and a stale vector store - and it adds facts, not skills.
  • RAG doesn't make the model smarter, it makes the model better informed.

Resources