7 Building a basic RAG system

From retrieved chunks to grounded answers

Chapter 6 ends with a search box: type a question, get back the most relevant chunks. But a person with a question wants an answer, whereas the search box hands back a stack of chunks. The move that turns one into the other is retrieval-augmented generation, or RAG: we retrieve the relevant chunks, hand them to a language model as context, and ask it to answer the question using only what we gave it. The result is an answer grounded in your documents, with citations, and, just as important, a system that says “I cannot find that” when the answer is not there, where a bare model would invent something. This is the first complete application in the book.

In the running system

This chapter assembles our desk’s first complete answer path, the document side that Chapter 25 puts into service as the /ask endpoint. Everything in Part III makes this same path more accurate.

Setup for this chapter

Run in the gaba-core environment, with an OPENROUTER_API_KEY set. This chapter introduces gaba.rag, which builds on the embeddings from Chapter 6. Building the corpus embeds the eight annual reports once (a few minutes on CPU, seconds on a GPU); answers are one model call each.

from dotenv import load_dotenv
load_dotenv()

True

7.1 The RAG pattern

RAG is three steps combined, and we have already built the first; the whole flow looks like this:

flowchart TB
  Q[Question] --> E[Embed query]
  E --> DB[(Vector DB)]
  DB --> C[Top-k chunks]
  Q --> P[Prompt: context plus question]
  C --> P
  P --> L[Language model]
  L --> A[Grounded answer + citations]

Figure 7.1: The RAG pattern. The question is embedded and used to fetch the nearest chunks; question and chunks together form the prompt, and the model composes an answer grounded in what was retrieved.

The key word is grounded. A language model on its own answers from its training, which may be stale, generic, or wrong about your specific documents, and it has no way to tell you where its answer came from. RAG changes the question we ask the model from “what do you know about this?” to “given exactly these passages, what is the answer?” This reframing is what makes the answer checkable and the system trustworthy. In the era of million-token context windows it is fair to ask why we retrieve at all when we could paste all eight reports into the prompt; Chapter 10 measures why not.

7.2 Building it

Two functions do the work, both in gaba.rag. retrieve is Chapter 6’s search, packaged: it embeds the question and returns the top-k nearest chunks (here the four nearest) with their source. rag_answer wraps a model call around it, with a system prompt, RAG_SYSTEM, that is sent with every call. It is short enough to read in full:

from gaba.rag import RAG_SYSTEM
print(RAG_SYSTEM)

Answer the question using only the provided context. Cite the company ticker in square brackets for each fact you use. If the answer is not in the context, say you cannot find it in the documents.

Three instructions, and each one maps to a behavior we can watch for in this chapter. “Using only the provided context” is the grounding itself: it makes the answer a function of the documents, so that nothing comes from the model’s memory; the next example demonstrates this property. “Cite the company ticker” produces the bracketed tickers you will see in the answers, the trail that lets a reader check each fact against its source. And “say you cannot find it” is the abstention that the Staying grounded section exercises and the evaluation at the end of the chapter scores.

from gaba.rag import retrieve, rag_answer

# Retrieval alone: the raw material the answer will be built from.
# n=4 here matches what rag_answer retrieves below.
hits = retrieve("What business segments does Amazon report?", n=4)
for h in hits:
    print(f"[{h['ticker']}] {h['text'][:90]}...")

[AMZN] We have organized our operations into three segments: North America, International, and Am...
[ATSG] We primarily operate through two reportable segments: Cargo Aircraft Management, Inc. ("CA...
[AMZN] In my annual letter over the last three years, I've tried to give shareholders more insigh...
[AMZN] *This Annual Report on Form 10-K and the documents incorporated herein by reference contai...

With the chunks in hand, we let the model turn them into an answer.

answer, sources = rag_answer("Which business segments does Amazon report results for?")
print(answer)
print("\nsources retrieved:", sorted({s["ticker"] for s in sources}))

Amazon reports results for three segments: North America, International, and Amazon Web Services ("AWS") [AMZN].

sources retrieved: ['AMZN', 'HRL']

The model answered from the retrieved passages and named the segments, and because we asked it to cite, the answer points back at the company it came from. Nothing here required the model to already “know” about Amazon; everything it said was in the chunks we handed it. This is the property we want: the answer is a function of the documents alone. Read the sources line carefully, though: retrieval also brought back an unrelated chunk from another company, and the grounded, Amazon-only prose means only that the model ignored it, since retrieval itself was not clean. Nothing forced that restraint, which is exactly why in Chapter 9 we measure faithfulness before we rely on it.

The assembled prompt deserves one direct look. rag_answer built one prompt from the pieces above, a Context: block holding every retrieved chunk tagged with its ticker, then the question, and we can rebuild its skeleton from the sources it returned (each chunk truncated here for display):

context_preview = "\n\n".join(f"[{s['ticker']}] {s['text'][:80]}..." for s in sources)
print(f"Context:\n{context_preview}\n\n"
      "Question: Which business segments does Amazon report results for?")

Context:
[AMZN] We have organized our operations into three segments: North America, Internation...

[AMZN] Last year at this time, I shared my enthusiasm and optimism for Amazon's future....

[AMZN] Further, Amazon's operating income and Free Cash Flow ("FCF") dramatically impro...

[HRL] The International segment processes, markets, and sells Company products interna...

Question: Which business segments does Amazon report results for?

This skeleton, with the system prompt riding above it, is the entire interface between retrieval and the model. Everything retrieval returned is in there, the off-topic chunk included, so the filtering that produced the clean answer happened in the model. When a RAG answer goes wrong, this assembled prompt is the first thing to print, because it shows exactly what the model was given to work with.

7.3 Staying grounded

The real test of a RAG system is what it does when the answer is absent. A model eager to be helpful will readily invent a plausible answer to a question your documents do not address, and a confident fabrication is dangerous precisely because nothing in it signals that it is wrong, whereas a grounded system refuses.

The baseline we are working against comes first: ask the bare model, with no retrieval and no context.

from gaba.llm import call_llm

# No retrieval: the model answers from memory, confidently and without a source.
print(call_llm("What is the capital of France?").text)

The capital of France is **Paris**.

The reply is confident, fluent, and unsupported by anything we can check. It happens to be right here, but the model sounds exactly this sure when it is wrong, and nothing in the reply lets us tell the two cases apart. Now the same question through the grounded system:

answer, sources = rag_answer("What is the capital of France?")
print(answer)

I cannot find the answer to your question in the provided documents.

The corpus is eight annual reports; it has nothing to say about French geography, and the system says so, although the model’s general knowledge could have supplied an answer. This abstention is a feature we require, although it can read at first as a limitation to be tolerated. A system that says “I cannot find that in the documents” is one you can trust with a question that matters, because its answers are tied to the source. One caveat: refusing a wildly off-topic question is the easy case. The harder test is a question that sounds in-domain but whose answer is simply absent, and whether the system abstains there too is exactly what the evaluation in Chapter 9 measures. This chapter shows the behavior; Chapter 9 proves it holds.

Don’t outsource this

An assistant can write the retrieve-and-prompt plumbing in a minute. It should not decide what your system does when the answer is absent. Whether it abstains, hedges, or routes the question to a human is a product and risk decision, which no coding assistant should make for you, and it is the line between a system you can trust with a real question and one you cannot.

7.4 Evaluating answers and refusals

We have a complete system, so we measure it, and a RAG system has to be measured on both halves of its job: answering what it can, and declining what it cannot.

Metric: fraction of cases handled correctly, where “correct” means an answerable question’s answer contains the expected fact, and an unanswerable question is refused.
Test set: three questions answerable from the reports and two that are not.
Baseline: the bare model with no retrieval, run through the same five cases and scored by the same checks; grounding is the thing we are testing, so the baseline is the system without it.

With an AI coding tool

Drafting the REFUSAL phrase list and the scoring loop below is a fast use of an assistant: it will happily generate plausible refusal phrases and a keyword-matching function. Read the list it returns carefully, because a phrase like “not available” can appear in a correct answer for reasons that have nothing to do with refusing, and a keyword match that looks tidy in the draft is exactly the crude check this chapter later admits can mark a right answer wrong.

import pandas as pd

# Refusal phrases that count as a correct abstention.
REFUSAL = ["cannot find", "could not find", "not in the", "unable", "does not",
           "not contain", "no information", "not available"]

cases = [
    ("Which business segments does Amazon report results for?", ["aws"], True),
    ("What kind of utility business does Black Hills operate?", ["electric", "gas"], True),
    ("What kinds of products does Hormel make?",
     ["meat", "turkey", "food", "pork", "spam"], True),
    # Easy abstention: wildly off-topic.
    ("What is the capital of France?", REFUSAL, False),
    # Hard abstention: sounds in-domain (Amazon, advertising revenue) but for a
    # year the 2023 report does not cover. This is the case that matters.
    ("What was Amazon's total advertising revenue in 2019?", REFUSAL, False),
]

rows = []
for question, expected, answerable in cases:
    rag_text, _ = rag_answer(question)
    bare_text = call_llm(question).text  # the baseline: no retrieval, no context
    rows.append(
        {
            "question": question[:40],
            "answerable": answerable,
            "RAG correct": any(e in rag_text.lower() for e in expected),
            "bare correct": any(e in bare_text.lower() for e in expected),
        }
    )

results = pd.DataFrame(rows)
results

	question	answerable	RAG correct	bare correct
0	Which business segments does Amazon repo	True	True	True
1	What kind of utility business does Black	True	True	True
2	What kinds of products does Hormel make?	True	True	True
3	What is the capital of France?	False	True	False
4	What was Amazon's total advertising reve	False	True	False

print(f"RAG:        {results['RAG correct'].sum()}/{len(results)} handled correctly")
print(f"bare model: {results['bare correct'].sum()}/{len(results)} handled correctly")

RAG:        5/5 handled correctly
bare model: 3/5 handled correctly

The RAG column shows the behavior we built for: the system answers the three questions it can, and refuses both kinds it cannot, the obviously off-topic France question and the harder one, a real question about Amazon’s own advertising revenue but for a year the 2023 report never covers. This second refusal is the property that matters, because it is the one a careless system gets wrong, confidently inventing a figure that looks right. Our system holds here, but abstention is never guaranteed; a subtler gap, or a weaker prompt, can still produce a fabrication, which is why in Chapter 9 we measure the abstention rate before we trust it.

The bare-model column deserves an equally careful read. On the answerable questions the bare model tends to score “correct” too: these are public companies, and facts about their segments and products sit in its training data. The columns part ways on the unanswerable rows, where the bare model produces an answer even though a refusal is the correct behavior. Its France answer happens to be right, and a plausible-sounding 2019 advertising figure may come straight from memory or from fabrication, but nothing in either reply lets you tell which, and a figure you cannot distinguish from a fabrication is not one you can put in front of a decision maker. This is the gain from grounding: although the model is no smarter, its answers are checkable and it refuses where checking is impossible. This evaluation is also still small and its “correct” check is a crude keyword match; a real RAG evaluation grades whether the answer is faithful to the sources and complete, a stricter standard than checking whether it contains a word. We build that in Chapter 9, which is the chapter this whole part has been building toward. For now we have something defensible: a system whose answers come from the documents, a baseline that shows what the grounding changed, and a check that confirms the behavior on both the easy refusal and the hard one.

Cost: what one answer costs

A RAG answer is one query embedding (local and free) plus one model call whose input is the question and a handful of retrieved chunks. Because that input runs a few hundred to a couple thousand tokens with a short answer out, one question costs a fraction of a cent on the default model. The corpus embedding is paid once and reused for every question after. What moves cost is how many chunks you place in the context and which model reads them, since the retrieval itself is essentially free; on self-hosted hardware the same two choices appear as latency and throughput, since more chunks per prompt means more input to process per answer and a larger model means fewer answers per second from the same GPU.

Compliance: the context is the exposure

RAG sends retrieved document text to the model on every call. Whatever is in your corpus, customer details, contract terms, internal figures, travels in the prompt. Grounding makes answers trustworthy, but it does not make the data less sensitive; the residency questions from Appendix D apply to every chunk that enters a prompt, whether or not the answer happens to use it.

7.5 Choosing your document stack

Part II built a document pipeline from components chosen for teaching: PyMuPDF for extraction, BGE-M3 for embeddings, and Chroma (the in-process vector store) for storage. Each is one option among several, and on a real project you will pick your own. The choice turns less on which name is best than on a few properties of the problem: how much document structure you need to preserve, whether you would rather run a model or call one, and how much scale and operations you are willing to take on.

Capability	Open-weight	Hosted or managed	Choose by
PDF to text	PyMuPDF (fast text), Unstructured, Docling or Marker (layout-aware)	AWS Textract, Azure Document Intelligence	how much table and column structure surrounds the numbers
Embeddings	BGE-M3, E5, Nomic	OpenAI text-embedding-3, Cohere, Voyage	running it locally and free per call against calling a hosted one
Vector store	Chroma, FAISS, LanceDB (in-process); Qdrant, Milvus (self-hosted)	Pinecone (managed); pgvector inside Postgres	a single app, scale and backups handled for you, or vectors beside existing data

Important

Measure retrieval before swapping any of these out. Chapter 8 shows that on an easy corpus the simplest stack is often already at the ceiling, so a fancier store can cost money and change nothing.
Keep one embedding model across a corpus. If you change it, re-embed everything, because old and new vectors sit in incomparable spaces.
Reach for a managed vector database only when scale or operations demand it. The crossover thinking from Chapter 4 applies: an in-process store handles more than most projects ever reach.

Common failure points

Chunking that splits a table leaves the model a column of numbers with no header; Chapter 5’s structured extraction is the guard.
Embedding the query with a different model than the corpus degrades every result with no error to warn you.
A cosine score carries meaning only as a relative ranking, so treating it as an absolute leads to bad cutoffs; Chapter 6’s similarity matrix is read by comparing entries with one another, and a reading against zero misleads.

The current open-source and vendor options, with trade-offs spelled out, are collected in the tooling-landscape appendix; model and provider selection is Appendix E. Both carry a date, because this layer changes faster than anything else in the book.

7.6 Exercises

7.6.1 Conceptual questions

What does the “augmented” in retrieval-augmented generation refer to?
1. The model is fine-tuned so its weights absorb facts from the documents
2. The embedding vectors are extended with extra dimensions of metadata
3. The user’s question is expanded into several alternative phrasings
4. The prompt carries retrieved passages the model must answer from
RAG changes the question we put to the model. From what, to what?
1. From “answer briefly” to “answer at length, citing your training data”
2. From “what do you know about this?” to “given exactly these passages, what is the answer?”
3. From “what is the answer?” to “which of these documents should I read?”
4. From “answer in your own words” to “quote the source documents verbatim”
A RAG answer cites the source it drew from. What does the citation give you?
1. A way to verify the answer against the document it came from
2. A guarantee that the answer contains no fabricated figures
3. A shorter prompt, since cited chunks can be dropped from the context
4. A faster call, because cited passages are cached by the provider
Why is behavior on an unanswerable question the more important test of a RAG system?
1. Unanswerable questions arrive far more often than answerable ones in production
2. Refusals are cheaper to grade, so the unanswerable test set can be larger
3. It shows whether the system abstains or invents, the most dangerous failure
4. Models answer unanswerable questions faster, which distorts latency metrics
The chapter calls “What was Amazon’s total advertising revenue in 2019?” a harder abstention test than “What is the capital of France?” Why?
1. Answering it would require arithmetic over figures the model cannot do reliably
2. It sounds in-domain, so the system is tempted to answer from plausible-looking chunks
3. The word France never appears in the corpus, so retrieval returns nothing to work with
4. Advertising revenue is spread across more chunks than a single geography fact would be
The chapter’s evaluation counts an answer correct if it contains an expected keyword. What is the weakness of that metric?
1. It rewards verbose answers, which contain more chances to match a keyword
2. It requires a hand-labeled test set, which makes it too slow to run often
3. It cannot recognize refusals, so correct abstentions always count as failures
4. An answer right in meaning but phrased in different words is marked wrong
You add ten more chunks to every prompt to give the model more to work with. What is the most likely effect?
1. Strictly better answers, since the model can ignore what it does not need
2. No change in cost, since providers bill only the tokens the model actually uses
3. A higher cost per answer, and possibly worse answers if the extra chunks are noise
4. A higher abstention rate, because added context makes the model more cautious
Why does the compliance callout say “the context is the exposure”?
1. Retrieved chunk text travels to the model in the prompt, used by the answer or not
2. Providers retain every prompt indefinitely, so context can never truly be deleted
3. Citations reveal internal document titles and paths to every end user
4. Grounding makes the data less sensitive, so only ungrounded calls carry exposure

7.6.2 Build lab

Change rag_answer to return the answer along with the exact chunks it cited. Then write a short check that flags any answer citing a company whose chunk was not actually retrieved. You may use an AI coding tool to draft the change; decide yourself what counts as a citation mismatch worth flagging.

7.6.3 Evaluate lab

Write five answerable questions and two unanswerable ones, run them through rag_answer, and score answering and abstention separately, reporting two numbers: how often it answers correctly and how often it correctly refuses. Then say which of the two you would weight more heavily for a system your company would deploy, and why.

Project ideas

You can now turn a set of documents into searchable text and vectors and answer questions against them, with a citation, or a refusal when the documents do not contain the answer. With these two projects, you apply this pipeline to a corpus of your choice.

A question-answering assistant over your documents. Run the pipeline over your organization’s policy PDFs, a product’s documentation, or a set of research papers. Write a small set of real questions and track two numbers: how often the answer cites a passage that supports it, and how often the system refuses when the corpus has no answer. Data to try: your own PDFs, arXiv papers in one category, or public agency reports on data.gov.
A semantic search tool with no generation step. Extract and embed a corpus, then return the top passages for a query without writing an answer. Compare what the embeddings surface against a keyword search on the same corpus, and note where meaning-based retrieval helps and where it misleads. Data to try: a folder of contracts or manuals, an arXiv category dump, or a Wikipedia subset.

Both of these lead into Part III, where we measure retrieval quality and work to improve it.

Where we go next

This system works, but its retrieval is the simplest possible: embed the query, take the nearest chunks, done. In Part III we make retrieval better, and in Chapter 8 we start with the techniques that matter most in practice: contextual retrieval, reranking, and combining meaning-based search with keyword search. Each one we add, we measure, to make sure it helped.

--- title: "Building a basic RAG system" subtitle: "From retrieved chunks to grounded answers" jupyter: gaba-core --- Chapter 6 ends with a search box: type a question, get back the most relevant chunks. But a person with a question wants an answer, whereas the search box hands back a stack of chunks. The move that turns one into the other is **retrieval-augmented generation**, or RAG: we retrieve the relevant chunks, hand them to a language model as context, and ask it to answer the question using only what we gave it. The result is an answer grounded in your documents, with citations, and, just as important, a system that says "I cannot find that" when the answer is not there, where a bare model would invent something. This is the first complete application in the book. ::: {.callout-note title="In the running system"} This chapter assembles our desk's first complete answer path, the document side that Chapter 25 puts into service as the `/ask` endpoint. Everything in Part III makes this same path more accurate. ::: ::: {.callout-note title="Setup for this chapter"} Run in the `gaba-core` environment, with an `OPENROUTER_API_KEY` set. This chapter introduces `gaba.rag`, which builds on the embeddings from Chapter 6. Building the corpus embeds the eight annual reports once (a few minutes on CPU, seconds on a GPU); answers are one model call each. ::: ```{python} from dotenv import load_dotenv load_dotenv() ``` ## The RAG pattern RAG is three steps combined, and we have already built the first; the whole flow looks like this: ```{mermaid} %%| echo: false %%| label: fig-rag-pattern %%| fig-cap: "The RAG pattern. The question is embedded and used to fetch the nearest chunks; question and chunks together form the prompt, and the model composes an answer grounded in what was retrieved." flowchart TB Q[Question] --> E[Embed query] E --> DB[(Vector DB)] DB --> C[Top-k chunks] Q --> P[Prompt: context plus question] C --> P P --> L[Language model] L --> A[Grounded answer + citations] ``` The key word is *grounded*. A language model on its own answers from its training, which may be stale, generic, or wrong about your specific documents, and it has no way to tell you where its answer came from. RAG changes the question we ask the model from "what do you know about this?" to "given exactly these passages, what is the answer?" This reframing is what makes the answer checkable and the system trustworthy. In the era of million-token context windows it is fair to ask why we retrieve at all when we could paste all eight reports into the prompt; Chapter 10 measures why not. ## Building it Two functions do the work, both in `gaba.rag`. `retrieve` is Chapter 6's search, packaged: it embeds the question and returns the top-k nearest chunks (here the four nearest) with their source. `rag_answer` wraps a model call around it, with a system prompt, `RAG_SYSTEM`, that is sent with every call. It is short enough to read in full: ```{python} from gaba.rag import RAG_SYSTEM print(RAG_SYSTEM) ``` Three instructions, and each one maps to a behavior we can watch for in this chapter. "Using only the provided context" is the grounding itself: it makes the answer a function of the documents, so that nothing comes from the model's memory; the next example demonstrates this property. "Cite the company ticker" produces the bracketed tickers you will see in the answers, the trail that lets a reader check each fact against its source. And "say you cannot find it" is the abstention that the Staying grounded section exercises and the evaluation at the end of the chapter scores. ```{python} from gaba.rag import retrieve, rag_answer # Retrieval alone: the raw material the answer will be built from. # n=4 here matches what rag_answer retrieves below. hits = retrieve("What business segments does Amazon report?", n=4) for h in hits: print(f"[{h['ticker']}] {h['text'][:90]}...") ``` With the chunks in hand, we let the model turn them into an answer. ```{python} answer, sources = rag_answer("Which business segments does Amazon report results for?") print(answer) print("\nsources retrieved:", sorted({s["ticker"] for s in sources})) ``` The model answered from the retrieved passages and named the segments, and because we asked it to cite, the answer points back at the company it came from. Nothing here required the model to already "know" about Amazon; everything it said was in the chunks we handed it. This is the property we want: the answer is a function of the documents alone. Read the sources line carefully, though: retrieval also brought back an unrelated chunk from another company, and the grounded, Amazon-only prose means only that the model ignored it, since retrieval itself was not clean. Nothing forced that restraint, which is exactly why in Chapter 9 we measure faithfulness before we rely on it. The assembled prompt deserves one direct look. `rag_answer` built one prompt from the pieces above, a `Context:` block holding every retrieved chunk tagged with its ticker, then the question, and we can rebuild its skeleton from the sources it returned (each chunk truncated here for display): ```{python} context_preview = "\n\n".join(f"[{s['ticker']}] {s['text'][:80]}..." for s in sources) print(f"Context:\n{context_preview}\n\n" "Question: Which business segments does Amazon report results for?") ``` This skeleton, with the system prompt riding above it, is the entire interface between retrieval and the model. Everything retrieval returned is in there, the off-topic chunk included, so the filtering that produced the clean answer happened in the model. When a RAG answer goes wrong, this assembled prompt is the first thing to print, because it shows exactly what the model was given to work with. ## Staying grounded The real test of a RAG system is what it does when the answer is absent. A model eager to be helpful will readily invent a plausible answer to a question your documents do not address, and a confident fabrication is dangerous precisely because nothing in it signals that it is wrong, whereas a grounded system refuses. The baseline we are working against comes first: ask the bare model, with no retrieval and no context. ```{python} from gaba.llm import call_llm # No retrieval: the model answers from memory, confidently and without a source. print(call_llm("What is the capital of France?").text) ``` The reply is confident, fluent, and unsupported by anything we can check. It happens to be right here, but the model sounds exactly this sure when it is wrong, and nothing in the reply lets us tell the two cases apart. Now the same question through the grounded system: ```{python} answer, sources = rag_answer("What is the capital of France?") print(answer) ``` The corpus is eight annual reports; it has nothing to say about French geography, and the system says so, although the model's general knowledge could have supplied an answer. This abstention is a feature we require, although it can read at first as a limitation to be tolerated. A system that says "I cannot find that in the documents" is one you can trust with a question that matters, because its answers are tied to the source. One caveat: refusing a wildly off-topic question is the easy case. The harder test is a question that *sounds* in-domain but whose answer is simply absent, and whether the system abstains there too is exactly what the evaluation in Chapter 9 measures. This chapter shows the behavior; Chapter 9 proves it holds. ::: {.callout-warning title="Don't outsource this"} An assistant can write the retrieve-and-prompt plumbing in a minute. It should not decide what your system does when the answer is absent. Whether it abstains, hedges, or routes the question to a human is a product and risk decision, which no coding assistant should make for you, and it is the line between a system you can trust with a real question and one you cannot. ::: ## Evaluating answers and refusals We have a complete system, so we measure it, and a RAG system has to be measured on both halves of its job: answering what it can, and declining what it cannot. **Metric**: fraction of cases handled correctly, where "correct" means an answerable question's answer contains the expected fact, and an unanswerable question is refused.\ **Test set**: three questions answerable from the reports and two that are not.\ **Baseline**: the bare model with no retrieval, run through the same five cases and scored by the same checks; grounding is the thing we are testing, so the baseline is the system without it. ::: {.callout-tip title="With an AI coding tool"} Drafting the `REFUSAL` phrase list and the scoring loop below is a fast use of an assistant: it will happily generate plausible refusal phrases and a keyword-matching function. Read the list it returns carefully, because a phrase like "not available" can appear in a correct answer for reasons that have nothing to do with refusing, and a keyword match that looks tidy in the draft is exactly the crude check this chapter later admits can mark a right answer wrong. ::: ```{python} import pandas as pd # Refusal phrases that count as a correct abstention. REFUSAL = ["cannot find", "could not find", "not in the", "unable", "does not", "not contain", "no information", "not available"] cases = [ ("Which business segments does Amazon report results for?", ["aws"], True), ("What kind of utility business does Black Hills operate?", ["electric", "gas"], True), ("What kinds of products does Hormel make?", ["meat", "turkey", "food", "pork", "spam"], True), # Easy abstention: wildly off-topic. ("What is the capital of France?", REFUSAL, False), # Hard abstention: sounds in-domain (Amazon, advertising revenue) but for a # year the 2023 report does not cover. This is the case that matters. ("What was Amazon's total advertising revenue in 2019?", REFUSAL, False), ] rows = [] for question, expected, answerable in cases: rag_text, _ = rag_answer(question) bare_text = call_llm(question).text # the baseline: no retrieval, no context rows.append( { "question": question[:40], "answerable": answerable, "RAG correct": any(e in rag_text.lower() for e in expected), "bare correct": any(e in bare_text.lower() for e in expected), } ) results = pd.DataFrame(rows) results ``` ```{python} print(f"RAG: {results['RAG correct'].sum()}/{len(results)} handled correctly") print(f"bare model: {results['bare correct'].sum()}/{len(results)} handled correctly") ``` The RAG column shows the behavior we built for: the system answers the three questions it can, and refuses both kinds it cannot, the obviously off-topic France question and the harder one, a real question about Amazon's own advertising revenue but for a year the 2023 report never covers. This second refusal is the property that matters, because it is the one a careless system gets wrong, confidently inventing a figure that looks right. Our system holds here, but abstention is never guaranteed; a subtler gap, or a weaker prompt, can still produce a fabrication, which is why in Chapter 9 we measure the abstention rate before we trust it. The bare-model column deserves an equally careful read. On the answerable questions the bare model tends to score "correct" too: these are public companies, and facts about their segments and products sit in its training data. The columns part ways on the unanswerable rows, where the bare model produces an answer even though a refusal is the correct behavior. Its France answer happens to be right, and a plausible-sounding 2019 advertising figure may come straight from memory or from fabrication, but nothing in either reply lets you tell which, and a figure you cannot distinguish from a fabrication is not one you can put in front of a decision maker. This is the gain from grounding: although the model is no smarter, its answers are checkable and it refuses where checking is impossible. This evaluation is also still small and its "correct" check is a crude keyword match; a real RAG evaluation grades whether the answer is faithful to the sources and complete, a stricter standard than checking whether it contains a word. We build that in Chapter 9, which is the chapter this whole part has been building toward. For now we have something defensible: a system whose answers come from the documents, a baseline that shows what the grounding changed, and a check that confirms the behavior on both the easy refusal and the hard one. ::: {.callout-tip title="Cost: what one answer costs"} A RAG answer is one query embedding (local and free) plus one model call whose input is the question and a handful of retrieved chunks. Because that input runs a few hundred to a couple thousand tokens with a short answer out, one question costs a fraction of a cent on the default model. The corpus embedding is paid once and reused for every question after. What moves cost is how many chunks you place in the context and which model reads them, since the retrieval itself is essentially free; on self-hosted hardware the same two choices appear as latency and throughput, since more chunks per prompt means more input to process per answer and a larger model means fewer answers per second from the same GPU. ::: ::: {.callout-important title="Compliance: the context is the exposure"} RAG sends retrieved document text to the model on every call. Whatever is in your corpus, customer details, contract terms, internal figures, travels in the prompt. Grounding makes answers trustworthy, but it does not make the data less sensitive; the residency questions from Appendix D apply to every chunk that enters a prompt, whether or not the answer happens to use it. ::: ## Choosing your document stack {#sec-document-stack} Part II built a document pipeline from components chosen for teaching: PyMuPDF for extraction, BGE-M3 for embeddings, and Chroma (the in-process vector store) for storage. Each is one option among several, and on a real project you will pick your own. The choice turns less on which name is best than on a few properties of the problem: how much document structure you need to preserve, whether you would rather run a model or call one, and how much scale and operations you are willing to take on. | Capability | Open-weight | Hosted or managed | Choose by | |---|---|---|---| | PDF to text | PyMuPDF (fast text), Unstructured, Docling or Marker (layout-aware) | AWS Textract, Azure Document Intelligence | how much table and column structure surrounds the numbers | | Embeddings | BGE-M3, E5, Nomic | OpenAI text-embedding-3, Cohere, Voyage | running it locally and free per call against calling a hosted one | | Vector store | Chroma, FAISS, LanceDB (in-process); Qdrant, Milvus (self-hosted) | Pinecone (managed); pgvector inside Postgres | a single app, scale and backups handled for you, or vectors beside existing data | ::: {.runlabel} **Important** ::: - Measure retrieval before swapping any of these out. Chapter 8 shows that on an easy corpus the simplest stack is often already at the ceiling, so a fancier store can cost money and change nothing. - Keep one embedding model across a corpus. If you change it, re-embed everything, because old and new vectors sit in incomparable spaces. - Reach for a managed vector database only when scale or operations demand it. The crossover thinking from Chapter 4 applies: an in-process store handles more than most projects ever reach. ::: {.runlabel .warn} **Common failure points** ::: - Chunking that splits a table leaves the model a column of numbers with no header; Chapter 5's structured extraction is the guard. - Embedding the query with a different model than the corpus degrades every result with no error to warn you. - A cosine score carries meaning only as a relative ranking, so treating it as an absolute leads to bad cutoffs; Chapter 6's similarity matrix is read by comparing entries with one another, and a reading against zero misleads. The current open-source and vendor options, with trade-offs spelled out, are collected in the [tooling-landscape appendix](../../appendices/i-tooling.qmd); model and provider selection is [Appendix E](../../appendices/e-vendor-selection.qmd). Both carry a date, because this layer changes faster than anything else in the book. ## Exercises ### Conceptual questions 1. What does the "augmented" in retrieval-augmented generation refer to? A. The model is fine-tuned so its weights absorb facts from the documents B. The embedding vectors are extended with extra dimensions of metadata C. The user's question is expanded into several alternative phrasings D. The prompt carries retrieved passages the model must answer from 2. RAG changes the question we put to the model. From what, to what? A. From "answer briefly" to "answer at length, citing your training data" B. From "what do you know about this?" to "given exactly these passages, what is the answer?" C. From "what is the answer?" to "which of these documents should I read?" D. From "answer in your own words" to "quote the source documents verbatim" 3. A RAG answer cites the source it drew from. What does the citation give you? A. A way to verify the answer against the document it came from B. A guarantee that the answer contains no fabricated figures C. A shorter prompt, since cited chunks can be dropped from the context D. A faster call, because cited passages are cached by the provider 4. Why is behavior on an *unanswerable* question the more important test of a RAG system? A. Unanswerable questions arrive far more often than answerable ones in production B. Refusals are cheaper to grade, so the unanswerable test set can be larger C. It shows whether the system abstains or invents, the most dangerous failure D. Models answer unanswerable questions faster, which distorts latency metrics 5. The chapter calls "What was Amazon's total advertising revenue in 2019?" a harder abstention test than "What is the capital of France?" Why? A. Answering it would require arithmetic over figures the model cannot do reliably B. It sounds in-domain, so the system is tempted to answer from plausible-looking chunks C. The word France never appears in the corpus, so retrieval returns nothing to work with D. Advertising revenue is spread across more chunks than a single geography fact would be 6. The chapter's evaluation counts an answer correct if it contains an expected keyword. What is the weakness of that metric? A. It rewards verbose answers, which contain more chances to match a keyword B. It requires a hand-labeled test set, which makes it too slow to run often C. It cannot recognize refusals, so correct abstentions always count as failures D. An answer right in meaning but phrased in different words is marked wrong 7. You add ten more chunks to every prompt to give the model more to work with. What is the most likely effect? A. Strictly better answers, since the model can ignore what it does not need B. No change in cost, since providers bill only the tokens the model actually uses C. A higher cost per answer, and possibly worse answers if the extra chunks are noise D. A higher abstention rate, because added context makes the model more cautious 8. Why does the compliance callout say "the context is the exposure"? A. Retrieved chunk text travels to the model in the prompt, used by the answer or not B. Providers retain every prompt indefinitely, so context can never truly be deleted C. Citations reveal internal document titles and paths to every end user D. Grounding makes the data less sensitive, so only ungrounded calls carry exposure ### Build lab Change `rag_answer` to return the answer along with the exact chunks it cited. Then write a short check that flags any answer citing a company whose chunk was not actually retrieved. You may use an AI coding tool to draft the change; decide yourself what counts as a citation mismatch worth flagging. ### Evaluate lab Write five answerable questions and two unanswerable ones, run them through `rag_answer`, and score answering and abstention separately, reporting two numbers: how often it answers correctly and how often it correctly refuses. Then say which of the two you would weight more heavily for a system your company would deploy, and why. ::: {.callout-tip title="Project ideas"} You can now turn a set of documents into searchable text and vectors and answer questions against them, with a citation, or a refusal when the documents do not contain the answer. With these two projects, you apply this pipeline to a corpus of your choice. - **A question-answering assistant over your documents.** Run the pipeline over your organization's policy PDFs, a product's documentation, or a set of research papers. Write a small set of real questions and track two numbers: how often the answer cites a passage that supports it, and how often the system refuses when the corpus has no answer. *Data to try:* your own PDFs, arXiv papers in one category, or public agency reports on data.gov. - **A semantic search tool with no generation step.** Extract and embed a corpus, then return the top passages for a query without writing an answer. Compare what the embeddings surface against a keyword search on the same corpus, and note where meaning-based retrieval helps and where it misleads. *Data to try:* a folder of contracts or manuals, an arXiv category dump, or a Wikipedia subset. Both of these lead into Part III, where we measure retrieval quality and work to improve it. ::: ::: {.callout-note title="Where we go next"} This system works, but its retrieval is the simplest possible: embed the query, take the nearest chunks, done. In Part III we make retrieval better, and in Chapter 8 we start with the techniques that matter most in practice: contextual retrieval, reranking, and combining meaning-based search with keyword search. Each one we add, we measure, to make sure it helped. :::