Evaluating a RAG System

Table of Contents

1. Why Measure It
2. The Eval Set
3. Hit Rate and MRR
4. Your First Numbers
5. Comparing Setups
6. Checking the Answers
7. Tuning the Refusal Threshold
8. A Failure Taxonomy
9. Regression Tests in CI
10. Do Not Overfit to Your Eval Set
11. LLM-as-Judge and Human Review
12. Growing the Eval Set from Real Questions
13. Recipe: The eval Module
14. Try It Yourself
15. Quick Review: Common Questions

1. Why Measure It

You now have a full RAG pipeline: load and chunk, embed, search, generate with citations, and improve retrieval. Each chapter ended with "this looks better". But looks better to whom, on which question?

Without measurement, tuning a RAG system is a game of whack-a-mole. You change the chunk size, the question you were testing improves, and three questions you forgot about quietly get worse. Nobody notices until a user does. A tiny evaluation set (a list of questions with known answers) turns "I think it is better" into "9 of 12 pass, up from 6".

This chapter builds that harness in src/eval.js. It answers three different questions, and it is important to keep them apart:

1. Did we find the right passage? (retrieval quality)
2. Did the model use it correctly? (answer quality)
3. Did it refuse when the documents have no answer? (honesty)

A wrong final answer can come from any of the three, and the fix is different for each. A system you cannot diagnose is a system you cannot improve.

2. The Eval Set

An eval set is just a file. Create eval/questions.json in raglab. Twelve questions about the club handbook: nine that have an answer and three that do not.

[
  { "id": "fee", "question": "How much does it cost to join each term?", "answerable": true,
    "source": "membership.md", "heading": "Membership fee", "evidence": "10 US dollars", "keyword": "10" },
  { "id": "scholarship", "question": "What can I do if I cannot pay the fee?", "answerable": true,
    "source": "membership.md", "heading": "Scholarships", "evidence": "scholarship membership", "keyword": "Dara" },
  { "id": "refund", "question": "Can I get a refund if I leave early?", "answerable": true,
    "source": "membership.md", "heading": "Refunds", "evidence": "half of", "keyword": "half" },
  { "id": "workshop", "question": "When is the Saturday workshop?", "answerable": true,
    "source": "schedule.md", "heading": "Saturday workshops", "evidence": "9:00 to 12:00", "keyword": "9:00" },
  { "id": "loan-days", "question": "How many days can I borrow a laptop?", "answerable": true,
    "source": "laptops.md", "heading": "How long", "evidence": "up to 7 days", "keyword": "7 days" },
  { "id": "laptop-count", "question": "How many laptops does the club own?", "answerable": true,
    "source": "laptops.md", "heading": "Who can borrow", "evidence": "eight laptops", "keyword": "eight" },
  { "id": "late-laptop", "question": "What happens if I return a laptop late?", "answerable": true,
    "source": "laptops.md", "heading": "Late returns", "evidence": "loses its borrowing rights", "keyword": "one month" },
  { "id": "showcase", "question": "How long is each team's presentation on Showcase Day?", "answerable": true,
    "source": "projects.md", "heading": "Showcase Day", "evidence": "7 minutes", "keyword": "7" },
  { "id": "room", "question": "Which room does the club meet in?", "answerable": true,
    "source": "about.md", "heading": "Where we meet", "evidence": "Room 204", "keyword": "Room 204" },
  { "id": "rust", "question": "Does the club teach Rust?", "answerable": false },
  { "id": "vice-president", "question": "Who is the vice president?", "answerable": false },
  { "id": "wifi-password", "question": "What is the Wi-Fi password?", "answerable": false }
]

Each answerable question records where the answer lives:

• source and heading say which file and section a human would open.
• evidence is a short phrase that must appear in a retrieved chunk. We match on the text (not the heading) so the same file can compare different chunking strategies, where the headings change.
• keyword is a word the final answer must contain.

The three unanswerable questions are deliberate. "Does the club teach Rust?" is close to the real text (the FAQ lists Python and JavaScript), and "What is the Wi-Fi password?" shares words with a real sentence about the Wi-Fi bill. They are the questions most likely to tempt a bad system into inventing something. A RAG system that answers everything is not smart, it is reckless.

Write the questions in the words a user would use, not the words of the document. Look at "What can I do if I cannot pay the fee?". The handbook says "scholarship", "treasurer" and "cannot pay". A student does not know the word "scholarship". If your eval set only echoes the documents, it will flatter your system.

How many questions? Twelve is enough to learn the ideas and to catch big mistakes. A real project starts at 30 to 50 and keeps growing (section 12). Treat any result from a small set as a rough signal, not a fact (section 10).

3. Hit Rate and MRR

For retrieval we ask one thing: did the right chunk appear in the top k results, and how high? Two numbers cover it.

Hit rate at k (hit@k) is the share of questions where the right chunk is somewhere in the top k. If 8 of 9 questions find it, hit@4 is 0.89. It answers "did we find it at all?"

Mean Reciprocal Rank (MRR) also rewards finding it early. For each question, take 1 divided by the position of the first right chunk (1st place scores 1, 2nd scores 0.5, 3rd scores 0.33, not found scores 0). Then average over questions. It answers "how close to the top was it?"

Why care about position? Because the model reads the top results first, and every extra chunk costs tokens and adds distractions. Finding the answer at rank 4 is far better than missing it, but it is worse than rank

1. A system with hit@4 = 1.00 and MRR = 0.40 finds everything but buries it.

Here is the function. It takes any search function, so you can test the vector search, BM25 and the hybrid with the same code:

// searchFn(question) -> [{ chunk, score }]
export async function evalRetrieval(searchFn, questions, { k = 4 } = {}) {
  const details = [];
  for (const q of questions.filter((x) => x.answerable)) {
    const hits = (await searchFn(q.question)).slice(0, k);
    const idx = hits.findIndex((h) => isHit(h.chunk, q));
    details.push({ id: q.id, rank: idx === -1 ? null : idx + 1, top: hits.map((h) => h.chunk.id) });
  }
  const hitRate = details.filter((d) => d.rank).length / details.length;
  const mrr = details.reduce((s, d) => s + (d.rank ? 1 / d.rank : 0), 0) / details.length;
  return { hitRate, mrr, details };
}

Only the nine answerable questions count here, because a question with no answer has no "right chunk". isHit is in the recipe (section 13): the chunk must come from the expected file and contain the evidence phrase.

4. Your First Numbers

Start the mock server (node mock-llm.js) and run the baseline: heading-based chunks, plain vector search. Save this as eval/baseline.js:

// Baseline retrieval numbers, per question, and a sweep over k.
import fs from "node:fs";
import { buildIndex } from "../src/indexer.js";
import { embed } from "../src/embed.js";
import { evalRetrieval } from "../src/eval.js";

const questions = JSON.parse(fs.readFileSync("eval/questions.json", "utf8"));
const store = await buildIndex("docs", {});
const search = async (q) => store.search((await embed([q]))[0], { k: 8 });

const base = await evalRetrieval(search, questions, { k: 4 });
console.log(`hit@4 = ${base.hitRate.toFixed(2)}   MRR = ${base.mrr.toFixed(2)}\n`);
for (const d of base.details) {
  console.log(`${d.id.padEnd(14)} rank ${String(d.rank ?? "-").padEnd(2)} top: ${d.top.slice(0, 3).join(", ")}`);
}

console.log("\nk   hit@k  mrr");
for (const k of [1, 2, 4, 8]) {
  const r = await evalRetrieval(search, questions, { k });
  console.log(`${String(k).padEnd(3)} ${r.hitRate.toFixed(2)}   ${r.mrr.toFixed(2)}`);
}

node eval/baseline.js
hit@4 = 0.89   MRR = 0.60

fee            rank 1  top: membership.md#1, projects.md#1, projects.md#2
scholarship    rank -  top: membership.md#3, membership.md#1, projects.md#4
refund         rank 1  top: membership.md#4, mentors.md#2, mentors.md#3
workshop       rank 3  top: laptops.md#2, projects.md#4, schedule.md#1
loan-days      rank 3  top: faq.md#2, laptops.md#3, laptops.md#2
laptop-count   rank 2  top: faq.md#2, laptops.md#1, laptops.md#3
late-laptop    rank 4  top: faq.md#2, laptops.md#1, conduct.md#4
showcase       rank 1  top: projects.md#2, faq.md#3, projects.md#1
room           rank 1  top: about.md#3, about.md#1, mentors.md#3

k   hit@k  mrr
1   0.44   0.44
2   0.56   0.50
4   0.89   0.60
8   1.00   0.62

All of these numbers are stable: the mock embeddings are deterministic, so you should get the same ones. Read the table, not just the summary:

• Four questions are found at rank 1 (fee, refund, showcase, room). Their words overlap strongly with the chunk.
• scholarship is missing entirely from the top four. The question says "cannot pay the fee", the right section talks about "scholarship membership" and the "treasurer". The toy embeddings match shared words, so "How to pay" and "Membership fee" win. A real embedding model would link "cannot pay" to "scholarship". This is the bag-of-words limit of the mock from chapter 3, now visible as a number.
• late-laptop is found only at rank 4. An FAQ entry about borrowing laptops outranks the section that has the real rule.

The k sweep at the bottom shows the cost of a small k. With k=1 you find the answer for fewer than half the questions. At k=8, hit rate is 1.00, so every answer was in the index, just ranked low. That is a useful diagnosis: the information is there, and the fix is ranking (better embeddings, hybrid search, reranking), not more documents.

But note the price of a bigger k: with 33 chunks, k=8 is a quarter of the whole corpus. Of course a quarter of the corpus contains the answer. Always read hit rate together with the fraction of the index you are returning.

5. Comparing Setups

Now the useful part: change one thing at a time and compare. This script tries three chunking settings and three retrieval methods (vector, BM25 and the hybrid from chapter 6), all at k=4. Save it as eval/retrieval.js:

// Compare retrieval setups on the eval set. Usage: node eval/retrieval.js
import fs from "node:fs";
import { buildIndex } from "../src/indexer.js";
import { embed } from "../src/embed.js";
import { Bm25 } from "../src/bm25.js";
import { hybridSearch } from "../src/hybrid.js";
import { evalRetrieval } from "../src/eval.js";

const questions = JSON.parse(fs.readFileSync("eval/questions.json", "utf8"));
const setups = {
  "heading": { strategy: "heading" },
  "size 300/50": { strategy: "size", size: 300, overlap: 50 },
  "size 600/100": { strategy: "size", size: 600, overlap: 100 },
};

console.log("setup          chunks  method   hit@4  mrr");
for (const [name, opts] of Object.entries(setups)) {
  const store = await buildIndex("docs", opts);
  const bm25 = new Bm25(store.items.map((i) => i.chunk));
  const methods = {
    vector: async (q) => store.search((await embed([q]))[0], { k: 4 }),
    bm25: async (q) => bm25.search(q, 4),
    hybrid: (q) => hybridSearch(q, { store, bm25, k: 4 }),
  };
  for (const [method, fn] of Object.entries(methods)) {
    const r = await evalRetrieval(fn, questions, { k: 4 });
    console.log(`${name.padEnd(14)} ${String(store.size).padEnd(7)} ${method.padEnd(8)} ${r.hitRate.toFixed(2)}   ${r.mrr.toFixed(2)}`);
  }
}

node eval/retrieval.js
setup          chunks  method   hit@4  mrr
heading        33      vector   0.89   0.60
heading        33      bm25     1.00   0.74
heading        33      hybrid   1.00   0.69
size 300/50    32      vector   1.00   0.75
size 300/50    32      bm25     1.00   1.00
size 300/50    32      hybrid   1.00   0.89
size 600/100   16      vector   1.00   0.81
size 600/100   16      bm25     1.00   0.93
size 600/100   16      hybrid   1.00   0.89

These numbers come from the mock, so they are deterministic on your machine too. Do not jump to "size 600 is best". Read them with care:

• Within every setup, BM25 is at least as good as the vector search. That is the mock again: its embeddings are word-hash vectors, so they are a blurry version of keyword matching. With a real embedding model the picture usually changes, because embeddings catch paraphrases that BM25 cannot. Run this script with a real provider before you believe any ranking between methods.
• Chunking matters as much as the search method. Heading chunks give 33 pieces, some only two sentences long, and vector search over them has the worst MRR (0.60). Larger chunks carry more words, so the question has more to match.
• The metric flatters big chunks. Our isHit asks "does the chunk contain the evidence?". A 600-character chunk contains more text, so it contains the evidence more often. And 16 chunks with k=4 means we return a quarter of the corpus. Bigger chunks also cost more tokens and dilute the prompt, which hit rate does not show. This is why retrieval numbers alone cannot pick the winner. Section 6 checks the final answers.
• One question is 0.11. With nine answerable questions, going from 0.89 to 1.00 means one question changed. Do not celebrate or panic over differences that small.

The habit to take away is the workflow, not these numbers: change one variable, re-run the same questions, look at which questions moved.

6. Checking the Answers

Retrieval is only half the system. evalAnswers runs the whole pipeline, ask() from chapter 5, and applies simple rules you can check by machine:

• An answerable question passes if the answer contains the expected keyword and cites a chunk from the expected file.
• An unanswerable question passes only if the system refuses.

When a question fails, it also asks the diagnostic question from section 1: was the evidence in the retrieved chunks? If yes, the problem is generation. If no, it is retrieval. Save this as eval/answers.js:

// Run the full pipeline on the eval set. Usage:
//   node eval/answers.js                      (heading chunks, k=4, minScore=0.2)
//   STRATEGY=size SIZE=600 OVERLAP=100 K=3 MIN_SCORE=0.2 node eval/answers.js
import fs from "node:fs";
import { buildIndex } from "../src/indexer.js";
import { ask } from "../src/rag.js";
import { evalAnswers } from "../src/eval.js";

const env = process.env;
const store = await buildIndex("docs", {
  strategy: env.STRATEGY ?? "heading",
  size: Number(env.SIZE ?? 500),
  overlap: Number(env.OVERLAP ?? 50),
});
const k = Number(env.K ?? 4);
const minScore = Number(env.MIN_SCORE ?? 0.2);
console.log(`chunks=${store.size} k=${k} minScore=${minScore}\n`);

const questions = JSON.parse(fs.readFileSync("eval/questions.json", "utf8"));
await evalAnswers((q) => ask(q, store, { k, minScore }), questions);

node eval/answers.js
chunks=33 k=4 minScore=0.2

PASS  fee             ok
FAIL  scholarship     missing "Dara" (evidence NOT retrieved: retrieval problem)
FAIL  refund          refused, though it retrieved the evidence
FAIL  workshop        missing "9:00" (evidence was retrieved: generation problem)
FAIL  loan-days       missing "7 days" (evidence was retrieved: generation problem)
FAIL  laptop-count    missing "eight" (evidence was retrieved: generation problem)
PASS  late-laptop     ok
FAIL  showcase        missing "7" (evidence was retrieved: generation problem)
PASS  room            ok
FAIL  rust            should have refused
PASS  vice-president  refused
FAIL  wifi-password   should have refused

4/12 passed

Only 4 of 12. Before you decide the system is broken, notice why each one failed. Several say "evidence was retrieved: generation problem". That is the mock model, not the retrieval: it is a canned stand-in that picks the single sentence with the most shared words and cites it. For "When is the Saturday workshop?" it chose a sentence about returning laptops "before the workshop begins". A real model reads all the passages and writes a fluent answer, and would usually do better. The mock's failures are useful anyway: the citation exposed the mistake. The answer cites laptops.md, a human sees that the schedule file was the right one, and the evalAnswers check "cites the right source" catches it too.

Two more failures are worth studying:

• rust and wifi-password should have been refused, and were not. The mock answered from "The fee pays for snacks, printing, and the Wi-Fi bill" and "Mekong Code Club is a student coding club". A weak system finds something vaguely related and answers. Real models make the same mistake less often, but not never. Unanswerable questions in your eval set are how you find out.
• scholarship is a pure retrieval failure, as section 4 predicted.

Now change one setting: chunks of about 600 characters and k=3 (the environment variables are read by the script):

STRATEGY=size SIZE=600 OVERLAP=100 K=3 node eval/answers.js
chunks=16 k=3 minScore=0.2

FAIL  fee             missing "10" (evidence NOT retrieved: retrieval problem)
FAIL  scholarship     missing "Dara" (evidence was retrieved: generation problem)
PASS  refund          ok
PASS  workshop        ok
FAIL  loan-days       missing "7 days" (evidence was retrieved: generation problem)
PASS  laptop-count    ok
PASS  late-laptop     ok
FAIL  showcase        missing "7" (evidence was retrieved: generation problem)
PASS  room            ok
PASS  rust            refused
PASS  vice-president  refused
PASS  wifi-password   refused

8/12 passed

From 4 to 8 passes, with no change to the model. The two unanswerable questions are now refused as well. I would not read a reason into that with a mock model: it is a different set of chunks and a different overlap pattern. That one experiment is the whole value of the harness: you can see which questions changed and why.

Be honest about what this measures. Everything above is the mock model and toy embeddings, so these exact numbers say nothing about a real provider. What transfers is the method: the table, the split between retrieval and generation failures, and the habit of re-running it.

7. Tuning the Refusal Threshold

Chapter 5 introduced minScore: if no chunk scores above it, ask() refuses before calling the model. It is a trade-off between two mistakes:

• set it too low and weak matches reach the model, which may invent an answer to an unanswerable question;
• set it too high and the system refuses questions it could have answered.

Do not guess the number. Sweep it. Save as eval/sweep.js:

// How does minScore trade wrong answers for wrong refusals?
import fs from "node:fs";
import { buildIndex } from "../src/indexer.js";
import { ask } from "../src/rag.js";
import { evalAnswers } from "../src/eval.js";

const questions = JSON.parse(fs.readFileSync("eval/questions.json", "utf8"));
const byId = Object.fromEntries(questions.map((q) => [q.id, q]));
const configs = {
  "heading chunks, k=4": [{}, 4],
  "size 600/100 chunks, k=3": [{ strategy: "size", size: 600, overlap: 100 }, 3],
};

for (const [name, [opts, k]] of Object.entries(configs)) {
  const store = await buildIndex("docs", opts);
  console.log(`\n${name}\nminScore  passed  answerable-refused  unanswerable-answered`);
  for (const minScore of [0, 0.2, 0.3, 0.4]) {
    const { passed, rows } = await evalAnswers((q) => ask(q, store, { k, minScore }), questions, { print: false });
    const wrongRefusals = rows.filter((r) => byId[r.id].answerable && r.why.startsWith("refused")).length;
    const wrongAnswers = rows.filter((r) => !byId[r.id].answerable && !r.pass).length;
    console.log(`${String(minScore).padEnd(9)} ${String(passed).padEnd(7)} ${String(wrongRefusals).padEnd(19)} ${wrongAnswers}`);
  }
}

node eval/sweep.js

heading chunks, k=4
minScore  passed  answerable-refused  unanswerable-answered
0         4       1                   2
0.2       4       1                   2
0.3       4       4                   0
0.4       4       6                   0

size 600/100 chunks, k=3
minScore  passed  answerable-refused  unanswerable-answered
0         8       0                   0
0.2       8       0                   0
0.3       6       3                   0
0.4       6       3                   0

Read the two extra columns. For heading chunks, raising minScore to 0.3 removes both wrongly answered unanswerable questions (the last column drops from 2 to 0), but three more answerable questions get refused (1 becomes 4), and at 0.4 it refuses six of nine. Total passes stay at 4, so a single "pass count" would have hidden this trade-off. For the 600 character setup the sweet spot is wide (0 to 0.2), and 0.3 starts to hurt.

Two lessons. First, report the two kinds of error separately, because which one is worse depends on your product. A homework helper can tolerate a few extra refusals. A medical or legal assistant should refuse more readily. Second, scores are not comparable between setups. A cosine score of 0.3 with these toy vectors means nothing for a real model, whose scores sit in a different range. Re-tune minScore whenever you change the embedding model.

8. A Failure Taxonomy

When a question fails, name the failure. Names turn a vague "it is bad" into a fixable to-do item. These are the six you will meet most:

The first four are things your code can fix. The last two are data problems, and no clever pipeline repairs wrong documents. If your evaluation shows a stale or contradictory answer, the correct action is to edit the documents, and tell someone to own that.

evalAnswers already separates retrieval misses from generation problems. A natural extension is to tag each failed question with one of these names in a column and count them. A bar of "bad chunk: 5, retrieval miss: 2" tells you exactly where to spend your next afternoon.

9. Regression Tests in CI

An eval you run by hand gets forgotten. Turn the important numbers into a regression test: a test that fails if quality drops, run automatically on every change (this is what CI, continuous integration, means: a server runs your tests whenever you push code). Node has a built-in test runner, so no packages. Save as test/eval.test.js:

import test from "node:test";
import assert from "node:assert/strict";
import fs from "node:fs";
import { startMock } from "../mock-llm.js";

// config.js reads the environment when first imported, so set it BEFORE importing.
process.env.LLM_BASE_URL = "http://localhost:8877";
const { buildIndex } = await import("../src/indexer.js");
const { embed } = await import("../src/embed.js");
const { ask } = await import("../src/rag.js");
const { evalRetrieval } = await import("../src/eval.js");

const mock = await startMock(8877);
test.after(() => mock.close());

const questions = JSON.parse(fs.readFileSync("eval/questions.json", "utf8"));
const store = await buildIndex("docs", {});

test("retrieval quality has not regressed", async () => {
  const search = async (q) => store.search((await embed([q]))[0], { k: 4 });
  const { hitRate, mrr } = await evalRetrieval(search, questions, { k: 4 });
  // Floors set a little below today's numbers (0.89 and 0.60) so noise does not fail the build.
  assert.ok(hitRate >= 0.85, `hit@4 fell to ${hitRate.toFixed(2)}`);
  assert.ok(mrr >= 0.55, `MRR fell to ${mrr.toFixed(2)}`);
});

test("when no passage is similar enough, we refuse without calling the model", async () => {
  const r = await ask("What is the Wi-Fi password?", store, { minScore: 0.99 });
  assert.equal(r.refused, true);
  assert.equal(r.usage, null); // null usage = we never called the model
});

test("the known-good answers still cite the right file", async () => {
  for (const id of ["fee", "room"]) {
    const q = questions.find((x) => x.id === id);
    const r = await ask(q.question, store);
    assert.ok(r.answer.includes(q.keyword), `${id}: ${r.answer}`);
    assert.equal(r.sources[0].source, q.source);
  }
});

Add "test": "node --test" to the scripts of package.json, then run it (npm test runs every test file in test/):

node --test --test-reporter=spec test/eval.test.js
✔ retrieval quality has not regressed (11.817584ms)
✔ when no passage is similar enough, we refuse without calling the model (0.877208ms)
✔ the known-good answers still cite the right file (1.730583ms)
ℹ pass 3
ℹ fail 0

The names and counts are stable, the millisecond numbers vary. Three decisions in that file are worth copying:

• The thresholds sit slightly below the current numbers (0.85 against today's 0.89). Today's results are the baseline, and the test fails only on a real drop, not on noise. When you genuinely improve things, raise the floor.
• We test the refusal path without calling the model. Setting minScore: 0.99 forces "no hit", and usage is null when the model was never called. That is cheap, deterministic and tests the money-saving logic from chapter 5.
• We use the mock in the test. CI should not spend money or depend on a provider's mood. A test that calls a live model is flaky (the answer differs between runs) and slow. A common split: the fast mock-based tests run on every push, and a bigger real-model eval runs on a schedule, for example nightly.

To prove the test is useful, I broke the pipeline on purpose by changing the chunking line to { strategy: "size", size: 40, overlap: 0 } (chunks of 40 characters, about six words) in a copy of the test:

✖ retrieval quality has not regressed
  AssertionError [ERR_ASSERTION]: hit@4 fell to 0.11
✖ the known-good answers still cite the right file
ℹ pass 1
ℹ fail 2

Without the test, that change might have shipped with the commit message "try smaller chunks". With it, the build fails and shows the number.

10. Do Not Overfit to Your Eval Set

There is a trap hiding in everything above. Every time you tweak a setting and re-run the same twelve questions, you learn a little about those twelve questions. After twenty tweaks you may have a system that is tuned to them and no better (or worse) on real users. This is overfitting to the eval set. It is the same problem as a student who memorises last year's exam.

Ways to stay honest:

• Keep some questions hidden. Split the set into a tuning part you look at while experimenting and a held-out part you only check occasionally. If both improve, good. If only the first does, you overfit.
• Prefer changes with a reason. "Heading chunks cut a policy in half, so I added overlap" is a fix. "Size 587 scored 0.01 higher" is noise chasing.
• Keep the set small but varied. Cover different documents, different question styles (short, long, misspelled, in Khmer), and unanswerable ones. Twelve questions that are all variations of "how much is X" teach less than twelve different ones.
• Distrust tiny differences. On nine questions, one flip is 0.11. A change from 0.89 to 1.00 is a hint, not a proof.
• Do not put eval questions into the documents. If you add the exact sentence to the handbook to make a test pass, you have measured your editing, not your system.

11. LLM-as-Judge and Human Review

Keyword checks are blunt. "The fee is ten dollars" fails a check for the keyword 10, though it is correct. And a keyword cannot judge whether an answer is complete, polite, or faithful to the passages. Two other ways exist.

LLM-as-judge means asking a second model call to grade the first. You send the question, the retrieved passages and the answer, with instructions such as: Is every claim in the answer supported by the passages? Reply yes or no and name any unsupported claim. It scales to many questions and handles paraphrases. Be careful, because it has weaknesses:

• The judge is a model too. It can be wrong, and can be biased, for example by favouring longer answers or answers that sound like its own style.
• Scores drift when you change the judge model or its wording, so you cannot compare numbers across judge versions.
• Judging costs money and adds a second failure point.
• It can share the blind spots of the model it grades.

Use it as a cheap first filter whose answers you sample and check, never as the final authority. I do not demo it here, because the mock cannot grade anything, and a fake grader would teach you nothing true.

Human review is slower and still the most reliable. Once a week, open 20 real questions with their answers and citations and mark each one good, wrong or unsupported. It teaches you things no metric does: how users phrase questions, what they expected, and what the answer felt like. Citations are what make this fast: the reviewer clicks the source and checks in seconds. If a person cannot verify an answer quickly, the system is hard to trust.

A sensible mix: a keyword eval in CI (cheap, strict, runs always), an LLM judge for broad checks (sampled, with a human checking the judge), and regular human reading of real traffic.

12. Growing the Eval Set from Real Questions

Your best questions are not the ones you imagine. They are the ones users actually ask, in their own words, including the odd ones: typos, Khmer mixed with English, half sentences, and questions you never thought the documents should answer.

So log questions, and treat each failure or thumbs-down as a candidate for a new eval case: find the correct passage, write evidence and keyword, and add it to questions.json. The eval set then grows along with the real problems. After a bug is fixed, its question stays in the set forever, so the bug cannot quietly return.

But a question log is personal data. People type names, phone numbers and private problems into chat boxes. Be conservative:

• Tell users that questions are stored, and why.
• Do not log by default what you do not need. The server in chapter 8 logs question length and scores, not the text. Turn text logging on deliberately, for a limited time, for a clear purpose.
• Remove names, numbers and emails before a question enters the eval set, and keep the raw log short-lived with limited access.
• Never copy real user questions into a public repository.

13. Recipe: The eval Module

The final src/eval.js. It has no dependencies, and it works with any retriever and any ask function, so it will still be useful when you swap the mock for a real provider.

// Measure retrieval and answers against a small set of known questions.

// A retrieved chunk is "right" if it comes from the expected file AND contains the
// evidence phrase. Using the text (not the heading) lets us compare chunking strategies.
const norm = (s) => s.replace(/\s+/g, " ");
const isHit = (chunk, q) => chunk.source === q.source && norm(chunk.text).includes(norm(q.evidence));

// searchFn(question) -> [{ chunk, score }]  (any retriever: vector, BM25, hybrid)
export async function evalRetrieval(searchFn, questions, { k = 4 } = {}) {
  const details = [];
  for (const q of questions.filter((x) => x.answerable)) {
    const hits = (await searchFn(q.question)).slice(0, k);
    const idx = hits.findIndex((h) => isHit(h.chunk, q));
    details.push({ id: q.id, rank: idx === -1 ? null : idx + 1, top: hits.map((h) => h.chunk.id) });
  }
  const hitRate = details.filter((d) => d.rank).length / details.length;
  const mrr = details.reduce((s, d) => s + (d.rank ? 1 / d.rank : 0), 0) / details.length;
  return { hitRate, mrr, details };
}

// askFn(question) -> { answer, sources, refused }
export async function evalAnswers(askFn, questions, { print = true } = {}) {
  const rows = [];
  for (const q of questions) {
    const r = await askFn(q.question);
    let pass, why;
    const retrieved = q.answerable ? (r.hits ?? []).some((h) => isHit(h.chunk, q)) : null;
    if (!q.answerable) {
      pass = r.refused;
      why = pass ? "refused" : "should have refused";
    } else if (r.refused) {
      pass = false; why = retrieved ? "refused, though it retrieved the evidence" : "refused, evidence not retrieved";
    } else {
      const cited = [...r.answer.matchAll(/\[(\d+)\]/g)].map((m) => r.sources[Number(m[1]) - 1]?.source);
      const hasWord = r.answer.toLowerCase().includes(q.keyword.toLowerCase());
      const rightSource = cited.includes(q.source);
      pass = hasWord && rightSource;
      why = pass ? "ok" : !hasWord ? `missing "${q.keyword}"` : `cited ${cited.join(",") || "nothing"}`;
      if (!pass) why += retrieved ? " (evidence was retrieved: generation problem)" : " (evidence NOT retrieved: retrieval problem)";
    }
    rows.push({ id: q.id, pass, retrieved, why });
  }
  const passed = rows.filter((r) => r.pass).length;
  if (print) {
    for (const r of rows) console.log(`${r.pass ? "PASS" : "FAIL"}  ${r.id.padEnd(15)} ${r.why}`);
    console.log(`\n${passed}/${rows.length} passed`);
  }
  return { passed, total: rows.length, rows };
}

Two details to notice. isHit normalises whitespace on both sides, so an evidence phrase that spans a line break in the Markdown still matches. And evalAnswers takes an askFn, not the store, so you can evaluate ask(), a hybrid pipeline, or a version with query rewriting by passing a different function. All it needs back is { answer, sources, hits, refused }.

14. Try It Yourself

1. Add your own questions. Write four more answerable questions in a student's words, such as "Can I bring my little sister?", and two more unanswerable ones. Find each evidence phrase in docs/. Re-run eval/baseline.js. Which of your questions fail, and why?
2. Hunt the paraphrase gap. Add a question that shares no content words with its answer, such as "What is the price of joining?" against "Membership costs 10 US dollars". Check which methods (vector, BM25, hybrid) find it. What would you expect a real embedding model to do?
3. A degenerate setup. Add { strategy: "size", size: 1200, overlap: 0 } to eval/retrieval.js. How many chunks are there? What does hit@4 mean when the whole corpus is a handful of chunks? What does that tell you about reading hit rate alone?
4. Count failure types. Extend evalAnswers so each failed row gets a label from the taxonomy (retrieval miss, generation) and the function prints a count per label. Which label dominates for heading chunks?
5. Add a held-out split. Mark three of the nine answerable questions "holdout": true. Make eval/retrieval.js print results for the tuning and held-out parts separately.
6. Break it on purpose. Change the prompt in buildPrompt so it no longer asks for citations. Which eval checks fail? Then restore it. This is how you learn what each check protects.

15. Quick Review: Common Questions

Questions beginners often ask about this chapter. Try answering each one yourself first, then click the question to open the answer.

Where to go next. You can now measure whether the pipeline works and catch regressions. The last step is to put it in front of people: a command-line tool, a small web endpoint with the safety rules you already know, and a checklist for access control, privacy, cost and freshness. The next chapter, Ship a Document Chatbot, does exactly that.