Why Models Hallucinate

Table of Contents

1. A Confident Wrong Answer
2. Why It Happens: Plausible, Not True
3. Gaps and Mistakes in the Training Data
4. No Built-in "I Don't Know"
5. Confidence Is Not Accuracy
6. The Many Faces of Hallucination
7. Fix 1: Give the Model Sources
8. Recipe: The grounding Module
9. Fix 2: Ask for Citations
10. Fix 3: Verify with Code and Tools
11. Fix 4: Lower the Temperature
12. Fix 5: Evals and Human Review
13. What This Toy Cannot Do
14. Try It Yourself
15. Quick Review: Common Questions

1. A Confident Wrong Answer

A student asks an AI assistant for a book about Cambodian history. The assistant answers with a title, an author, a year, and even a short summary. It all sounds perfect. The student goes to the library and finds out the book does not exist. Nobody lied. The model made up a believable title because believable titles are exactly what it is good at writing.

This is called a hallucination: output that is fluent and confident but not true, or not backed by any source. The word is borrowed from human experience, and it is a bit misleading, because the model is not "seeing things". It is doing its normal job. This chapter explains why that job produces false statements, and then you build a small tool that shows the main cure: make the model answer from evidence, and say "I don't know" when there is none.

You need the toolkit from the earlier chapters, especially the tiny language model from Build a Tiny Language Model and the shared corpus from What Is an LLM?. If you haven't followed along, the key ideas are all repeated below.

2. Why It Happens: Plausible, Not True

Remember the loop: predict the next token, add it, repeat. The model was trained to make its predictions match the text it saw (see How a Model Is Trained). The training goal was "predict text that looks like real text". It was never "say only true things".

Most of the time plausible and true go together, because most of the text people write is about true things. If you ask for the capital of France, the most plausible next word is also the correct one. The trouble starts when the two come apart: a rare fact, a recent event, a very specific detail. Then the model still produces the most plausible continuation, and plausible is not the same as correct.

You can see this with your own tiny model. It writes sentences that were never in its training text. Save this as demo9b.js (it needs src/bigram.js and src/rng.js from chapter 3):

import { corpus, words } from "./src/corpus.js";
import { trainBigram, nextWordProbs, generate } from "./src/bigram.js";
import { mulberry32 } from "./src/rng.js";

const tokens = words(corpus);
const model = trainBigram(tokens);
const text = tokens.join(" ");

for (const seed of [1, 5, 6]) {
  const sentence = generate(model, "sokha", 9, mulberry32(seed));
  console.log(sentence);
  console.log("  appears in the training text:", text.includes(sentence));
}

// Walk through seed 6 and print how likely each chosen word was.
const rng = mulberry32(6);
let word = "sokha";
for (let i = 0; i < 8; i++) {
  const options = nextWordProbs(model, word);
  let r = rng();
  let pick = options.at(-1);
  for (const option of options) {
    if (r < option.prob) { pick = option; break; }
    r -= option.prob;
  }
  console.log(word.padEnd(8), "->", pick.word.padEnd(8), pick.prob.toFixed(2));
  word = pick.word;
}
// sokha opens the app saves time every morning the
//   appears in the training text: false
// sokha opens the students laugh and a small app
//   appears in the training text: false
// sokha makes iced tea the river and dara walks
//   appears in the training text: false
// sokha    -> makes    0.20
// makes    -> iced     1.00
// iced     -> tea      0.40
// tea      -> the      0.50
// the      -> river    0.13
// river    -> and      0.33
// and      -> dara     0.27
// dara     -> walks    0.14

Because the random generator is seeded, you get exactly these lines on every run.

Read the third sentence: "sokha makes iced tea the river and dara walks". The start ("sokha makes iced tea") is close to something the text says, and then it drifts: every pair of neighbouring words is something the model really saw, so every step looks reasonable, but the whole sentence is a claim that was never written anywhere, and nobody checked it. The second sentence, "sokha opens the students laugh and a small app", is fluent-looking nonsense. A real model is far better than this toy, so its nonsense is much harder to spot, but the mechanism is the same: each step is locally plausible, and nothing checks the whole against reality.

3. Gaps and Mistakes in the Training Data

A model can only learn patterns from what it was shown. That leads to several kinds of trouble:

• Missing knowledge. If a topic appeared rarely or never, the model has little to go on. Small local topics (a village, a small company, your school's history) are often thin in the training data.
• Old knowledge. Training stops at some date. The model does not know about anything after that unless you tell it in the prompt.
• Wrong knowledge. The internet contains mistakes, rumours, and jokes. A model that learned from them may repeat them.
• Blurry knowledge. A model does not store a neat list of facts. It stores billions of numbers that nudge predictions. A fact that was seen only a few times can end up half-remembered, like a word on the tip of your tongue, and the model fills the gap with something that fits.

Your toy model shows the "blurry" part clearly. It has no idea who owns the coffee shop; it only knows which words tend to follow which. If you ask it a question about something it has never seen, it will still produce words.

4. No Built-in "I Don't Know"

A language model always has an answer to "what comes next?". The probabilities always add up to 1, and one token always gets picked. There is no special "I have no evidence" signal inside the loop. Compare that with a database query, which can return an empty result.

So if a model says "I don't know", it is because it learned that pattern from text (people do write "I'm not sure" in many places), or because its makers trained it to do so for certain kinds of questions during fine-tuning. That behaviour is learned, and it is imperfect. Models can still guess confidently where a careful human would say "I'm not sure".

This is why the fix for hallucination is usually not "make the model smarter" but "put a system around the model": give it evidence, check its answers, and allow it to refuse. You will build a toy version of that in a moment.

5. Confidence Is Not Accuracy

Look again at the printout in section 2. The step makes -> iced has probability 1.00: in the training text, "makes" was always followed by "iced". The step the -> river has probability 0.13. Yet there is no sign in the output of which parts are reliable. A fluent tone does not mean the claim is true, and a confident tone does not mean the model "knows".

Two common mistakes:

1. Trusting the tone. A model writes in a polished, certain voice even when it is guessing, because polished certain text is what most of its training text looks like.
2. Asking the model how sure it is. The model can write "I am 95% sure", but that is also just generated text. Some research suggests models carry a useful internal sense of uncertainty, and some systems try to read it out, but a sentence of self-reported confidence is not a measurement you can rely on.

A good habit for any user or developer: judge an answer by its evidence, not by its tone.

6. The Many Faces of Hallucination

Hallucination is not one problem. Recognising the types helps you decide how to defend against each:

Developers meet the invented-API one daily: the model suggests array.removeItem(x) because that is a natural name, but JavaScript has no such method. The cure is the same as for facts: run the code, read the docs, check the real source.

7. Fix 1: Give the Model Sources

The single most effective defence is to stop asking the model to answer from memory. Instead, hand it the relevant text and tell it to answer only from that text. If the answer is not in the text, it should say so. This pattern is often called grounding, and when the text is found automatically by searching a library of documents, it is called retrieval-augmented generation (RAG). A later series covers RAG properly; here you build the idea in miniature.

The recipe has three steps:

1. Retrieve: find the documents most related to the question.
2. Decide: if nothing relevant is found, answer "I don't know".
3. Answer from the evidence, and say which document it came from.

In a real system, step 1 uses the embeddings you met earlier (searching by meaning), and step 3 is done by the model reading the retrieved text. In our toy, step 1 counts shared keywords and step 3 returns the matching sentence. The shape of the pipeline is the same, and so is the key behaviour: no evidence, no answer.

8. Recipe: The grounding Module

Create src/grounding.js. It reuses words from src/corpus.js.

import { words } from "./corpus.js";

// Very common words carry no evidence, so we ignore them when matching.
const STOP = new Set([
  "a", "an", "the", "is", "are", "was", "and", "or", "of", "to", "in",
  "on", "at", "by", "for", "with", "what", "who", "when", "where", "how",
  "does", "do", "did", "it", "that", "this", "s", "after", "near",
]);

export function keywords(text) {
  return words(text).filter((w) => !STOP.has(w));
}

// Find the doc that shares the most keywords with the question.
export function retrieve(question, docs, minOverlap = 2) {
  const q = new Set(keywords(question));
  let best = null;
  for (const doc of docs) {
    const d = new Set(keywords(doc.text));
    const shared = [...q].filter((w) => d.has(w));
    if (shared.length >= minOverlap && (!best || shared.length > best.score)) {
      best = { doc, score: shared.length, shared };
    }
  }
  return best;
}

export function groundedAnswer(question, docs) {
  const hit = retrieve(question, docs);
  if (!hit) return { answer: "I don't know.", source: null };
  return { answer: hit.doc.text, source: hit.doc.id };
}

// Crude check: what share of the answer's keywords appear in the docs?
export function checkSupported(answer, docs, threshold = 0.8) {
  const known = new Set(docs.flatMap((d) => keywords(d.text)));
  const kws = keywords(answer);
  const missing = kws.filter((w) => !known.has(w));
  const coverage = kws.length ? (kws.length - missing.length) / kws.length : 1;
  return { supported: coverage >= threshold, coverage, missing };
}

// Show the reader where an answer came from.
export function withCitation({ answer, source }) {
  return source ? `${answer} [source: ${source}]` : answer;
}

The exports are:

• keywords(text) returns the meaningful words of a text (lowercase, no punctuation, no stop words).
• retrieve(question, docs, minOverlap = 2) returns the best matching document as { doc, score, shared }, or null if fewer than minOverlap keywords match. The minOverlap rule is the "no evidence" gate.
• groundedAnswer(question, docs) returns { answer, source }, with "I don't know." and source: null when there is no evidence.
• checkSupported(answer, docs, threshold = 0.8) returns { supported, coverage, missing }: the share of the answer's keywords found somewhere in the documents, and the keywords that were not found.
• withCitation({ answer, source }) appends [source: id] to an answer.

Next, a small library of documents. Create docs.js in the project folder (not in src/). Each document has an id and some text:

export const docs = [
  { id: "shop-1", text: "Sokha opens the small coffee shop near the river every morning." },
  { id: "shop-2", text: "Sokha makes iced coffee with milk and iced tea with lime." },
  { id: "shop-3", text: "The students come to the coffee shop after school." },
  { id: "dara-1", text: "Dara is a student and Dara writes code at the table by the window." },
  { id: "app-1", text: "The app helps the coffee shop take orders on a phone." },
];

These are real sentences from the corpus, which makes them our "trusted library". Try the module in demo9.js:

import { docs } from "./docs.js";
import { retrieve, groundedAnswer, checkSupported, keywords } from "./src/grounding.js";

console.log(keywords("Who opens the coffee shop?"));
const hit = retrieve("Who opens the coffee shop?", docs);
console.log(hit.doc.id, hit.score, hit.shared);
console.log(groundedAnswer("Who opens the coffee shop?", docs));
console.log(groundedAnswer("What is the wifi password?", docs));

console.log(checkSupported("Sokha opens the coffee shop near the river.", docs));
console.log(checkSupported("Sokha opens the shop at six o'clock with a robot.", docs));
console.log(checkSupported("Dara opens the coffee shop every morning.", docs));
// [ 'opens', 'coffee', 'shop' ]
// shop-1 3 [ 'opens', 'coffee', 'shop' ]
// {
//   answer: 'Sokha opens the small coffee shop near the river every morning.',
//   source: 'shop-1'
// }
// { answer: "I don't know.", source: null }
// { supported: true, coverage: 1, missing: [] }
// {
//   supported: false,
//   coverage: 0.42857142857142855,
//   missing: [ 'six', 'o', 'clock', 'robot' ]
// }
// { supported: true, coverage: 1, missing: [] }

All of this output is stable (no randomness). Notice three things:

1. The question about the wifi password has no keywords in common with any document, so the answer is I don't know. The system refuses instead of guessing.
2. The invented sentence "Sokha opens the shop at six o'clock with a robot" is flagged: six, o, clock, and robot appear nowhere in the library. (The o and clock come from how words splits o'clock at the apostrophe.)
3. The last line is an honest failure. "Dara opens the coffee shop every morning" is false (Sokha opens it), but checkSupported says it is supported, because every word does appear in the library. The check looks at words, not meaning. It catches invented details, and it misses swapped facts.

That third point matters beyond the toy. Real support checks (sometimes done by a second model, sometimes by dedicated tools) are much better than word counting, but none are perfect. Treat them as a safety net, not a guarantee.

9. Fix 2: Ask for Citations

A grounded answer should carry its source, so a human can check it in seconds instead of trusting it. withCitation does this in the toy:

import { docs } from "./docs.js";
import { groundedAnswer, withCitation } from "./src/grounding.js";

console.log(withCitation(groundedAnswer("Who opens the coffee shop?", docs)));
console.log(withCitation(groundedAnswer("What is the wifi password?", docs)));
// Sokha opens the small coffee shop near the river every morning. [source: shop-1]
// I don't know.

With a real model you do this in the prompt: "Answer using only the documents below. After each claim, cite the document id in brackets. If the documents do not contain the answer, say you don't know." You will write prompts like this in the next series.

One honest warning: models can invent citations too. A model may write [source: shop-9] for a document that does not exist, or cite a real document that does not support the claim. So your code should check that every cited id exists in the list you sent, and (where it matters) that the cited text really supports the claim. Citations make checking easy. They do not make checking unnecessary.

10. Fix 3: Verify with Code and Tools

Some claims can be checked by a machine, so check them by machine. Models are fluent with numbers but can slip on exact arithmetic, especially long calculations done "in their head". Do not let them do what a calculator does perfectly.

// Verify a model's claim with code instead of trusting it.
const claim = { text: "17 * 24 = 418", a: 17, b: 24, said: 418 };
const real = claim.a * claim.b;
console.log(claim.text, "->", real === claim.said ? "correct" : `wrong, it is ${real}`);
// 17 * 24 = 418 -> wrong, it is 408

This pattern shows up everywhere in AI engineering:

• Calculations: call a calculator function instead of asking the model.
• Code the model writes: run the tests, run the linter, run it in a sandbox (see Docker Sandbox Crash Course).
• Facts that live in a database: query the database and pass the result to the model.
• Dates and times: use the clock, not the model.
• Output format: parse the JSON and validate the keys, as you did in Prompts, Roles and Structured Output.

Many hosted models can be given tools (functions the model may ask your program to call). Then the model decides when to call the calculator or the database and your code does the exact work. This is the next series' territory.

11. Fix 4: Lower the Temperature

From Probabilities, Temperature and Sampling you know that a low temperature makes the model pick the most likely token more often. For tasks where you want faithful, consistent answers (extraction, classification, answering from a document), a low temperature usually reduces random slips.

But be careful about what it does not do. A low temperature does not add knowledge. If the most likely continuation is a wrong one, the model will pick the wrong one every time, only more consistently. Lowering temperature trims some random mistakes; it does not fix gaps in knowledge. Grounding, verification, and evals matter more.

12. Fix 5: Evals and Human Review

How do you know your system is getting better and not worse? You measure it. An eval (evaluation) is a fixed list of test questions with known correct outcomes, run every time you change something. It is a unit test for AI behaviour.

Include unanswerable questions in your list. A system that answers every answerable question but also makes up answers to the unanswerable ones is dangerous. Create evals.js:

import { docs } from "./docs.js";
import { groundedAnswer, retrieve } from "./src/grounding.js";

const cases = [
  { q: "Who opens the coffee shop?", source: "shop-1" },
  { q: "What does Sokha make with lime?", source: "shop-2" },
  { q: "Where does Dara write code?", source: "dara-1" },
  { q: "What is the wifi password?", source: null },
  { q: "When does the shop close on Sunday?", source: null },
];

// A bot that always answers, even with no evidence (like a model with no "I don't know").
function eagerAnswer(q) {
  const hit = retrieve(q, docs, 0);
  return { answer: hit.doc.text, source: hit.doc.id };
}

function runEval(name, answerFn) {
  let passed = 0;
  console.log(`--- ${name} ---`);
  for (const c of cases) {
    const got = answerFn(c.q);
    const ok = got.source === c.source;
    if (ok) passed++;
    console.log(ok ? "PASS" : "FAIL", c.q, "->", got.source ?? "I don't know");
  }
  console.log(`${passed}/${cases.length} passed`);
  return passed;
}

runEval("grounded", (q) => groundedAnswer(q, docs));
runEval("eager", eagerAnswer);

console.log(groundedAnswer("How much does iced tea cost?", docs));
// --- grounded ---
// PASS Who opens the coffee shop? -> shop-1
// PASS What does Sokha make with lime? -> shop-2
// PASS Where does Dara write code? -> dara-1
// PASS What is the wifi password? -> I don't know
// PASS When does the shop close on Sunday? -> I don't know
// 5/5 passed
// --- eager ---
// PASS Who opens the coffee shop? -> shop-1
// PASS What does Sokha make with lime? -> shop-2
// PASS Where does Dara write code? -> dara-1
// FAIL What is the wifi password? -> shop-1
// FAIL When does the shop close on Sunday? -> shop-1
// 3/5 passed
// {
//   answer: 'Sokha makes iced coffee with milk and iced tea with lime.',
//   source: 'shop-2'
// }

The output is deterministic. Both bots pass the three answerable questions. Only the grounded bot passes the two unanswerable ones; the "eager" bot always has an answer, exactly like a model that has no built-in way to refuse. A score of 3/5 hides how bad that is, which is why you should look at which cases fail, not only the total.

The last line is another honest failure. "How much does iced tea cost?" shares the words iced and tea with the shop-2 document, so the retrieval step passes it, and the system returns a sentence that mentions iced tea but says nothing about a price. The eval did not catch it because we never wrote that question into cases. The lesson: an eval is only as good as the questions in it. When you find a failure in real use, add it to the list so it can never come back unnoticed. Try exactly that in the exercises.

Finally, for anything high-stakes (health, law, money, school grades, safety) keep a human in the loop. Evals and checks reduce risk; a person who understands the subject catches what they miss. A good rule: the more it would hurt to be wrong, the more a human should check.

13. What This Toy Cannot Do

Be clear about what is simplified here:

• Real retrieval searches by meaning with embeddings, so it finds "price" when a document says "cost". Ours only matches exact words.
• A real model writes a fresh answer from the retrieved text. Ours returns the matching sentence unchanged.
• Real support checks use stronger methods than counting keywords, as Dara opens the coffee shop showed.
• Real systems still hallucinate, even when grounded. A model can misread the document, blend two documents, or add a detail from its own memory. Grounding lowers the rate, it does not remove it.

The lesson to keep is not the code. It is the design principle: do not trust a model's memory for facts that matter; give it evidence, ask for sources, check what you can check, and measure.

14. Try It Yourself

1. Spot the hallucination. Run demo9b.js with seeds 1 to 8 (change the for list). For each sentence, decide whether it is true, false, or meaningless according to the corpus. How many of them would you mark as "fluent but not true"?
2. Add a document. The corpus says the coffee shop is quiet in the morning and busy at lunch. Add a document with id shop-4 that states the lunch fact in your own words. Ask "When is the coffee shop busy?" and check that shop-4 is the document returned.
3. Fix the price failure. Make "How much does iced tea cost?" return I don't know. Hint: a question word like cost that appears in no document is a sign the evidence is missing. Try requiring that the winning document contains more than half of the question's keywords, not just two. Add the question to cases in evals.js and watch it pass (and check the other cases still pass).
4. Break the checker. Write three false sentences that checkSupported still marks as supported (reuse words from the docs). Then write a stricter check (for example, require each pair of neighbouring keywords in the answer to appear together in a document) and test it.
5. Validate citations. Write validateCitations(text, docs) that finds every [source: id] in a string and returns the ids that are not in docs. Test it with "... [source: shop-9]".
6. Grow the eval. Add five more cases to evals.js, at least three of them unanswerable. Does the grounded bot still pass everything? If not, which rule needs to change?

15. Quick Review: Common Questions

Questions beginners often ask about this chapter. Try answering each one yourself first, then click the question to open the answer.

Where to go next. You now know why models make things up and what a careful developer does about it. The last chapter pulls the whole series together: tokens, cost, context limits, the questions your audience will ask, and a small command-line report you can demo: Limits, Cost and Common Questions.