How a Model Is Trained

Table of Contents

1. Counting Is Not How Big Models Learn
2. Parameters: The Knobs of a Model
3. Loss: A Score for How Wrong You Are
4. Gradient Descent: Turn the Knobs Downhill
5. The Training Loop in Code
6. Watching the Loss Fall
7. What the Model Learned
8. The Learning Rate
9. Memorizing vs Learning: Unseen Text
10. Scaling Up: Billions of Parameters
11. Pretraining, Fine-Tuning, and Preference Tuning
12. Why Training Is Expensive and Inference Is Cheap
13. Training Data, Copyright, and Bias
14. Recipe: The train Module
15. Try It Yourself
16. Quick Review: Common Questions

1. Counting Is Not How Big Models Learn

In Build a Tiny Language Model you made a model by counting: how often does "coffee" follow "the"? Count every pair once, divide, done. No learning loop, no guessing.

Counting works for a toy but not for a real LLM. Real models read hundreds or thousands of earlier tokens, so the number of possible contexts is astronomically bigger than any text you could count, and they must answer sensibly for contexts they have never seen.

So real models do not store counts. They store a large pile of numbers (called parameters or weights) and tune those numbers, slowly, until the model predicts text well. The tuning process is called training.

In this chapter you do exactly that, on a model small enough to read. The model will end up almost the same as your counting table, which is the point: you will see what training does without any math beyond school arithmetic.

2. Parameters: The Knobs of a Model

Think of a mixing desk with many knobs: each knob is a number, and all the positions together decide what comes out. A model is a set of numbers like that.

Our tiny model is a table of scores (the series calls them logits, as you saw in Probabilities, Temperature and Sampling). It has one row for every word in the vocabulary and one column for every word that could come next:

            next: coffee  tea  table  ...
current: the         ?     ?     ?
         coffee      ?     ?     ?
         students    ?     ?     ?

To predict what follows "the", look at the "the" row, pass the scores through softmax to get probabilities, and pick. Every cell is one parameter. With a vocabulary of V words the table has V * V parameters.

Let's look at the data and count them:

// demo-data.js
import { corpus, words } from "./src/corpus.js";
import { buildVocab, makePairs } from "./src/train.js";

const tokens = words(corpus);
const { vocab, index } = buildVocab(tokens);
const pairs = makePairs(tokens, index);

console.log("tokens:", tokens.length, "vocab:", vocab.length, "pairs:", pairs.length);
console.log("parameters:", vocab.length * vocab.length);
console.log("first pair:", pairs[0], vocab[pairs[0][0]], "->", vocab[pairs[0][1]]);
// tokens: 182 vocab: 67 pairs: 181
// parameters: 4489
// first pair: [ 52, 39 ] sokha -> opens

The corpus has 182 words and 67 different ones. Each neighbouring pair (current, next) is one training example: "after sokha, the true next word is opens". There are 181 of them. The table has 67 * 67 = 4,489 parameters. (The code for buildVocab and makePairs is in the recipe at the end of the chapter; you will paste it there.)

The numbers 52 and 39 are word indexes (the vocabulary is sorted alphabetically). The model works on numbers, not strings.

What is stable: all four numbers come from the shared corpus, so you should see exactly these on any computer.

Simplification. This toy uses whole words; real models use sub-word tokens and compute scores from the whole context with layers of arithmetic, not one row per word. Chapter 6 shows a glimpse.

3. Loss: A Score for How Wrong You Are

Training needs a way to say "how bad is the model right now?" with one number. That number is the loss. Lower is better.

For next-word prediction the loss is built from a simple idea. For one example, look at the probability the model gave to the true next word:

• gave it 0.9: great, small penalty.
• gave it 0.1: poor, bigger penalty.
• gave it 0.001: terrible, huge penalty.

The penalty is -log(p), where p is that probability. You do not need logarithms to use this. Just remember the shape:

The penalty grows quickly as the model becomes less sure of the right answer. This is cross-entropy loss; the loss for the whole dataset is the average penalty over all examples.

A model that knows nothing gives every one of the 67 words the same chance, 1/67. Its loss is -log(1/67), which is log(67):

// demo-loss.js
import { corpus, words } from "./src/corpus.js";
import { buildVocab, makePairs, lossOf } from "./src/train.js";

const tokens = words(corpus);
const { vocab, index } = buildVocab(tokens);
const pairs = makePairs(tokens, index);

const zero = vocab.map(() => vocab.map(() => 0)); // all scores equal
console.log("start loss:", lossOf(zero, pairs).toFixed(4));
console.log("ln(vocab):", Math.log(vocab.length).toFixed(4));
// start loss: 4.2047
// ln(vocab): 4.2047

The two lines match: a model that guesses at random has loss log(vocabulary size). That is the number to beat. (Math.log in JavaScript is the natural logarithm, which is what we use throughout.)

4. Gradient Descent: Turn the Knobs Downhill

Now the clever part. We have knobs (parameters) and a score (loss). How do we turn the knobs to lower the score?

Imagine you are standing on a foggy hill and want to reach the valley. You cannot see far, but you can feel the slope under your feet. So you take a small step downhill, feel the slope again, step again. Slowly you reach the bottom.

That is gradient descent:

1. Run the model on the data and compute the loss.
2. For each parameter, work out its gradient: "if I raise this number a tiny bit, does the loss go up or down, and by how much?"
3. Move every parameter a small step in the direction that lowers the loss.
4. Repeat, thousands or millions of times.

The size of each step is the learning rate: small is safe but slow, huge can overshoot the valley.

Working out the gradient for a big network needs calculus and a trick called backpropagation. We are lucky: for our table, with softmax and cross-entropy loss, the gradient is simple enough to write on one line. For one example, with true next word t:

gradient of the score for word j  =  probability(j)  -  (1 if j is t, else 0)

Read that in plain words. For the true word, the gradient is probability - 1, which is negative, so the step raises its score. For every wrong word, the gradient is probability, which is positive, so the step lowers its score, more for words the model liked too much. The model is pushed up where it should be right and pushed down where it was wrong. That single line is the whole learning rule of the toy.

Real frameworks compute gradients automatically for billions of parameters; the idea is the same: small steps downhill.

5. The Training Loop in Code

Here is the core, written in about 35 lines. It builds on softmax from the previous chapter.

// src/train.js (core)
import { softmax } from "./sampling.js";

export function lossOf(logits, pairs) {
  let total = 0;
  for (const [cur, next] of pairs) {
    total += -Math.log(softmax(logits[cur])[next]);
  }
  return total / pairs.length;
}

export function step(logits, pairs, lr) {
  const grad = logits.map((row) => row.map(() => 0));
  for (const [cur, next] of pairs) {
    const probs = softmax(logits[cur]);
    for (let j = 0; j < probs.length; j++) {
      grad[cur][j] += (probs[j] - (j === next ? 1 : 0)) / pairs.length;
    }
  }
  for (let i = 0; i < logits.length; i++) {
    for (let j = 0; j < logits[i].length; j++) logits[i][j] -= lr * grad[i][j];
  }
}

Read it slowly:

• lossOf is the cross-entropy from section 3: for every example, take the probability the model gives the true next word, and average -log of it.
• step is one round of gradient descent. It makes a grad table the same shape as logits, filled with zeros. For every example it adds probability - onehot to the right row (the one-line rule from section 4, divided by the number of examples to make an average).
• The last loop moves every parameter by -lr * gradient: a small step in the downhill direction.

Then the loop that calls step again and again:

export function train(tokens, { steps = 200, lr = 5, every = 20 } = {}) {
  const { vocab, index } = buildVocab(tokens);
  const pairs = makePairs(tokens, index);
  const logits = vocab.map(() => vocab.map(() => 0));
  const history = [{ step: 0, loss: lossOf(logits, pairs) }];
  for (let s = 1; s <= steps; s++) {
    step(logits, pairs, lr);
    if (s % every === 0) history.push({ step: s, loss: lossOf(logits, pairs) });
  }
  return { vocab, index, logits, history };
}

All scores start at 0, so the model starts knowing nothing. Then it repeats step, recording the loss now and then in history. This is a training loop, and the same shape sits inside every deep learning framework: predict, measure loss, compute gradient, update, repeat.

We use full-batch training: every step uses all 181 examples. Real training uses small random batches, because the dataset is huge.

6. Watching the Loss Fall

Run the training loop for 300 steps and print the loss every 50:

// demo-train.js
import { corpus, words } from "./src/corpus.js";
import { train } from "./src/train.js";

const tokens = words(corpus);
const model = train(tokens, { steps: 300, lr: 5, every: 50 });
for (const h of model.history) {
  console.log(String(h.step).padStart(4), h.loss.toFixed(4));
}
//    0 4.2047
//   50 2.5158
//  100 1.8059
//  150 1.4594
//  200 1.2882
//  250 1.2001
//  300 1.1502

This is what "learning" looks like to a computer: a number going down. From 4.20 (random guessing among 67 words) to 1.15 (much better than random) in 300 small steps. Nobody told the model about coffee or students. The numbers moved because moving them made the predictions on the data less wrong.

Notice the shape too: fast progress at first (4.20 to 2.52 in 50 steps), slow at the end (1.20 to 1.15 in the last 50). Real training curves look like this too.

Every line is deterministic: there is no random number in this code, so you should see exactly these numbers.

7. What the Model Learned

Train longer, then peek inside the table. Compare the learned probabilities with the counted ones from the bigram model of chapter 3:

// demo-learned.js
import { corpus, words } from "./src/corpus.js";
import { train, lossOf, makePairs } from "./src/train.js";
import { softmax } from "./src/sampling.js";
import { trainBigram, nextWordProbs } from "./src/bigram.js";

const tokens = words(corpus);
const model = train(tokens, { steps: 1000, lr: 5, every: 250 });
for (const h of model.history) {
  console.log(String(h.step).padStart(4), h.loss.toFixed(4));
}

function learned(word) {
  const probs = softmax(model.logits[model.index.get(word)]);
  return probs
    .map((p, i) => [model.vocab[i], p])
    .sort((a, b) => b[1] - a[1])
    .slice(0, 4)
    .map(([w, p]) => `${w} ${p.toFixed(3)}`);
}

const counts = trainBigram(tokens);
const fmt = ([w, p]) => `${w} ${p.toFixed(3)}`;
console.log("learned after 'the':", learned("the").join(", "));
console.log("counted after 'the':", nextWordProbs(counts, "the").slice(0, 4).map(fmt).join(", "));
console.log("learned after 'dara':", learned("dara").join(", "));
//    0 4.2047
//  250 1.2001
//  500 1.0722
//  750 1.0430
// 1000 1.0305
// learned after 'the': students 0.207, coffee 0.207, app 0.124, river 0.124
// counted after 'the': coffee 0.208, students 0.208, river 0.125, app 0.125
// learned after 'dara': writes 0.280, drinks 0.137, is 0.137, says 0.137

After 1000 steps the learned probabilities after "the" are almost identical to the counted ones (0.207 against 0.208): coffee and students most likely, then river and app. The model found, by nudging numbers, what the counting table computed directly. For this tiny model there is a perfect answer (the counts), and training is walking toward it.

How low can it go? The best possible loss for this model is the loss of the counting table itself:

// continue demo-learned.js
let best = 0;
const pairs = makePairs(tokens, model.index);
for (const [c, n] of pairs) {
  const row = nextWordProbs(counts, model.vocab[c]);
  best += -Math.log(row.find(([w]) => w === model.vocab[n])[1]);
}
console.log("loss of counted model:", (best / pairs.length).toFixed(4));
// loss of counted model: 1.0004

Training reached 1.0305 and is heading to 1.0004. It cannot go lower than that, because a model that only sees one previous word cannot be less confused than the counts. After "dara" there are several legitimate next words, and no amount of training can say which one comes next. That floor, the unavoidable uncertainty, is also why language models need more context to get lower loss: the same reason a trigram beat a bigram in chapter 3.

One more honest note. Sampling from this table gives text like "sokha cleans the window dara thinks about", where every adjacent pair really appears in the corpus but the sentence is not good English, because one word of context is too little. The model learned the text's habits, not meaning. That distinction matters in the hallucination chapter.

8. The Learning Rate

The step size, lr, is the most important setting in training. Try four values for 100 steps each:

// demo-lr.js
import { corpus, words } from "./src/corpus.js";
import { train } from "./src/train.js";

const tokens = words(corpus);
for (const lr of [0.5, 5, 50, 500, 5000]) {
  const m = train(tokens, { steps: 100, lr, every: 100 });
  console.log("lr", String(lr).padEnd(5), "loss after 100 steps:", m.history[1].loss.toFixed(4));
}
// lr 0.5   loss after 100 steps: 3.7206
// lr 5     loss after 100 steps: 1.8059
// lr 50    loss after 100 steps: 1.0300
// lr 500   loss after 100 steps: 4.3847
// lr 5000  loss after 100 steps: 61.9882

Read it like a story: 0.5 is safe but far too slow; 5 makes steady progress; 50 is fastest here. At 500 the loss is worse than where it started (4.38 against 4.20): the steps are so big that the model jumps over the valley and lands higher on the other side. At 5000 it has diverged.

A good learning rate is "as big as possible without diverging". Our toy is a gentle, smooth problem, so 50 works. Real networks are bumpier and need much smaller steps, plus tricks like starting small, then slowing down. Picking learning rates is a big part of the craft of training real models.

9. Memorizing vs Learning: Unseen Text

A low loss on the training text can fool you. A model could do well on text it has seen because it essentially memorized it, and do badly on new text. The honest test is to hold back some data, never train on it, and measure the loss on it afterwards.

// demo-heldout.js
import { corpus, words } from "./src/corpus.js";
import { train, makePairs, lossOf } from "./src/train.js";

const tokens = words(corpus);
const cut = Math.floor(tokens.length * 0.8);
const m = train(tokens.slice(0, cut), { steps: 1000, lr: 5, every: 1000 });

// Pairs from the last 20% of the text, which the model never saw.
const unseen = makePairs(tokens.slice(cut - 1), m.index);
const usable = unseen.filter(([a, b]) => a !== undefined && b !== undefined);

console.log("train loss:   ", m.history[1].loss.toFixed(4));
console.log("unseen pairs: ", usable.length, "usable of", unseen.length);
console.log("unseen loss:  ", lossOf(m.logits, usable).toFixed(4));
// train loss:    0.9436
// unseen pairs:  23 usable of 37
// unseen loss:   2.9710

We train on the first 80% of the words and test on the rest. The training loss is 0.94, but the loss on the unseen text is 2.97, three times worse. Even that flatters the model: 14 of the 37 unseen pairs had to be thrown away because they contain a word that never appeared in the training part, so this toy cannot even score them.

A large gap between training loss and unseen loss is called overfitting. Real teams always keep a validation set the model never trains on, and watch that loss too.

10. Scaling Up: Billions of Parameters

Our table has 4,489 parameters. Real language models are described by their parameter counts, for example "7 billion" or "70 billion" (these are example sizes, not claims about any particular product). The parameters are not a lookup table of words. They are the numbers inside many layers of arithmetic (the Transformer, which chapter 6 introduces), which compute the next-token scores from the whole context.

What does a parameter count mean in practice? Mostly memory. Each number takes space, often 2 bytes in common formats:

// demo-size.js
for (const p of [4489, 7e9, 70e9]) {
  console.log(String(p).padStart(12), "params ->", (p * 2) / 1e9, "GB at 2 bytes each");
}
//         4489 params -> 0.000008978 GB at 2 bytes each
//   7000000000 params -> 14 GB at 2 bytes each
//  70000000000 params -> 140 GB at 2 bytes each

A 7-billion-parameter model needs about 14 GB just to hold its numbers (at 2 bytes each; smaller formats shrink that). A 70-billion one needs about 140 GB, more memory than any normal laptop or a single typical graphics card has, which is why big models run on special servers. Smaller models can run on your own machine, which is a real option you will meet in the final chapter.

Bigger is not automatically better, but with enough data and compute, larger models have tended to predict text better. That is an observed trend, not a guarantee.

11. Pretraining, Fine-Tuning, and Preference Tuning

The training you just did is pretraining in miniature: predict the next token on a huge pile of text. Chat assistants are made in stages. This is a conceptual picture; labs differ in the details, and many of them keep their exact recipes private.

Stage 1: Pretraining. Train a model from random numbers to predict the next token over a very large collection of text (web pages, books, code, and so on). The result is a base model. It is a very good text continuer. Give it "Q: What is the capital of Cambodia? A:" and it may continue with an answer, but it may also continue with five more questions, because on the internet that is what often follows a question. It has no idea it is supposed to be a helpful assistant.

Stage 2: Fine-tuning (instruction tuning). Keep training the same model, now on a much smaller set of carefully made examples of conversations: a question, then a good answer. Same loss, same gradient descent, different data. The model learns the format and style of being an assistant.

Stage 3: Preference tuning (for example RLHF). Show people (or another model) two answers to the same question and ask which is better. Then adjust the model to make the preferred kind of answer more likely. RLHF stands for reinforcement learning from human feedback, and it is one family of ways to do this. The goal is answers that are more helpful, more honest, and less harmful.

random numbers
      |
      |  pretraining: predict next token on huge text
      v
  base model        (continues text, not yet an assistant)
      |
      |  fine-tuning: examples of good conversations
      v
  assistant-style model
      |
      |  preference tuning: learn which answers people prefer
      v
  chat assistant

Two honest cautions. Tuning changes behaviour, but it does not install a fact-checker: a tuned model can still be confidently wrong, the subject of the hallucination chapter. And preference tuning teaches what raters liked, and raters can prefer polite, confident answers over correct ones. Treat tuned behaviour as "usually helpful", not "always right".

For most application work you will not train anything: you will write good prompts and give the model the right information.

12. Why Training Is Expensive and Inference Is Cheap

You may have heard that training an LLM costs a great deal, while using one costs little per request. Our toy shows why, with plain counting.

Inference (using a trained model) means: run the arithmetic once per generated token. The parameters are fixed. You just do multiplications.

Training means doing the same arithmetic over the entire dataset, many times, plus computing gradients (roughly the same amount of work again, or more) and updating every parameter on every step.

In our toy, one training step is 181 softmaxes plus 4,489 parameter updates; 1,000 steps is 181,000 softmaxes.

For a real model both numbers get enormous: the dataset is trillions of tokens, each token goes through billions of parameters, and training makes one or more passes over it. The total is far more arithmetic than one laptop could finish in a lifetime, so labs use thousands of specialized chips running for weeks or months, with a large electricity bill. Training is a big one-time cost for the lab.

After that, serving one request is a few seconds of one slice of that hardware. The training cost is spread over millions of requests. That is why a lab can offer a hosted model for a small price per request, and why you should almost never train your own from scratch. (Hosted services still charge per token, because answering is real work. You will estimate that cost in Tokens and Tokenizers and again in the final chapter.)

A trained model is also frozen: while you chat, its parameters do not change, and its knowledge stops at the end of its training data. That is why a model alone does not know yesterday's news.

13. Training Data, Copyright, and Bias

A model can only learn patterns that were in its data. This has consequences worth saying plainly.

Where the data comes from. Large models are usually trained on huge collections of web text, books, and code. The exact datasets are often not published in detail.

Copyright and permission. Much of that text was written by people who were never asked. Whether training on copyrighted work is allowed, and under what conditions, is being argued in courts and parliaments in different countries, and the answer is not settled. Some models are trained only on text whose owners agreed, or on openly licensed text. As a developer, check the license and terms of any model you use.

Bias. If the data contains stereotypes, or mostly represents some people, languages, and viewpoints, the model absorbs that. Text in English is much more common online than text in Khmer, so many models are noticeably stronger in English and weaker for smaller languages. If you build for Khmer speakers, test with real Khmer examples, and do not assume the quality you see in English carries over.

None of this means "do not use LLMs". It means treat a model like a widely read but unreliable colleague whose reading list you cannot see.

14. Recipe: The train Module

Create src/train.js in your llmlab folder. It needs src/sampling.js from Probabilities, Temperature and Sampling (for softmax). Start the file with these two helpers, then paste lossOf, step and train from section 5 below them (and add export before each function):

// src/train.js
import { softmax } from "./sampling.js";

// Unique words, sorted, plus a word -> index lookup.
export function buildVocab(tokens) {
  const vocab = [...new Set(tokens)].sort();
  const index = new Map(vocab.map((w, i) => [w, i]));
  return { vocab, index };
}

// Every (current word, next word) pair as two index numbers.
export function makePairs(tokens, index) {
  const pairs = [];
  for (let i = 0; i < tokens.length - 1; i++) {
    pairs.push([index.get(tokens[i]), index.get(tokens[i + 1])]);
  }
  return pairs;
}

The exports are buildVocab(tokens), makePairs(tokens, index), lossOf(logits, pairs), step(logits, pairs, lr) and train(tokens, { steps, lr, every }). The core logic (loss, step, and loop) is under 40 lines. Save any demo above as a demo-*.js file next to src/ and run it with node demo-train.js.

15. Try It Yourself

1. Plot the curve. Train for 2000 steps with every: 100 and print the loss each time. Where does the curve flatten? Compare with the floor of 1.0004 from section 7.
2. Learning rate hunt. Run 100 steps with lr set to 10, 20, 100, 200, 300. At which value does the loss stop falling and start to get worse than the start loss of 4.2047? Why?
3. Different sentence, different habits. Replace the corpus text with five sentences about your own life (use words() from src/corpus.js). Train, then print the top four predictions after a common word. Do they match what you wrote?
4. Overfit it. Train on only the first 40 words of the corpus for 2000 steps and measure the loss on the remaining words (as in section 9). How many unseen pairs are usable? Is the unseen loss higher or lower than before, and why?
5. Compare with counts. After training, compute the average absolute difference between the learned probabilities and the counted probabilities (from trainBigram) for the words that have counts. Does it shrink as you add steps?

16. Quick Review: Common Questions

Questions beginners often ask about this chapter. Try answering each one yourself first, then click the question to open the answer.

Where to go next. You have now seen the full training loop: a table of numbers, a loss, and small steps downhill. But our model treats coffee and tea as two unrelated labels, which is not how real models see words. Next you will see how words become lists of numbers where similar words are close together, and how a model lets words look at each other: Embeddings and Attention.