Statistics and the Big Picture

Table of Contents

1. Mean, Variance, and Standard Deviation
2. Standardizing: Why Scale Matters
3. Train and Test: Don't Grade Yourself on the Homework
4. Overfitting: Memorizing Is Not Learning
5. Randomness and Seeds
6. Reading a Loss Curve
7. The One-Page Map: Which Math Appears Where
8. Recipe: The split and index Modules
9. Try It Yourself
10. Quick Review: Common Questions
11. Talk Wrap-Up: Audience Q&A

1. Mean, Variance, and Standard Deviation

Two classes both have an average exam score of 50. In class A everybody scored exactly 50. In class B the scores are 10, 30, 50, 70, 90. The averages are equal, but the classes are very different. A single number, the mean, hides how spread out the data is.

Here is how to measure spread, in three school-level steps:

1. Find the mean: add everything, divide by how many.
2. For each value, take its distance from the mean and square it (squaring makes negatives positive, just like the loss in chapter 5).
3. The variance is the mean of those squared distances. The standard deviation is its square root, which puts the spread back in the original units.

class A: 50 50 50 50 50      spread = 0       (no spread at all)
class B: 10 30 50 70 90      spread = about 28 points either side of 50
         |----|----|----|----|

The stats.js module from Probability, Logs and Softmax already has mean, variance, std, and standardize. Try them (save this as demo-stats.js in the root of your mathai folder):

// demo-stats.js
import { mean, variance, std, standardize } from "./src/stats.js";

const a = [50, 50, 50, 50, 50];
const b = [10, 30, 50, 70, 90];
for (const [name, xs] of [["a", a], ["b", b]]) {
  console.log(name, "mean", mean(xs), "variance", variance(xs), "std", std(xs).toFixed(3));
}
const z = standardize(b);
console.log("standardized b:", z.map((v) => v.toFixed(3)));
console.log("new mean:", mean(z).toFixed(3), "new std:", std(z).toFixed(3));
// output
// a mean 50 variance 0 std 0.000
// b mean 50 variance 800 std 28.284
// standardized b: [ '-1.414', '-0.707', '0.000', '0.707', '1.414' ]
// new mean: 0.000 new std: 1.000

Check variance of b by hand: the distances from 50 are -40, -20, 0, 20, 40. Squares: 1600, 400, 0, 400, 1600. They add to 4000, and 4000 divided by 5 is 800. The square root of 800 is about 28.28. All output here is stable.

One honest note: statisticians sometimes divide by n - 1 instead of n when the data is only a sample of something bigger. Our variance divides by n, which is what we want for describing the numbers we have.

2. Standardizing: Why Scale Matters

Standardizing a list means rescaling it so its mean is 0 and its standard deviation is 1: subtract the mean, then divide by the standard deviation. In the output above, class B's scores became -1.414, -0.707, 0, 0.707, 1.414. Each number now says "how many spreads away from the average", whatever the original unit was. A score of 90 and a height of 180 centimetres can then be compared fairly.

Why does an AI person care? Imagine one input is "hours studied" (1 to 10) and another is "minutes studied" (60 to 600). Gradient descent updates a weight by learning rate * slope, and the slope for a feature grows with the size of that feature. A huge feature makes the landscape very steep in one direction, so a step size that suits small numbers throws you far across the valley.

Test it. We take the study data from chapter 1, turn hours into minutes, and fit a line with fitLine from chapter 5. First with the raw minutes, then with standardized minutes:

// demo-scale.js
import { study } from "./src/data.js";
import { fitLine } from "./src/descent.js";
import { standardize } from "./src/stats.js";

const minutes = study.map(([h]) => h * 60); // 60 ... 600
const scores = study.map(([, s]) => s);

const raw = minutes.map((m, i) => [m, scores[i]]);
const scaled = standardize(minutes).map((m, i) => [m, scores[i]]);

function report(name, points, lr) {
  const { history } = fitLine(points, { lr, steps: 200 });
  const last = history[history.length - 1];
  console.log(
    `${name} lr=${lr}: loss ${last.loss.toExponential(2)}  w=${last.w.toFixed(3)}  b=${last.b.toFixed(2)}`
  );
}
report("raw   ", raw, 0.01);
report("raw   ", raw, 0.00001);
report("raw   ", raw, 0.0000001);
report("scaled", scaled, 0.1);
// output
// raw    lr=0.01: loss NaN  w=NaN  b=NaN
// raw    lr=0.00001: loss 1.19e+103  w=-9.253100910238202e+48  b=-2.2031226705115405e+46
// raw    lr=1e-7: loss 4.10e+2  w=0.187  b=0.00
// scaled lr=0.1: loss 1.41e+0  w=14.379  b=71.20

Read the four lines:

• With raw minutes and the learning rate that worked in chapter 5 (0.01), the numbers explode until JavaScript gives up and prints NaN ("not a number").
• Even 0.00001 still diverges, to a loss with 103 digits.
• A learning rate small enough to be safe, 1e-7, does not blow up, but after 200 steps the loss is still 410 and the intercept b has barely moved from zero.
• With standardized minutes, a normal learning rate 0.1 reaches a loss of 1.4 and a sensible line (a slope of about 14 points per spread, and a starting level near 71) in the same 200 steps.

The data, the model, and the algorithm are identical. Only the scale of the input differed. These numbers are stable. In real pipelines, standardizing (or a similar rescaling) of the inputs is a routine first step, and large networks also normalize numbers inside the network for the same reason: keep the numbers in a friendly range so the slopes behave.

3. Train and Test: Don't Grade Yourself on the Homework

A student who memorizes last year's exam answers scores 100 on last year's exam, and may fail a new one. If you test a model on the same data it learned from, you only learn how well it memorized. What you want to know is how it does on data it has never seen.

So before training, set some data aside:

all data:   [ ......................................... ]
split:      [ ...... train (about 70%) ...... ][ test 30% ]
train on the left part, grade yourself ONLY on the right part

split.js does this with two small functions: shuffle mixes the items randomly (the Fisher-Yates method, using the seeded random numbers from chapter 1) and trainTestSplit(data, fraction, rng) returns { train, test }. Shuffling first matters: if the data is sorted, say all the "coder" rows first, taking the last 30% would test only on gamers.

4. Overfitting: Memorizing Is Not Learning

Overfitting means a model fits the training data very closely, including its noise and accidents, and then does worse on new data. Here is a small honest demo. We use k-NN from chapter 2, which labels a new point by a vote among the k closest training points. We make two overlapping groups of students ([hours coding, hours gaming]): coders tend to code more, gamers tend to game more, but plenty of each group looks like the other. No method can be perfect here.

With k = 1 the model copies the single nearest neighbour, which means it memorizes every quirk. With a larger k it averages over more neighbours and ignores the quirks.

// demo-split.js
import { mulberry32 } from "./src/rng.js";
import { trainTestSplit } from "./src/split.js";
import { knnClassify } from "./src/similarity.js";

// Two overlapping groups of students: [hours coding, hours gaming].
function makeData(n, rng) {
  const noise = () => (rng() + rng() + rng() - 1.5) * 4; // roughly bell-shaped
  return Array.from({ length: n }, (_, i) => {
    const coder = i % 2 === 0;
    const centre = coder ? [6, 4] : [4, 6];
    return { x: [centre[0] + noise(), centre[1] + noise()], label: coder ? "coder" : "gamer" };
  });
}
const accuracy = (rows, train, k) =>
  rows.filter((r) => knnClassify(r.x, train, k) === r.label).length / rows.length;

const rng = mulberry32(42);
const { train, test } = trainTestSplit(makeData(300, rng), 0.3, rng);
console.log("train size", train.length, "test size", test.length);
for (const k of [1, 5, 15, 45]) {
  console.log(`k=${k}: train ${accuracy(train, train, k).toFixed(2)}  test ${accuracy(test, train, k).toFixed(2)}`);
}
// output
// train size 210 test size 90
// k=1: train 1.00  test 0.54
// k=5: train 0.80  test 0.64
// k=15: train 0.75  test 0.68
// k=45: train 0.69  test 0.72

Look at the pattern, not just the numbers:

              train accuracy     test accuracy
k = 1             1.00     <-- perfect?       0.54   barely better than a coin
k = 45            0.69                        0.72   honest and steady

With k = 1, the training score is a perfect 1.00. Of course: each training point's nearest neighbour is itself. But on new students, it scores 0.54, hardly better than guessing. As k grows, the training score falls and the test score rises, until they meet. The gap between the two scores is the signal that you are overfitting.

Honest caveats. This is one split of 90 test points, so each score wobbles by several points: do not read much into the difference between 0.64 and 0.68. With other seeds the test numbers change (try it in the exercises), though the perfect-on-training, weaker-on-test pattern for k = 1 stays. Real projects repeat the experiment with different splits, or keep a third "validation" set for choosing settings like k. Also, the best k depends on this made-up data. It is not a general rule.

The idea transfers to large models. Training loss almost always keeps going down as a model gets bigger or trains longer. What matters is the loss on held-out data. A model can drive the training loss lower by memorizing, so the held-out loss is the honest number. Big language models are trained on huge amounts of text partly so that memorizing is harder than learning general patterns, but they can still memorize parts of their training text. This is why evaluation data must be kept separate.

5. Randomness and Seeds

Computers do not produce true randomness by themselves. A pseudo-random number generator is a formula: it takes a number called the seed, and produces a stream of numbers that look random but are completely decided by the seed. Same seed, same stream. That is what mulberry32(seed) from chapter 1 does.

Why does AI use randomness at all? To choose starting weights, to shuffle data, to split it, and to sample the next word (see Probabilities, Temperature and Sampling). Because the starting weights are random, two training runs can end in different places. Seeds let you repeat a run exactly, which is essential for debugging and for sharing results.

// demo-seeds.js
import { mulberry32 } from "./src/rng.js";
import { xor } from "./src/data.js";
import { makeNet, trainEpoch } from "./src/nn.js";

const a = mulberry32(5);
const b = mulberry32(5);
const c = mulberry32(6);
console.log("seed 5:", a().toFixed(4), a().toFixed(4));
console.log("seed 5 again:", b().toFixed(4), b().toFixed(4));
console.log("seed 6:", c().toFixed(4), c().toFixed(4));

// Same data, same learning rate, different starting weights.
for (const seed of [1, 2, 3, 4, 5, 6]) {
  const net = makeNet(2, seed);
  let loss = 0;
  for (let epoch = 0; epoch < 4000; epoch++) loss = trainEpoch(net, xor, 0.5);
  console.log(`2 hidden neurons, seed ${seed}: final loss ${loss.toFixed(4)}`);
}
// output
// seed 5: 0.6898 0.7727
// seed 5 again: 0.6898 0.7727
// seed 6: 0.5257 0.0023
// 2 hidden neurons, seed 1: final loss 0.0004
// 2 hidden neurons, seed 2: final loss 0.0004
// 2 hidden neurons, seed 3: final loss 0.1337
// 2 hidden neurons, seed 4: final loss 0.0004
// 2 hidden neurons, seed 5: final loss 0.0005
// 2 hidden neurons, seed 6: final loss 0.1337

The first three lines show the idea: seed 5 gives the same numbers twice, seed 6 gives different ones. All output is stable on any machine.

The last six lines are the surprise. Nothing changed except the starting weights, yet seeds 3 and 6 end with a loss of 0.13, stuck partway, while the other four reach 0.0004. The network got a bad start and the training never escaped. A tiny network is fragile like this. Real large networks are much less fragile, and practitioners often run several seeds to check that a result is not luck.

Do not use Math.random() for this series' experiments. It cannot be seeded, so you could never reproduce a number you saw. And never use these generators for secrets: they are predictable by design.

6. Reading a Loss Curve

The loss curve is the heartbeat of training: loss on the vertical axis, epochs (or steps) along the horizontal axis. You will see these everywhere in AI. Reading it is a skill with only a few shapes to learn. Train the same XOR network with three learning rates and print the loss at a few epochs:

// demo-curve.js
import { xor } from "./src/data.js";
import { makeNet, trainEpoch } from "./src/nn.js";

const marks = [0, 100, 200, 400, 800, 1600, 3200];
for (const lr of [0.01, 0.5, 20]) {
  const net = makeNet(4, 1);
  const row = [];
  for (let epoch = 0; epoch <= 3200; epoch++) {
    const loss = trainEpoch(net, xor, lr);
    if (marks.includes(epoch)) row.push(loss.toFixed(3));
  }
  console.log(`lr ${String(lr).padEnd(4)}`, row.join("  "));
}
// output
// lr 0.01 0.266  0.253  0.252  0.251  0.251  0.251  0.249
// lr 0.5  0.297  0.279  0.273  0.245  0.005  0.001  0.000
// lr 20   0.587  0.500  0.490  0.225  0.314  0.234  0.222

The columns are epochs 0, 100, 200, 400, 800, 1600, 3200. Output is stable. Three curves, three stories:

loss
0.3 |-----------------------        lr 0.01: too slow. Flat, barely moves.
    |
    |-----.                         lr 0.5: plateau, then a sudden drop
    |      '-.__                             to nearly zero. Healthy.
    |            '-----.
0.0 |                   '-----
    |  /\/\/\/\_____                lr 20: jumps around, never settles.
    +------------------------> epoch

• Flat from the start (lr 0.01): the steps are too small, or the network is stuck. The loss sits near 0.25, the "always say 0.5" level.
• Plateau, then a cliff (lr 0.5): the loss hovers near 0.25 until around epoch 400, then drops. Many training runs have such a flat stretch before the network finds a useful pattern. Do not give up too early.
• Noisy and not settling (lr 20): steps too big, so the weights jump past the valley and back. The numbers go up and down. With an even bigger rate (try 50) it saturates and sits at 0.5 forever.
• Going to NaN or infinity: steps so big the numbers explode, like the raw-minutes demo in section 2.
• Training loss falls, test loss rises: overfitting, from section 4. Always plot both.

A sudden spike in the middle of a long, healthy run usually means a bad batch of data or a step that was too large. Those are the practical things people watch for when a big model trains for days.

7. The One-Page Map: Which Math Appears Where

Here is the summary to keep. For each part of how a language model works, the table names the math, the chapter that builds it, and what the idea means in plain words. Real models are far more elaborate, so read this as a map of the ideas, not a blueprint.

What the table leaves out: the real transformer architecture (see Embeddings and Attention), how tokens are made (Tokens and Tokenizers), how training is organized (How a Model Is Trained), and the engineering that makes billions of parameters trainable. The math here is the vocabulary. Those pages are the sentences.

The whole story in one breath: text becomes vectors; layers of matrices and bends turn them into scores; softmax turns scores into probabilities; the loss measures how wrong the right answer's probability was; the gradient says how to nudge every weight; and gradient descent repeats that over an enormous amount of text.

8. Recipe: The split and index Modules

Save the first file as src/split.js. It has shuffle and trainTestSplit, which you met in section 3.

// Shuffle a copy of the array (Fisher-Yates) using a seeded rng.
export function shuffle(items, rng) {
  const a = items.slice();
  for (let i = a.length - 1; i > 0; i--) {
    const j = Math.floor(rng() * (i + 1));
    [a[i], a[j]] = [a[j], a[i]];
  }
  return a;
}

// Split data into a training part and a held-out test part.
export function trainTestSplit(data, testFraction, rng) {
  const shuffled = shuffle(data, rng);
  const testSize = Math.max(1, Math.round(shuffled.length * testFraction));
  return { test: shuffled.slice(0, testSize), train: shuffled.slice(testSize) };
}

Save the second as src/index.js. It re-exports every module you built in this series, so one import gives you the whole toolkit:

// One import for the whole toolkit:  import * as ai from "./src/index.js"
export * as vector from "./vector.js";
export * as similarity from "./similarity.js";
export * as matrix from "./matrix.js";
export * as calculus from "./calculus.js";
export * as descent from "./descent.js";
export * as stats from "./stats.js";
export * as nn from "./nn.js";
export * as split from "./split.js";
export { mulberry32 } from "./rng.js";
export * as data from "./data.js";

Now your toolkit is complete. Here is a quick check that everything is wired up (run it from the mathai folder):

// demo-index.js
import * as ai from "./src/index.js";

console.log(Object.keys(ai));
console.log(ai.vector.dot([1, 2], [3, 4]), ai.nn.sigmoid(0));
// output
// [
//   'calculus',   'data',
//   'descent',    'matrix',
//   'mulberry32', 'nn',
//   'similarity', 'split',
//   'stats',      'vector'
// ]
// 11 0.5

The list is alphabetical and stable. The two numbers are 1*3 + 2*4 = 11 and sigmoid(0) = 0.5. Each module is a namespace, so you write ai.matrix.matMul(...). Because stats.js and vector.js both have a function called mean, namespaces keep them from colliding.

9. Try It Yourself

1. In demo-stats.js, try the list [1, 2, 3, 4, 100]. Compare the mean and standard deviation with and without the 100. You should see one outlier drag both numbers a long way.
2. In demo-scale.js, run fitLine on the raw minutes with lr set to 1e-6, 5e-6, and 1e-5. You should see the first two stay stuck at a loss of about 410, and 1e-5 explode. The safe window is tiny, far below what the scaled data accepts.
3. In demo-split.js, change the seed from 42 to 7 and 99. You should see k = 1 stay perfect on training and weaker on test, but the test scores for the other k values shuffle around. One split is a noisy judge.
4. In demo-split.js, set the test fraction to 0.05. You should see the test scores jump around more, because a tiny test set is a noisy judge.
5. In demo-seeds.js, try 12 seeds with 2 hidden neurons, then with 4. Count how many get stuck. You should see 4 of 12 stuck with 2 hidden neurons and none with 4.
6. In demo-curve.js, try the learning rates 2, 10, and 50. You should see 2 and 10 settle quickly, while 50 is stuck at 0.5 for good. Where does the curve change from "healthy" to "broken"?

10. Quick Review: Common Questions

Questions beginners often ask about this chapter. Try answering each one yourself first, then click the question to open the answer.

11. Talk Wrap-Up: Audience Q&A

You have built the toolkit and walked through every idea. Here are questions a mixed room tends to ask, with answers short enough to say out loud. Where the truth is "it depends" or "nobody knows", the answer says so.

1. "Is AI just math?" The core of it is, yes: lists of numbers, multiplications, slopes, and probabilities, as you just built. What makes it feel like more is the scale of the data and compute, and what the numbers end up capturing.

2. "Do I need to be good at math to use AI?" No, to use it you do not. To explain it, debug it, or build it, the ideas in this series are what help. School algebra plus patience gets you far.

3. "Is a vector the same as an embedding?" An embedding is a vector, one that was learned so that useful things (like similar meanings) end up as nearby numbers. Not every vector is an embedding.

4. "How many dimensions do real embeddings have?" Hundreds to thousands in many systems, though it varies by model. Our examples used 2 or 3 so you could see them, but the arithmetic is the same in any number of dimensions.

5. "What is a parameter?" One number the model can change while learning, mostly weights and biases. "A model with billions of parameters" means billions of such numbers, all set by training.

6. "What is the loss?" A single score for how wrong the model currently is on its examples. Training tries to make it small. For language models, it is built from the probability given to the correct next token.

7. "What is a gradient?" The slope of the loss with respect to every weight at once. It tells you which direction makes the loss go up fastest, so training steps the opposite way.

8. "What is a learning rate?" The size of each downhill step. Too small and training crawls, too big and it overshoots or explodes. It is one of the first things people tune.

9. "Why do models need so much data?" The model has a huge number of numbers to set, and each example only nudges them a little. Lots of varied data also makes memorizing less useful than learning general patterns. How much is enough depends on the task and nobody has a single rule.

10. "Why do they need GPUs?" Because the work is mostly giant matrix multiplications, and those can be split into many small calculations done at the same time, which is what graphics hardware is built for. A laptop can still train our toy network in a second.

11. "Why does the answer change when I ask twice?" The model produces probabilities for the next token and then samples from them, so there is randomness. A lower temperature makes the choice more predictable. See the temperature chapter linked in section 5.

12. "Why do models sometimes make things up?" A language model is trained to produce text that is likely given the text so far, not to look facts up. When it lacks the knowledge, a likely sounding answer can still come out. Retrieval, which finds real documents first, reduces this but does not remove it.

13. "Does the model understand what it says?" It captures real patterns of language and the world well enough to be very useful. Whether that is "understanding" is an open question that experts disagree on. It is fair to say it is doing math that works, and to leave the word open.

14. "Can I train my own model?" A small one, yes, as you just did, with no special hardware. Training a large language model from scratch needs far more data and compute than an individual has. Most people start from an existing model and adjust how they use it.

15. "Is the model copying its training data?" It learns patterns from the data rather than storing it as a database, but it can reproduce some passages that it saw often. That is a known issue and an active area of work.

16. "Will AI replace programmers (or teachers)?" Nobody can say for sure. What we can see is that AI is a tool that changes how much of the routine work needs a person. People who understand how it works, including its limits, are in a better position to use it well and check it.

17. "What should I learn next?" Linear algebra (vectors and matrices), calculus (derivatives), and probability and statistics are the three standard foundations, and you have now met the beginner version of each. Then build things: a chatbot, a retrieval app, an agent.

18. "How do I know the AI's answer is right?" You do not, by default. Check against sources, test the code it writes, and treat its output as a draft. Understanding the math gives you a reason to be careful: it is an estimate of likely text, not a guarantee.

19. "Is bigger always better?" Bigger models tend to do more, but not always better for a given job, and they cost more to run. The held-out test is what tells you, not the size.

20. "Why did you build it from scratch if libraries exist?" So that the library stops being magic. Once you have written a forward pass, a loss, and a gradient step yourself, you can read any framework's documentation and know what it is doing for you.

Where to go next. You now have the math, and the code, to explain how an AI model learns. The next step is to see these ideas used inside real applications. RAG: Chat with Your Own Documents uses vectors, cosine similarity, and nearest-neighbour search to let a model answer from your files. Build a Chatbot with an LLM API shows how to connect a model to an app, and AI Agents with Tools and MCP lets a model take actions. You can also revisit Embeddings and Attention with fresh eyes, because every operation in it is one you have now built.