Matrices and Layers

Table of Contents

1. A Table of Numbers
2. A Matrix Is a Machine
3. Matrix Times Vector, Step by Step
4. Shapes and the Shape-Mismatch Error
5. Matrices That Move Points
6. Transpose: Flip the Table
7. Matrix Times Matrix
8. A Linear Layer: y = Wx + b
9. Batches: Many Inputs at Once
10. Stacking Layers, and a Warning
11. Why GPUs Love Matrix Multiplication
12. What Is Simplified Here
13. Recipe: The matrix Module
14. Try It Yourself
15. Quick Review: Common Questions

1. A Table of Numbers

You already know a vector from Vectors: Lists of Numbers: one student is [hours studied, hours slept]. What if you have a whole class? You stack the students, one per row. That table is a matrix.

                  studied  slept
   student 0  [     2        6   ]
   student 1  [     5        7   ]
   student 2  [     8        5   ]

A matrix is a list of rows, and each row is a vector. In JavaScript it is an array of arrays, and m[row][col] picks one number. We say this matrix has 3 rows and 2 columns, or shape 3 by 2. Always say rows first.

Save src/matrix.js from the recipe at the end of this chapter first, then create demo-table.js in your mathai folder.

// demo-table.js
import { shape, matVec } from "./src/matrix.js";

// Each row is one student: [hours studied, hours slept]
const students = [
  [2, 6],
  [5, 7],
  [8, 5],
];
console.log(shape(students));
console.log(students[1][0]);

// Scores = 4 points per study hour + 3 points per sleep hour (an invented rule).
const rule = [[4, 3]];
console.log(matVec(rule, students[0]));
console.log(matVec(rule, students[1]));
// [ 3, 2 ]
// 5
// [ 26 ]
// [ 41 ]

shape returns [rows, columns]. Row 1, column 0 is 5, the hours that student 1 studied. The last two lines used a matrix with one row as a scoring rule: 4*2 + 3*6 = 26. The next section explains how that worked.

All four lines are exact and stable. The scoring rule is invented for the demo; it is not a real law about students.

2. A Matrix Is a Machine

Here is the idea that makes matrices useful. Think of a matrix as a machine. A vector goes in, a different vector comes out.

               +-----------+
 [4, 5, 6] --> |  matrix   | --> [16, 21]
   3 numbers   |  2 rows   |      2 numbers
               |  3 cols   |
               +-----------+

The machine has one row per output number. Each row is a little recipe: "take the input, multiply each position by my numbers, add it up." That recipe is the dot product from Dot Product and Similarity. So:

Matrix times vector = take the dot product of every row with the vector. The answers, in order, are the output vector.

A real-life picture is a school grade report. Your three marks go in as [math, khmer, english]. The machine has two rows: the first adds all three (the total), the second adds all three and divides by 3 (the average). Two numbers come out.

Why does AI care? A neural network layer is exactly this machine. It takes a vector of numbers in (say, an embedding) and produces a new vector out. Learning means finding good numbers for the table.

3. Matrix Times Vector, Step by Step

Let us do one by hand. The machine has two rows and three columns:

   [ 1  0  2 ]     [4]
   [ 0  3  1 ]  x  [5]
                   [6]

 row 0 . v = 1*4 + 0*5 + 2*6 = 4 + 0 + 12 = 16
 row 1 . v = 0*4 + 3*5 + 1*6 = 0 + 15 +  6 = 21

The answer is [16, 21]. Now let the computer do it, and also try the grade-report machine:

// demo-matvec.js
import { matVec } from "./src/matrix.js";

// A machine with 2 outputs and 3 inputs.
const m = [
  [1, 0, 2],
  [0, 3, 1],
];
const v = [4, 5, 6];
console.log(matVec(m, v));
// by hand
console.log(1 * 4 + 0 * 5 + 2 * 6, 0 * 4 + 3 * 5 + 1 * 6);

// A grade-report machine: [math, khmer, english] -> [total, average]
const report = [
  [1, 1, 1],
  [1 / 3, 1 / 3, 1 / 3],
];
const marks = [90, 75, 60];
console.log(matVec(report, marks).map((x) => x.toFixed(1)));
// [ 16, 21 ]
// 16 21
// [ '225.0', '75.0' ]

The first two lines agree. The report machine turns [90, 75, 60] into a total of 225 and an average of 75.

Look at how matVec is written in the recipe. It is one line: m.map((row) => dot(row, v)). The whole idea fits in one line because it really is just "dot every row". All outputs here are exact and stable.

4. Shapes and the Shape-Mismatch Error

A machine with 3 columns expects 3 numbers. Give it 2 and the recipe has nothing to multiply with the third column. So there is a rule:

Matrix times vector works only when the number of columns equals the length of the vector.

  matrix  (2 x 3)    vector (3)       OK, 3 = 3 -> output has 2 numbers
  matrix  (2 x 3)    vector (2)       ERROR, 3 != 2

Our matVec checks this and throws a clear message. Try it:

// demo-mismatch.js
import { matVec } from "./src/matrix.js";

const m = [
  [1, 0, 2],
  [0, 3, 1],
];
try {
  matVec(m, [1, 2]);
} catch (e) {
  console.log(e.message);
}
// shape mismatch: matrix 2,3 times vector [2]

The message prints the matrix shape as 2,3 (2 rows, 3 columns) and the vector length as [2]. This exact line is stable.

You will meet this error constantly when you work with real machine learning libraries. When your code breaks, the usual cause is that you wired two layers whose shapes do not fit. The habit that saves hours: write the shape of every table in a comment. A 3-in, 2-out layer has a weight matrix of shape 2 x 3: output count first, input count second.

5. Matrices That Move Points

Matrices that move points are easier to feel than abstract tables. A point [x, y] on a map is a vector with 2 numbers. A 2 x 2 matrix moves it somewhere else. Some famous machines:

 identity   [1 0]    does nothing
            [0 1]

 scale by 2 [2 0]    doubles x and y
            [0 2]

 stretch x  [2 0]    doubles x only
            [0 1]

 rotate 90  [0 -1]   turns the point a quarter turn
 degrees    [1  0]   counter-clockwise

Try each on the point [3, 2]:

// demo-transform.js
import { matVec, identity } from "./src/matrix.js";

const point = [3, 2];
console.log(matVec(identity(2), point));
console.log(matVec([[2, 0], [0, 2]], point));
console.log(matVec([[2, 0], [0, 1]], point));
console.log(matVec([[0, -1], [1, 0]], point));

// Rotate by 30 degrees using cos and sin.
const c = Math.cos(Math.PI / 6);
const s = Math.sin(Math.PI / 6);
console.log(matVec([[c, -s], [s, c]], [1, 0]).map((x) => x.toFixed(3)));
// [ 3, 2 ]
// [ 6, 4 ]
// [ 6, 2 ]
// [ -2, 3 ]
// [ '0.866', '0.500' ]

The identity matrix returns the point unchanged, which is why it is called that: identity(n) has 1 on the diagonal and 0 elsewhere, the matrix version of "multiply by 1". The doubling matrix moves [3, 2] to [6, 4]. The stretch matrix changes only the first number. The 90-degree matrix turns [3, 2] into [-2, 3]. The last line turns the point [1, 0] by 30 degrees and lands on [0.866, 0.5], a point on the same circle.

All five lines are stable. Nothing here is random.

The AI connection: a layer of a neural network is a more flexible cousin of these. Instead of 2 numbers it moves hundreds of numbers, and instead of a rotation we picked, it learns the table from data.

6. Transpose: Flip the Table

The transpose of a matrix flips it along its diagonal: rows become columns and columns become rows. A 2 x 3 table becomes a 3 x 2 table.

   [1 2 3]          [1 4]
   [4 5 6]    ->    [2 5]
                    [3 6]

// demo-transpose.js
import { transpose, shape } from "./src/matrix.js";

const m = [
  [1, 2, 3],
  [4, 5, 6],
];
console.log(shape(m), shape(transpose(m)));
console.log(transpose(m));
console.log(transpose(transpose(m)));
// [ 2, 3 ] [ 3, 2 ]
// [ [ 1, 4 ], [ 2, 5 ], [ 3, 6 ] ]
// [ [ 1, 2, 3 ], [ 4, 5, 6 ] ]

Flipping twice gives the original, as you would expect. Why do we need it? Two reasons you will see soon: matrix times matrix needs the columns of the second table, and many libraries store layer weights the other way around from how we write them, so you transpose to make the shapes agree.

7. Matrix Times Matrix

What if you feed a whole table of vectors through a machine? Or what if you put two machines one after the other? Both are matrix times matrix.

The rule: the number at row i, column j of the answer is the dot product of row i of the left matrix with column j of the right matrix.

   A (2 x 2)     B (2 x 2)         A x B
   [1 2]         [0 1]             [1*0+2*1  1*1+2*0]   [2 1]
   [3 4]    x    [1 0]      =      [3*0+4*1  3*1+4*0] = [4 3]

And the shape rule is the one you already know, applied to columns:

   (a x b)  times  (b x c)  =  (a x c)
        ^^^^^^^^^^
        the inner two numbers must match; the outer two survive

// demo-matmul.js
import { matMul, matVec, identity, shape } from "./src/matrix.js";

const a = [[1, 2], [3, 4]];
const b = [[0, 1], [1, 0]];
console.log(matMul(a, b));
console.log(matMul(b, a));
console.log(matMul(a, identity(2)));

const x = [[1, 2, 3], [4, 5, 6]];
const y = [[1, 0], [0, 1], [1, 1]];
const p = matMul(x, y);
console.log(shape(x), shape(y), shape(p), p);

try { matMul(y, y); } catch (e) { console.log(e.message); }

// Two machines in a row equal one combined machine.
const double = [[2, 0], [0, 2]];
const flip = [[0, 1], [1, 0]];
const v = [3, 5];
console.log(matVec(double, matVec(flip, v)));
console.log(matVec(matMul(double, flip), v));
// [ [ 2, 1 ], [ 4, 3 ] ]
// [ [ 3, 4 ], [ 1, 2 ] ]
// [ [ 1, 2 ], [ 3, 4 ] ]
// [ 2, 3 ] [ 3, 2 ] [ 2, 2 ] [ [ 4, 5 ], [ 10, 11 ] ]
// shape mismatch: 3,2 times 3,2
// [ 10, 6 ]
// [ 10, 6 ]

Four things to notice:

• matMul(a, b) and matMul(b, a) are different. Order matters, unlike with plain numbers. Here b swaps columns when it is on the right, and swaps rows when it is on the left.
• Multiplying by identity changes nothing.
• A 2 x 3 times a 3 x 2 gives a 2 x 2: the inner 3s matched and vanished.
• Multiplying 3 x 2 by 3 x 2 fails, because the inner numbers are 2 and 3. The error message has the same format as before.

The last two lines are the important idea: sending a vector through flip, then through double, gives [10, 6], and sending it through the single combined matrix double x flip gives the same answer. A chain of machines can be merged into one. We use that in section 10.

Every number printed here is exact and stable.

8. A Linear Layer: y = Wx + b

Now the main event. One layer of a neural network does two things:

1. Multiply the input vector x by a table of weights W.
2. Add a vector of biases b (a fixed shift for each output).

   y  =  W x  +  b

   x : the input vector      (n numbers)
   W : the weights           (m rows, n columns)
   b : the biases            (m numbers)
   y : the output vector     (m numbers)

A picture from daily life: three inputs [hours studied, hours slept, hours on the phone] go in, two outputs [predicted score, predicted mood] come out. Each row of W says how much each input pushes that output up or down. The bias is the starting value when all inputs are zero.

// demo-layer.js
import { matVec } from "./src/matrix.js";
import { add } from "./src/vector.js";

// y = W x + b
function linear(W, b, x) {
  return add(matVec(W, x), b);
}

// 3 inputs: [hours studied, hours slept, hours on phone]
// 2 outputs: [predicted score, predicted mood]  (invented weights)
const W = [
  [5, 2, -3],
  [0, 1, -1],
];
const b = [20, 3];
console.log(linear(W, b, [4, 7, 2]));
console.log(linear(W, b, [4, 7, 8]));
// [ 48, 8 ]
// [ 30, 2 ]

First student: score 5*4 + 2*7 - 3*2 + 20 = 48. The second student studied and slept the same but spent 8 hours on the phone instead of 2, and the negative weight -3 pulls the score down to 30. The weights are invented to show the mechanism; no real data says phones cost 3 points per hour.

Here is the crucial step. A real network does not have its weights chosen by a person. They start as random numbers and are learned: gradient descent nudges every number in W and b until the outputs get better. The layer's shape (how many numbers in and out) is chosen by the designer. The values are learned.

Parameters are the numbers that get learned. This layer has 2*3 + 2 = 8 of them. A layer taking 1000 numbers in and producing 1000 out has 1000*1000 + 1000, which is just over a million. That is how models reach billions of parameters: many big tables, stacked.

9. Batches: Many Inputs at Once

In practice you rarely process one example at a time. You send a batch (many examples) through the layer together. Put one example per row to get an input matrix X of shape batch x inputs.

The trick: if W has shape outputs x inputs, then X x W^T (W transposed) has shape batch x outputs, one output row per example.

   X (4 x 3)  times  W^T (3 x 2)  =  Y (4 x 2)
   4 students        3 -> 2          4 results

// demo-batch.js
import { matVec, matMul, transpose, shape } from "./src/matrix.js";

const W = [
  [5, 2, -3],
  [0, 1, -1],
];
// 4 students, one per row: shape [4, 3]
const X = [
  [4, 7, 2],
  [4, 7, 8],
  [1, 5, 1],
  [9, 3, 0],
];

// Row by row: 4 separate matVec calls.
console.log(X.map((row) => matVec(W, row)));

// All at once: X (4x3) times W^T (3x2) gives 4x2.
const Wt = transpose(W);
const Y = matMul(X, Wt);
console.log(shape(X), shape(Wt), shape(Y));
console.log(Y);
// [ [ 28, 5 ], [ 10, -1 ], [ 12, 4 ], [ 51, 3 ] ]
// [ 4, 3 ] [ 3, 2 ] [ 4, 2 ]
// [ [ 28, 5 ], [ 10, -1 ], [ 12, 4 ], [ 51, 3 ] ]

(We left the bias out so the point is clear; you would add b to each row.) The row-by-row loop and the single matrix product give the same answer. The single product is not just shorter to write: it hands the computer one big, regular job instead of many small ones, which is the reason for the next two sections.

All three lines are exact and stable.

10. Stacking Layers, and a Warning

A network has many layers, each one feeding the next. With our tools: take the output of layer 1 and use it as the input of layer 2. The shapes must chain: if layer 1 outputs 4 numbers, layer 2 must take 4.

Now a surprise. Remember from section 7 that two machines in a row can be merged into one matrix. Watch what happens with two linear layers:

// demo-collapse.js
import { mulberry32 } from "./src/rng.js";
import { matVec, matMul, randomMatrix, shape } from "./src/matrix.js";

const rng = mulberry32(7);
const W1 = randomMatrix(4, 3, rng); // 3 numbers in, 4 out
const W2 = randomMatrix(2, 4, rng); // 4 numbers in, 2 out
const x = [1, 2, 3];

const twoSteps = matVec(W2, matVec(W1, x));
const oneStep = matVec(matMul(W2, W1), x);
console.log(twoSteps.map((n) => n.toFixed(6)));
console.log(oneStep.map((n) => n.toFixed(6)));
console.log(shape(W1), shape(W2), shape(matMul(W2, W1)));
// [ '1.223858', '-2.199598' ]
// [ '1.223858', '-2.199598' ]
// [ 4, 3 ] [ 2, 4 ] [ 2, 3 ]

mulberry32(7) makes the "random" numbers the same on every run, so your output matches this page. Two layers of random weights, and the result is identical to one layer whose matrix is W2 x W1. A stack of purely linear layers is secretly a single linear layer, no matter how tall the stack.

That is a problem, because a single linear layer is a weak machine: it can only do rotations, stretches and sums. The fix, which is the whole point of Build a Neural Network, is to put a small bend between layers (an activation function). The bend stops the layers from merging and lets the network learn curved, complicated patterns. For now, remember: matrices do the mixing; activations add the bends.

11. Why GPUs Love Matrix Multiplication

You may have heard that AI needs graphics cards (GPUs). Here is the reason, in plain words, without numbers we cannot back up.

Count the work in a matrix product. Multiplying two n x n tables takes n * n * n multiplications, because each of the n * n answers needs a dot product of length n. Run this to see the count:

// demo-count.js
import { mulberry32 } from "./src/rng.js";
import { matMul, randomMatrix } from "./src/matrix.js";

const rng = mulberry32(1);
const n = 100;
const A = randomMatrix(n, n, rng);
const B = randomMatrix(n, n, rng);

const start = performance.now();
const C = matMul(A, B);
const ms = Math.round(performance.now() - start);

console.log("result size:", C.length, "x", C[0].length);
console.log("multiplications:", n * n * n);
console.log("took", ms, "ms (your number will differ)");
// result size: 100 x 100
// multiplications: 1000000
// took 4 ms (your number will differ)

The first two lines are stable. The time depends on your computer, so it will differ every run; ours is just an example.

One million multiplications for two small tables, and a real model does this for tables far larger and many times per answer. But look at the structure of the work: every one of those dot products is independent of the others. None waits for another. That is the perfect shape for hardware that can do a very large number of simple calculations at the same time, which is what a GPU is built for. A CPU is good at doing one complicated thing after another. A GPU is good at doing many simple things side by side. Matrix multiplication is the second kind of job.

This is why the same math you run here in plain JavaScript, one number at a time, can run enormously faster on suitable hardware. The ideas are the same. Only the speed changes.

12. What Is Simplified Here

• Our matMul is the plain textbook method. Real libraries use tuned code, special hardware, and compact typed arrays.
• Real layers have thousands of inputs and outputs, not 2 or 3. The idea and the shape rules are identical.
• Many libraries write the batch version as Y = XW + b with weights stored as inputs x outputs. Same math, flipped bookkeeping, which is why transpose appears so often.
• Weights are learned, not chosen. Here we invented them to see the mechanism. Chapters 5 and 7 show the learning.

13. Recipe: The matrix Module

Save this as src/matrix.js in your mathai folder. It imports dot from the vector module you built in Vectors: Lists of Numbers.

// A matrix is an array of rows: m[row][col].
import { dot } from "./vector.js";

export const shape = (m) => [m.length, m[0].length];
export const zeros = (r, c) => Array.from({ length: r }, () => Array(c).fill(0));
export const identity = (n) =>
  Array.from({ length: n }, (_, i) => Array.from({ length: n }, (_, j) => (i === j ? 1 : 0)));

export const transpose = (m) => m[0].map((_, j) => m.map((row) => row[j]));

// matrix (r x c) times vector (c) -> vector (r)
export function matVec(m, v) {
  if (m[0].length !== v.length) {
    throw new Error(`shape mismatch: matrix ${shape(m)} times vector [${v.length}]`);
  }
  return m.map((row) => dot(row, v));
}

// matrix (r x c) times matrix (c x k) -> matrix (r x k)
export function matMul(a, b) {
  if (a[0].length !== b.length) {
    throw new Error(`shape mismatch: ${shape(a)} times ${shape(b)}`);
  }
  const bt = transpose(b);
  return a.map((row) => bt.map((col) => dot(row, col)));
}

export function randomMatrix(rows, cols, rng, scaleBy = 1) {
  return Array.from({ length: rows }, () =>
    Array.from({ length: cols }, () => (rng() * 2 - 1) * scaleBy)
  );
}

Notes on the code:

• matMul transposes b first so that its columns become rows, then every output number is a dot of a row of a with a row of the flipped b. That is the "row times column" rule written once.
• randomMatrix takes a random function rng (use mulberry32(seed) from src/rng.js) and fills the table with numbers from -scaleBy to scaleBy. A seed makes your results repeatable.
• zeros makes a blank table of a given shape, used in later chapters.

14. Try It Yourself

1. Hand check. Compute [[2, 1], [0, 3]] times [4, 5] on paper, then with matVec. You should see [13, 15].
2. A grade machine. Build a matrix with two rows that turns [math, khmer, english] into [weighted score, plain average]. For the first row use weights 0.5, 0.3, 0.2. Try it on [90, 75, 60] and say in one sentence what the weights mean.
3. Break it on purpose. Call matVec with a 3 x 4 matrix and a vector of length 3, then matMul with shapes 2 x 3 and 2 x 3. Read both error messages and write the correct shapes that would fix each one.
4. Rotate a square. Take the four corners [0,0] [1,0] [1,1] [0,1] and rotate each by 45 degrees with the rotation matrix from section 5. Check that the distance from [0,0] to the corner [1,0] stays 1 using norm from the vector module.
5. Order matters. Find two 2 x 2 matrices A and B where matMul(a, b) and matMul(b, a) differ, and one pair where they are equal. (Hint: the identity, or two copies of the same scaling matrix.)
6. Count parameters. A network has layers of shape 784 to 128, then 128 to 10 (weights plus biases). How many parameters is that? You should get 101,770.

15. Quick Review: Common Questions

Questions beginners often ask about this chapter. Try answering each one yourself first, then click the question to open the answer.

Where to go next. You can now say what a matrix is, how a layer uses it, and why shapes must match. A layer has numbers that must be learned, and to learn them we need to know which way to nudge each one. That needs slopes: Slopes and Derivatives.