Dot Product and Similarity

Table of Contents

1. How Much Do Two People Agree?
2. The Dot Product
3. Parallel, Perpendicular, Opposite
4. The Problem: Dot Product Rewards Size
5. Cosine Similarity
6. The Angle Formula
7. Gotcha: Parallel Vectors That Print 0.9999999999999998
8. Cosine vs Distance
9. Nearest Neighbour Search
10. A Tiny Search Engine by Meaning
11. k-Nearest-Neighbours: Classify by Voting
12. The AI Connection
13. Recipe: The similarity Module
14. Try It Yourself
15. Quick Review: Common Questions

1. How Much Do Two People Agree?

Three friends rate how much they like three kinds of movies, from 0 to 5: action, comedy, romance.

            action  comedy  romance
  Sokha        5       1       0
  Vireak       4       2       1
  Dara         0       1       5

Sokha and Vireak look alike: both love action and like comedy a bit. Sokha and Dara look different: where one is high, the other is low. You could sense that by eye. Now we want a single number that says "how much do these two agree?", so a program can rank thousands of people, not just three.

A natural recipe: for each kind of movie, multiply the two ratings, then add up the results. If both rate a genre high, the product is big. If one is high and the other is zero, the product is zero, so a mismatch adds nothing. Agreement adds points, disagreement does not.

That recipe has a name: the dot product. This chapter turns it into the tool behind search by meaning, recommendation, and a large part of what happens inside a neural network.

Say it out loud: "The dot product multiplies two lists position by position and adds the results. It is big when the two lists agree."

2. The Dot Product

In symbols, for two vectors a and b:

dot(a, b) = a[0]*b[0] + a[1]*b[1] + a[2]*b[2] + ...

In code it is one line, and it already lives in your vector.js from the previous chapter. Let us try it on the movie lovers:

// demo-dot.js
import { dot } from "./src/vector.js";

// ratings of [action, comedy, romance] on a 0..5 scale (invented)
const sokha = [5, 1, 0];
const vireak = [4, 2, 1];
const dara = [0, 1, 5];

console.log(dot(sokha, vireak));
console.log(dot(sokha, dara));
console.log(dot([1, 0], [0, 1]));
console.log(dot([1, 0], [-1, 0]));
console.log(dot([2, 3], [2, 3]));

// output:
// 22
// 1
// 0
// -1
// 13

By hand: dot(sokha, vireak) = 5*4 + 1*2 + 0*1 = 20 + 2 + 0 = 22. And dot(sokha, dara) = 5*0 + 1*1 + 0*5 = 1. Sokha and Vireak score 22, Sokha and Dara score 1. The program agrees with your eyes.

The last three lines are tiny 2D cases that show the three behaviors you need to know. [1, 0] and [0, 1] (one points right, one points up) give 0. [1, 0] and [-1, 0] (opposite directions) give a negative number. A vector with itself, [2, 3] and [2, 3], gives 4 + 9 = 13, which is its length squared. That is why norm(a) = sqrt(dot(a, a)).

3. Parallel, Perpendicular, Opposite

The dot product has a beautiful geometric meaning. Think of two arrows starting from the same point. The angle between them decides the sign of the dot product:

   same direction        at right angles         opposite
   (angle 0 deg)         (angle 90 deg)          (angle 180 deg)

   a ----> b ---->            ^ b                <---- b   a ---->
                              |
   dot is positive            +----> a           dot is negative
   (they agree)               dot is zero
                              (they ignore each other)

In short: parallel arrows give the biggest positive dot product, right angles give exactly 0, and opposite arrows give a negative number.

"Perpendicular" means at a right angle, like the corner of a page. In AI language you will also hear orthogonal: unrelated on this measure.

4. The Problem: Dot Product Rewards Size

There is a catch. The dot product depends on how long the vectors are, not only on where they point. Take the vector [1, 0] and compare it with three vectors that all point exactly the same way, just with different lengths:

// demo-size.js
import { dot, cosine } from "./src/vector.js";

const a = [1, 0];
console.log(dot(a, [1, 0]), dot(a, [100, 0]), dot(a, [0.01, 0]));
console.log(cosine(a, [1, 0]), cosine(a, [100, 0]), cosine(a, [0.01, 0]));

// output:
// 1 100 0.01
// 1 1 1

The dot products are 1, 100 and 0.01, even though all three arrows point the same way. The dot product mixes two things together: "how well do they point the same way" and "how big are they". The second line uses a function called cosine, which gives 1 for all three, because it only looks at direction. Let us see how it works.

5. Cosine Similarity

To measure only the direction, divide the dot product by the lengths of both vectors. This is the same as normalizing both vectors first (the previous chapter's normalize) and then taking the dot product:

cosine(a, b) = dot(a, b) / (norm(a) * norm(b))

The result is always between -1 and 1:

It is called "cosine" because it is the cosine of the angle between the two arrows, a function you may have met in school trigonometry. You do not need trigonometry to use it; just remember the table.

// demo-angles.js
import { cosine } from "./src/vector.js";

console.log(cosine([1, 0], [1, 0]));
console.log(cosine([1, 0], [5, 0]));
console.log(cosine([1, 0], [1, 1]).toFixed(3));
console.log(cosine([1, 0], [0, 1]));
console.log(cosine([1, 0], [-1, 0]));
console.log(cosine([0, 0], [1, 1]));

const deg = (c) => ((Math.acos(c) * 180) / Math.PI).toFixed(1);
console.log(deg(cosine([1, 0], [1, 1])), deg(cosine([1, 0], [0, 1])), deg(-1));

// output:
// 1
// 1
// 0.707
// 0
// -1
// 0
// 45.0 90.0 180.0

Line by line: the same direction gives 1 (and [5, 0] is just a longer arrow in that direction, still 1). The diagonal [1, 1] is halfway between right and up, an angle of 45 degrees, and its cosine is 0.707. Perpendicular gives 0. Opposite gives -1.

The sixth line is a design choice: the zero vector has no direction, so our cosine returns 0 instead of NaN, like normalize.

The last line turns cosines back into degrees with Math.acos: 45, 90, 180, matching the picture.

6. The Angle Formula

Everything fits together in one formula. For any two vectors:

dot(a, b) = norm(a) * norm(b) * cosine(a, b)

The dot product is "length times length times how aligned". Cosine removes the first two factors. Check it with [3, 4] and [4, 3], two arrows of length 5 that point in similar directions:

// demo-angle-formula.js
import { dot, norm, cosine } from "./src/vector.js";

const a = [3, 4];
const b = [4, 3];

console.log(dot(a, b));
console.log(norm(a) * norm(b));
console.log(dot(a, b) / (norm(a) * norm(b)));
console.log(cosine(a, b));
const degrees = (Math.acos(cosine(a, b)) * 180) / Math.PI;
console.log(degrees.toFixed(1));

// output:
// 24
// 25
// 0.96
// 0.96
// 16.3

dot = 24 and norm * norm = 25, so the cosine is 24 / 25 = 0.96. The angle between them is about 16.3 degrees, a small angle, so a high cosine. The movie lovers work the same way:

// demo-movies.js
import { dot, cosine } from "./src/vector.js";

// ratings for [action, comedy, romance], 0 to 5. Invented numbers.
const sokha = [5, 1, 0];
const vireak = [4, 2, 1];
const dara = [0, 1, 5];

console.log(dot(sokha, vireak), dot(sokha, dara));
console.log(cosine(sokha, vireak).toFixed(3), cosine(sokha, dara).toFixed(3));

// output:
// 22 1
// 0.942 0.038

Sokha and Vireak have a cosine of 0.942: very similar taste. Sokha and Dara have 0.038, almost perpendicular: nothing in common. Now you can say it with one number.

7. Gotcha: Parallel Vectors That Print 0.9999999999999998

Computers store decimals in a way that cannot represent every number exactly. Tiny rounding errors appear in the last digit. You will see them for the first time right here. Two vectors that point exactly the same way should have cosine exactly 1. Let us test one pair:

// demo-float.js
import { cosine } from "./src/vector.js";

const a = [1, 1, 8];
const b = [10, 10, 80]; // exactly 10 times a: same direction

console.log(cosine(a, b));
console.log(cosine(a, b) === 1);
console.log(Math.abs(cosine(a, b) - 1) < 1e-9);

const c = cosine([1, 1, 2], [10, 10, 20]);
console.log(c);
console.log(Math.acos(c));
console.log(Math.acos(Math.min(1, c)));

// output:
// 0.9999999999999998
// false
// true
// 1.0000000000000002
// NaN
// 0

The first line prints 0.9999999999999998, not 1. The second line shows that === 1 is false. The error is about 0.0000000000000002, far below anything that matters, and the third line shows the safe way to compare: check that two decimals are very close, never exactly equal.

The pair [1, 1, 2] and [10, 10, 20] is worse in a different way: rounding pushes the cosine above 1 (1.0000000000000002), a value a real cosine can never take. Feed it to Math.acos and you get NaN (not a number). The last line shows the usual fix: clamp the value with Math.min(1, c) before calling acos. In practice you rarely need acos, because you can rank things by cosine directly.

These digits are stable: they come from how JavaScript numbers work, so you will see them on any computer. The lesson: never test floating point results with ===. Use a tolerance, or round for display with .toFixed(3).

8. Cosine vs Distance

In the previous chapter you measured the distance between vectors. Now you have a second measure, cosine. They answer different questions, and the difference matters for text.

Count how often the words [cat, dog, car] appear in three texts.

// demo-cosine-vs-distance.js
import { cosine, distance } from "./src/vector.js";

// counts of [cat, dog, car] words. Invented numbers.
const shortNote = [1, 0, 0];
const longEssay = [30, 2, 1];
const carNote = [0, 0, 1];

console.log(distance(shortNote, longEssay).toFixed(2));
console.log(distance(shortNote, carNote).toFixed(2));
console.log(cosine(shortNote, longEssay).toFixed(3));
console.log(cosine(shortNote, carNote).toFixed(3));

// output:
// 29.09
// 1.41
// 0.997
// 0.000

By distance, the cat note is far from the long essay (29.09) and close to the car note (1.41). That is silly: the essay is mostly about cats, and the car note is about cars. Distance is fooled because the essay is simply long, so every count is bigger.

By cosine, the cat note and the essay are almost identical (0.997), and the cat note and the car note have nothing in common (0.000). That matches what a human would say.

A rule of thumb: use cosine when you care about direction (topic, meaning, taste) and size is noise, like text length. Use distance when size is meaningful (hours studied, pixel color) and the numbers are on comparable scales. Some systems normalize every vector first, and then cosine, dot product and distance all give the same ranking.

9. Nearest Neighbour Search

Now the payoff. Given one query vector and a list of items, which items are most similar? Compute the cosine of the query against every item, sort from highest to lowest, and keep the top few. This is called nearest neighbour search, and it is only a few lines in the toolkit:

nearest(query, items, k)  ->  the k best matches, best first

Each item is an object { id, vector }, and the answer is a list of { id, score }. The full file is in the recipe at the end. First, a hand-made vocabulary with four invented axes: food, drink, tech, animal.

// demo-search.js
import { nearest } from "./src/similarity.js";

// Invented 4-number word vectors. Axes: [food, drink, tech, animal]
const words = {
  coffee: [0.7, 0.9, 0.0, 0.0],
  tea: [0.6, 0.9, 0.0, 0.0],
  rice: [0.9, 0.0, 0.0, 0.0],
  soup: [0.9, 0.3, 0.0, 0.0],
  laptop: [0.0, 0.0, 0.9, 0.0],
  phone: [0.0, 0.0, 0.8, 0.0],
  dog: [0.2, 0.0, 0.0, 0.9],
  cat: [0.1, 0.0, 0.0, 0.8],
};

const items = Object.entries(words).map(([id, vector]) => ({ id, vector }));

console.log("neighbours of coffee:");
for (const r of nearest(words.coffee, items, 3)) {
  console.log(" ", r.id.padEnd(7), r.score.toFixed(3));
}

// output:
// neighbours of coffee:
//   coffee  1.000
//   tea     0.997
//   soup    0.832

The best match for coffee is coffee itself (cosine 1), then tea, then soup: the closest in the "food and drink" world. The function always returns k results, even when nothing is really similar, so read the scores, not just the ranks. A low score means "this was the best we had", not "this is a good match". The output is the same on every run; do not depend on the order of exactly tied scores.

10. A Tiny Search Engine by Meaning

To search texts instead of single words, make one vector per text. A simple, honest method: take the average of the word vectors (the centroid from the previous chapter) of the words it contains.

// demo-doc-search.js
import { mean } from "./src/vector.js";
import { nearest } from "./src/similarity.js";

const words = {
  coffee: [0.7, 0.9, 0.0, 0.0],
  tea: [0.6, 0.9, 0.0, 0.0],
  rice: [0.9, 0.0, 0.0, 0.0],
  soup: [0.9, 0.3, 0.0, 0.0],
  laptop: [0.0, 0.0, 0.9, 0.0],
  phone: [0.0, 0.0, 0.8, 0.0],
  dog: [0.2, 0.0, 0.0, 0.9],
  cat: [0.1, 0.0, 0.0, 0.8],
};

// A text becomes the average of its known word vectors.
function embed(text) {
  const known = text.split(" ").filter((w) => words[w]);
  return mean(known.map((w) => words[w]));
}

const docs = [
  "coffee tea",
  "rice soup",
  "laptop phone",
  "dog cat",
  "coffee laptop",
];
const items = docs.map((text) => ({ id: text, vector: embed(text) }));

for (const query of ["soup", "tea rice", "phone cat"]) {
  console.log(`query: ${query}`);
  for (const r of nearest(embed(query), items, 2)) {
    console.log(" ", r.id.padEnd(14), r.score.toFixed(3));
  }
}

// output:
// query: soup
//   rice soup      0.988
//   coffee tea     0.812
// query: tea rice
//   rice soup      0.930
//   coffee tea     0.919
// query: phone cat
//   dog cat        0.709
//   laptop phone   0.704

The query soup finds rice soup (0.988) first: same topic, no shared sentence, only shared meaning in the numbers. The query tea rice sits between two topics, and the scores 0.930 and 0.919 are almost tied, which is exactly what a mixed query should look like. The query phone cat mixes two unrelated ideas, and both results score about 0.7: weak matches. The honest reading is "nothing here is a great answer".

This is a toy: averaging ignores word order, our vectors were typed by hand, and the vocabulary has eight words. But the shape of the program is what a real semantic search does: embed the texts, embed the query, rank by cosine.

11. k-Nearest-Neighbours: Classify by Voting

The same idea can predict a label. To decide whether a new student is a coder or a gamer, find the k labeled students nearest to them and let them vote. This is the k-nearest-neighbours (k-NN) classifier. It uses distance, since our two numbers (hours coding and hours gaming) are on the same scale.

k-NN is about the simplest "machine learning" there is: no training at all, just remember the data and look around.

// demo-knn.js
import { knnClassify } from "./src/similarity.js";
import { students } from "./src/data.js";

console.log(knnClassify([9, 2], students, 3));
console.log(knnClassify([2, 8], students, 3));
console.log(knnClassify([6, 4], students, 3));
console.log(knnClassify([4, 6], students, 3));

// output:
// coder
// gamer
// coder
// gamer

[9, 2] (codes a lot, games little) is a coder, [2, 8] is a gamer. [6, 4] is closer to the coder side and [4, 6] to the gamer side. The function took the three closest labeled students and returned the majority label.

Why k matters: a single odd data point can mislead a classifier that only looks at one neighbour. Let us add one mistake to the data, a heavy coder who was wrongly recorded as a gamer:

// demo-knn-noise.js
import { knnClassify } from "./src/similarity.js";
import { students } from "./src/data.js";

// one mislabeled point: a heavy coder recorded as a "gamer" by mistake
const noisy = [...students, { x: [9, 2.2], label: "gamer" }];
const query = [9, 2.3];

console.log(knnClassify(query, students, 1));
console.log(knnClassify(query, noisy, 1));
console.log(knnClassify(query, noisy, 3));
console.log(knnClassify(query, noisy, 5));

// output:
// coder
// gamer
// coder
// coder

With clean data, k = 1 says coder. With the mistake added, k = 1 follows the single closest point, the bad one, and says gamer. With k = 3 or k = 5, the majority outvotes the mistake. This is a trade-off, not a rule: a bigger k is steadier but blurs the boundary between groups, and a very large k just predicts whichever label is most common. People usually try several k values and compare (chapter 8 shows how to test fairly).

Two more details from the code:

• Ties. With an even k the vote can tie. In our knnClassify a tie goes to the label whose neighbour was closest, because that label was counted first. With all 8 students voting (4 coders, 4 gamers), [6, 4] gets coder and [4, 6] gets gamer. Choosing an odd k for two labels avoids the problem.

12. The AI Connection

Everything you built in this chapter is the working core of several famous AI ideas.

Embeddings and semantic search. An embedding model turns a text, an image or a song into a vector. To find the best match for a query, embed the query and rank all stored vectors by cosine similarity. That is nearest(). See Embeddings and Attention for the words-as-points picture, and the RAG series (linked below) for the full app. Real embedding vectors are learned by a model and have hundreds or thousands of dimensions, not 4. Search over millions of vectors uses cleverer indexes than our "compare to everything" loop, but the similarity measure is the same.

Neurons. One neuron in a neural network computes a dot product of its inputs and its weights: "how much does this input match the pattern I look for?" You build one in Build a Neural Network.

Attention. In the Transformer, each word scores how relevant every other word is with dot products, then blends the words by those scores. You now know the ingredient that makes it work.

Honest caveats.

• Cosine similarity measures how alike two sets of numbers are. Whether that equals "similar meaning" depends on how good the embedding model is. A high score is evidence, not proof.
• Scores from different embedding models cannot be compared with each other. Always embed the query and the documents with the same model.
• There is no universal threshold like "0.8 means relevant". It depends on the data and the model. Treat any cut-off as something you tune and test.

13. Recipe: The similarity Module

Save this as src/similarity.js in your mathai project. It imports cosine and distance from the vector.js you saved in the previous chapter.

import { cosine, distance } from "./vector.js";

// Rank items by cosine similarity to a query vector.
// items: [{ id, vector }]  ->  [{ id, score }] best first
export function nearest(query, items, k = 3) {
  return items
    .map((item) => ({ id: item.id, score: cosine(query, item.vector) }))
    .sort((a, b) => b.score - a.score)
    .slice(0, k);
}

// k-nearest-neighbours: majority vote among the k closest labeled points.
// labeled: [{ x: [..], label }]
export function knnClassify(point, labeled, k = 3) {
  const closest = labeled
    .map((item) => ({ label: item.label, d: distance(point, item.x) }))
    .sort((a, b) => a.d - b.d)
    .slice(0, k);
  const votes = new Map();
  for (const { label } of closest) votes.set(label, (votes.get(label) ?? 0) + 1);
  return [...votes.entries()].sort((a, b) => b[1] - a[1])[0][0];
}

nearest(query, items, k) ranks { id, vector } items by cosine similarity and returns the best k as { id, score }. knnClassify(point, labeled, k) takes { x, label } points and returns the winning label among the k nearest by distance.

Every output in this chapter is stable: nothing uses time or Math.random().

14. Try It Yourself

1. Your own taste. Write your own movie (or food, or music) ratings as a 3-number vector, plus two friends'. Compute cosine between all pairs. Who is most alike? Does the answer match your gut feeling?
2. Same direction, different size. Compute cosine([2, 3], [4, 6]) and cosine([2, 3], [20, 30]). You should see 1 (or a number within a hair of 1). Then print dot for both pairs. Why do the dot products differ?
3. Perpendicular by design. Find a second vector, not [0, 0], whose dot product with [3, 4] is exactly 0. Hint: swap the numbers and flip one sign. Confirm with cosine.
4. Grow the vocabulary. Add juice: [0.8, 0.9, 0, 0] and monkey: [0.1, 0, 0, 0.7] to the word list in demo-search.js. Print nearest for both. Do the neighbours look sensible?
5. Break k-NN. In demo-knn-noise.js, move the query to [5, 5] and try k from 1 to 8. Which k values flip the answer, and why does the tie rule matter?

15. Quick Review: Common Questions

Questions beginners often ask about this chapter. Try answering each one yourself first, then click the question to open the answer.

Where to go next. A single dot product compares two vectors. A neural network does millions of them at once, and the tool for that is the matrix, a table of numbers that transforms vectors. That is next in Matrices and Layers. To see where cosine search ends up in a real app, look at the RAG: Chat with Your Own Documents series, and for the probability side of the story read Probability Basics.