Tokens and Tokenizers

Table of Contents

1. Models Do Not Read Words
2. Three Ways to Cut Text
3. Characters and Words in Code
4. The Idea of Byte-Pair Encoding
5. Training BPE by Hand
6. Encoding and Decoding
7. Recipe: The Tokenizer Module
8. Run It on the Corpus
9. Strange Tokenizer Effects
10. Khmer and Other Languages
11. Tokens Are Money
12. Try It Yourself
13. Quick Review: Common Questions

1. Models Do Not Read Words

In chapter 1 we said a model "predicts the next word". That was a simplification. A real model does not see words, and it does not see letters either. It sees tokens.

A token is a piece of text, and every model has a fixed list of all the pieces it knows. That list is the vocabulary. A piece can be a whole word ( the), part of a word (ing), a single character (z), a punctuation mark, or even a space plus a word. Each piece has a number, called its token ID. The model never touches the text; it reads a list of IDs and predicts the ID of the next token.

The program that turns text into IDs and back is the tokenizer:

"Dara drinks iced tea"
        |  tokenizer (encode)
        v
[ "Dar", "a", " drinks", " iced", " tea" ]      <- pieces (illustrative)
        |
        v
[ 8501, 64, 20832, 39874, 8887 ]                <- token IDs (made up)
        |
        v
    the model
        |
        v
  next token ID  --->  tokenizer (decode)  --->  " and"

The pieces and IDs above are illustrative, not from any real tokenizer. Different models cut the same sentence differently, because each tokenizer is built for its own model.

Why does this matter to you as a developer?

• Cost. Hosted models usually charge per token, not per word or per character.
• Limits. A model's memory (the context window, chapter 7) is counted in tokens.
• Speed. A model produces one token at a time, so more tokens means a longer wait.
• Strange behavior. Many odd model mistakes, such as miscounting letters in a word, start with tokenization.

2. Three Ways to Cut Text

There are three obvious ways to break text into pieces.

Characters. One token per character: c, o, f, f, e, e.

• Tiny vocabulary: a few hundred characters cover almost any text.
• Never meets an unknown word.
• But sequences get very long. A 5-letter word is 5 tokens, so the model sees little in each step and must do much more work.

Words. One token per word: coffee.

• Short sequences, and each token carries meaning.
• But the vocabulary has to contain every word, in every form: walk, walks, walked, walking, plus names, typos, and new words. Vocabulary would be huge and still incomplete. A word that is not in the list is an unknown word, and the model cannot handle it.

Sub-words. Pieces between the two: common words are one token, rare words are several pieces, and any unusual text can still be built from small pieces.

• coffee might be one token. unbreakable might be un, break, able. A made-up word falls back to smaller pieces, down to single characters if needed.

Real LLMs use sub-word tokens. That is why we said in chapter 1 that models predict "the next token". Our tiny models in chapters 3 to 5 use whole words only, because that is simpler to read, and we will flag that as a simplification when it comes up.

3. Characters and Words in Code

We already have the word splitter from chapter 1 (words in src/corpus.js). Character tokens are even simpler: a string is already iterable, so spreading it with [...text] gives one element per character.

Here is what the two look like on a short phrase:

import { words } from "./src/corpus.js";

const sentence = "the students order iced coffee";
console.log([...sentence].length);
console.log(words(sentence));
console.log(words(sentence).length);

30
[ 'the', 'students', 'order', 'iced', 'coffee' ]
5

Thirty character tokens against five word tokens for the same sentence, a six times difference. Sub-word tokens will land between them. The output is stable.

One detail: [...text] splits by Unicode code points, which is closer to "characters" than text.split("") is. For ordinary English the two are the same. We come back to why this matters for Khmer in section 10.

4. The Idea of Byte-Pair Encoding

How does a tokenizer decide which sub-words to keep? A popular method, used in many real tokenizers, is byte-pair encoding (BPE). The idea is simple enough to run on paper:

1. Start with the text as single characters.
2. Count every pair of neighbouring tokens.
3. Take the most common pair and glue it into one new token.
4. Replace that pair everywhere in the text.
5. Repeat steps 2 to 4 a fixed number of times.

Each repetition is called a merge, and the ordered list of merges is the recipe. Frequent letter groups such as th and ing get merged early and become tokens. Whole common words get built up over several merges. Rare words never get merged much, so they stay as small pieces.

The vocabulary is therefore: every single character, plus one new token for every merge. If you choose 50,000 merges, you get a vocabulary of about 50,000 plus the starting characters. The number of merges is a design choice. More merges give shorter sequences but a bigger vocabulary.

Real tokenizers add details: they often start from bytes instead of characters (we explain why in section 10), they split text into chunks first, and they are trained on enormous text. Our toy keeps the core idea and removes the rest. One real feature we keep: we split text into chunks first, where each chunk is a word with the space in front of it. A merge never crosses a chunk border. Without this rule, a merge could glue the end of one word to the start of the next.

5. Training BPE by Hand

Before the code, run the idea on a very small text. Take this text, where each chunk is a word with its leading space:

"low", " lower", " lowest"

Start with single characters: l o w, space l o w e r, space l o w e s t. That is 16 tokens in total.

• Merge 1. The pair l + o appears three times, the most common (together with o + w, which also appears three times; the first one found wins). Merge it into lo.
• Merge 2. lo + w appears three times. Merge into low.
• Merge 3. space + low appears twice. Merge into low.
• Merge 4. low + e appears twice. Merge into lowe.
• After that, no pair appears more than once, so we stop. A pair seen only once is not worth a token, because it never saves any space.

Our program will do exactly this. After you build the module in section 7, this snippet prints the token list after each merge:

import { trainBpe, encode } from "./src/tokenizer.js";

const text = "low lower lowest";
const merges = trainBpe(text, 10);
for (let k = 0; k <= merges.length; k++) {
  const t = encode(text, merges.slice(0, k));
  console.log(String(k).padStart(2), t.length, JSON.stringify(t));
}

The output is stable:

 0 16 ["l","o","w"," ","l","o","w","e","r"," ","l","o","w","e","s","t"]
 1 13 ["lo","w"," ","lo","w","e","r"," ","lo","w","e","s","t"]
 2 10 ["low"," ","low","e","r"," ","low","e","s","t"]
 3 8 ["low"," low","e","r"," low","e","s","t"]
 4 6 ["low"," lowe","r"," lowe","s","t"]

The first column is the number of merges applied, the second is the token count. Sixteen tokens shrank to six. Notice that the recipe is based only on counting, so a different training text gives a different vocabulary.

6. Encoding and Decoding

Training produces the list of merges. Using the tokenizer later has two parts.

Encoding turns new text into tokens. Start from single characters and replay the merges in the same order. The order matters: a later merge might use a token that an earlier merge created. Encoding text the tokenizer has never seen still works. Pieces that never merged stay as single characters.

Decoding turns tokens back into text. This is easy: because every token is just a string, you glue them together with join(""). A good tokenizer is lossless: decode(encode(text)) gives back exactly the text you started with. We will test that.

7. Recipe: The Tokenizer Module

Create src/tokenizer.js. This is the module later chapters import:

import { words } from "./corpus.js";

// 1. Character tokens: every character is one token.
export function charTokens(text) {
  return [...text];
}

// 2. Word tokens: the words() helper from corpus.js.
export function wordTokens(text) {
  return words(text);
}

// 3. Toy byte-pair encoding (BPE).
// First split the text into chunks: a word together with the space in
// front of it. Merges never cross a chunk boundary, so " the" can become
// one token but "e t" cannot. (Real tokenizers also pre-split like this.)
function chunks(text) {
  return (text.match(/ ?[^ ]+| +/g) ?? []).map((chunk) => [...chunk]);
}

// Training: start from single characters, then repeat numMerges times:
// find the most common neighbouring pair and glue it into one token.
export function trainBpe(text, numMerges) {
  let pieces = chunks(text);
  const merges = [];
  for (let m = 0; m < numMerges; m++) {
    const counts = new Map();
    for (const chunk of pieces) {
      for (let i = 0; i < chunk.length - 1; i++) {
        const key = chunk[i] + "\u0000" + chunk[i + 1];
        counts.set(key, (counts.get(key) ?? 0) + 1);
      }
    }
    let best = null;
    let bestCount = 1; // a pair seen only once is not worth merging
    for (const [key, count] of counts) {
      if (count > bestCount) {
        best = key;
        bestCount = count;
      }
    }
    if (best === null) break;
    const pair = best.split("\u0000");
    merges.push(pair);
    pieces = pieces.map((chunk) => applyMerge(chunk, pair));
  }
  return merges;
}

function applyMerge(tokens, [a, b]) {
  const out = [];
  for (let i = 0; i < tokens.length; i++) {
    if (tokens[i] === a && tokens[i + 1] === b) {
      out.push(a + b);
      i++; // skip the second half of the pair
    } else {
      out.push(tokens[i]);
    }
  }
  return out;
}

// Encoding: split into chunks, replay the merges in the same order.
export function encode(text, merges) {
  return chunks(text).flatMap((chunk) => {
    let tokens = chunk;
    for (const pair of merges) tokens = applyMerge(tokens, pair);
    return tokens;
  });
}

// Decoding: just glue the pieces back together.
export function decode(tokens) {
  return tokens.join("");
}

// Cost = tokens / 1,000,000 * price per million tokens.
export function estimateCost(tokenCount, pricePerMillion) {
  return (tokenCount / 1_000_000) * pricePerMillion;
}

Walk through the parts.

• chunks(text) uses the regular expression / ?[^ ]+| +/g: "an optional space followed by one or more non-spaces, or else a run of spaces". "low lower" becomes the chunks low and lower. ?? [] handles empty text, where match returns null.
• trainBpe keeps pieces, a list of chunks, each chunk a list of tokens. In each round it counts neighbouring pairs across all chunks, picks the pair with the highest count, remembers it in merges, and applies it everywhere. When two pairs tie, the one counted first wins, which is why the output is reproducible.
• The pair is stored as a string with a null character in the middle ("\u0000") so it can be a Map key. The null character is never part of normal text, so it is safe as a separator.
• applyMerge walks the list. When it finds the pair, it pushes the glued token and skips ahead by one extra step. Skipping stops the second half of the pair from being used again.
• encode and decode are the two directions from section 6.
• estimateCost is plain arithmetic, used in section 11.

The signatures, for later chapters:

charTokens(text)             -> string[]
wordTokens(text)             -> string[]
trainBpe(text, numMerges)    -> [string, string][]   (ordered merges)
encode(text, merges)         -> string[]
decode(tokens)               -> string
estimateCost(tokenCount, pricePerMillion) -> number  (in currency units)

8. Run It on the Corpus

Create tokens-demo.js in the root of llmlab:

import { corpus, words } from "./src/corpus.js";
import { charTokens, wordTokens, trainBpe, encode, decode } from "./src/tokenizer.js";

const training = words(corpus).join(" ");
const merges = trainBpe(training, 60);

console.log("characters:", charTokens(training).length);
console.log("words:", wordTokens(training).length);
console.log("bpe (60 merges):", encode(training, merges).length);

const sentence = "dara drinks iced coffee and the students order iced tea";
const tokens = encode(sentence, merges);
console.log(JSON.stringify(tokens));
console.log(tokens.length, "tokens for", sentence.length, "characters");
console.log("round trip ok:", decode(tokens) === sentence);

Run node tokens-demo.js:

characters: 934
words: 182
bpe (60 merges): 396
["d","ar","a"," d","ri","n","k","s"," iced"," coffee"," and"," the"," students"," ","or","de","r"," iced"," t","e","a"]
21 tokens for 55 characters
round trip ok: true

Everything here is stable, because there is no randomness. Study it:

• The same text is 934 tokens as characters, 182 as words, and 396 with 60 BPE merges. BPE lands in between, as promised.
• Common words from the corpus, such as iced, coffee, the and students, became single tokens, each with a leading space.
• Words the corpus rarely used, such as dara at the start of the text or order, are split into several small pieces: d, ar, a.
• It still decodes to exactly the original text.

Now more merges. With 200 requested merges the training stops early, at 118 merges, because no pair repeats any more (our corpus is only 934 characters). The same sentence then needs only 15 tokens:

const bigger = trainBpe(training, 200);
console.log(bigger.length);
console.log(JSON.stringify(encode(sentence, bigger)));

118
["d","ar","a"," d","ri","n","ks"," iced"," coffee"," and"," the"," students"," order"," iced"," tea"]

That is 15 tokens instead of 21. More merges mean fewer tokens but a bigger vocabulary. Real tokenizers choose a size by balancing exactly that. Your numbers match ours because the algorithm and the corpus are fixed.

Finally, text the tokenizer never saw during training:

console.log(JSON.stringify(encode("sokha drinks bubble tea", merges)));

["s","okha"," d","ri","n","k","s"," b","u","b","b","le"," t","e","a"]

bubble was never in our corpus, but it does not break anything: it is built from single characters and the small piece le. This is the big advantage of sub-words: no unknown words, only less efficient ones.

9. Strange Tokenizer Effects

Real tokenizers are trained once, on a large text, and then frozen. The consequences show up everywhere. Our toy reproduces several of them.

console.log(JSON.stringify(encode("coffee", merges)));
console.log(JSON.stringify(encode(" coffee", merges)));
console.log(JSON.stringify(encode("Coffee", merges)));
console.log(JSON.stringify(encode("2026", merges)));
console.log(JSON.stringify(encode("strawberry", merges)));

["c","o","f","f","ee"]
[" coffee"]
["C","o","f","f","ee"]
["2","0","2","6"]
["s","t","r","a","w","b","er","r","y"]

What these teach:

• A space belongs to a token. coffee and coffee are different tokens (five pieces against one). In real tokenizers too, the same word at the start of a line and in the middle of a sentence can have different token IDs.
• Capitals are different tokens. Coffee is not coffee, and our corpus was all lowercase, so the capital C is a stray character. Real tokenizers have separate pieces for Coffee and coffee.
• Numbers split in odd places. Our toy never saw digits, so 2026 becomes four characters. Real tokenizers handle numbers better, but usually still in chunks that do not line up with place value. That is one reason models make arithmetic mistakes with long numbers.
• The model does not see letters. If a model sees strawberry as two or three tokens, then "how many r's are in strawberry?" asks about letters inside a token it only knows as a single ID. It has learned the answer from text, not by looking at the spelling. That explains why models are often unreliable at counting letters, even though the task looks trivial. When you need exact work such as counting, use code.

Our toy is also lossless on odd text:

const odd = "  two  spaces";
console.log(decode(encode(odd, merges)) === odd);

true

Because whitespace is part of the tokens, nothing is thrown away.

10. Khmer and Other Languages

A tokenizer is trained on some text. If most of that text is English, English gets long, efficient merges, and other languages get fewer. When a language has few merges in the tokenizer, its text is cut into many small pieces, so the same message uses more tokens, costs more, and uses up more of the context window.

We should be careful about numbers here. How many extra tokens a Khmer sentence needs depends on the specific tokenizer, and tokenizers change. Do not trust a number from an article, including this one. Test it yourself. Search the web for the "tokenizer" or "token counter" page published by the company whose model you plan to use, paste the same message in English and in Khmer, and compare the counts. Do the same when you change models. In many tokenizers Khmer text does come out as noticeably more tokens than the same meaning in English, and you can plan around that once you have measured it.

One thing we can measure with our own code is the raw size of the text in bytes, which is where real tokenizers often start. Create bytes.js:

const samples = { english: "coffee", khmer: "កាហ្វេ", greeting: "សួស្តី" };
for (const [name, text] of Object.entries(samples)) {
  console.log(name.padEnd(9), [...text].length, "characters,", Buffer.byteLength(text, "utf8"), "UTF-8 bytes");
}
console.log(Buffer.from("coffee"));
console.log(Buffer.from("កា"));
console.log([..."កាហ្វេ"].map((c) => c.codePointAt(0).toString(16)));

english   6 characters, 6 UTF-8 bytes
khmer     6 characters, 18 UTF-8 bytes
greeting  6 characters, 18 UTF-8 bytes
<Buffer 63 6f 66 66 65 65>
<Buffer e1 9e 80 e1 9e b6>
[ '1780', '17b6', '17a0', '17d2', '179c', '17c1' ]

All of this output is stable. Here is how to read it.

• The English word coffee is 6 code points and 6 bytes: one byte per letter.
• The Khmer word for coffee, កាហ្វេ, is also 6 code points, but each Khmer code point takes 3 bytes in UTF-8, so it is 18 bytes. The first two code points, ក (hex 1780) and ា (hex 17b6), became the six bytes e1 9e 80 e1 9e b6.
• Note that "6 characters" is how JavaScript counts code points. A Khmer reader may count syllables or clusters differently, because Khmer uses vowel signs and the subscript sign ្ (hex 17d2) that are separate code points but are not separate letters to a reader.

Why does this matter? Many real tokenizers work on bytes instead of characters. That guarantees any text, in any language, or even emoji, can be encoded: the worst case is one token per byte, so there are never unknown characters. But it also means that text which is under-represented in the training of the tokenizer starts from a longer byte sequence and has fewer helpful merges. The result is not a rule of nature. It is a consequence of what text the tokenizer designers used, and newer tokenizers are often better. It is a reason to measure rather than assume.

Our toy tokenizer works on characters, not bytes, so a Khmer character it has never seen simply stays as itself. Try it yourself: run encode("កាហ្វេ", merges) and you will get six single-character tokens, because our corpus had no Khmer to learn merges from.

Practical advice if you build apps for Khmer users:

• Measure real token counts with the tokenizer for your chosen model.
• Budget for more tokens when you estimate cost and context size.
• Test the quality of replies in Khmer, not just the cost. Fewer merges is one issue; how much Khmer text the model saw in training is a separate one.

11. Tokens Are Money

Most hosted models charge for the tokens you send and the tokens the model writes back, often at different prices. The formula is simple:

cost = tokens / 1,000,000 * price per million tokens

That is what estimateCost computes. The prices below are made-up examples, only to show the arithmetic. Always read the current price page of the service you use.

import { estimateCost } from "./src/tokenizer.js";

const inputPrice = 1.0;  // example price: $1.00 per million input tokens
const outputPrice = 4.0; // example price: $4.00 per million output tokens

// One chat: 2,000 tokens sent in, 500 tokens written back.
const perChat = estimateCost(2000, inputPrice) + estimateCost(500, outputPrice);
console.log(perChat.toFixed(4));
console.log((perChat * 10000).toFixed(2));

0.0040
40.00

At these example prices, one chat costs 0.004 dollars, less than half a cent. Ten thousand chats cost 40 dollars. Small numbers become real money at scale. That is also why developers care about:

• Sending less. Do not paste a whole document when five paragraphs would do.
• Asking for shorter answers. Output tokens often cost more than input tokens, so a reply of 100 tokens is cheaper than 1,000.
• Remembering that history is resent. In a chat, every new message also pays for the earlier ones (chapter 7).
• Using a smaller model for easy jobs. Prices differ a lot between models.

A rough rule of thumb for English is that one token is a little under one word on average, but it varies widely by tokenizer, language, and type of text (code, numbers and rare words use more). Use it only for a first guess, and use the real tokenizer when the number matters.

12. Try It Yourself

1. Merge by hand. On paper, run BPE on "aa aab aab" for three merges. Then check your answer with trainBpe("aa aab aab", 3).
2. Watch the size shrink. Run trainBpe on the corpus for 0, 10, 30, 60, 100 and 200 merges, and print encode(training, merges).length each time. Does the count keep shrinking? Where does it stop?
3. Train on different text. Train on a text of your own (a paragraph from your notes), then encode a sentence from the coffee corpus. Compare the token count with the one from the corpus-trained tokenizer. Which is more efficient on which text?
4. Prove it is lossless. Write a loop that checks decode(encode(line, merges)) === line for every line in corpus, split with corpus.trim().split("\n"). Print how many pass.
5. Find the unknown. Encode a sentence that contains emoji or Khmer with encode. What do the pieces look like? Then explain in one sentence why a byte-based tokenizer would never run into an unknown character.
6. Measure for real. Pick any hosted model's public tokenizer or token-counter page. Paste the same short message in English and in Khmer. Write down both counts and the ratio. Repeat with a second model's tokenizer if you can, and see whether the ratio is the same.

13. Quick Review: Common Questions

Questions beginners often ask about this chapter. Try answering each one yourself first, then click the question to open the answer.

Where to go next. You can now turn text into pieces and count them. The next step is the one that makes a model generate: learning which piece tends to follow which. In Build a Tiny Language Model we count word pairs in the corpus, turn the counts into probabilities, and write new text with a model we built ourselves.