Context Windows and Memory

Table of Contents

1. The Model Has No Memory
2. A Stateless Fake Model
3. Memory Is Just Resending the Chat
4. The Context Window
5. Counting Tokens: An Estimate
6. When the Window Fills Up
7. Trimming: Keep the System Message and the Newest Turns
8. Summarizing Old Turns
9. Cost Grows With History
10. Lost in the Middle
11. Prompt Caching: The Idea
12. Long Context vs Finding the Right Pieces
13. Recipe: The context Module
14. Try It Yourself
15. Quick Review: Common Questions

1. The Model Has No Memory

You tell an AI assistant, "My name is Sokha." Ten messages later you ask, "What is my name?" and it answers correctly. It feels like the assistant remembers you, the way a friend does.

Here is the surprise: the model itself remembers nothing. In the last chapters you saw what a model does: it reads text, scores the next token, picks one, and repeats (Probabilities, Temperature and Sampling). When it finishes, it is done. Its internal numbers (the weights you met in How a Model Is Trained) do not change. Nothing about your conversation is stored inside it.

Think of a very smart person with no memory at all, who wakes up fresh for every question. If you hand them a notebook with the whole conversation written down, they read it and answer as if they remembered. If you hand them a blank page, they know nothing about you.

A chat app does exactly that. Every time you press Send, the app packs the entire conversation so far into one big piece of text and sends it to the model again. The "memory" lives in the app, not in the model. This design is called stateless: each request stands alone.

This chapter shows that trick in code, explains the limit it runs into (the context window), and teaches the tools every chatbot developer needs: counting tokens, trimming old messages, summarizing, and thinking about cost.

2. A Stateless Fake Model

We cannot call a real model (no keys, no network in this series), so we build a fake one with one important property: it is stateless and only knows what is inside the messages array you hand it. It is a toy that can answer two questions, "What is my name?" and "What is my favorite drink?", by searching the earlier messages for the answer.

Make a new file src/context.js and add this first part. You will add four more parts as the chapter goes on.

// src/context.js (part 1: the fake model)

// A stateless fake model. It sees ONLY the messages you pass in.
// Statements get "Noted." It answers two questions by searching the
// earlier user messages.
export function fakeModel(messages) {
  const last = messages[messages.length - 1].content;
  if (!last.includes("?")) return "Noted.";
  const question = last.toLowerCase();
  const earlier = messages
    .slice(0, -1)
    .filter((m) => m.role === "user")
    .map((m) => m.content);

  if (question.includes("my name")) {
    for (const text of earlier.reverse()) {
      const m = text.match(/my name is (\w+)/i);
      if (m) return `Your name is ${m[1]}.`;
    }
    return "I don't know your name. You have not told me.";
  }
  if (question.includes("favorite drink")) {
    for (const text of earlier.reverse()) {
      const m = text.match(/i like (?:to drink )?([\w ]+?)[.!]?$/i);
      if (m) return `You like ${m[1]}.`;
    }
    return "I don't know your favorite drink.";
  }
  return "I can only answer questions about your name and favorite drink.";
}

A message here is an object with a role and some content. The roles are "system" (instructions from the developer), "user" (what the person typed) and "assistant" (what the model said). The next chapter, Prompts, Roles and Structured Output, is all about them. For now just know that a conversation is an array of these objects, oldest first.

Test it. Ask the question with nothing else in the array, then ask again with the earlier turn included:

// demo.js
import { fakeModel } from "./src/context.js";

console.log(fakeModel([{ role: "user", content: "What is my name?" }]));
// I don't know your name. You have not told me.

const history = [
  { role: "system", content: "You are a helpful assistant." },
  { role: "user", content: "Hi, my name is Dara." },
  { role: "assistant", content: "Nice to meet you, Dara." },
  { role: "user", content: "What is my name?" },
];
console.log(fakeModel(history));
// Your name is Dara.

Same question, same "model", two different answers. The only difference is what we put in the array. That is the whole secret of chat memory. Both lines are stable, because the fake model uses no randomness.

A real model is far more capable than this regex toy, but the principle is the same: it can only use what is in front of it, plus general knowledge from training (not facts about you).

3. Memory Is Just Resending the Chat

Now build the loop a chat app runs. Keep one array. On every turn, add the user's message, send the whole array to the model, then add the reply to the array too.

import { fakeModel } from "./src/context.js";

const chat = [{ role: "system", content: "You are a helpful assistant." }];

function say(text) {
  chat.push({ role: "user", content: text });
  const reply = fakeModel(chat); // the WHOLE history goes in every time
  chat.push({ role: "assistant", content: reply });
  console.log("You:", text);
  console.log("Bot:", reply);
}

say("My name is Sokha.");
say("I like iced tea.");
say("What is my name?");
say("What is my favorite drink?");
console.log(chat.length);
// You: My name is Sokha.
// Bot: Noted.
// You: I like iced tea.
// Bot: Noted.
// You: What is my name?
// Bot: Your name is Sokha.
// You: What is my favorite drink?
// Bot: You like iced tea.
// 9

The array grew from 1 message (the system message) to 9: four user messages and four assistant replies, plus the system message. Every call to fakeModel received a longer array than the last.

Two consequences follow: the array has a size limit (the context window), and you pay for the whole array on every turn.

If your app loses the array (new chat, server restart, a bug), the model "forgets" everything. The memory is only as good as your code.

4. The Context Window

The context window is the maximum amount of text (measured in tokens, see Tokens and Tokenizers) that a model can look at in one request. It includes everything:

• the system message,
• all earlier user and assistant messages,
• any documents you pasted in,
• the new question,
• and the answer the model is writing right now.

Think of it as the size of the desk. A model can only work with papers that fit on the desk at once. A bigger desk lets you spread out more papers, but the desk is never infinite.

Window sizes differ between models and change over time as new models arrive. As examples only, a model might have a window of 8,000 tokens, or 128,000, or even more. Check the documentation of the model you use; never trust a number from a blog post, including this one.

A few honest points about windows:

• The window is shared between input and output. If the window is 8,000 tokens and your prompt uses 7,900, the answer can only be about 100 tokens long.
• Bigger is not free. More tokens means a higher bill and usually a slower reply (section 9).
• Fitting is not the same as using well. A model can read a long prompt but does not always use every part equally (section 10).
• Why is there a limit at all? In the attention idea from Embeddings and Attention, every token looks at the other tokens, so the work grows quickly with length. Engineers keep stretching windows, but there is always a limit.

5. Counting Tokens: An Estimate

To stay inside the window you must count tokens. The exact count depends on the tokenizer of the specific model. A real app uses the tokenizer the model provider supplies, or asks the API to count for it.

For learning, a simple rule of thumb is good enough: in ordinary English, one token is roughly 4 characters. So we estimate:

// src/context.js (part 2)

// Rough estimate: English text averages about 4 characters per token.
// Real tokenizers differ, so treat this as a ballpark, never a bill.
export function estimateTokens(text) {
  return Math.ceil(text.length / 4);
}

// A message costs its text plus a few tokens of wrapper (role, markers).
export function messageTokens(message) {
  return estimateTokens(message.content) + 4;
}

export function totalTokens(messages) {
  return messages.reduce((sum, m) => sum + messageTokens(m), 0);
}

The + 4 is a made-up allowance for the "wrapper" that APIs add around each message (the role and some markers). Real overhead varies by provider. The point is that messages cost slightly more than their text.

Try it on the shared corpus from chapter 1, and on a longer document made by repeating it:

import { corpus } from "./src/corpus.js";
import { estimateTokens } from "./src/context.js";

console.log(estimateTokens("Hello"));
// 2
console.log(corpus.length, estimateTokens(corpus));
// 951 238

const windowSize = 8000; // an example window, not a real model's
console.log(((estimateTokens(corpus) / windowSize) * 100).toFixed(1) + "%");
// 3.0%

const doc = corpus.repeat(40);
console.log(estimateTokens(doc), estimateTokens(doc) > windowSize);
// 9510 true

The small coffee-shop story uses about 3% of an example 8,000-token window. Forty copies of it do not fit.

Be careful with other languages. The 4-characters rule is for English. Many tokenizers split Khmer and other non-English text into more tokens per character, so the same meaning can use more of the window and cost more. Do not guess: paste your text into the token counter on a real tokenizer web page (many providers publish one) and look. This is a good habit for any app that serves Cambodian users.

6. When the Window Fills Up

What happens if the conversation grows past the window? It depends on the provider, but the options are:

1. The API returns an error. "Your request is too long." Your user sees a failure.
2. Your app cuts the history before sending (this chapter).
3. The provider truncates for you, silently dropping some text. This is the dangerous one: the model answers as if the dropped part never existed, and nobody tells you.

Because you cannot count on the provider doing the right thing, a careful developer decides what to drop, in code. There are three common strategies:

• Trim: drop the oldest turns. Simple, but old facts vanish.
• Summarize: replace old turns with a short summary. Keeps the gist, but the summary can lose details or add mistakes.
• Retrieve: store everything outside and fetch only the relevant pieces (section 12). More moving parts.

You can combine them. Let's build the first two.

7. Trimming: Keep the System Message and the Newest Turns

The rule: always keep the system message (it holds the rules for the whole conversation), then keep the newest turns that fit in the budget, and drop the oldest.

// src/context.js (part 3)

// Keep the system message and as many of the NEWEST turns as fit.
export function trimHistory(messages, maxTokens) {
  const system = messages.filter((m) => m.role === "system");
  const rest = messages.filter((m) => m.role !== "system");
  let budget = maxTokens - totalTokens(system);
  const kept = [];
  for (let i = rest.length - 1; i >= 0; i--) {
    const cost = messageTokens(rest[i]);
    if (cost > budget) break;
    kept.unshift(rest[i]);
    budget -= cost;
  }
  return [...system, ...kept];
}

We walk backwards from the newest message, adding messages while they fit, and stop at the first one that does not. unshift puts each one at the front so the original order is preserved.

Now watch it forget. This chat has the name early on, and then some small talk:

import { messageTokens, totalTokens, trimHistory, fakeModel } from "./src/context.js";

const chat = [
  { role: "system", content: "You are a helpful assistant." },
  { role: "user", content: "My name is Sokha." },
  { role: "assistant", content: "Noted." },
  { role: "user", content: "I like iced tea." },
  { role: "assistant", content: "Noted." },
  { role: "user", content: "The coffee shop is quiet in the morning." },
  { role: "assistant", content: "Noted." },
  { role: "user", content: "What is my name?" },
];

console.log(chat.map(messageTokens));
// [
//   11,  9, 6, 8,
//    6, 14, 6, 8
// ]
console.log(totalTokens(chat));
// 68

const trimmed = trimHistory(chat, 30);
console.log(totalTokens(trimmed), trimmed.length);
// 25 3
for (const m of trimmed) console.log(m.role.padEnd(9), m.content);
// system    You are a helpful assistant.
// assistant Noted.
// user      What is my name?

console.log(fakeModel(trimmed));
// I don't know your name. You have not told me.
console.log(fakeModel(trimHistory(chat, 200)));
// Your name is Sokha.

With a budget of 30 tokens only the system message and the last two messages survive. The line "My name is Sokha." was dropped, so the model honestly says it does not know. With a budget of 200 everything fits and it answers correctly.

Two things to notice:

• The model did not "forget". Your code threw the information away. When a chatbot forgets something from long ago, that is usually a design decision in the app, not a flaw in the model.
• Trimming can start in the middle of a pair. Our trimmed chat begins with an assistant message right after the system message. Some APIs want the first non-system message to be a user message, so a production version often drops that leading assistant turn too. That is a good exercise (number 3 below).

8. Summarizing Old Turns

Trimming forgets old facts completely. A gentler idea: replace the old turns with a short summary and keep the newest turns word for word. In a real app, you ask the model itself to write the summary ("Summarize this conversation in 5 lines, keeping names, numbers, and decisions"). Here we write a placeholder that only shows the shape of the idea:

// src/context.js (part 4)

// Placeholder: a real app would ask a model to write this summary.
// Here we just note how many old messages were folded away.
export function summarizeOld(messages, keepLast) {
  const system = messages.filter((m) => m.role === "system");
  const rest = messages.filter((m) => m.role !== "system");
  const old = rest.slice(0, Math.max(0, rest.length - keepLast));
  const recent = rest.slice(-keepLast);
  if (old.length === 0) return messages;
  const note = {
    role: "system",
    content: `Summary of earlier chat: ${old.length} messages (replace with a real summary).`,
  };
  return [...system, note, ...recent];
}

import { summarizeOld } from "./src/context.js";
// (reuse the 8-message `chat` array from section 7)

const folded = summarizeOld(chat, 2);
for (const m of folded) console.log(m.role.padEnd(9), m.content);
// system    You are a helpful assistant.
// system    Summary of earlier chat: 5 messages (replace with a real summary).
// assistant Noted.
// user      What is my name?

Five old messages became one short note. If the real summary said "The user's name is Sokha and they like iced tea", the model could still answer the name question after the old turns were gone.

Summaries have honest weaknesses: details get lost, the summarizer is a model that can get facts wrong (and a wrong fact in the summary is then trusted), and summarizing is itself a request you pay for.

A common pattern is to keep important facts in a separate, small "profile" list that your app edits explicitly (name, language, preferences) and always includes in the system message. That is more reliable than hoping a summary keeps them.

9. Cost Grows With History

Most hosted models charge per token, with a price for tokens you send (input) and tokens the model writes (output). Prices differ by model and change; in this chapter we use the made-up example of 1.00 per million input tokens, only to show the shape of the cost.

Here is the trap: because every turn resends the full history, the tokens you pay for grow faster than the number of turns. If each turn adds about 200 tokens, then:

• turn 1 sends 200 tokens,
• turn 2 sends 400 (the old 200 plus the new 200),
• turn 3 sends 600,
• and turn t sends 200 * t.

Add them up and the total over T turns is 200 * (1 + 2 + ... + T), which is roughly 200 * T * T / 2. It grows like T^2, the same quadratic shape from Big-O Notation. Doubling the length of a chat roughly quadruples its input cost.

// src/context.js (part 5)

// Total input tokens you pay for over a whole conversation, because
// every turn resends everything before it.
export function resendCost(tokensPerTurn, turns, pricePerMillion) {
  let tokens = 0;
  for (let t = 1; t <= turns; t++) tokens += tokensPerTurn * t;
  return { tokens, cost: (tokens / 1_000_000) * pricePerMillion };
}

import { resendCost } from "./src/context.js";

for (const turns of [1, 10, 50, 100]) {
  const r = resendCost(200, turns, 1); // example price: 1.00 per million
  console.log(turns, r.tokens, r.cost.toFixed(4));
}
// 1 200 0.0002
// 10 11000 0.0110
// 50 255000 0.2550
// 100 1010000 1.0100

A 10-turn chat sent 11,000 input tokens in total, not 2,000. A 100-turn chat sent over a million, which is ten times more turns than the 10-turn chat but about ninety times more tokens. Real chats are a little different (turns have different lengths, and outputs cost extra), but the shape is real.

Long history also makes replies slower, because the model must read all of it before it writes the first word. So trimming and summarizing are not only about fitting the window: they save money and time too. Keeping the prompt short is a skill.

10. Lost in the Middle

If a model can fit 100 pages, will it use all 100 pages equally well? Researchers and developers have commonly observed that it does not always. Facts placed at the very beginning or the very end of a long prompt are often found more reliably than facts buried in the middle. This is called the lost in the middle effect.

Be careful how you state this. It is a commonly observed tendency, not a law of nature. It varies from model to model, it can improve as models improve, and it depends on the task. But it is common enough that careful developers plan for it:

• Put the most important instructions at the start, and consider repeating a short reminder at the end.
• Do not pile in everything "just in case". A prompt with the three relevant paragraphs usually beats a prompt with fifty pages where the answer is hidden.
• Test with your own data. Put a known fact in different places and see whether your model finds it. An evaluation you run yourself is worth more than any claim in an article.

The lesson is the opposite of "the window is big, so dump everything in": choose what goes in on purpose.

11. Prompt Caching: The Idea

Look again at the resend loop in section 3. Turn 2 sends the same first messages as turn 1, plus a bit more. The start of the prompt is identical every time. Re-reading identical text over and over is wasteful, and some providers offer prompt caching to avoid it.

The idea, at a conceptual level: the provider remembers the result of processing a prefix of your prompt (the beginning). If your next request starts with exactly the same prefix, it can reuse that work instead of redoing it, often at a lower price and faster speed. Details, prices, and rules differ by provider, so read their documentation. We only want you to know the idea, because it changes how you should design prompts.

We can see what "same prefix" means with a few lines of code:

import { totalTokens } from "./src/context.js";

const turn1 = [
  { role: "system", content: "You are a helpful assistant." },
  { role: "user", content: "My name is Sokha." },
];
const turn2 = [
  ...turn1,
  { role: "assistant", content: "Noted." },
  { role: "user", content: "I like iced tea." },
];

function sharedPrefix(a, b) {
  let i = 0;
  while (i < a.length && i < b.length &&
         a[i].content === b[i].content && a[i].role === b[i].role) i++;
  return i;
}

console.log(sharedPrefix(turn1, turn2), turn2.length);
// 2 4
console.log(totalTokens(turn1), totalTokens(turn2));
// 20 34

Turn 2 starts with the 2 messages (20 tokens) of turn 1. A cache could reuse those and only process the 14 new tokens. For a long system prompt or a big pasted document this matters a lot.

The design rule that follows: put the stable parts first and the changing parts last. Instructions, examples, and reference documents go at the start; the user's new question goes at the end. And do not change the beginning without a reason: even a small edit (a timestamp, a random ID) at the top of the prompt breaks the shared prefix, and the cache cannot help.

12. Long Context vs Finding the Right Pieces

When your documents are bigger than the window, or you do not want to pay to resend them every turn, there are two broad approaches:

• Stuff it all in. Use a model with a huge window and paste everything. Simple, but costly, slower, and subject to the middle problem.
• Retrieve only what is relevant. Keep documents outside the model. For each question, search for the few pieces that matter and put only those in the prompt. This uses the vector-similarity idea you saw in Embeddings and Attention.

The second approach is called retrieval-augmented generation, usually shortened to RAG. It is how most "chat with your documents" products work, and it is also a good defense against making things up, which is the topic of Why Models Hallucinate. A later series covers building it for real. Neither approach is always better: a short, well-chosen prompt is cheap and sharp, and a long window is convenient when you truly need the whole document at once.

13. Recipe: The context Module

Your src/context.js now holds the five parts from this chapter, in this order: fakeModel (section 2), the three token counters (section 5), trimHistory (section 7), summarizeOld (section 8), and resendCost (section 9). Every function is exported, and the file has no imports and no dependencies.

Exports, with what each does:

Remember what this module is and is not: fakeModel is a toy, and estimateTokens is a guess. The real value is the shape of the code (an array of messages, a budget, trimming from the oldest end, a summary note), which carries over unchanged to a real API in the next series.

14. Try It Yourself

1. Prove it is stateless. Call fakeModel twice in a row with the same two-message array. Then call it again after pushing a new user message into a different array. Explain, in one sentence, why the first two calls cannot affect each other.
2. Budget the window. Pick an example window of 4,000 tokens. Using estimateTokens, how many copies of the corpus fit? How many tokens are left for the answer if you paste 10 copies?
3. Fix the leading assistant. After trimHistory, if the first non-system message is from "assistant", drop it. Write trimHistoryStrict(messages, maxTokens) and show that it never starts with an assistant turn.
4. Always keep the first question. Change trimHistory so it also keeps the first user message (often the task description), plus the newest turns that fit. What new problem appears when the middle goes missing? Connect your answer to section 10.
5. Plot the bill. Using resendCost, find how many turns of 200 tokens make the total input exceed 1,000,000 tokens. Then change the turn size to 100 and find it again. Does halving the turn size halve the number of turns? Why not?
6. A better summarizer. Replace the placeholder in summarizeOld with a version that collects every my name is X and I like X from the old user messages into the summary note, so fakeModel can still answer after trimming. What kinds of facts would this approach miss?

15. Quick Review: Common Questions

Questions beginners often ask about this chapter. Try answering each one yourself first, then click the question to open the answer.

Where to go next. You now know that a chat is an array of messages that your code resends, trims and pays for. But what goes inside those messages, and who gets to say what? The next chapter, Prompts, Roles and Structured Output, covers system and user roles, few-shot examples, getting reliable JSON out of a model, and the biggest security risk of all: prompt injection.