How LLMs Work

The 30-second version

That's it. There's no database of facts inside, no plan for the whole answer, no memory between calls. A chat reply that looks like it was written in one go was really built one small piece at a time, and each piece was chosen by asking: given everything so far, what usually comes next?

Those pieces are the tokens: a common word, part of a word, or a punctuation mark (§02). Step through one reply:

Why this one idea matters to you as an engineer

Almost every surprising thing about LLMs follows from the loop:

This page is the foundation for the rest of the AI Engineering area: Embeddings, Prompting, RAG and LLMs in Production all build on it. No math beyond a little arithmetic, and no training math at all: this is about using models as a software engineer.

Tokens: what the model actually reads

The tokenizer has a fixed list of every token it knows, called its vocabularyVocabulary (tokenizer)The fixed list of every token a model knows, each with a number (its token id). Modern tokenizers have about 100,000–200,000 entries. The model can only read and write tokens from this list., usually 100,000 to 200,000 entries. Common words like " the" are one token. Rare words are split into pieces that are in the list. Try it on real text:

The quick brown fox jumps over the lazy dog.

What to notice

1. Spaces are part of tokens. "hello" and " hello" (with a leading space) are different tokens with different ids. So are " Hello" and " HEL"+"LO". To the model, these are unrelated numbers that it has learned are related.
2. English is cheap, other languages often aren't. The English sentence averages over 4 characters per token. Vietnamese and Japanese pack fewer characters into each token, especially with the older cl100k_base, where the Vietnamese sentence takes twice as many tokens.
3. Code and JSON are full of small tokens. Punctuation like {", ": and ;\n each cost a token. Indentation costs tokens too.
4. Numbers are chopped into chunks. 1234567 becomes 123 · 456 · 7. The model never sees "one million two hundred…", which is one reason LLMs are unreliable at arithmetic.
5. One emoji can be two tokens. Most emoji are 4 bytes in UTF-8UTF-8The standard way to turn text into bytes. Plain English letters take 1 byte each; accented letters, other scripts and emoji take 2–4 bytes., and some tokens hold only some of those bytes. Some emoji are several characters glued together: 👩‍💻 is 👩 + an invisible "zero-width joiner" + 💻, and the joiner is a token of its own. (Bytes, code points and UTF-8 are explained in Text & Encodings.)

Why not just use letters, or whole words?

• Letters would make every text very long: this paragraph would be hundreds of steps instead of dozens. The model runs once per token (§01), so that would be slow and expensive, and it would have to learn spelling before it could learn meaning.
• Whole words need an impossibly large list (every name, typo, URL, word in every language, every variable name in every codebase), and anything not in the list can't be read at all.

Tokens are the compromise: frequent text gets short, single tokens; anything else can still be spelled out from smaller pieces, down to single bytes. §03 shows how that list is built.

Where tokens come from, and why counts matter

Here is the whole algorithm, run for real on a tiny word list. Watch low, new, er and est appear:

A real tokenizer does exactly this, with three differences: it starts from the 256 possible bytes (so any text, in any language, can always be encoded), it learns from a huge text collection, and it stops after roughly 100,000–200,000 merges. The merge list is then frozen. Encoding new text means applying the same merges in the same order, so the same text always gives the same tokens.

The merge list depends on the text it was learned from. o200k_base has whole-word tokens like " trời" and " đẹp", so it encodes the Vietnamese sentence in §02 with half the tokens of cl100k_base, which has to spell those words from smaller pieces. A larger vocabulary, built from a broader mix of languages, buys shorter token sequences.

Why token counts matter

Limits are in tokens. The Context windowContext windowThe maximum number of tokens a model can consider at once: the system prompt, the conversation so far, any documents you pasted in, and the reply it is writing. Anything outside the window simply doesn't exist for the model. (§06) and the maximum reply length are counted in tokens, not characters or words.

Cost is in tokens. Model APIs bill per token, usually at one price for input tokens and a higher one for output tokens. The same document can cost noticeably more in one language than another, or with one model family than another, purely because of tokenization.

A rough rule of thumb, for English prose only: about 4 characters, or ¾ of a word, per token. It's fine for a back-of-the-envelope estimate ("a 3,000-word document is about 4,000 tokens"). It's wrong for code, JSON, numbers and most other languages, as §02 showed. Never use it to decide whether a request will fit; count with the provider's tokenizer.

The strawberry problem

Ask a model how many r's are in "strawberry" and it may well say two. In §02's explorer, " strawberry" is one token: id 101830 in o200k_base. The model doesn't see ten letters; it sees one number. To count the r's, it has to remember how that token is spelled, from patterns in its training text, and that memory can be wrong.

The same blind spot shows up as trouble with reversing strings, counting words exactly, rhyming, and spotting a typo inside a long word. Newer models are trained to work around it, and often get these right, but the fix in your application is simple: if code can compute it, let code compute it, and give the model the result.

Predicting the next token: scores to probabilities

What "the model" is, in one paragraph

An LLM is a neural networkNeural networkA program made of many layers of simple arithmetic (multiply by weights, add, apply a simple function). Data flows through the layers; training tunes the weights until the outputs are useful. An LLM is a very large one.: each token id is first swapped for a long list of numbers that stands for that token (an embedding, see Embeddings), and then layer after layer of simple arithmetic (multiply, add) is applied to those numbers. The numbers it multiplies by are its weightsWeights (parameters)The billions of numbers inside a model that training adjusts. Running the model means doing arithmetic with them on your input. Everything the model “knows” is stored in them, in a form no one can read like a database., billions of them, set during training (§08). Pushing your text through all the layers once is called a forward passForward passOne run of input through all the layers of the model to produce output scores. Generating a reply takes one forward pass per output token.. You don't need to know what happens in each layer to build with LLMs; you need to know what comes out.

What comes out

What comes out of one forward pass is a list of about 200,000 numbers, one per vocabulary token, called logitsLogitsThe raw scores a model outputs, one per token in its vocabulary, before they're turned into probabilities. Higher means more likely. They can be negative and don't add up to anything.. A higher logit means "this token is more likely to come next". Here are 8 of them, for the text The cat sat on the:

Walk through the three stages in the widget. The arithmetic is the whole of softmax:

1. Raise e (about 2.718) to the power of each score. Negative scores become small positive numbers; big scores become much bigger.
2. Add them all up.
3. Divide each one by that total. Now every value is between 0 and 1, and they sum to 1.

function softmax(logits) {
  const exps = logits.map((x) => Math.exp(x));    // step 1
  const sum = exps.reduce((a, b) => a + b, 0);    // step 2
  return exps.map((e) => e / sum);                // step 3
}

console.log(softmax([5.0, 3.6, 3.2]).map((p) => p.toFixed(3)));
// [ '0.708', '0.175', '0.117' ]

Then: pick one

The simplest way to choose is to always take the most likely token. That's called greedy decodingGreedy decodingAlways picking the single most likely next token. Same as temperature 0. Predictable, but can get repetitive., and here it picks " mat". But notice the model doesn't know the answer is "mat": it says "mat" about 60% of the time, "floor" about 15%, and so on. The probabilities describe what text like this usually continues with, not what is true. Keep that in mind for §09.

Most of the time, apps don't pick greedily; they sample, which is the next section.

Choosing a token: temperature and top-p

Why not always pick the top token? Because greedy text tends to be bland and can get stuck repeating itself ("the cat sat on the mat. the cat sat on the mat…"). A little randomness gives more natural, varied text. It's also why the same prompt can give different answers.

Temperature: divide the scores, then softmax

Temperature is one number, T. Before softmax, every logit is divided by T:

• T below 1 makes the gaps between scores bigger, so the favorite gets even more likely (sharper).
• T = 1 leaves the model's probabilities as they are.
• T above 1 shrinks the gaps, so unlikely tokens get a real chance (flatter).
• T = 0 would mean dividing by zero, so it's defined as "always take the top token": greedy.

Top-p (also called nucleus sampling) runs after that: sort tokens by probability, keep the smallest group whose probabilities add up to at least p (say 0.9), drop the rest, and scale the survivors back up to 100%. It stops a rare, silly token from ever being picked, without making the common choices less varied.

Try both, and press Sample 100× at a few temperatures:

Here's the same thing as code. It's short because it's just the widget's math:

function sampleNext(logits, temperature) {
  if (temperature === 0) return logits.indexOf(Math.max(...logits)); // greedy
  const scaled = logits.map((x) => x / temperature);  // ← temperature is just this
  const probs = softmax(scaled);                      // from §04
  let r = Math.random();                              // a number in [0, 1)
  for (let i = 0; i < probs.length; i++) {
    r -= probs[i];                                    // walk along the bars…
    if (r < 0) return i;                              // …until r runs out
  }
  return probs.length - 1;                            // guard against rounding
}

A real language model you can run: bigrams

Everything above used hand-written scores. Here's the same loop with a real model: a bigram modelBigram modelThe simplest language model: it predicts the next word from only the one word before it, using counts of which word followed which in its training text., trained by counting which word follows which in 21 short sentences. It looks at only one previous word. Generate a few sentences at temperature 0, 1 and 2:

At temperature 0 it gets stuck: “the cat sat on the cat sat on the…”, because after “the” the top word is “cat” (tied with “dog” at 6 each; ties go alphabetically), then “sat”, then “on”, then “the” again. That is the repetition problem from the top of this section, in miniature. At 2 it wanders into odd but grammatical-ish combinations. An LLM is this same loop, with a context of thousands of tokens instead of one word, and billions of learned weights instead of a table of counts.

Which settings to use

Change one of temperature or top-p, not both; they interact. And check what your API allows: as of October 2026, Anthropic's Messages API returns an error if you send temperature, top_p or top_k (top-k: keep only the k most likely tokens, a cruder cousin of top-p) with API version 2026-10-21, so all requests use the default settings. Several other providers' reasoning-focused models accept only the default value too. The mechanism still runs on the server; you just can't tune it there.

Why temperature 0 still isn't perfectly repeatable

Temperature 0 removes randomness from the choice. It doesn't guarantee the scores are identical every time. Servers process many users' requests together in a batch, and the batch size changes with load. Different batch sizes can make the GPU add up the same numbers in a different order, and with floating-point numbers, a different order can give slightly different results (see Numbers in Computers). When two tokens are nearly tied, a tiny difference flips the pick, and every token after that differs.

In a September 2025 write-up, researchers at Thinking Machines Lab reported sending the same prompt 1,000 times at temperature 0 to an open model served with the vLLM inference server, and getting 80 different completions. They traced it to exactly this batch effect and showed that specially written "batch-invariant" GPU code made all 1,000 identical, at a cost in speed. Hosted APIs don't generally promise identical outputs.

The context window: the model's entire world

The model has no memory

LLM APIs are statelessStateless (API)The server keeps no memory between calls: each request must contain everything needed to answer it. LLM APIs work like this, so a chat app re-sends the whole conversation on every call.: each request stands alone, and the server forgets it once the reply is sent. So how does a chat app remember your name? Your app keeps the conversation and sends all of it, every time:

Call 1:  [system] + [user: Hi, I'm Mai]                                     → "Hi Mai!"
Call 2:  [system] + [user: Hi, I'm Mai] + [assistant: Hi Mai!] + [user: …]  → …
Call 3:  [system] + … every earlier message … + [user: What's my name?]     → "Mai"

The System promptSystem promptInstructions given to a model before the user's messages, usually by the app developer: its role, rules, tone and output format. The model weighs it more heavily than user text, but it is not a security boundary. (the app's standing instructions) goes in front of every call too. Three consequences follow:

1. Every message makes later calls bigger. Message 30 of a chat re-sends messages 1–29. You pay for those input tokens again on every call, and long inputs are slower to start (§11).
2. The window fills up. Once system prompt + history + room for the reply exceeds the limit, something has to go.
3. "Memory" features are your code. If a product remembers things across chats, it's storing notes somewhere and pasting them into the prompt.

When the window is full

Play the conversation below. The window sizes are tiny so you can see it happen; the token counts are real.

The common strategies:

• Drop the oldest messages (a "sliding window"). Simple, and recent context is usually what matters. But a fact from early on, like the user's name, silently disappears.
• Summarize the oldest messages into a short note and keep that instead. Keeps the gist, loses detail, and costs an extra model call to write the summary. Some APIs can do this server-side for you.
• Keep the important things pinned: the system prompt always; key facts in a short "notes" block you maintain in code.
• Retrieve instead of including everything. Store documents outside the prompt and paste in only the relevant parts per question. That's RAG.

Whichever you choose, if you don't handle it, the API does it for you in the least friendly way: an error saying the request is too long.

How big are windows?

Big, and growing. As of October 2026, Anthropic's current models, for example, accept up to 1 million tokens of input per request, and other providers offer windows from about a hundred thousand to over a million tokens. Check the model's documentation; the limit is per model, not per provider.

A bigger window doesn't mean you should fill it. More tokens cost more and take longer to process (§11). And models don't use every part of a long context equally well: a 2023 study, "Lost in the Middle" (Liu et al., published in TACL 2024), found that the models it tested answered best when the relevant information was at the start or end of a long input, and noticeably worse when it was buried in the middle. Newer models have improved, but the practical advice stands: put the instructions and the most important material where they're easy to find, and don't paste in what isn't needed.

Attention: which earlier tokens matter

Take The cat sat on the mat because it was tired. To predict what comes after "it", the model has to work out what "it" refers to. The word "it" alone says nothing; the information is in "cat", several tokens back. Attention is the mechanism that pulls it in.

The intuition

At each layer of the network, every token does three things:

1. Asks a question, like "I'm a pronoun: which noun am I about?"
2. Scores every earlier token (and itself) on how well it answers that question. The scores go through a softmax (§04), so they add up to 100%.
3. Mixes in information from those tokens in those proportions: 50% of what "cat" carries, 20% of "mat", a little of the rest.

After that, the "it" token isn't just "it" anymore; it carries "it, probably the cat". The next layer builds on that, and so on through dozens of layers.

What the picture shows

• The empty triangle. A token can only attend to tokens before it (and itself). When the model is predicting the next token, the future doesn't exist yet, so it's masked out during training too. That's why it's called a "causal" model.
• Many heads. Each layer runs many attention headsAttention headOne independent copy of the attention step inside a layer. A layer has many heads, and each learns to look for something different (the previous word, the subject of the sentence, a matching bracket…). side by side. Each one asks a different question: one tracks what pronouns refer to, one looks at the previous word, one matches an opening bracket to its closing bracket. Nobody programs these. They emerge from training, and researchers study them after the fact.
• The weights aren't fixed. Unlike the model's weights (§04), attention scores are computed fresh for every input. Change the sentence to "…because it was soft." and a well-trained model shifts attention from "cat" toward "mat".

Why it matters for you

This design is called a transformerTransformerThe neural-network design behind today's LLMs (introduced in 2017). Its key part is attention, which lets every token pull in information from the other tokens in the context. It processes all prompt tokens in parallel, which suits GPUs. (from a 2017 paper, "Attention Is All You Need"), and it's what every major LLM is built on. Two practical facts come straight out of it:

• Every token can reach every earlier token, however far back. That's what makes long documents, long code files and long conversations workable at all.
• The work grows with length. Each new token attends to all the tokens before it, so a longer context means more work per token and more memory. That's one reason long prompts cost more and why §11 exists.

Training vs. inference

Pretraining is where nearly all the knowledge comes from. The model is shown a vast amount of text, and at every position it predicts the next token, compares its guess with the real one, and nudges its weights slightly toward the right answer. Repeated over trillions of tokens, that single task forces it to pick up grammar, facts, coding patterns and a lot of reasoning-shaped behavior, because all of those help predict text. The result of pretrainingPretrainingThe first, most expensive training stage: the model reads a huge amount of text and learns by predicting each next token and being corrected, adjusting its weights billions of times. The result is a base model. is a base model: give it "Dear Sir," and it continues the letter. Ask it a question and it might continue with three more questions, because that's what text on the internet often does.

Tuning turns that into an assistant. The base model is trained further on a much smaller set of example conversations (a prompt and a good answer), and then on feedback about which of two answers people, or another model, preferred. This fine-tuningFine-tuning (instruction tuning)Extra training after pretraining on a much smaller, curated set: example conversations, and feedback on which answers people prefer. It turns a text-continuer into an assistant that follows instructions. teaches it to answer instead of continue, to follow instructions and formats, and to decline some requests. It adds behavior far more than knowledge.

What this means when you call it

• It doesn't learn from your conversation. If it gets something wrong and you correct it, it uses the correction for the rest of this conversation, because the correction is in the context (§06). Next conversation, the mistake is back.
• It has a knowledge cutoffKnowledge cutoffThe date after which the model's training text stops. It knows nothing about later events unless you put that information in the prompt.. The training text stops at some date. Ask about anything later (a new library version, last week's outage, today's date) and it either says it doesn't know or, worse, makes up something plausible (§09). If the answer depends on fresh or private information, put that information in the prompt.
• Its knowledge is fuzzy, not stored. There's no row in a table saying "the capital of France is Paris". There are weights that make "Paris" very likely after that phrase. Popular facts are reliable; rare ones (a small library's API, a specific paper's page numbers) are shaky.
• Your data and training are a policy question. Whether a provider may use your API traffic to train future models is set by their terms and your account settings, not by how a call works. Check before sending sensitive data.

Why LLMs hallucinate

Nothing in the loop from §01 looks anything up. When the model "knows" a fact well, the true continuation has a high probability and wins. When it doesn't, there's no alarm, no empty result, no null: the loop still has to output some token, and it picks one that fits the pattern. Watch a citation get built that way:

Why it happens

1. Likely isn't true. The model learned what text looks like. A made-up citation looks exactly like a real one: same format, plausible names, a reasonable year.
2. "I don't know" isn't the default. Saying "I'm not sure" is just another continuation, and it competes with confident-sounding ones. Tuning makes models admit uncertainty more often, but they still can't reliably tell what they know from what they're guessing.
3. Each token commits. Once "Chen et al." is written, the rest of the answer is built to be consistent with it. The model doesn't go back and revise; it continues what's there.
4. Rare facts are fuzzy. Popular facts appear thousands of times in training text; a specific paper's title or a function in a niche library might appear a handful of times or never (§08).

Where it bites hardest

• Citations, quotes, URLs and references: exact strings the model can't verify. Treat every one as unverified until checked.
• Numbers: statistics, prices, dates, version numbers, line numbers. Tokenized in chunks (§02), easy to get "almost right".
• APIs and code: function names and flags that should exist, by analogy with similar libraries. The code looks idiomatic and fails to compile, or worse, compiles and does something else.
• Anything after the knowledge cutoff or private to your company: the model has never seen it.

What reduces it

You can't switch hallucination off with a setting (a low temperature only makes the model more consistent, including consistently wrong). You design around it:

• Ground the answer. Put the facts in the prompt (retrieved documents, database rows, tool results) and instruct the model to answer only from them and to say when they don't contain the answer. That's groundingGroundingGiving the model the facts it should use, in the prompt (retrieved documents, tool results, database rows), and asking it to answer from them, instead of relying on what its weights remember., and RAG is the standard way to do it.
• Use tools for facts and arithmetic. Let the model call a search, a database query or a calculator and use the result (Tool callingTool calling (function calling)A model API feature where you describe functions (name, purpose, JSON Schema for arguments) and the model can reply with a structured request to call one. Your code runs the function; the model never executes anything itself.; see Agents & Tool Calling).
• Verify in code. Check that cited documents exist and contain the quoted text, that generated JSON matches a schema, that code compiles and tests pass.
• Measure. Run a set of questions with known answers on every prompt or model change, and track how often it's right (Evaluating LLM Apps).
• Design the UI for it. Show sources, let users check them, and don't present generated text as authoritative where a mistake is costly.

One API call, end to end

Here's a minimal chat call to Anthropic's Messages API, as documented in October 2026 (API version 2026-10-21). Other providers name things differently (messages vs input, max_tokens vs max_output_tokens), but every one has the same parts.

let history = []; // the whole conversation lives in YOUR program

async function ask(userText) {
  const messages = [...history, { role: "user", content: userText }];
  const res = await fetch("https://api.anthropic.com/v1/messages", {
    method: "POST",
    headers: {
      "content-type": "application/json",
      "x-api-key": process.env.ANTHROPIC_API_KEY, // never hard-code a key
      "anthropic-version": "2026-10-21",
    },
    body: JSON.stringify({
      model: "claude-opus-5-5",
      max_tokens: 16000, // cap on the reply's length, in tokens
      system: "You are a helpful assistant for a bike shop. Answer briefly.",
      messages, // ← the WHOLE conversation, every call
    }),
    signal: AbortSignal.timeout(120_000), // replies can take a while; never wait forever
  });
  if (!res.ok) throw new Error(`HTTP ${res.status}: ${await res.text()}`);
  const msg = await res.json();
  console.log(msg.usage.input_tokens, msg.usage.output_tokens, msg.stop_reason);
  if (msg.stop_reason === "refusal") throw new Error("the model declined this request"); // history unchanged
  if (msg.stop_reason === "max_tokens") console.warn("reply was cut off at max_tokens"); // show it as incomplete
  history = [...messages, { role: "assistant", content: msg.content }]; // success: keep both turns
  return msg.content.filter((b) => b.type === "text").map((b) => b.text).join("");
}

console.log(await ask("My name is Mai. Which brake pads fit a 2019 Trek FX 3?"));

Not run here (it needs an API key, and this site never calls a model). The Go version compiles with go vet; its imports are bytes, encoding/json, errors, fmt, io, log, net/http, os, slices and time.

Three details make it safer than the shortest possible version. It sets a timeout (2 minutes here), because a long reply can take a while and a hung connection shouldn't hang your app. It adds the new turns to history only after a successful reply, so a failed call doesn't leave a dangling user message. And it acts on stop_reason (explained below): a refusal becomes an error, and a cut-off reply is flagged. Keep the key in an environment variable or a secrets manager (Secrets), never in code or in the browser.

Reading the response

The response is JSON shaped like this (the values here are illustrative):

{
  "role": "assistant",
  "content": [{ "type": "text", "text": "Your FX 3 has Shimano hydraulic disc brakes…" }],
  "stop_reason": "end_turn",
  "usage": { "input_tokens": 41, "output_tokens": 63 }
}

• usage is the token bill for this call. input_tokens grows with every message in history; that's §06's "every message makes later calls bigger", in numbers. Log both counts on every call.
• stop_reason says why generation ended. end_turn means the model picked its end token (§01). max_tokens means it hit your max_tokensmax_tokensA request setting that caps how many tokens the model may generate in its reply. If it's reached, the reply is cut off mid-sentence and the API says so (e.g. stop_reason: "max_tokens"). cap and the reply is cut off mid-sentence: show that to users as incomplete, or raise the cap. Other values cover tool calls, stop strings you defined and refusals.
• No temperature. In this API version you can't send it (§05), so the request has none. On APIs that accept it, it's one more field next to max_tokens.

Trimming history to a budget

The code above grows history forever. Here's the "drop oldest" strategy from §06, runnable without an API key. It works on the same Message shape as ask(): user turns are plain strings, assistant turns are the block arrays the API returned, so it counts only the text inside them. (Skip that step and an array's length gets estimated instead of its text, and nothing is ever trimmed.) It uses a rough estimate (§03's ~4 characters per token) so it can run anywhere; for real numbers, call the provider's token counter (Anthropic's is POST /v1/messages/count_tokens, as of October 2026).

// The text in a message: content is a string (what we send) or an array of
// blocks (what the API returns). Only "text" blocks count here.
const textOf = (content) =>
  typeof content === "string"
    ? content
    : content.filter((b) => b.type === "text").map((b) => b.text).join("");

// Rough estimate: about 4 characters per token for English text.
// For real numbers, use your provider's token-counting endpoint.
const estimateTokens = (content) => Math.ceil(textOf(content).length / 4);

// Keep the newest messages that fit in `budget` tokens; drop the oldest.
function fitHistory(messages, budget) {
  const kept = [];
  let used = 0;
  for (let i = messages.length - 1; i >= 0; i--) { // newest → oldest
    const cost = estimateTokens(messages[i].content);
    if (used + cost > budget) break;               // ← everything older is dropped
    kept.unshift(messages[i]);
    used += cost;
  }
  while (kept.length && kept[0].role !== "user") { // the API wants a user message first
    used -= estimateTokens(kept.shift().content);
  }
  return { kept, used };
}

// A history shaped like ask() builds it: user turns are strings,
// assistant turns are the block arrays the API returned.
const user = (text) => ({ role: "user", content: text });
const assistant = (text) => ({ role: "assistant", content: [{ type: "text", text }] });
const history = [
  user("Hi! My name is Mai and I ride a 2019 Trek FX 3."),
  assistant("Hi Mai! The FX 3 is a great hybrid. How can I help?"),
  user("My rear brake squeaks when it rains. What should I check?"),
  assistant("Wet rim or disc brakes often squeak. Clean the rotor with isopropyl alcohol and check the pads for glazing or wear."),
  user("The pads look shiny. Should I replace them?"),
  assistant("Shiny pads are glazed. You can sand them lightly, but new resin pads are cheap and usually fix it."),
  user("What was my name again?"),
];

const { kept, used } = fitHistory(history, 70);
console.log(`kept ${kept.length} of ${history.length} messages, ~${used} tokens`);
for (const m of kept) console.log(`${m.role.padEnd(9)} ${textOf(m.content).slice(0, 40)}`);

Real output (both versions print the same):

kept 3 of 7 messages, ~42 tokens
user      The pads look shiny. Should I replace th
assistant Shiny pads are glazed. You can sand them
user      What was my name again?

And there's the bug from §06, in real code: the message with Mai's name was dropped, so the model will be asked "What was my name again?" without it.

Staff-level: why tokens come out one at a time

Two phases: prefill and decode

PrefillPrefillThe first phase of generating a reply: the model processes all the prompt tokens at once, in parallel, and produces the first output token. Longer prompts mean a longer prefill and a later first token. handles the whole prompt. Every prompt token is already known, so the model can process all of them in parallel, which a GPUGPUGraphics processor: a chip with thousands of small cores that run the same simple calculation in parallel. Browsers use it to draw and combine page layers; password crackers use it to test billions of fast hashes per second; AI models run on them because their maths is the same simple operations repeated billions of times. (the kind of chip AI models run on) does very efficiently. Prefill ends by producing the first output token. Its length grows with the prompt, so it dominates time to first tokenTime to first token (TTFT)How long a user waits from sending a request until the first token of the reply arrives. With streaming, this — not total generation time — is what makes an LLM feature feel fast or slow..

DecodeDecode (phase)The second phase of generation: one forward pass per new token, each needing the token before it, so they can't run in parallel. This sets the tokens-per-second speed you see while text streams. produces everything after that, one token per pass. These passes are sequential: token 7 can't be computed before token 6 exists, because token 6 is part of its input. Decode speed (tokens per second) sets how fast text streams after it starts.

The KV cache

In each pass, every token computes keys and values for attention (§07): what it offers to tokens after it, and what it passes along. Those don't change once computed, because a token never attends to tokens after it. So the server keeps them in GPU memory, in the KV cacheKV cacheMemory on the GPU where the server keeps the intermediate results (attention “keys” and “values”) for every token already processed, so each new token only needs its own computation. It grows with context length., and each decode pass only computes the newest token. That turns the quadratic "recompute everything" work from the widget into roughly one position's worth of computation per new token.

The cache isn't free:

• It costs memory, proportional to the number of tokens in context, for every request being served at once. Long contexts mean fewer requests fit on one GPU, which is part of why long-context requests can be priced higher.
• Each decode step still reads the whole cache. Attention looks at every earlier token, so decode slows down a bit as the context grows.

What this means for your app

Some models also generate hidden "reasoning" tokens before the visible answer. Those are ordinary decode steps too: they add to the wait before the first visible token and are usually billed as output. LLMs in Production covers streaming, caching, retries and cost control in depth.

Pitfalls gallery

Check yourself

Predict the answer before you click. Q1 is covered in §02–§03, Q2 in §06, Q3 in §05, Q4 in §11 and Q5 in §09.

Cheat sheet