LLMs in Production

The 30-second version

Those are the three facts, and each one maps to a set of tools:

One request, from your server's point of view, looks like this. The long part is the model writing:

What this page covers

Where the seconds go

Three numbers, not one

A normal web request has one latency number. A model call has three you should know apart:

1. Time to first tokenTime to first token (TTFT)How long a user waits from sending a request until the first token of the reply arrives. With streaming, this — not total generation time — is what makes an LLM feature feel fast or slow. (TTFT). From sending the request until the first piece of the reply arrives. The model spends it reading your prompt. A longer prompt means a longer TTFT.
2. Tokens per secondTokens per second (output speed)How fast a model writes its reply once it has started, e.g. 50 tokens per second. Together with time to first token it sets how long a reply takes: total ≈ TTFT + output tokens ÷ tokens per second.. How fast the reply comes out once it has started. The model writes one token at a time, each one depending on the token before it, so this rate is roughly steady for a given model and load.
3. Total time. When the last token arrives. That's TTFT plus the time to write every output token.

Why are the two phases so different? Reading the prompt can be done for all tokens at once, in parallel. Writing can't, because token 7 needs token 6 to exist first. How LLMs Work → inference shows this in detail; it calls the two phases prefillPrefillThe first phase of generating a reply: the model processes all the prompt tokens at once, in parallel, and produces the first output token. Longer prompts mean a longer prefill and a later first token. and decodeDecode (phase)The second phase of generation: one forward pass per new token, each needing the token before it, so they can't run in parallel. This sets the tokens-per-second speed you see while text streams..

A worked example

Assume (these are illustrative numbers, not measurements; real speeds vary by model, provider and load):

• a 2,000-token prompt with a TTFT of 0.8 s,
• an output speed of 50 tokens per second,
• a 400-token reply.

Total ≈ 0.8 s + 400 ÷ 50 = 0.8 + 8 = 8.8 s. Now try the two obvious fixes:

Output length dominates. If a feature is slow, first ask whether the model needs to write that much. Ask for a short answer, a list of IDs instead of prose, or JSON with only the fields you use.

Streaming: same total, much shorter wait

You can't make the model write faster, but you can show each token as soon as it exists. That's Streaming responseStreaming responseSending an HTTP response to the client in pieces as they become ready (live logs, server-sent events, AI tokens), instead of building the whole body and sending it at the end. Each piece must be flushed: pushed out of the server's buffer onto the network.: the API sends the reply in small pieces while it is still being written, instead of one response at the end. The total time doesn't change. What changes is how long the user stares at a spinner.

Drag the sliders. With a long reply, the non-streamed version keeps the user waiting for many seconds; the streamed version starts after the TTFT whatever the length. People also read slower than many models write, so a streamed answer often feels finished as soon as it starts.

Streaming from the model API

The format: server-sent events

Model APIs stream with server-sent events (SSE)Server-sent events (SSE)A simple one-way stream from server to browser over one long-lived HTTP response. Each message is a line like data: hello, sent (and flushed) as soon as it's ready.: one long HTTP response whose body is plain text, a few field: value lines per event, each event ended by a blank line. WebSockets & SSE → SSE explains the format and the browser's EventSource; here we only need to read such a stream from server code.

Here is the sequence Anthropic's Messages API sends (event names and fields as documented in October 2026; OpenAI and Google also stream over SSE, with different event names):

event: message_start
data: {"type":"message_start","message":{"id":"msg_…","content":[],"usage":{"input_tokens":25,"output_tokens":1}}}

event: content_block_start
data: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}}

event: ping
data: {"type":"ping"}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"Hello"}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"!"}}

event: content_block_stop
data: {"type":"content_block_stop","index":0}

event: message_delta
data: {"type":"message_delta","delta":{"stop_reason":"end_turn"},"usage":{"output_tokens":15}}

event: message_stop
data: {"type":"message_stop"}

• The reply text arrives in deltasDelta (stream event)One small piece of a streamed reply, such as {"type": "text_delta", "text": " with"}. Your code glues the deltas together, in order, to rebuild the full reply.: each content_block_delta carries the next small piece in delta.text. Glue them together in order.
• message_delta near the end carries the Stop reasonStop reasonA field in the API response that says why the model stopped writing: it finished, it hit max_tokens, it wants to call a tool, or it declined. Check it before trusting the text. (end_turn, max_tokens, refusal, …) and the output token count so far. The docs note the counts in it are cumulative: the total, not an increment.
• ping events can appear anywhere. They keep the connection busy so proxies don't close it; ignore them.
• An error event can arrive after the 200 status, for example overloaded_error when the provider runs out of capacity mid-reply. The docs also say new event types may be added, so ignore types you don't know instead of failing.
• On models that think first, a thinking block streams before the text block, with thinking_delta and signature_delta events (the thinking text is often empty, when the reasoning isn't shown). Anthropic's newest models always think (as of October 2026), so these will arrive. Act only on text_delta; reading delta.text from a thinking delta would append the string undefined.
• message_stop is the only sign that the reply is complete. If the connection closes without it, the answer was cut off, even though no error event arrived.

Parsing it: why you need a buffer

The network doesn't deliver events one by one. It delivers chunks of bytes, and a chunk can hold three events, or half of one. So the parser appends every chunk to a buffer, cuts off complete events (everything up to a blank line), and leaves any incomplete tail for the next chunk.

1let buffer = "", text = "", stopReason = null;
2for await (const chunk of body) { // text, as it arrives
3  buffer += chunk;
4  let end;
5  while ((end = buffer.indexOf("\n\n")) !== -1) {
6    const raw = buffer.slice(0, end);   // one whole event
7    buffer = buffer.slice(end + 2);     // keep the rest
8    const ev = JSON.parse(dataLineOf(raw)); // the JSON after "data: "
9    if (ev.type === "content_block_delta" && ev.delta.type === "text_delta") {
10      text += ev.delta.text;            // skip thinking_delta etc.
11      show(ev.delta.text);
12    } else if (ev.type === "message_delta") {
13      stopReason = ev.delta.stop_reason;
14    } else if (ev.type === "error") {
15      throw new Error(ev.error.type);
16    } else if (ev.type === "message_stop") {
17      return { text, stopReason };      // the only successful end
18    } // ping, message_start, …: ignore
19  }
20}
21throw new Error("stream ended early");

The code

To run this for real without an API keyAPI keyA long random string a service gives you so your program can call its API. Whoever sends the key is treated as you, so it is a password for a program., we wrote a small fake API (shown further down) that streams Anthropic-shaped events: a 300 ms pause, then 11 text deltas 40 ms apart. With ?fail=1 it sends an overloaded_error event halfway; with ?cut=1 it simply hangs up halfway, with no error at all. The client does four runs: a normal one, the two broken ones, and one the "user" cancels after 400 ms.

type event struct {
	Type  string `json:"type"`
	Delta struct {
		Type       string `json:"type"`
		Text       string `json:"text"`
		StopReason string `json:"stop_reason"`
	} `json:"delta"`
	Usage struct {
		OutputTokens int `json:"output_tokens"`
	} `json:"usage"`
	Error struct {
		Type string `json:"type"`
	} `json:"error"`
}

type result struct {
	Text         string
	StopReason   string
	OutputTokens int
}

var errEndedEarly = errors.New("stream ended early") // closed before message_stop

// streamReply sends the request with "stream": true and calls onText for every text delta.
func streamReply(ctx context.Context, url string, body map[string]any, onText func(string)) (result, error) {
	var r result
	body["stream"] = true // note: this adds a key to the caller's map
	payload, err := json.Marshal(body)
	if err != nil {
		return r, err
	}
	req, err := http.NewRequestWithContext(ctx, "POST", url, bytes.NewReader(payload)) // ← ctx cancels it
	if err != nil {
		return r, err
	}
	req.Header.Set("content-type", "application/json")
	res, err := http.DefaultClient.Do(req)
	if err != nil {
		return r, err
	}
	defer res.Body.Close()
	if res.StatusCode != http.StatusOK {
		msg, _ := io.ReadAll(res.Body)
		return r, fmt.Errorf("HTTP %d: %s", res.StatusCode, msg)
	}

	sc := bufio.NewScanner(res.Body) // reads the body line by line, as it arrives
	sc.Buffer(make([]byte, 64*1024), 1024*1024)
	for sc.Scan() {
		line := sc.Text()
		if !strings.HasPrefix(line, "data: ") {
			continue // "event: …" lines and the blank line between events
		}
		var ev event
		if err := json.Unmarshal([]byte(line[len("data: "):]), &ev); err != nil {
			return r, err
		}
		switch ev.Type {
		case "content_block_delta":
			if ev.Delta.Type == "text_delta" { // skip thinking_delta, signature_delta, …
				r.Text += ev.Delta.Text
				onText(ev.Delta.Text)
			}
		case "message_delta":
			r.StopReason = ev.Delta.StopReason
			r.OutputTokens = ev.Usage.OutputTokens // cumulative
		case "message_stop":
			return r, nil // ← the only way a stream ends successfully
		case "error": // ← errors can arrive AFTER a 200 status
			return r, fmt.Errorf("stream error: %s", ev.Error.Type)
		}
	}
	if err := sc.Err(); err != nil {
		return r, err // a cancelled ctx or a broken connection
	}
	return r, errEndedEarly // the server hung up cleanly, but before message_stop
}

func main() {
	url := "http://localhost:8787/v1/messages"
	newBody := func() map[string]any { // a fresh map per call (streamReply adds "stream")
		return map[string]any{
			"model":      "claude-opus-5-5",
			"max_tokens": 1024,
			"messages":   []map[string]string{{"role": "user", "content": "Checkout got slow. What first?"}},
		}
	}
	show := func(s string) { fmt.Print(s) } // print each piece as it arrives

	start := time.Now()
	r, err := streamReply(context.Background(), url, newBody(), show)
	if err != nil {
		fmt.Println("\nerror:", err)
		return
	}
	fmt.Printf("\nstop_reason=%s output_tokens=%d total=%dms\n", r.StopReason, r.OutputTokens, time.Since(start).Milliseconds())

	r, err = streamReply(context.Background(), url+"?fail=1", newBody(), show)
	fmt.Printf("\n%v; kept %q\n", err, r.Text)

	r, err = streamReply(context.Background(), url+"?cut=1", newBody(), show)
	fmt.Printf("\n%v; kept %q\n", err, r.Text)

	ctx, cancel := context.WithTimeout(context.Background(), 400*time.Millisecond) // user clicks "Stop"
	defer cancel()
	_, err = streamReply(ctx, url, newBody(), show)
	fmt.Println("\nstopped:", errors.Is(err, context.DeadlineExceeded))
}

Real output against the fake API (Go 1.24, Node 22; the Go file also has the usual package main and imports, left out above). The total time varies a little between runs:

Start with what changed on Tuesday. Compare p95 latency.
stop_reason=end_turn output_tokens=11 total=760ms
Start with what changed
stream error: overloaded_error; kept "Start with what changed"
Start with what changed
stream ended early; kept "Start with what changed"
Start with what
stopped: true

Look at the second and third runs: the failure came after four pieces of text had already been shown. Both versions keep that partial text and return it with the error; §04 covers what to do with it. The third run is the sneaky one. A proxy restart or a server crash can close the connection cleanly, so the read loop simply ends with no error. Without the message_stop check, the code would report a truncated answer as a success.

The fake API, for completeness. It's the server side of the same format: write event: and data: lines, end each event with a blank line.

import http from "node:http";

const REPLY = ["Start", " with", " what", " changed", " on", " Tuesday", ".", " Compare", " p95", " latency", "."];
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

http.createServer(async (req, res) => {
  const url = new URL(req.url, "http://x");
  // ?fail=1 sends an error event halfway; ?cut=1 just hangs up halfway.
  res.writeHead(200, { "content-type": "text/event-stream", "cache-control": "no-cache" });
  const send = (type, data) => res.write(`event: ${type}\ndata: ${JSON.stringify({ type, ...data })}\n\n`);
  await sleep(300); // "prefill": the wait before the first token
  send("message_start", { message: { id: "msg_fake", role: "assistant", content: [], usage: { input_tokens: 16, output_tokens: 1 } } });
  send("content_block_start", { index: 0, content_block: { type: "text", text: "" } });
  send("ping", {});
  for (let i = 0; i < REPLY.length; i++) {
    if (i === 4 && url.searchParams.get("fail")) {
      send("error", { error: { type: "overloaded_error", message: "Overloaded" } });
      return res.end();
    }
    if (i === 4 && url.searchParams.get("cut")) return res.end(); // a clean close, no error event
    send("content_block_delta", { index: 0, delta: { type: "text_delta", text: REPLY[i] } });
    await sleep(40); // "decode": one token at a time
  }
  send("content_block_stop", { index: 0 });
  send("message_delta", { delta: { stop_reason: "end_turn" }, usage: { output_tokens: REPLY.length } });
  send("message_stop", {});
  res.end();
}).listen(8787);

Pointing it at the real API

Only the URL and three headersHTTP headerA Name: value line sent before the body of an HTTP request or response, carrying metadata — Content-Type: application/json, Authorization: Bearer …, Cache-Control: max-age=60. change. Not run here: it needs an API key, which must stay on your server.

req.Header.Set("content-type", "application/json")
req.Header.Set("x-api-key", os.Getenv("ANTHROPIC_API_KEY")) // never hard-code the key
req.Header.Set("anthropic-version", "2026-10-21")
// url = "https://api.anthropic.com/v1/messages"
// body: "max_tokens": 64000 is a reasonable cap when streaming

On to the browser: relay, cancel, recover

Why a second stream, and why your own format

Your server sits in the middle for good reasons: it holds the API keyAPI keyA long random string a service gives you so your program can call its API. Whoever sends the key is treated as you, so it is a password for a program., checks who the user is, applies limits, and logs usage. It also shouldn't pass the provider's events straight through. Translate them into a tiny format of your own, say text, done and error. The browser code then doesn't change when you switch providers, and internal details (token counts, request IDs) don't leak to the client.

The relay handler

Each handler reads the user's message, opens the upstream stream with streamReply from §03, and writes one SSE event per text piece. The important lines are the flush and the cancellation.

func chat(w http.ResponseWriter, r *http.Request) {
	var in struct {
		Message string `json:"message"`
	}
	if err := json.NewDecoder(r.Body).Decode(&in); err != nil {
		http.Error(w, "bad request", http.StatusBadRequest)
		return
	}
	w.Header().Set("content-type", "text/event-stream")
	w.Header().Set("cache-control", "no-cache")
	rc := http.NewResponseController(w)

	send := func(event string, v any) {
		data, _ := json.Marshal(v)
		fmt.Fprintf(w, "event: %s\ndata: %s\n\n", event, data)
		rc.Flush() // ← push it to the browser now, don't wait for more
	}

	body := map[string]any{
		"model":      "claude-opus-5-5",
		"max_tokens": 64000,
		"messages":   []map[string]string{{"role": "user", "content": in.Message}},
	}
	// r.Context() is cancelled when the browser disconnects or aborts,
	// which cancels the upstream request too: we stop paying for tokens nobody reads.
	res, err := streamReply(r.Context(), upstream, body, func(text string) {
		send("text", map[string]string{"text": text})
	})
	if err != nil { // an error event, a cut connection, or "stream ended early": never "done"
		log.Printf("stream failed after %d chars: %v", len(res.Text), err)
		send("error", map[string]string{"message": "The answer was interrupted. Please try again."})
		return
	}
	send("done", map[string]string{"stop_reason": res.StopReason})
}

// registered with: http.HandleFunc("POST /chat", chat)

We ran both against the fake API from §03 and read them with curl -N:

• Closing curl after a few events made the Go server log stream failed after 23 chars: context canceled. The browser-side disconnect really did cancel the upstream call.
• Against the ?cut=1 stream, both relays logged stream ended early and sent an error event, never done: a truncated answer is not reported as finished.
• A malformed body (curl -d nope) gets a 400 from both. The Node version needs the try/catch around JSON.parse. Without it, the exception escapes the async handler as an unhandled rejection, which by default crashes the whole Node process: one bad request would take the server down for everyone.

The browser side

EventSource, the browser's built-in SSE client, can only send GET requests with no body. A chat message needs a POST, so read the stream with fetch, using the same buffer-and-split loop as §03. This is browser code (not run here):

let controller = null;

async function ask(message, outputEl) {
  controller = new AbortController();
  let state = "interrupted"; // stays this way unless a "done" event arrives
  try {
    const res = await fetch("/chat", {
      method: "POST",
      headers: { "content-type": "application/json" },
      body: JSON.stringify({ message }),
      signal: controller.signal, // ← the Stop button calls controller.abort()
    });
    if (!res.ok) throw new Error(`HTTP ${res.status}`); // e.g. 400 for a bad body, 429 from your own limits
    let buffer = "";
    for await (const chunk of res.body.pipeThrough(new TextDecoderStream())) {
      buffer += chunk;
      let end;
      while ((end = buffer.indexOf("\n\n")) !== -1) {
        const raw = buffer.slice(0, end);
        buffer = buffer.slice(end + 2);
        const event = raw.match(/^event: (.*)$/m)?.[1];
        const data = JSON.parse(raw.match(/^data: (.*)$/m)?.[1] ?? "{}");
        if (event === "text") outputEl.textContent += data.text; // textContent: never innerHTML for model output
        if (event === "done") {
          state = { end_turn: "done", max_tokens: "cut-off", refusal: "declined" }[data.stop_reason] ?? "done";
        }
      }
    }
  } catch (err) {
    state = err.name === "AbortError" ? "stopped" : "interrupted"; // Stop button, or a network failure
  }
  outputEl.dataset.state = state; // CSS shows "stopped", "cut off", "declined" or "interrupted" labels
}

stopButton.onclick = () => controller?.abort(); // browser → server → model API: all three stop

Three details matter. Every ending gets a label: the answer is only marked done when a done event arrives, so an error event, a dropped connection or a stream that just stops all leave it marked "interrupted", and a refusal or max_tokens stop reason is shown as such. Appending with textContent treats the model's output as plain text: if it ever contains <script> or an <img onerror=…>, it is shown, not run (see Injection). And a Stop button is a cost feature, not just a nicety: with the cancellation chain above, stopping a long answer stops the billing for tokens that were never generated.

When the stream breaks halfway

Errors mid-stream are normal: an overloaded provider, a deploy that restarts your server, a phone switching networks. By then the user has already read part of the answer. Your options:

If you really need to continue an interrupted answer, Anthropic's streaming docs (October 2026) describe sending a new request with the partial text and asking the model to continue from where it stopped. That costs a second request's input tokens and is only worth it for long outputs.

Timeouts for streams

A single overall timeout doesn't fit a stream. A healthy long answer can take two minutes, while a dead connection should fail in seconds. Use two limits:

• an idle timeout: if no event (not even a ping) arrives for, say, 30 s, give up. Reset the timer on every event;
• an overall deadline generous enough for your longest legitimate answer.

Caching whole answers (and why it's risky)

There are three different "LLM caches", and people mix them up:

Exact match: a normal cache with a careful key

This is ordinary cache-aside: before calling the model, look up a key; on a miss, call the model and store the answer. The only hard part is the Cache keyCache keyThe name a cached value is stored under, like product:42. Two requests share a cached answer only if they produce the same key, so everything the answer depends on must be in it.. The answer depends on everything in the request, so everything goes into the key: model name, system prompt, the full message list, tools, output settings, and your prompt version. Serialize them in a fixed order and hash the result with SHA-256SHA-256The standard cryptographic hash function today: it turns any data into a 32-byte (64 hex characters) fingerprint, and changing one byte of the input changes the fingerprint completely. Part of the SHA-2 family (2001); no practical attacks are known.:

type Message struct {
	Role    string `json:"role"`
	Content string `json:"content"`
}

// Everything that can change the answer goes into the key.
type keyParts struct {
	Model         string            `json:"model"`
	PromptVersion string            `json:"prompt_version"`
	System        string            `json:"system"`
	Tools         []json.RawMessage `json:"tools"` // each tool definition as JSON, in a fixed order
	Messages      []Message         `json:"messages"`
	MaxTokens     int               `json:"max_tokens"`
}

func cacheKey(p keyParts) string {
	b, _ := json.Marshal(p) // struct fields always marshal in the same order
	sum := sha256.Sum256(b)
	return "llm:" + hex.EncodeToString(sum[:])
}

Both versions produce the same key for the same request (we ran them: llm:7e3aa642… for a small test request), so a Go service and a Node service can share one cache.

Why it rarely hits in chat: the key includes the whole conversation, and no two conversations are identical. One extra space and it's a miss. Where it does pay off:

• Fixed inputs: generating a description for each of 10,000 products, then regenerating only the changed ones.
• Repeated identical requests: the same document summarized for many users, a classification run over a dataset with duplicates.
• Development: re-running an eval suite without paying again for unchanged cases (but don't let it hide real variation: Evaluating LLM Apps runs cases several times on purpose).

Note that a cached answer is frozen: the same request always gets the same text. That's usually what you want, but it also means a bad answer stays bad until it expires. Give entries a TTL and include the prompt version in the key, so a prompt change invalidates old answers automatically (Caching → invalidation).

Semantic caching: matching by meaning

Users rarely type exactly the same thing twice. A Semantic cacheSemantic cacheA response cache that looks answers up by meaning instead of exact text: it embeds the new question and reuses the stored answer of the most similar earlier question if the similarity is above a threshold. Fast and cheap, but it can return an answer to a different question. tries to catch paraphrases: turn the new question into an EmbeddingEmbeddingA list of numbers (a vector, often hundreds long) that an embedding model produces for a piece of text, such that texts with similar meaning get vectors that point in similar directions., find the most similar stored question by Cosine similarityCosine similarityA score from −1 to 1 for how closely two vectors point in the same direction, ignoring their length. For embeddings, higher means more similar meaning. For unit-length vectors it equals the dot product (multiply matching numbers and add them up)., and if the score is above a threshold, return that question's stored answer. (Embeddings and Vector Databases cover the machinery.)

The failure mode is the whole story. Embeddings measure "about the same topic, with similar words", not "has the same answer". Drag the threshold and watch what gets served:

No threshold is both safe and useful here, and that's typical. The pairs that break it are exactly the ones that matter: cancel vs renew, export vs import, my subscription vs my teammate's. The cache doesn't just fail; it answers instantly and confidently, so nobody notices until a user follows the wrong instructions.

Provider prompt caching

What is actually cached

Not the answer. The model still writes a fresh reply every time. What's reused is the result of reading the prompt (the "prefill" work from §02, stored as the KV cacheKV cacheMemory on the GPU where the server keeps the intermediate results (attention “keys” and “values”) for every token already processed, so each new token only needs its own computation. It grows with context length., see How LLMs Work → inference). That's why it helps most when a long, identical beginning repeats: a big system prompt, tool definitions, a long document you ask several questions about, or the earlier turns of a conversation.

Because reading token 500 depends on tokens 1 to 499, the reused part must be an exact prefix. Change token 3 and every later token has to be processed again.

How Anthropic's API does it (as of October 2026)

Other providers have their own versions, some fully automatic; the prefix rule is the same everywhere. On Anthropic's Messages API:

• Order. The prompt is assembled as tools → system → messages. A change in the tool list invalidates everything after it, including the system prompt.
• BreakpointsCache breakpointA marker in a request (cache_control on Anthropic's API) that says "cache the prompt up to here". A later request whose prompt is identical up to that point reads the cached part instead of processing it again.. You mark where a cacheable prefix ends by adding cache_control: {"type": "ephemeral"} to a content block, up to 4 per request. There is also a top-level cache_control that places the breakpoint on the last block automatically.
• Minimum size. Prefixes shorter than a model-dependent minimum (512 to 4,096 tokens) are simply not cached. There is no error; you just see zero cache tokens.
• Lifetime. An entry lives 5 minutes by default, and every read restarts the timer. A 1-hour lifetime is available ("ttl": "1h"), at a higher write price.
• Scope. Caches are per model and per workspace (or organization): switching models means starting cold.
• Billing. Writing a prefix into the cache costs somewhat more than normal input; reading it costs much less. So a prefix that's written and never read again is a net loss.

Watch it work (or not)

The three requests below share 747 tokens of tool definitions and system rules. Each also carries a timestamp, a classic way to break caching by accident:

The request

A system prompt sent as a list of blocks, with the breakpoint on the last stable block. The per-request details go into the user message, after the breakpoint. Not run here (needs an API key); the request shape is as documented in October 2026.

body := map[string]any{
	"model":      "claude-opus-5-5",
	"max_tokens": 16000,
	"tools":      tools, // ← same list, same order, every request (sort by name)
	"system": []map[string]any{{
		"type":          "text",
		"text":          systemRules, // ← frozen: no dates, names or IDs in here
		"cache_control": map[string]string{"type": "ephemeral"}, // ← cache everything up to here
	}},
	"messages": []map[string]any{{
		"role":    "user",
		"content": fmt.Sprintf("Current time: %s\n\n%s", now.Format(time.RFC3339), question), // ← changing parts last
	}},
}
// After the call: log usage.cache_read_input_tokens and usage.cache_creation_input_tokens.

For a chat, put a second breakpoint on the latest message (or use the automatic top-level one). Each new turn then reads the whole earlier conversation from the cache and only processes the new message, which directly attacks the cost growth you'll see in §09.

Silent cache breakers

Each of these changes the prefix on every request, and none of them causes an error:

Retries that don't make things worse

Model APIs fail more often than most dependencies: providers run near full capacity, and your account has rate limits. The general retry rules are taught in Idempotency → retries and Rate Limiting → HTTP; this section applies them to model calls.

What to retry

Error codes from Anthropic's API (as of October 2026); other providers use the same HTTP meanings, though not always the same numbers:

A retried model call is usually safe: generating text twice has no side effect except paying twice. It stops being safe when your code acts on the reply (sends the email the model drafted, runs the tool it asked for). §08 covers that.

How long to wait

Three rules, all from Idempotency → retries:

1. Exponential backoffExponential backoffWaiting longer after each failed retry, typically doubling: 1 s, 2 s, 4 s, 8 s… up to a cap. It gives a struggling server room to recover instead of hammering it.: the wait limit doubles each attempt (200 ms, 400 ms, 800 ms…, up to a cap).
2. Full jitterFull jitterA backoff rule where each retry waits a random time between 0 and the current backoff limit (which doubles each attempt), instead of exactly that limit. It spreads retries out so clients don't all come back at the same moment.: wait a random time between 0 and that limit, so thousands of clients don't come back at the same instant.
3. retry-after wins: if the server says when to come back, don't come back sooner.

The simulator shows why. Sixty clients hit a provider that is down for a second and then can only serve 4 requests per 100 ms:

Without jitter, the clients stay synchronized and arrive in waves far above capacity, the shape of a Retry stormRetry stormWhen a struggling service gets even more traffic because every caller retries its failed requests — often at several layers at once — so a small slowdown becomes a full outage.. With jitter, the same capacity serves every client within a few seconds, with far fewer wasted requests.

A retry wrapper, run against a flaky fake

The fake server from §03 has a /flaky route that answers 429 (with retry-after: 1), then 529, then 500, then success. It answers 400 to a request with max_tokens: -1, and imitates the spend-cap 429 (no retry-after, enforced_spend_limit_reached in the body) for a message containing spend-cap-test. A 429 alone doesn't tell you which kind it is, so both wrappers read the body of every 429.

var retryable = map[int]bool{408: true, 429: true, 500: true, 502: true, 503: true, 504: true, 529: true} // 529 = "overloaded"

// isSpendCap reports whether a 429 body is Anthropic's monthly spend cap, which retrying can't fix.
func isSpendCap(body []byte) bool {
	var e struct {
		Error struct {
			Details struct {
				ErrorCode string `json:"error_code"`
			} `json:"details"`
		} `json:"error"`
	}
	return json.Unmarshal(body, &e) == nil && e.Error.Details.ErrorCode == "enforced_spend_limit_reached"
}

// callWithRetry POSTs body and retries failures that might go away. ctx carries the overall deadline.
func callWithRetry(ctx context.Context, url string, body []byte) ([]byte, error) {
	const maxAttempts = 5
	base, maxWait := 200*time.Millisecond, 8*time.Second
	client := &http.Client{Timeout: 30 * time.Second} // ← limit per attempt

	for attempt := 1; ; attempt++ {
		req, err := http.NewRequestWithContext(ctx, "POST", url, bytes.NewReader(body))
		if err != nil {
			return nil, err
		}
		req.Header.Set("content-type", "application/json")
		res, err := client.Do(req)

		what, retryAfter := "network error", 0
		if err != nil {
			if ctx.Err() != nil || attempt == maxAttempts {
				return nil, err // out of time or attempts
			}
			// network error or per-attempt timeout: retry
		} else {
			data, readErr := io.ReadAll(res.Body)
			res.Body.Close()
			if res.StatusCode == http.StatusOK {
				return data, readErr
			}
			if !retryable[res.StatusCode] || isSpendCap(data) { // ← 400, 401, 404…, and the spend cap
				return nil, fmt.Errorf("HTTP %d (not retrying): %s", res.StatusCode, data)
			}
			what = strconv.Itoa(res.StatusCode)
			retryAfter, _ = strconv.Atoi(res.Header.Get("retry-after")) // seconds; 0 if absent
		}
		if attempt == maxAttempts {
			return nil, fmt.Errorf("gave up after %d attempts (last: %s)", attempt, what)
		}

		limit := min(maxWait, base<<(attempt-1))         // 200 ms, 400 ms, 800 ms… up to 8 s
		wait := time.Duration(rand.Int64N(int64(limit))) // "full jitter": random, from 0 up to the limit
		note := ""
		if retryAfter > 0 {
			wait, note = time.Duration(retryAfter)*time.Second, " (retry-after)"
		}
		fmt.Printf("attempt %d: %s → wait %d ms%s\n", attempt, what, wait.Milliseconds(), note)

		select {
		case <-time.After(wait):
		case <-ctx.Done(): // the overall deadline passed while we were waiting
			return nil, ctx.Err()
		}
	}
}

func main() {
	url := "http://localhost:8787/flaky"
	ctx, cancel := context.WithTimeout(context.Background(), time.Minute) // ← one budget for ALL attempts
	defer cancel()

	data, err := callWithRetry(ctx, url, []byte(`{"model":"claude-opus-5-5","max_tokens":1024,"messages":[{"role":"user","content":"Hi"}]}`))
	if err != nil {
		fmt.Println("failed:", err)
		return
	}
	fmt.Println("success:", string(data))

	_, err = callWithRetry(ctx, url, []byte(`{"model":"claude-opus-5-5","max_tokens":-1,"messages":[]}`))
	fmt.Println(err)

	_, err = callWithRetry(ctx, url, []byte(`{"model":"claude-opus-5-5","max_tokens":1024,"messages":[{"role":"user","content":"spend-cap-test"}]}`))
	fmt.Println(err)
}

Real output (the fake server was restarted before each run; the jittered waits are random, so yours will differ):

attempt 1: 429 → wait 1000 ms (retry-after)
attempt 2: 529 → wait 56 ms
attempt 3: 500 → wait 36 ms
success: {"content":[{"type":"text","text":"Hello!"}],"stop_reason":"end_turn","usage":{"input_tokens":12,"output_tokens":3}}
HTTP 400 (not retrying): {"type":"error","error":{"type":"invalid_request_error","message":"max_tokens: must be positive"}}
HTTP 429 (not retrying): {"type":"error","error":{"type":"rate_limit_error","message":"You have reached your API usage limits.","details":{"error_code":"enforced_spend_limit_reached"}}}

Timeouts for long generations

A model call can legitimately take minutes. Pick limits per feature from the longest answer you allow, not from your usual 5-second API default:

• Non-streaming: the per-attempt TimeoutTimeoutA limit on how long to wait for something (a connection, a response) before giving up with an error, so a slow or dead peer can't hang your program forever. must cover TTFT plus max_tokens ÷ tokens per second. A 4,000-token cap at 40 tokens/s is 100 s before you even count the prompt. If that's too long for your users, stream (§04) or ask for less.
• Streaming: an idle timeout between events, plus an overall deadline (§04).
• The overall deadline covers all attempts and the waits between them. A user waiting on a page won't wait for five 100-second attempts; a nightly job might.

When retries aren't enough

Fallback models

When the main model keeps failing (overloaded, or the provider is down), a Fallback modelFallback modelA second model (or provider) your code calls when the main one keeps failing or is overloaded. It keeps the feature working, usually with somewhat different output, so it needs its own testing. can keep the feature alive: a different model from the same provider, or a different provider. It's simple to write and easy to get subtly wrong:

• Prompts don't port perfectly. A prompt tuned for one model can produce different formats, lengths or refusals on another. Run your eval set against the fallback too (Evaluating LLM Apps), and keep its prompt variant in source control.
• The fallback is cache-cold. Prompt caches belong to one model (§06), so the first minutes on the fallback cost more and start slower.
• Capacity and limits are separate. The fallback has its own rate limits. If all your traffic moves there at once, you can hit them immediately.
• Some failures shouldn't fall back. A 400 means your request is wrong and will be wrong everywhere. A refusal is a policy decision, not an outage (some providers offer their own refusal-fallback features; treat those as a product decision).
• Record which model answered, in logs and ideally in your data, so a quality complaint can be traced to the fallback.

Circuit breakers

If the provider is down, each request still waits through timeouts and retries before failing. With long timeouts, that ties up connections and memory until your own service falls over: a Cascading failureCascading failureOne slow or broken service makes its callers slow or broken (they wait, pile up requests, run out of connections), which breaks their callers in turn, and so on up the chain.. A Circuit breakerCircuit breakerA wrapper around calls to another service that stops calling it for a while after too many failures (“opens”), failing fast instead of piling up waiting requests, then tries again carefully. watches the recent failure rate and, past a threshold, stops calling for a while:

While the breaker is open, requests fail immediately (or go straight to the fallback) instead of waiting. After a pause it lets a test request through; success closes it again. Notification system → providers builds one for SMS and email providers; the same pattern applies per model.

Pair it with a Retry budgetRetry budgetA cap on how much extra load retries may add (say, 10% of normal traffic), so retries can't multiply an outage.: retries may add at most, say, 10% on top of normal traffic. When the budget is used up, failures are returned instead of retried, so retries can't multiply an outage.

Side effects must happen once

A model call itself is safe to repeat. The danger is what your code does with the answer. If the model asks to call a tool like create_return or send_email, and the request is retried after a timeout, the tool can run twice. (Tool calling and agent loops get their own topic: Agents.)

• Give each action an Idempotency keyIdempotency keyA unique ID the client sends with a request (e.g. an Idempotency-Key header) so the server can recognise a repeat and do the work only once. derived from the conversation and step (for example conv_812:step_3:create_return), and have the tool refuse a key it has already completed. Idempotency → keys shows how.
• Separate "the model suggested it" from "we did it": record the action, then execute it once, outside the retry loop.
• Never retry a streamed answer that a user has already read by appending (§04).

Degrade, don't disappear

Decide per feature what "the model is down" looks like before it happens:

Background work should go through a job queue so an outage delays it instead of losing it. Interactive features need a message the user understands and, ideally, a way to get help without the model.

What a request really costs

The formula

For one request:

cost = uncached_input × input_price
     + cache_writes   × cache_write_price
     + cache_reads    × cache_read_price
     + output         × output_price          (prices are per token; lists show them per million)

Every term comes straight from the usage fieldsUsage fieldsThe token counts a model API returns with every reply: input tokens, output tokens, and (with prompt caching) tokens written to or read from the cache. They are what you are billed for, so log them. in the response, so you can compute it exactly for every request and add it up per user and per feature (§11). Two facts make this formula behave differently from what people expect:

• Output tokens usually cost several times more than input tokens, and they're also what makes requests slow (§02). A wordy reply is expensive twice.
• Thinking tokens are output. On models that reason before answering, those tokens are usually billed as output even when you never see them.

Chat history is the silent multiplier

Model APIs are stateless (Prompting → messages): to continue a chat, you send the whole conversation again. So call 8 pays for the system prompt and calls 1–7 all over again:

The total input grows roughly with the square of the number of turns: each call is a bit longer than the last, and you pay for every call. Your options:

• Prompt caching (§06): the repeated part is read from the cache at a fraction of the price. The biggest lever for chat.
• Trim: drop or shorten old turns once the conversation is long, or replace them with a short summary. Do it in rare, big steps rather than every turn, because every edit of early history breaks the cache.
• Keep tool results small: a tool that returns a 5,000-token JSON blob adds 5,000 tokens to every later call in that conversation.

Levers, roughly in order

1. Ask for less output. Tell the model the length you want, ask for IDs or JSON fields instead of prose, and set max_tokensmax_tokensA request setting that caps how many tokens the model may generate in its reply. If it's reached, the reply is cut off mid-sentence and the API says so (e.g. stop_reason: "max_tokens"). as a safety cap, not a target. If a reply hits the cap, it's cut off (stop_reason: "max_tokens") and you've paid for a useless answer, so don't set it tight. On models with built-in thinking, the thinking counts toward the cap too.
2. Cache what repeats (§06), and don't re-send what the model doesn't need: trim retrieved documents to the relevant parts (RAG).
3. Batch offline work. For work nobody is waiting on (nightly classification, backfills, evals), providers offer a Batch APIBatch APIA provider endpoint where you submit many requests at once and collect the results later (minutes to hours), instead of waiting for each one. Providers bill batch work at a discount because they can run it when they have spare capacity.. As of October 2026, Anthropic's Message Batches API processes requests asynchronously, usually within an hour and at most 24 hours, at a discounted per-token rate. Results come back in any order, so match them by your own ID.
4. Choose the model per task. Smaller models are cheaper and faster per token; larger ones handle harder tasks and may need fewer retries or follow-ups. It's a trade-off you measure with your eval set, not a ranking: compare cost per successful task, not cost per request. On models with an effort setting, lowering it is another way to trade depth for tokens.
5. Count before you send. Providers offer token-counting endpoints (Anthropic: POST /v1/messages/count_tokens) so you can reject or trim an oversized request before paying for it.

Put numbers on it

Rate limits, queues and capacity

What's limited

Every provider caps how much one account can use; exceed it and you get a Rate limitRate limitA cap on how many requests are allowed per unit of time — e.g. 100 per second. It says nothing about how many may be running at once. error (429). As of October 2026, Anthropic measures three things for each model (a few related models share one limit), each with a Token bucketToken bucketA rate-limit algorithm: a bucket refills with tokens at a steady rate (say 10 per second) up to a maximum. Each request takes one token; no token, no request. The maximum allows short bursts. that refills continuously rather than resetting at the top of the minute. Careful with the word "token" here: the bucket's tokens are just units of allowance. For RPM, each request takes one unit; for ITPM and OTPM, each LLM token takes one unit.

• RPM: requests per minute.
• ITPM: input tokens per minute. For most current models, tokens read from the prompt cache don't count toward it, only uncached input and cache writes. Caching raises your effective limit as well as cutting cost.
• OTPM: output tokens per minute, counted as tokens are actually generated. max_tokens doesn't count against it, so a generous cap costs you nothing here.

Those last two are tokens-per-minute limitsTokens-per-minute limit (ITPM / OTPM)A rate limit on how many tokens you may send (input tokens per minute) or receive (output tokens per minute), on top of the usual requests-per-minute limit. A few huge requests can hit it long before the request limit., and they surprise people: 20 requests with 50,000-token prompts each can use a million input tokens in a minute while barely touching the request limit. A limit of 60 RPM may also be enforced as roughly 1 per second, so short bursts can be rejected even when the minute's total is fine.

Responses carry headers showing where you stand, for example anthropic-ratelimit-input-tokens-remaining and anthropic-ratelimit-requests-reset (an RFC 3339 time), plus retry-after on a 429. Log them, and graph "remaining" during load tests (Rate Limiting → HTTP explains this family of headers in general).

How many calls are open at once?

Little's lawLittle's LawL = λ × W: the average number of things in a system equals the rate they arrive times how long each one stays. 100 requests/s × 0.2 s each = 20 in progress at any moment.: calls in flight = arrival rate × time each call takes. Model calls take seconds, so this number gets big fast:

Every one of those is an open connection on your server and one in-progress request at the provider. Size HTTP connection pools, memory, and your server's stream limits for the peak, not the average (the calculator in §09 does this sum).

Limit yourself before the provider does

Rather than sending everything and handling a flood of 429s, cap your own concurrency slightly below what your limits allow:

• A SemaphoreSemaphoreA counter of free slots. acquire() takes a slot (waiting if there are none left); release() gives it back. A semaphore with 10 slots lets at most 10 tasks through at once. per model, e.g. "at most 50 calls in flight" (Worker Pools → why limit shows how in Go and JS).
• A client-side token bucket that spends estimated tokens, not just requests, so a burst of huge prompts waits its turn.
• Priorities: interactive requests first; background jobs only use what's left.

Queue what can wait

Work that no user is staring at (summarizing yesterday's tickets, embedding new documents, running evals) belongs in a job queue, drained by a fixed number of workers. When the provider is slow or rate-limited, the queue gets longer instead of requests failing, and workers retry with backoff. That's BackpressureBackpressurePushing back on whoever sends you work when you're full — making them wait or rejecting them — instead of piling it up in memory.: the slowdown is absorbed in one visible place you can monitor. For very large offline jobs, the batch API (§09) is usually cheaper still.

Logging and metrics for model calls

One line per call

A structured logStructured loggingWriting log lines as key–value fields (often JSON) instead of free text — {"level":"error","order_id":42,"msg":"payment failed"} — so tools can filter and count them. line can be grouped and summed later, which is what turns "the bill doubled" into "the triage feature's output tokens doubled after Tuesday's prompt change". Log what you'll want to group by and what you'll want to add up:

type usage struct{ Input, CacheRead, CacheWrite, Output int }

func logLLMCall(log *slog.Logger, feature, userID, prompt string, u usage, ttft, total time.Duration, stop string) {
	sum := sha256.Sum256([]byte(prompt))
	log.Info("llm_call",
		"feature", feature, // ← cost per feature = sum of tokens grouped by this
		"model", "claude-opus-5-5",
		"prompt_version", "triage-v7",
		"user_id", userID, // an internal ID, never an email address
		"prompt_sha", hex.EncodeToString(sum[:])[:12], // ← groups identical prompts without storing the text
		"input_tokens", u.Input,
		"cache_read_tokens", u.CacheRead,
		"cache_write_tokens", u.CacheWrite,
		"output_tokens", u.Output,
		"ttft_ms", ttft.Milliseconds(),
		"total_ms", total.Milliseconds(),
		"stop_reason", stop, // "end_turn", "max_tokens", "refusal", …
	)
}

func main() {
	log := slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
		ReplaceAttr: func(_ []string, a slog.Attr) slog.Attr {
			if a.Key == slog.TimeKey {
				return slog.Attr{} // drop the timestamp so this demo's output is stable
			}
			return a
		},
	}))
	logLLMCall(log, "ticket-triage", "u_4821", "Ticket: my invoice is wrong…",
		usage{Input: 212, CacheRead: 747, CacheWrite: 0, Output: 96},
		840*time.Millisecond, 2910*time.Millisecond, "end_turn")
}

Both print the same line (real output; the Go file's imports are left out above):

{"level":"INFO","msg":"llm_call","feature":"ticket-triage","model":"claude-opus-5-5","prompt_version":"triage-v7","user_id":"u_4821","prompt_sha":"8a871a49595e","input_tokens":212,"cache_read_tokens":747,"cache_write_tokens":0,"output_tokens":96,"ttft_ms":840,"total_ms":2910,"stop_reason":"end_turn"}

Keep the text out of ordinary logs

Prompts and replies contain whatever users typed: names, addresses, health details, passwords pasted by mistake. That's Personal dataPersonal data (PII)Personally identifiable information: anything that points to a real person, such as a name, email, address, phone number or IP address., and logs are copied to more places with more readers than any database (Secrets → leak paths).

• Default: log numbers, not text. The line above is enough for cost, latency and error dashboards.
• If you need text for debugging or evals, store it separately: a restricted store with a short retention period and access logs, written for a sample of requests, with RedactionRedactionReplacing a sensitive value with a placeholder like [REDACTED] before it is printed, logged or sent anywhere. of obvious identifiers (emails, card numbers, phone numbers) first. Redaction by pattern misses things; treat the store as sensitive anyway.
• Never log the API key or full request headers. An x-api-key header in a debug log is a leaked key.
• Tell users what you keep, and make sure "delete my data" also reaches these stores.

The dashboard

Quality needs its own signals (thumbs up or down, user edits, escalations), which Evaluating LLM Apps → online evaluation covers.

Staff level: upgrades, providers, privacy, self-hosting

Model upgrades are deploys

Switching to a newer model changes behavior everywhere at once without a line of your code changing: output length, format details, refusals, tool-calling habits, even token counts (a new tokenizer can change your bill). Providers also retire old models on a schedule, so upgrades aren't optional; they're just best done on your schedule.

1. Pin the exact model ID in config, never a moving alias, so a change is always a reviewed config change.
2. Run the eval set on the candidate: several runs per case, compared against the current model (Evaluating LLM Apps → CI). Read a sample of outputs side by side; aggregate scores hide the weird ones.
3. Re-check the operational numbers: tokens per request, TTFT, output speed, cache behavior, rate limits for the new model.
4. Roll out gradually. Send a copy of real traffic to the new model without showing users its answers (Shadow trafficShadow traffic (dark launch)Sending a copy of real requests to a new system and comparing its answers with the old one's, while users still get the old system's answers.), then move a small share of users to it (a Canary releaseCanary releaseSending a small slice of real traffic (say 1%) to a new version first, watching its errors and latency, then increasing the share if it looks good.) behind a Feature flagFeature flagA runtime on/off switch (a config value) that decides whether new code runs, so you can ship code turned off and enable it gradually — or turn it off instantly if it breaks., watching the §11 dashboard and quality signals (Deployment strategies → canary).
5. Keep the old one switchable until the new one has survived real traffic for a while.

The same process applies to big prompt changes, since a prompt change is also a behavior change (Prompting).

Multi-provider abstraction: what it costs

An LLM gatewayLLM gatewayA layer in front of one or more model providers that gives your services one interface and handles shared concerns: keys, retries, fallbacks, rate limits, logging and cost tracking. or a thin interface over several providers promises freedom from Vendor lock-inVendor lock-inHow hard (and costly) it would be to move off a provider, because your system depends on its unique services, APIs or data formats. and a fallback for outages (§08). It's often worth it, but be clear about the price:

• Lowest common denominator. Each provider's best features (prompt caching controls, structured outputs, tool details, batch APIs, thinking settings) differ in shape. A uniform interface either hides them or grows provider-specific escape hatches.
• Prompts are not portable. The same prompt behaves differently on another model; each provider needs its own tuned and evaluated variant.
• Semantics differ. Error codes, streaming event types, token counting, rate-limit headers and cache rules all vary. Your retry and logging code must handle each one correctly.

A reasonable middle ground: one internal interface for the operations you need (generate, stream, count tokens), shared concerns in one place (keys, retries, logging, cost), and per-provider adapters that don't pretend the providers are the same.

Data retention and privacy

Before sending user data to any model API, find out (from the provider's current terms and your contract, not from blog posts) what happens to it. As of October 2026, the questions to ask are broadly the same everywhere:

• RetentionData retention (model APIs)How long a provider keeps the prompts and replies you send it, and for what (abuse monitoring, debugging, features like batches). It is set by the provider's terms and your contract, so check it before sending personal data.: how long are prompts and outputs stored, and why (abuse monitoring, debugging, features that keep results like batches or file uploads)? For example, Anthropic's Message Batches results stay downloadable for 29 days.
• Training: is API data used to train models by default? Many providers say no for API traffic, but check.
• Zero data retentionZero data retention (ZDR)An arrangement where the provider does not store your prompts and replies after returning the response, except where required by law or for abuse checks. Usually a contract option, and some features don't work under it. and region options: available? On which contract? Which features stop working under them? (Some require a minimum retention period. For example, as of October 2026, Anthropic documents that Claude Fable 5.1 requires 30-day data retention and isn't available under zero data retention unless Anthropic authorizes it, and that Message Batches aren't eligible for zero data retention.)
• Your side: what you log (§11), where it goes, and how deletion requests reach it. Usually the larger risk is in your own logs, not at the provider.

Send the minimum: strip fields the model doesn't need, replace identifiers with placeholders where the task allows, and keep secrets out of prompts entirely. AI Security covers prompt injection and data leaks through the model itself.

Self-hosting vs an API

Self-hostingSelf-hosting a modelRunning a model whose weights you can download on your own GPUs (or rented ones), instead of calling a provider's API. You control the data and the version, and you take on the hardware, scaling and upgrades. an open-weight model gives you full control over data, version and availability, and no per-token bill. In return, you take on what the provider did for you:

Rules of thumb that last: APIs win for spiky or modest traffic, for access to the most capable models, and for small teams. Self-hosting starts to make sense with strict data-residency needs, steady very high volume of a task a smaller model handles well, or a need for a frozen model. Either way, the techniques on this page (streaming, caching, retries, budgets, logging, evals) still apply: they're properties of the workload, not of who runs the GPU.

Check yourself

Predict the answer before you click. Q1 is covered in §02, Q2 in §07, Q3 in §06, Q4 in §09, Q5 in §05, Q6 in §04 and Q7 in §10.

// Called for every request, a few seconds apart.
const system = [{
  type: "text",
  text: `Today is ${new Date().toISOString()}.\n` + RULES, // RULES: 3,000 tokens
  cache_control: { type: "ephemeral" },
}];

Cheat sheet