# How Prompt Caching Works

By Saurabh Kumar · Jun 21, 2026 · Tags: llm, performance

Source: https://saurabh-kumar.com/articles/2026/06/how-prompt-caching-works/

> What LLM APIs cache, what breaks the cache, and why long agent loops get 5–10× cheaper with it.

Today I learned how prompt caching actually works under the hood in LLM APIs
(Anthropic, OpenAI, Bedrock, etc.), and why it makes long agent loops 5–10×
cheaper and noticeably faster.

## The problem it solves

In an agent / chat loop, every request to the model looks almost identical to
the previous one:

- The same big **system prompt** (rules, tool descriptions, skills…)
- The same set of **tool definitions**
- The same **history** of past user/assistant/tool messages, with only a few
  new tokens appended at the end

Without caching, the provider has to re-tokenize and run a full attention
pre-fill over all of that prefix on every turn. For a 100k-token context, that
prefill dominates both cost and latency.

## What gets cached: the KV state

When a transformer processes a prompt, it computes a **key/value tensor pair
for every token in every attention layer**. That KV state is what attention
queries against on the next decoded token.

Prompt caching is, mechanically, "save those KV tensors for a prefix on the
server, and reuse them next time the same prefix shows up." Output tokens are
never cached — only the prompt-side KV state.

## The lifecycle

1. **Mark a cache breakpoint.** The client places a `cache_control` marker
   (Anthropic) or relies on automatic prefix detection (OpenAI). Everything
   *before* the marker is the cacheable prefix.
2. **First request → cache write.** The provider hashes the cacheable prefix
   (model id + tools + messages + cache key, byte-exact), runs the prefill,
   and stores the resulting KV tensors keyed by that hash. You're billed
   `cache_write` tokens, usually a bit *more* than normal input
   (~1.25× for Anthropic 5-minute cache, ~2× for the 1-hour "extended"
   cache).
3. **Next request → cache read.** If the new request starts with exactly the
   same prefix, the provider looks up the hash, loads the stored KV state,
   and only runs fresh compute for the suffix after the last hit. You're
   billed `cache_read` tokens — often **~10% of the normal input price**.
4. **TTL.** Entries expire after a TTL (Anthropic: 5 min default, 1 hour
   opt-in; OpenAI: a few minutes, automatic). Each hit typically refreshes
   the TTL, so an active session keeps the cache warm.

## What invalidates the cache

The hash is over the **exact bytes** of the prefix. Anything that changes
those bytes forces a fresh `cache_write`:

- Editing the system prompt (even reordering sentences or adjusting
  whitespace)
- Adding, removing, or reordering a tool definition
- Inserting/editing/reordering an earlier message in the history
- Switching model id or provider
- Changing the cache key (some providers expose one for tenant isolation)

The corollary: to maximize hit rate, agents should **only append at the end**
and keep system prompt and tools byte-stable across turns.

**Figure:** The earlier the edit in the prompt, the more blocks are written to the cache again; appending rewrites only the new turn. Blocks are conceptual, not token counts. The new turn is always written once.


## Reading the usage block

Most providers return a usage object on every response with at least these
fields:

```json
{
  "input": 312,          // fresh, non-cached prompt tokens this turn
  "output": 845,         // generated tokens
  "cacheRead": 47210,    // prompt tokens served from cache
  "cacheWrite": 0        // prompt tokens written to cache this turn
}
```

A useful per-turn metric is the **cache hit ratio**:

```
cacheHitRate = cacheRead / (input + cacheRead + cacheWrite)
```

- `0%` → nothing reused. Normal on the very first turn, or right after the
  prefix changed (tool list updated, history compacted, model swapped).
- `70–95%` → healthy steady state for a long agent loop.
- `cacheWrite > 0, cacheRead = 0` → you just primed the cache; the next turn
  should be cheap if you don't break the prefix.

## Pricing back-of-the-envelope

Rough Anthropic numbers (Claude Sonnet, illustrative):

| Token type     | Price multiplier vs. input |
|----------------|---------------------------|
| `input`        | 1×                        |
| `cache_write`  | ~1.25× (5 min) / ~2× (1h) |
| `cache_read`   | ~0.1×                     |
| `output`       | ~5×                       |

So a 100k-token prefix that's hit 20 times across a session costs roughly:

```
1 × 1.25 × 100k   (one write)
+ 19 × 0.1 × 100k (nineteen reads)
= 1.25 × 100k + 1.9 × 100k
≈ 3.15 × 100k input-equivalent

vs. uncached:
20 × 1 × 100k = 20 × 100k
```

A ~6× saving on the prompt side, plus a big latency win because the server
skips the prefill.

## Gotchas

- **Minimum prefix size.** Most providers require the cacheable prefix to be
  above some threshold (e.g. 1024 tokens for Anthropic) before caching kicks
  in at all.
- **Position of the breakpoint matters.** Putting it after volatile content
  (like the latest user message) means nothing ever gets reused.
- **Tool definitions are part of the prefix.** Dynamically generated tool
  lists with shuffled order will silently destroy your hit rate.
- **Streaming vs. non-streaming** use the same cache; cache fields show up
  in the final usage event.
- **Cost accounting:** `cache_write` is more expensive than `input`, so a
  flaky prefix that constantly re-writes can be *worse* than no caching at
  all.