research on Saurabh Kumar

How Prompt Caching Works

Sun, 21 Jun 2026 00:00:00 +0000

Today I learned how prompt caching actually works under the hood in LLM APIs (Anthropic, OpenAI, Bedrock, etc.), and why it makes long agent loops 5–10× cheaper and noticeably faster.

The problem it solves

In an agent / chat loop, every request to the model looks almost identical to the previous one:

The same big system prompt (rules, tool descriptions, skills…)
The same set of tool definitions
The same history of past user/assistant/tool messages, with only a few new tokens appended at the end

Without caching, the provider has to re-tokenize and run a full attention pre-fill over all of that prefix on every turn. For a 100k-token context, that prefill dominates both cost and latency.

What gets cached: the KV state

When a transformer processes a prompt, it computes a key/value tensor pair for every token in every attention layer. That KV state is what attention queries against on the next decoded token.

Prompt caching is, mechanically, “save those KV tensors for a prefix on the server, and reuse them next time the same prefix shows up.” Output tokens are never cached — only the prompt-side KV state.

The lifecycle

Mark a cache breakpoint. The client places a cache_control marker (Anthropic) or relies on automatic prefix detection (OpenAI). Everything before the marker is the cacheable prefix.
First request → cache write. The provider hashes the cacheable prefix (model id + tools + messages + cache key, byte-exact), runs the prefill, and stores the resulting KV tensors keyed by that hash. You’re billed cache_write tokens, usually a bit more than normal input (~1.25× for Anthropic 5-minute cache, ~2× for the 1-hour “extended” cache).
Next request → cache read. If the new request starts with exactly the same prefix, the provider looks up the hash, loads the stored KV state, and only runs fresh compute for the suffix after the last hit. You’re billed cache_read tokens — often ~10% of the normal input price.
TTL. Entries expire after a TTL (Anthropic: 5 min default, 1 hour opt-in; OpenAI: a few minutes, automatic). Each hit typically refreshes the TTL, so an active session keeps the cache warm.

What invalidates the cache

The hash is over the exact bytes of the prefix. Anything that changes those bytes forces a fresh cache_write:

Editing the system prompt (even reordering sentences or adjusting whitespace)
Adding, removing, or reordering a tool definition
Inserting/editing/reordering an earlier message in the history
Switching model id or provider
Changing the cache key (some providers expose one for tenant isolation)

The corollary: to maximize hit rate, agents should only append at the end and keep system prompt and tools byte-stable across turns.

Reading the usage block

Most providers return a usage object on every response with at least these fields:

{
 "input": 312, // fresh, non-cached prompt tokens this turn
 "output": 845, // generated tokens
 "cacheRead": 47210, // prompt tokens served from cache
 "cacheWrite": 0 // prompt tokens written to cache this turn
}

A useful per-turn metric is the cache hit ratio:

cacheHitRate = cacheRead / (input + cacheRead + cacheWrite)

0% → nothing reused. Normal on the very first turn, or right after the prefix changed (tool list updated, history compacted, model swapped).
70–95% → healthy steady state for a long agent loop.
cacheWrite > 0, cacheRead = 0 → you just primed the cache; the next turn should be cheap if you don’t break the prefix.

Pricing back-of-the-envelope

Rough Anthropic numbers (Claude Sonnet, illustrative):

Token type	Price multiplier vs. input
`input`	1×
`cache_write`	~1.25× (5 min) / ~2× (1h)
`cache_read`	~0.1×
`output`	~5×

So a 100k-token prefix that’s hit 20 times across a session costs roughly:

1 × 1.25 × 100k (one write)
+ 19 × 0.1 × 100k (nineteen reads)
= 1.25 × 100k + 1.9 × 100k
≈ 3.15 × 100k input-equivalent

vs. uncached:
20 × 1 × 100k = 20 × 100k

A ~6× saving on the prompt side, plus a big latency win because the server skips the prefill.

Gotchas

Minimum prefix size. Most providers require the cacheable prefix to be above some threshold (e.g. 1024 tokens for Anthropic) before caching kicks in at all.
Position of the breakpoint matters. Putting it after volatile content (like the latest user message) means nothing ever gets reused.
Tool definitions are part of the prefix. Dynamically generated tool lists with shuffled order will silently destroy your hit rate.
Streaming vs. non-streaming use the same cache; cache fields show up in the final usage event.
Cost accounting: cache_write is more expensive than input, so a flaky prefix that constantly re-writes can be worse than no caching at all.

Notes on Understanding Hebbian Learning

Wed, 05 Nov 2025 00:00:00 +0000

Hebbian learning describes how correlated neural activity can strengthen connections. It is named after psychologist Donald Hebb, who proposed it in 1949.

The entire concept boils down to one beautiful phrase:

“Neurons that fire together, wire together.”

That is a useful shorthand, not a complete account of learning. Timing and the synapses involved matter.

Breaking It Down

Repeated, correlated activity can strengthen the connection between two neurons. The effect depends on timing and the synapse, rather than simultaneous activity always producing the same change.

Think of it like this:

First time: “Oh, these two things happened together”
Fifth time: “Okay, these definitely seem related”
Hundredth time: “These are basically the same signal now”

Real Examples That Make It Click

Learning to Type

Remember learning to type? At first, finding the ‘Q’ key required conscious thought. Your brain neurons for “want to type Q” and “move left pinky to top left” fired together… over and over. Now? It’s automatic. Those neurons are strongly wired.

Pavlov’s Dogs (The Classic)

Bell rings → food appears → dogs salivate

Do this enough times, and the neurons for “bell sound” and “food is coming” become so tightly linked that just the bell makes dogs salivate. No food needed.

Your Morning Coffee Ritual

Why does the smell of coffee make you feel more awake even before you drink it? Neurons fired together thousands of times:

Smell of coffee + feeling of alertness
Sound of coffee maker + anticipation of energy
Taste of first sip + actual caffeine effect

Now they’re all wired together into one mega-circuit.

The Actual Mechanism (Simplified)

What’s physically happening:

Neuron A fires
Neuron B fires (around the same time)
The synapse (connection) between them releases more receptors
The connection physically strengthens
Next time A fires, it’s easier to activate B

It’s literally changing the structure of your brain. Learning is physical.

Why This Matters

For Understanding Yourself

Habits: They form because neurons wire together through repetition
Memory: Retrieving memories strengthens those neural pathways
Skills: Practice literally builds stronger brain connections
Trauma: Unfortunately, negative associations work the same way

For AI and Machine Learning

Hebbian learning inspired artificial neural networks. Modern AI doesn’t use pure Hebbian learning anymore (it’s evolved into more sophisticated algorithms), but it started here.

The basic idea - connections that prove useful should get stronger - is still foundational.

The Flip Side: “Use It or Lose It”

Here’s the thing nobody tells you: it works in reverse too.

Neurons that STOP firing together will gradually UNWIRE.

This is why:

You forget languages you don’t practice
Old skills get rusty
Memories fade
Bad habits can actually be broken

The Math (Optional)

For the nerds: The basic Hebbian rule can be expressed as:

Δw = η × x × y

Where:

Δw = change in connection strength
η = learning rate
x = activity of first neuron
y = activity of second neuron

If both are active (positive values), the connection strengthens. Simple, elegant, powerful.

Practical Takeaways

For Learning Anything:

Repetition isn’t just helpful - it’s literally how learning works
Practice needs to be consistent (neurons need to fire together repeatedly)
Spacing matters (distributed practice > cramming)
Context matters (learning in similar contexts strengthens related neural patterns)

For Breaking Bad Habits:

You need to stop the neurons from firing together
Replace the pattern with a new one (create competing neural pathways)
It takes time - you’re literally rewiring your brain

For Building Skills:

Deliberate practice works because it forces specific neurons to fire together
Feedback loops help strengthen the right connections
Muscle memory is real - it’s Hebbian learning in motor cortex

Cool Extensions of This Idea

Spike-timing-dependent plasticity (STDP): Order matters too. In this simplified account, neuron A firing before neuron B (within ~20ms) can strengthen the connection, while the reverse order can weaken it. The timing and effect depend on the synapse.

Competitive learning: When neurons wire together, they can also suppress other connections. This is how your brain specializes - certain neuron clusters become experts at specific patterns.

Associative memory: Hebbian associations offer a way to think about connections between faces, names, and songs. They are not a complete explanation of memory.

If you want to dive deeper:

Donald Hebb’s original work: “The Organization of Behavior” (1949)
Modern neuroscience textbooks on synaptic plasticity
Research on long-term potentiation (LTP) - the molecular mechanism behind Hebbian learning
AI/ML papers on Hebbian learning algorithms