Part I · Foundations
The KV cache, and why decoding would crawl without it
A language model writes one token at a time, and each new token has to look back over everything already written. That backward glance is where almost all the cost of generating text hides. The KV cache is the single idea that keeps it from spiralling out of control.
What gets cached & why
Inside attention, each token becomes three vectors: a query, a key, and a value. To pick the next word, the model takes the newest token's query, checks it against the key of every earlier token to decide how much each one matters, then blends their values into the answer.
When the model generates token n, it attends over every token before it. Recomputing their keys and values at every step is the O(n²) trap. The KV cache stores each token's key and value once, so a new token only projects its own K and V and reads the rest straight from memory.
See it · one decode step, cache on vs. off
Interactive · KV cacheCache ON
Sequence
AGPUserves
KV cache
3 tokens · 3 cachedKV
reusedwrittenrecomputed
Compute this step
3 tokens prefilled
O(n): each new token adds just one K/V row.
Turn the cache off and every earlier token lights up amber again: the model repeats work it already finished. Turn it on and only the newest row is written each step.
Memory-cost math
That speed is bought with memory. Every token adds one key and one value to the cache, in every attention head of every layer, and nothing is discarded mid-generation, so the footprint is completely predictable, and easy to underestimate.
Cache size
kv_bytes = 2 · layers · heads · d_head · seq_len · batch · dtype
Try the numbers · cache sizeFP16FP8INT4
≈ 21 GB
70B-class · 64K context · batch 1 · FP16
Still smaller than the model. Push context or batch higher to see it flip.
Fixed: 80 layers · 8 KV heads · head-dim 128.
Cache layout
Physically, the cache is two tensors per layer, a K cache and a V cache, indexed by attention head and by position. A new token writes one slot per head, per layer, then reads the whole column back to attend over the past.
Prefill · once
Read the whole prompt
Every prompt token is processed together in a single pass, and its key and value are written into the cache up front.
Decode · every step
Add one token at a time
Each new token computes only its own K and V, appends one row to the cache, and attends over everything stored so far.
Eviction & paging
In a plain decode loop nothing is ever evicted, so the cache only grows. That is fine for a short chat. Push the context long enough and the bottleneck flips: the cache, not the model, is what runs you out of memory.
Real systems refuse to just let it grow. They hand out cache memory in fixed blocks so it does not fragment (paging), and they drop, compress, or offload the entries that matter least (eviction).