The KV Cache by Hand

What the K and V Matrices Look Like at Token 1, Token 2, Token 3. Until Now. With the Arithmetic.

🔓 Free link for non-members — click here to read without a paywall

TL;DR

Every time you generate a token with an LLM, the model runs attention.

Attention requires K and V vectors for every token that came before.

Without a cache: recompute every K and V vector from scratch at every step. For token 1000, that is 999 redundant matrix multiplications.

With the KV cache: store the K and V vectors the first time. Never recompute them.

That storage is not free.

For GPT-3 at 2048-token context: 800 GB of cache. Per request.

For Llama-2–7B at 4096 tokens: 2 GB. Per request.

For GPT-4 at 128K context: roughly 128 GB. Per request.

This is why inference is expensive. Not the model weights. The cache.

This article shows you — by hand, with the same toy numbers from the hand-tracing series — exactly what gets cached, what gets reused, and how to derive the memory formula from first principles.