The Cache Is the Model

A model with 8 billion parameters can be served on one GPU. The same model with 64 concurrent users at 32K context can run out of memory. The weights did not change. The cache did.

What is actually stored

Attention needs keys and values for every position that a new token can attend to. Rather than recompute them, the serving engine stores them. For each layer, each KV head, each token position, it keeps a key vector and a value vector of size $d_h$ .

The cache size per token is:

M_{\text{kv per token}} = L \times 2 \times H_{kv} \times d_h \times b.

Where:

$L$ = number of layers,
the 2 is for keys and values,
$H_{kv}$ = number of KV heads,
$d_h$ = head dimension,
$b$ = bytes per element (2 for BF16, 1 for FP8, 0.5 for INT4).

For total cache memory, multiply by the number of cached tokens per request and the number of concurrent requests:

M_{\text{kv total}} = M_{\text{kv per token}} \times T_{\text{context}} \times C.

This linear relationship is the whole story. Doubling context doubles memory. Doubling concurrency doubles memory. Quantizing the cache to FP8 halves it. Using fewer KV heads (GQA/MQA) divides it by the group ratio. Compressing to a latent (MLA) can shrink it further.

Figure 4.1KV cache growth

An 8B GQA model in BF16 uses about 128 KiB per token. At 32K context and 16 concurrent requests, the cache alone is 64 GiB — larger than the weights.

A worked table

Take four common architectures, all in BF16. The per-token cache assumes the stated number of layers, KV heads and head dimension.

Model	Params	Layers	KV heads	Head dim	KV / token	8K request	32K request
Llama 3.1 8B	8B	32	8	128	128 KiB	1 GiB	4 GiB
Mistral 7B	7B	32	8	128	128 KiB	1 GiB	4 GiB
Llama 3.3 70B	70B	80	8	128	320 KiB	2.5 GiB	10 GiB
DeepSeek-V3	671B total / 37B active	61	MLA	—	~70 KiB (approx.)	—	~2.2 GiB

The DeepSeek row is deliberately rough: MLA does not store conventional per-head K/V tensors. The number is an approximation for intuition, not an implementation spec. The lesson is that architecture choices can change the cache by an order of magnitude.

Now add concurrency. A 70B model on two H200s has about 282 GB of HBM. BF16 weights take ~141 GB, leaving ~141 GB for cache and workspace. At 10 GiB per 32K request, that is roughly 14 requests — before overhead, fragmentation and any safety margin. If your product needs 100 concurrent 32K conversations, a naive deployment will not fit, no matter how fast the GPU is.

Why fragmentation matters

A naive allocator reserves a contiguous block for the maximum context length of each request. If a request is capped at 32K tokens but only uses 3K, most of that reservation is wasted. If requests arrive with variable lengths, the allocator leaves holes. Over time, free memory is fragmented: there is enough total memory, but not enough contiguous space for the next long request.

PagedAttention borrows the virtual-memory playbook. The KV cache is split into fixed-size blocks (for example, 16 tokens each). Each request has a block table mapping logical blocks to physical blocks. Blocks do not need to be contiguous. The allocator can fill holes, share blocks between requests with a common prefix, and free blocks individually.

Figure 4.2Paged KV cache

A logical sequence maps through a block table to physical KV blocks. Shared prefixes can point at the same physical blocks; copy-on-write handles divergence.

This does three things:

Reduces internal fragmentation. A request only pays for blocks it actually uses, plus at most one partially filled block.
Enables prefix sharing. Two requests with the same system prompt can point at the same physical blocks. Reference counting decides when a block can be freed.
Enables preemption. Under memory pressure, the scheduler can swap or recompute a request’s cache instead of failing the whole batch.

The block size is a tradeoff. Larger blocks reduce table overhead but waste more space at the tail of each sequence. Smaller blocks are more flexible but add indirection and metadata. Most engines use 16-token blocks as a default; the right value depends on sequence-length distribution and kernel support.

Prefix caching: the free lunch with conditions

Many workloads repeat the same prefix: a long system prompt, a set of tool definitions, a retrieved document, a few-shot example set. If the engine can detect that a new request shares a prefix with a cached one, it can reuse the KV blocks instead of recomputing prefill.

The savings can be dramatic. A 4,000-token system prompt that is identical across requests costs 4,000 tokens of prefill once, not once per request. TTFT drops, compute drops, and the cache becomes a shared asset.

But prefix caching is not automatic magic:

The prefix must match exactly, including whitespace and special tokens.
The cache must have room; long unique prompts evict shared prefixes.
Cache hits interact with scheduling: a request that could reuse a prefix may still wait behind a long prefill.
Multi-tenant isolation matters. Sharing physical blocks across tenants can leak information if the mapping is observable. Engines handle this with per-tenant namespaces and careful eviction.

SGLang’s RadixAttention is a well-known design that organizes prefix reuse around a radix tree. vLLM exposes automatic prefix caching as a configurable feature. The implementation differs, but the principle is the same: repeated prefill is wasted work.

Cache quantization and compression

The cache is a memory-bandwidth and capacity consumer. Reducing its precision is attractive:

FP8 KV cache halves the bytes with modest quality impact on many models. It requires kernels that support FP8 attention and careful scaling.
INT8 KV cache is similar in size but uses integer quantization; quality depends on calibration.
MLA compresses keys and values into a latent vector, reducing storage at the architecture level.
Eviction and sparsity methods keep only the tokens deemed important. They can help long-context workloads, but they change what the model can attend to and are not a free win.

Quantizing the cache affects both memory and bandwidth. In decode, reading 64 GiB of cache instead of 128 GiB can nearly double the token rate if bandwidth is the limit. But the attention kernel may be slower at low precision, and quality regressions can be subtle. Measure on your own prompts.

The concurrency equation

A useful capacity-planning identity:

C_{\max} \approx \frac{M_{\text{usable}} - M_{\text{weights}} - M_{\text{workspace}}}{M_{\text{kv per token}} \times T_{\text{context}}}.

For an 80 GB H100 at 90% usable memory (72 GB), an 8B BF16 model (16 GB), 4 GB of workspace, and 128 KiB per token at 8K context:

C_{\max} \approx \frac{72 - 16 - 4}{0.000128 \times 8192} \approx 49.

That is an optimistic ceiling: it ignores fragmentation, scheduler headroom, and the fact that not every request uses the full context. Real concurrency will be lower, especially with long-tail prompts. But the identity tells you which lever matters. If you need 200 concurrent requests, you need a smaller cache per token, a shorter context, more memory, or a different model.

What to watch in production

KV-cache occupancy as a fraction of allocated blocks. A server at 95% occupancy is one long prompt away from preemption.
Preemption and recompute counts. These are direct signals of memory pressure.
Cache hit rate for prefix caching, split by tenant and route.
TTFT by prompt length bucket. A single average hides the long-prompt tail.
Tokens per request vs. context limit. If most requests use 5% of the window, you are paying for headroom you rarely use.

Next: batching and SLOs — how a scheduler turns spare memory into throughput without destroying the latency users feel.