KV-Cache Engineering: The Memory Wall of LLM Serving

The first thing to unlearn about LLM serving is that the model weights are the memory problem. They are not. The weights are a fixed cost you pay once when the model loads, and quantization has made them cheap to hold. The thing that actually decides how many users you can serve, how long their contexts can be, and how fast their tokens stream is the KV cache, the per-token state that attention keeps so it does not recompute the past on every step. At long context the KV cache consumes 70 to 90% of GPU memory and accounts for the majority of wall-clock time, which makes it the single largest cost lever in production inference today. A serving stack is, to a first approximation, a KV cache management system with a model attached.

This is the systems counterpart to the inference optimization overview: not how one request runs, but how a fleet of them shares a GPU. The whole discipline is moving the memory wall, and there are exactly five levers that do it.

Why the cache explodes

Recall the mechanism. To generate token N, attention needs the keys and values of tokens 1 through N-1. Recomputing them every step would be quadratic, so you cache them: the KV cache. It works, and then it grows, because its size scales with everything you want more of.

The size of the cache is concrete arithmetic:

KV bytes = 2 (K and V) × layers × kv_heads × head_dim × seq_len × bytes_per_elem × batch

Run it for a mid-sized model with 64 layers, 8 KV heads (already using grouped-query attention), a head dimension of 128, in FP16. Per token that is 2 × 64 × 8 × 128 × 2 bytes, about 256 KB. A single 128K-token context then needs roughly 32 GB of KV cache, for one sequence. That is more memory than the quantized weights of many models. Now multiply by the batch size you need for throughput, and the problem is obvious: the weights are fixed, but the cache is weights-scale memory per concurrent long-context user. The cache, not the model, sets your concurrency ceiling.

The consequence is a wall. As context lengths climbed past 32K and batch sizes climbed for throughput, KV cache went from a footnote to the dominant term, and every serving advance of the last two years has been an attack on it from a different angle.

Two costs: concurrency and decode speed

The cache hurts in two ways, and conflating them leads to the wrong fix.

The first is concurrency. Take an 80GB GPU serving a model whose quantized weights occupy about 24GB, leaving roughly 52GB for KV cache after runtime overhead. At 256 KB per token, that is room for about 200,000 token-slots of cache. You can spend that budget as one user at a 200K context, or about 25 users at 8K each, or over 1,500 users at a few hundred tokens each. That single number is your concurrency ceiling, and it is set entirely by the cache, not the weights. Double the per-token cache, with twice the KV heads or FP16 instead of FP8, and you halve every one of those figures.

The second is decode speed, and it is the one people miss. Because decode is memory-bandwidth bound, every generated token reads not just the weights but the entire KV cache for that sequence. A longer context is therefore slower per token, not merely heavier in memory, which is the latency dimension of the advertised-versus-effective context gap: a model does not just get less accurate deep into a long context, it gets slower there too. This is why cache size and cache compression matter twice. Shrinking the cache raises concurrency and speeds up decode at the same time, which makes it the rare lever that improves capacity and latency together rather than trading one for the other.

The five levers

Everything so far has been diagnosis. These five techniques are the treatment, and they compose rather than compete.

Lever 1: PagedAttention, the memory substrate

The first attack was on waste. Naive KV cache implementations reserve a contiguous block of memory per sequence, sized for the maximum possible length, because you do not know in advance how many tokens a request will generate. That over-allocation is internal fragmentation, and the scattered gaps left between sequences as they start and finish are external fragmentation. Together they wasted 60 to 80% of KV memory in pre-2023 systems, which directly capped how many sequences a server could hold.

PagedAttention, the algorithm behind vLLM, borrowed the fix from operating systems: paging. Instead of one contiguous allocation per sequence, the cache is split into fixed-size blocks that can live anywhere in memory, with a block table mapping each sequence's logical positions to physical blocks, exactly like virtual memory maps pages. Internal fragmentation is bounded by one block, external fragmentation disappears, and waste drops under 4%. The reported result was up to 24 times higher throughput on the same hardware, because the reclaimed memory becomes batch capacity. PagedAttention is not an optimization you turn on; it is the substrate modern serving is built on, and the reason vLLM became the default open serving engine. The design is not unchallenged: later work like vAttention argues that paging adds software indirection and that CUDA's own virtual-memory primitives can deliver contiguous-looking cache without a block table. But PagedAttention's model won the ecosystem, and it remains both the default implementation and the right mental model for what follows.

Paging also unlocked the next lever, because once the cache is blocks with a mapping table, blocks can be shared.

Lever 2: Prefix caching, computing the suffix only

Most production traffic is not unique. A chat app sends the same system prompt on every request. A few-shot pipeline sends the same exemplars. A RAG system sends the same retrieved chunk to many users. An agent loop resends a growing conversation that shares its whole history with the previous turn. In all of these, a large prefix of the input is identical to something the server already processed, and recomputing its KV cache is pure waste.

Prefix caching eliminates that waste. The server identifies the longest common prefix between a new request and cached sequences, and on a hit it loads the stored KV tensors and computes new ones only for the suffix. vLLM calls this automatic prefix caching; SGLang generalizes it as RadixAttention, organizing cached prefixes in a radix tree so overlapping prefixes share storage and lookups are fast. The payoff is largest exactly where it is needed: a long shared system prompt or RAG context that would dominate prefill is computed once and reused across every request that shares it, cutting time-to-first-token and prefill cost dramatically. Concretely, a support bot with a 2,000-token system prompt serving 10,000 requests a day recomputes that prefix 10,000 times without caching and once with it, turning 20 million prefill tokens into 2,000, and the savings land on the most expensive phase (prefill) and the most latency-sensitive metric (time-to-first-token). For any workload with repeated prefixes, which is most of them, this is among the highest-return changes you can make.

Lever 3: Architectural compression, fewer bytes per token

The first two levers manage the cache without changing its size. The third shrinks the size at the source by changing attention itself, and it is where the cache formula's kv_heads term gets attacked.

The progression is a steady trade of KV footprint against a little quality or complexity:

• Multi-query attention (MQA) shares a single key-value head across all query heads, slashing the cache but costing quality.
• Grouped-query attention (GQA) is the production compromise now used almost everywhere: a small number of KV heads shared across groups of query heads, typically a 4 to 8 times cache reduction with negligible quality loss. The 8-KV-head model in the arithmetic above is already a GQA model.
• Multi-head latent attention (MLA), introduced by DeepSeek, goes further by storing a low-rank latent projection of the keys and values instead of the full tensors, reconstructing them on the fly. It reaches roughly 7 to 14 times compression, and it is a large part of why DeepSeek's V4 line can serve million-token contexts economically while GQA-based competitors carry several times the cache.

This lever is chosen at model-design time, not serving time, so it is not something you toggle. But it is why model choice is a serving decision: an MLA model and a GQA model with the same parameter count have very different concurrency profiles at long context, and the difference is entirely in the cache.

Lever 4: Quantizing the cache

The KV cache is activations, so it can be quantized like any other tensor, and it is one of the highest-value places to do it because it is so large. Storing the cache in FP8 instead of FP16 roughly doubles how much context and how many sequences fit, at minimal accuracy cost, and INT8 and newer rotation-based, data-free methods push further. This is the same machinery covered in the quantization deep dive, pointed at the cache specifically, and the caveat is the same: error in the cache accumulates over long contexts, so validate on your longest sequences rather than trusting an average. For most deployments, an FP8 cache is close to free capacity and a sensible default.

Lever 5: Offloading, when it still will not fit

When the cache exceeds VRAM even after compression, the last resort is to move it off the GPU: to CPU RAM, to NVMe, or to a peer GPU. Systems now offload on the order of gigabytes of KV per GPU to host memory, and disk-aware schemes extend on-device long-context inference past what RAM allows. The cost is the one this whole stack keeps running into: bandwidth. Fetching a cache block back from CPU memory or disk is far slower than reading it from VRAM, so offloading trades latency for capacity and only pays when the alternative is not serving the request at all. It is a capacity lever, not a speed one, and treating it as the latter is how a system that "supports" long context becomes unusably slow on it.

The architecture shift: disaggregating prefill and decode

The newest lever is structural, and it reframes the whole serving problem around the cache. Generation has two phases with opposite hardware appetites. Prefill, processing the prompt, is compute-bound: it crunches the whole input in parallel and saturates the GPU's math units. Decode, generating tokens one at a time, is memory-bandwidth-bound: it streams the weights and the growing KV cache for each token and barely touches the math units. Run both on the same GPU, as traditional serving does, and they interfere: a long prefill stalls the decodes of other users, and SLOs blow up under load.

Disaggregated serving splits them onto separate worker pools. DistServe (OSDI 2024) showed the payoff: serving 7.4 times more requests, or meeting 12.6 times tighter latency targets, on the same hardware budget, by letting prefill and decode scale independently and stop fighting. The catch is that the KV cache computed during prefill must now travel to the decode worker, which is why the leading systems are KV-cache-centric by design. Mooncake, the platform behind Moonshot's Kimi and a FAST 2025 best-paper, builds the entire architecture around a disaggregated KV cache store and transfer engine, and it now interoperates with LMCache and vLLM. The 2025 inflection toward this design was economic, not technical: the papers existed in 2024, but it took products at millions-of-users scale, where every SLO violation is a measurable revenue event, to justify the engineering.

When the cache fills: eviction and scheduling

Everything above assumes the cache fits. Under real load it does not always, and what the serving engine does at that boundary decides whether latency stays smooth or falls off a cliff. When KV memory is exhausted, the scheduler has three moves, each a latency tradeoff: stop admitting new requests (queueing, which protects the running ones but grows wait time), evict cached prefix blocks by least-recently-used (cheap until you evict a hot shared prefix and pay to recompute it), or preempt an in-flight sequence by swapping its cache to host memory or dropping and recomputing it later.

The policy interacts with prefix caching in a way that bites if you ignore it: an eviction policy blind to which prefixes are hot will throw away exactly the cache that was saving the most work, turning a high cache-hit workload into a thrashing one. This is why production serving is a scheduling problem as much as a memory one, and why mature engines expose knobs for cache reservation, admission control, and preemption mode rather than treating the cache as an infinite pool. If your tail latency degrades sharply only under load, the cause is usually here.

How to choose your levers

These compose, and the right subset depends on what is breaking.

• Low concurrency, fragmentation waste: PagedAttention. This is table stakes; use a serving engine built on it.
• Repeated system prompts, few-shot, or RAG context: prefix caching. Highest return for the least effort on most real traffic.
• Long context that strains memory: an MLA-class model if you are choosing one, plus an FP8 cache, plus offloading as the capacity backstop.
• Latency SLOs under heavy, mixed load: disaggregated prefill and decode, once you are at the scale where prefill interference is measurably costing you.

The through-line is that serving cost in 2026 is KV cache cost. The weights are a solved, fixed expense; the cache is the variable that scales with your users and your context lengths, and engineering it is most of what separates a serving stack that is cheap and fast at scale from one that falls over at the first long-context spike. If you profile one thing in your inference deployment, profile the cache.

Key Takeaways

1. The KV cache, not the weights, is the serving bottleneck. At long context it is 70 to 90% of GPU memory and the majority of wall-clock time. Weights are a fixed one-time cost; the cache scales with batch and context.
2. The cache is weights-scale memory per long-context user. A single 128K context can need ~32 GB of FP16 KV cache, more than many models' weights. Concurrency is capped by the cache, not the model.
3. PagedAttention is the substrate. Paging the cache like OS virtual memory cut waste from 60-80% to under 4% and unlocked up to 24x throughput. Build on an engine that has it.
4. Prefix caching computes only the suffix. Reusing the KV of shared prefixes (system prompts, few-shot, RAG, multi-turn) via vLLM APC or SGLang RadixAttention is the highest-return change for typical repeated-prefix traffic.
5. Architecture sets the per-token size. MQA, then GQA (4-8x reduction, now standard), then DeepSeek's MLA (7-14x via low-rank latent KV). An MLA model and a GQA model have very different concurrency at long context, so model choice is a serving decision.
6. Quantize the cache. FP8 KV roughly doubles capacity at minimal cost and is a sensible default, but cache error accumulates over long contexts, so validate on your longest sequences.
7. Offloading is a capacity lever, not a speed one. Moving the cache to CPU, NVMe, or a peer GPU buys room at the cost of bandwidth. Use it when the alternative is not serving the request, not to go faster.
8. Disaggregate prefill and decode at scale. Prefill is compute-bound, decode is bandwidth-bound; splitting them onto separate workers served 7.4x more requests or 12.6x tighter SLOs in DistServe, which is why leading systems like Mooncake are built around a disaggregated KV cache.

The Acing AI newsletter covers the serving stack the way this piece covers the KV cache: the system behind the single request. Subscribe for the parts the model card leaves out.