The key-value (KV) cache speeds up autoregressive language model generation by storing previously computed attention key-value pairs so models never recompute them at later steps. That same cache grows linearly with model size, batch size, and sequence length, which is why long-context or high-concurrency inference can turn it into the dominant memory bottleneck. Quantization and PagedAttention are documented countermeasures: a quantized KV cache extends capacity to 128k tokens versus 40k tokens in half precision on the same 80GB A100 GPU.
How Does the KV Cache Accelerate Token-by-Token Generation?
Hugging Face states that a key-value cache eliminates the inefficiency of recomputing attention outputs by storing key-value pairs derived from the attention layers of previously processed tokens, so later tokens reuse them directlyCITE:E1. NVIDIA describes the same mechanism from the hardware side: rather than recomputing key and value tensors for every token at each generation step, an inference system can cache those tensors in GPU memoryCITE:E2. Together, the two descriptions point to the same underlying trade: the KV cache trades stored memory for skipped computation, which is what makes token-by-token generation faster than recalculating attention from scratch at every step.
Why Does KV Cache Memory Grow Linearly With Sequence Length and Batch Size?
NVIDIA states that the KV cache grows linearly with the size of the language model, the number of batched requests, and the sequence context lengthCITE:E4. In an earlier post, NVIDIA had already flagged the same dynamic on November 17, 2023, noting that KV cache memory requirements grow linearly with batch size and sequence length and can "quickly scale"CITE:E3. The two NVIDIA posts, published roughly fourteen months apart, describe the identical linear-growth mechanism from two angles — one framed around batch size and sequence length, the other adding model size as a third linear factor — which means every one of those three variables (model size, batch count, sequence length) independently multiplies the cache's memory footprint rather than any single variable acting alone.
When Does the KV Cache Turn From Accelerator Into Bottleneck?
Hugging Face states that even though the KV cache speeds up autoregressive generation, it can become a memory bottleneck under long context length or high batch size, in a post dated May 16, 2024CITE:E5. This is the same accelerate-then-constrain mechanism described in the first two sections: the identical storage mechanism that removes redundant computation (E1, E2) is the identical linear-growth mechanism (E3, E4) that, once sequence length or batch size crosses a given GPU's memory capacity, converts a speed advantage into a memory ceiling.
What Mitigation Techniques Address the KV Cache Memory Problem?
Hugging Face documents cache quantization as one mitigation, and the PagedAttention research team documents paged memory management as anotherCITE:E6CITE:E7. On the same May 16, 2024 post, Hugging Face reports that a quantized KV cache can support up to 128k tokens with Flash Attention enabled on an 80GB A100 GPU, compared with a maximum capacity of 40k tokens for a cache kept in half precision on the same hardwareCITE:E6. Separately, in a paper dated September 12, 2023, the PagedAttention research team describes an attention algorithm inspired by the classical virtual memory and paging techniques used in operating systemsCITE:E7.
| Configuration | Hardware | Token capacity |
|---|
| Half-precision KV cache | 80GB A100 | 40k tokens |
| Quantized KV cache + Flash Attention | 80GB A100 | 128k tokens |
On identical 80GB A100 hardware, quantization alone raises the documented capacity from 40k to 128k tokens — a 3.2x increase without adding GPU memory, which is the clearest evidence-backed illustration of how a mitigation technique acts directly against the linear-growth mechanism described above rather than working around it.
What This Means
The same storage-and-reuse design that lets a KV cache skip redundant attention computationCITE:E1CITE:E2 is, by NVIDIA's own description, the design that scales linearly with model size, batch size, and sequence lengthCITE:E3CITE:E4 and can therefore flip into a memory bottleneck at long context or high concurrencyCITE:E5. The two documented mitigations sit on opposite ends of the stack: quantization shrinks what each cached entry costs, moving capacity on an 80GB A100 from 40k to 128k tokensCITE:E6, while PagedAttention borrows operating-system virtual memory and paging concepts to manage how cache entries are laid out and reusedCITE:E7. Both target the same linear-growth mechanism from different angles, without changing the fact that the growth itself is linear.
Author's Take・EffectStory 編輯部
The 3.2x jump from 40k to 128k tokens on the identical 80GB A100 GPU is the single most useful number in this evidence set, because it isolates what quantization buys you when hardware is held constant: more context capacity without more memory. That framing matters because the underlying problem — linear growth across model size, batch size, and sequence length — is not something quantization or PagedAttention removes; both are documented as mitigations that push the ceiling higher, not as fixes that make the cache stop growing linearly. For anyone evaluating long-context serving infrastructure, the concrete metric to track is quantized-cache token capacity per GPU class, not raw context-window advertising, since the 40k-vs-128k comparison shows the same context-length claim can mean very different memory bills depending on precision. The open variable worth watching next is how paged memory management, borrowed from operating-system virtual memory design, performs once batch size and sequence length are both pushed up simultaneously, since the evidence here documents each mitigation individually but not their combined effect.