# KV cache (key–value cache and prompt caching)

> VI: https://5ac.vn/wiki/kv-cache/ · EN: https://5ac.vn/en/wiki/kv-cache/ · lang: en · spec_as_of 2026-10-21 (spec no formal spec) · cluster C17

Where an engine keeps the key–value pairs it already computed: paging, prefix reuse, an 890-byte-per-token footprint and the pricing tier for cache-hit requests.

---
## Definition

The KV cache is the memory an inference engine keeps the key–value pairs of attention in, once per token, so the next token does not have to rebuild the whole history from scratch. Generating one token at a time while recomputing every earlier token at every step is the expensive part of inference; the cache turns that work into a read. The buffer grows with conversation length, making it both the speed lever and the standing memory problem of any serving system.

## Mechanism

PagedAttention (arXiv:2309.06180, Kwon et al., SOSP 2023) is the starting point: the paper audited earlier serving systems and found only 20.4–38.2% of KV cache memory actually holding real tokens, the rest wasted by upfront block allocation. PagedAttention carves memory into fixed cells and allocates on demand, cutting waste below 4% and lifting throughput 2–4× over FasterTransformer and Orca. When several outputs shared the same prompt, memory fell by 55 percent and throughput gained 2.2x; SGLang reuses cache through a radix tree for up to 6.4×. The footprint is a variable too: DeepSeek's documentation computes the KV cache at 890 bytes per token.

How an engine implements this — KV management, prefix caching, KV quantization — belongs to the [Inference engine](/en/wiki/inference-engine/) entry.

## Application context

In production the number that matters is the hit rate: a 4x-H200 field test measured 96% of input as cache hits, DeepSeek holds its KV tier for more than 72 hours (a 3-day TTL for agent traffic), and roughly 14.4 TB per GPU at logical size. Caching, not kernels, is the pricing story — per million tokens, input costs $0.30 when it misses the cache but $0.006 when it hits, so one team paid $440.88 a day renting 4x-H200 for work the official API carried for $184–223 a day. That price tier is explained in the [Token cost control](/en/wiki/ai-token-cost/) entry.

Self-hosted, the cache is real RAM: a 24 GB card adds 10–30% VRAM for the KV cache per run — the formula lives in the [Local LLM Deployment](/en/wiki/local-llm-deployment/) entry — while on a DGX Spark the weights plus an NVFP4 KV cache exceed 128 GB of unified memory, leaving decode bound by roughly 273 GB/s of bandwidth. Cache hits also travel with context length; that half sits in the [Context window](/en/wiki/context-window/) entry.

## Sources fetched

- [DeepSeek Open-Sources Its Production Inference Stack: What Changes for Self-Hosters](/en/blog/deepseek-opens-production-inference-stack/) — 96% cache hits, TTL beyond 72 hours, 14.4 TB per GPU, 890 bytes per token, the $0.006 price line — accessed 2026-10-21
- [Deploy DeepSeek V4 Flash on 2× DGX Spark — recipe](/en/blog/deploy-deepseek-v4-flash-2x-dgx-spark/) — NVFP4 KV cache, 128 GB unified memory, ~273 GB/s bandwidth — accessed 2026-10-21
- [GPU Kernel Fusion: Why Your AI Bill Is Really a Memory Problem](/en/blog/gpu-kernel-fusion/) — KV waste statistics, PagedAttention, RadixAttention — accessed 2026-10-21
- <https://arxiv.org/abs/2309.06180> — the PagedAttention paper behind vLLM (v1) — verified via arXiv API 2026-10-21

## Related

- [Inference engine](/en/wiki/inference-engine/)
- [Local LLM Deployment](/en/wiki/local-llm-deployment/)
- [Token cost control](/en/wiki/ai-token-cost/)
- [Context window](/en/wiki/context-window/)