# NVFP4 (4-bit floating-point format for weights and KV cache)

> VI: https://5ac.vn/wiki/nvfp4/ · EN: https://5ac.vn/en/wiki/nvfp4/ · lang: en · spec_as_of 2026-10-22 (spec no formal spec) · cluster C17

NVIDIA's FP4 format for weights and KV cache alike: the fastest line in the GB10 benchmark, and it needs the Marlin backend because SM121 has no native FP4 cores.

---
## Definition

NVFP4 is NVIDIA's FP4 format — FP4 meaning 4-bit floating point — used to quantize both the weights and the KV cache of a model. Squeeze a model down to 4 bits and it fits in less memory while every generated token reads fewer bytes; those two things are what decide speed on hardware bound by memory bandwidth. In 5ac's measurements the name always shows up in one of three forms: NVFP4 weights, NVFP4 KV cache, and NVFP4 + MTP. How a serving engine slots the format into its wider quantization pipeline belongs to the [Inference engine](/en/wiki/inference-engine/) entry; this one keeps only the GB10 field notes.

## Mechanism

The GB10 benchmark (Kubesimplify, 14–17/08/2026) puts NVFP4 in the fast group: NVFP4 on vLLM reaches 11.5 tok/s single-stream and 84.3 tok/s aggregate, but it requires the Marlin backend; NVFP4 + MTP climbs to 22.0 tok/s single-stream and 105.8 tok/s aggregate — the fastest line on the table, at the price of two hard reboots under long context and high concurrency. On the memory side, the NVFP4 KV cache runs as nvfp4_ds_mla, text util 0.835 (~2.49M tokens). The ceiling sits in the chip: SM121 has no native FP4 tensor cores, so NVFP4 goes through the flashinfer/triton backend instead of a native path.

## Application context

Picking NVFP4 trades stability for speed. On the same GB10, plain FP8 only makes 8.2 tok/s but stays clean and stable; FP8 + DFlash reaches 17.2 tok/s for free generation and about 65.6 tok/s aggregate; NVFP4 + MTP tops the aggregate table and has already rebooted a machine. For an agent running all shift, one mid-shift crash costs more than the few tok/s on the table.

NVFP4 is also what makes a 1-million-token window realistic at a workable memory budget: DeepSeek V4 Flash on two DGX Spark boxes uses an NVFP4 KV cache to hold 1M context while weights plus cache still exceed one node's memory, which is why two boxes is the minimum viable configuration (the model quantization table sits in the [Local LLM Deployment](/en/wiki/local-llm-deployment/) entry). A quantized KV cache still grows with conversation length — that half is the [KV cache](/en/wiki/kv-cache/) entry — and holding one set of NVFP4 weights fixed across A/B runs is what the [Speculative decoding](/en/wiki/speculative-decoding/) entry covers.

## Sources fetched

- [DSpark vs MTP on DGX Spark — a practical benchmark for speculative decoding](/en/blog/dspark-vs-mtp-benchmark-gb10/) — the FP8 / NVFP4 / NVFP4 + MTP table, the Marlin backend, two hard reboots — accessed 2026-10-22
- [Deploy DeepSeek V4 Flash on 2× DGX Spark — the complete, measured recipe](/en/blog/deploy-deepseek-v4-flash-2x-dgx-spark/) — nvfp4_ds_mla, text util 0.835, SM121 with no native FP4 cores — accessed 2026-10-22

## Related

- [Inference engine](/en/wiki/inference-engine/)
- [KV cache](/en/wiki/kv-cache/)
- [Local LLM Deployment](/en/wiki/local-llm-deployment/)
- [Speculative decoding](/en/wiki/speculative-decoding/)