# Speculative decoding (draft–verify inference)

> VI: https://5ac.vn/wiki/speculative-decoding/ · EN: https://5ac.vn/en/wiki/speculative-decoding/ · lang: en · spec_as_of 2026-10-21 (spec no formal spec) · cluster C17

A smaller model drafts a few tokens and the main model verifies them in one pass — same output, fewer serial steps; the gain is biggest on memory-bound hardware.

---
## Definition

Speculative decoding speeds up token generation like this: a smaller model — or a multi-token prediction head — guesses the next few tokens, then the main model checks them in a single pass; tokens that match are kept, the first mismatch is discarded and recomputed. The point the original paper makes is that the trick buys speed without changing the answer — it computes several tokens in parallel while leaving the output distribution untouched. On T5-XXL the paper measured a 2× to 3× speedup over the standard T5X run with identical outputs (arXiv:2211.17192, ICML 2023 Oral).

## Mechanism

The piece on DeepSeek's production inference stack spells out the loop: the draft produces 5 tokens per step, a confidence head decides how many of them to verify as the sequence length varies, and the whole apparatus adds +4 to +7% throughput at the production operating point. The recipe for deploying DeepSeek V4 Flash on 2× DGX Spark rebuilds the same idea with MTP_NUM_TOKENS=5 — predict many tokens, verify once — and according to that piece the approach wins biggest on hardware bound by memory bandwidth, like the GB10, where every generation step has to read the weights again. llama-server in llama.cpp also lists speculative decoding among its built-in features, next to continuous batching and JSON schema.

## Application context

There is no single winner: a practical benchmark on the DGX Spark (GB10), five runs per mode on the same NVFP4 weights, has DSpark ahead on code (51.5 vs 34.5 tok/s, about 1.5×) and on short chat (23.2 vs 21.0), while MTP wins long essays (24.1 vs 18.3 tok/s, about 1.3×). The floor for comparison is vLLM with speculative decoding off: 8.2 tok/s on one stream, 57.9 tok/s across 10 streams. When the model runs in thinking mode the token sequence gets harder to predict, so the advantage shrinks.

Worth keeping in mind when picking a configuration for agents: the fastest configuration measured (NVFP4 plus MTP, 105.8 tok/s combined) is also the one that hard-rebooted the machine under long context and high concurrency. For an agent that runs all day, one crash mid-shift costs more than a few tok/s on the table.

## Sources fetched

- [DeepSeek Open-Sources Its Production Inference Stack: What Changes for Self-Hosters](/en/blog/deepseek-opens-production-inference-stack/) — 5 draft tokens per step, confidence head, +4–7% throughput — accessed 2026-10-21
- [DSpark vs MTP on DGX Spark — practical benchmark](/en/blog/dspark-vs-mtp-benchmark-gb10/) — n=5 on NVFP4, 51.5/34.5 and 24.1/18.3 tok/s, 8.2 tok/s baseline — accessed 2026-10-21
- [Deploy DeepSeek V4 Flash on 2× DGX Spark — recipe](/en/blog/deploy-deepseek-v4-flash-2x-dgx-spark/) — MTP_NUM_TOKENS=5, the win on bandwidth-bound hardware — accessed 2026-10-21
- <https://arxiv.org/abs/2211.17192> — the original speculative decoding paper, ICML 2023 Oral (v2) — verified via arXiv API 2026-10-21

## Related

- [Inference engine](/en/wiki/inference-engine/)
- [Local LLM Deployment](/en/wiki/local-llm-deployment/)