# Inference engine

> VI: https://5ac.vn/wiki/inference-engine/ · EN: https://5ac.vn/en/wiki/inference-engine/ · lang: en · spec_as_of 2026-10-19 (spec no formal spec) · cluster C23

The serving runtime that actually runs a trained model — KV-cache paging, continuous batching, kernel fusion, quantization. vLLM, SGLang, TensorRT-LLM, llama.cpp.

---
## Definition

An inference engine is the software layer that actually runs a trained model: loading the weights, grouping requests into batches, managing the KV-cache memory as conversations grow, and turning a graphics card into tokens per second at some real cost. It is neither the model itself nor the whole local LLM deployment project — the engine is one component inside that project. This entry covers the serving runtimes: vLLM, SGLang, TensorRT-LLM, llama.cpp and the ExLlama family.

## Mechanism

Three levers separate one engine from another. Memory management: vLLM's PagedAttention (arXiv:2309.06180) keeps the KV cache in memory pages instead of one contiguous block, which is what makes continuous batching, chunked prefill and prefix caching possible. Kernel fusion: collapsing many small math steps into a single kernel so data stays in fast on-chip memory instead of round-tripping to HBM after each step — FlashAttention v1 measured 7.6× faster than a standard PyTorch implementation on an A100, and CUDA Graphs showed that roughly 70% of a short kernel's runtime can be launch overhead rather than computation (NVIDIA, 2019). Quantization: FP8, GPTQ, AWQ and GGUF weights plus NVFP4 KV caches — TensorRT-LLM on H100+ reports double the throughput and half the memory of 16-bit, paid for with lost portability.

## Application context

Engines fit situations: llama.cpp for laptops and edge devices, MLX for Macs, ExLlama for one to a few NVIDIA cards, vLLM as the default open-source production serving runtime, SGLang strongest on long context and MoE; Ollama is convenient for trials, but 5ac's piece warns against it for production serving. A real 2026 rig: DeepSeek V4 Flash on two DGX Spark boxes with a vLLM image rebuilt for the GB10 chip — 1M-token context, six concurrent requests, NVFP4 KV cache — with an expected 62–83 tok/s per chat under 128K and roughly 160–190 tok/s across six sessions (README figures, flagged by the author as not independently measured). The golden rule from the same articles: never rank engines on a single tokens/s number — a usable benchmark lists model, quantization, engine version, hardware, input/output mix, concurrency, TTFT, TPOT, p50/p95/p99, memory and cost per 1M tokens.

## Sources fetched

- [Local AI 2026: From an RTX 3090 to Enterprise On-Prem Infrastructure](/en/blog/local-ai-2026-from-rtx3090-to-enterprise/) — engine decision table, benchmark discipline — accessed 2026-10-19
- [GPU Kernel Fusion: Why Your AI Bill Is Really a Memory Problem](/en/blog/gpu-kernel-fusion/) — FlashAttention 7.6×, CUDA Graphs overhead share — accessed 2026-10-19
- [Deploy DeepSeek V4 Flash on 2× DGX Spark — recipe](/en/blog/deploy-deepseek-v4-flash-2x-dgx-spark/) — vLLM configuration on GB10, expected throughput — accessed 2026-10-19
- <https://arxiv.org/abs/2309.06180> — the PagedAttention paper behind vLLM (v1) — verified via arXiv API 2026-10-19

## Related

- [Local LLM deployment](/en/wiki/local-llm-deployment/)
- [Benchmark suite](/en/wiki/benchmark-suite/)