Your GPU Was Never the Slow Part

Rent a GPU by the hour, and you learn fast where the money goes. Your AI assistant answers one short question, then sits quiet for eight seconds. Most owners blame the model at that point. Yet the model is the one part of the stack that moves the least.

Here is the twist that reshaped how we plan AI budgets at 5ac.vn. Keep the same model. Keep the same GPU. Change only the software serving that model, including the kernel fusion at its core, and published tests show throughput rising up to 24 times (vLLM launch post, June 2023). The recipe never changed. The kitchen did.

That gap is the subject of this article, and kernel fusion sits at its center.

TL;DR

• Kernel fusion merges many small GPU steps into one job. On the same A100 chip, it made attention 7.6 times faster (Dao et al., 2022).
• NVIDIA timed a short kernel at 2.9 microseconds. Paying for its launch cost 9.6 microseconds (NVIDIA, 2019).
• Across published benchmarks, the engine around the model explains gaps of an order of magnitude. The model rarely does.

Context: why this lands on a Vietnamese SMB desk in 2026

Vietnamese SMBs meet GPU costs in two shapes today. Some rent cloud GPU hours by the minute. Others buy one workstation card for the office. Both paths ask the same question: how many tokens does one dollar buy?

The answers moved quickly. Token prices kept falling through 2025 and into 2026, which we covered in Token Cheap as Electricity. Local hardware became a workable option at the same time, as our guide to local AI hardware explains. Compute is no longer the scarce item it was two years ago.

So the pressure moved somewhere else. If you already own the chip, or rent it at a fixed price, the question becomes simple. Where does extra speed come from? The answer lives below the model, in two things most buyers never see. Those things are the inference engine and the GPU kernels it runs.

What do the published measurements actually say?

Every number below comes from a paper, a vendor engineering post, or a project blog that we opened and read. Each row names its test setup, because a benchmark without a setup proves very little.

Why is a faster chip not making your AI feel faster?

Attention looks like heavy math. In practice it is heavy traffic. Each step of a Transformer reads and writes large blocks of memory, and that traffic costs more time than the arithmetic does. FlashAttention proved the point in 2022.

The Stanford team cut the work into tiles that fit on an on-chip buffer called SRAM. They computed the softmax step by step, and never wrote the full attention matrix to HBM. The whole operation became one fused CUDA kernel. Attention then ran 7.6 times faster on an A100 in FP16 (Dao et al., 2022).

The same paper reports up to 9x fewer reads and writes to HBM. Memory for attention fell from 17,024 MB to 836 MB at a 4K sequence. Notice what did not happen: the model did not get smaller. The arithmetic actually grew, because the kernel recomputes a few values instead of storing them.

Anyscale reached the same conclusion from the serving side. Their engineers state that LLM inference waits on memory IO, not on compute. Loading one megabyte to the compute cores takes longer than doing math on that megabyte. The FlashAttention-2 paper adds a useful ratio for the same chip: matrix multiply runs up to 16x faster than non-matrix work.

A kitchen makes the pattern easy to see. HBM is the pantry down the hall. SRAM is the counter beside the stove. A cook who walks to the pantry after every step cooks slowly, no matter how sharp the knife is. Fusion keeps the ingredients on the counter.

Where does the time leak out between steps?

During generation, the GPU does not run one large job. It runs dozens of tiny ones for every token. NVIDIA measured a plain kernel at 2.9 microseconds on a Tesla V100, then measured the same kernel with a launch and a synchronize call around it. The effective time was 9.6 microseconds (NVIDIA, 2019). Roughly 70 percent of that figure was service, not cooking.

CUDA Graphs attack that gap. NVIDIA recorded 20 short kernels per timestep and replayed them as one graph. The effective time per kernel dropped from 3.8 microseconds to 3.4 microseconds, close to the pure kernel time. Fewer handoffs, less waiting.

The same idea applies inside a single step. LinkedIn's Liger Kernel fused normalization, rotary embeddings, activation, and loss into Triton kernels. Training throughput rose about 20 percent, and GPU memory use fell about 60 percent on LLaMA 3-8B across 8 A100s (Liger Kernel, 2024). Their HuggingFace baseline ran out of memory at a 4,096-token context. The fused version reached 16K context on the same hardware.

That last detail matters for small teams. Solving an out-of-memory wall with a kernel change is far cheaper than solving it with a new card.

Where does kernel fusion pay off most in production?

For most teams, the largest win sits in the KV cache. That cache stores the key and value tensors which let a model remember earlier tokens. Older serving stacks reserved memory in big fixed blocks. Berkeley researchers found that only 20.4 to 38.2 percent of that memory held real tokens (Kwon et al., SOSP 2023). One LLaMA-13B conversation can hold 1.7 GB of cache by itself (vLLM blog, 2023).

PagedAttention borrowed a trick from operating systems. It cut the cache into fixed blocks and handed them out on demand. Waste fell under 4 percent. Throughput rose 2 to 4 times at equal latency. When several outputs shared the same prompt, memory fell by 55 percent and throughput gained 2.2x.

Scheduling adds another jump. Orca scheduled work at the level of each token instead of each batch, and reached 36.9x the throughput of FasterTransformer on GPT-3 175B at equal latency (Yu et al., OSDI 2022). Anyscale measured 8x from continuous batching alone against naive batching. SGLang went further with a radix tree that reuses shared prefixes, up to 6.4x on multi-turn, JSON, and agent workloads.

Quantization layers on top of all this. SpQR keeps weights in 3 to 4 bits plus a few outliers, and its GPU kernel dequantizes at runtime. A 33B model ran on one 24GB consumer card, 15 percent faster than the 16-bit baseline, with over 4x compression (Dettmers et al., 2023). AQLM pushed to 2 bits with a custom kernel that matched or beat FP16 on Llama 2.

Read that list again as an owner, not an engineer. Each item is a different way to get more work out of hardware you already pay for.

What is the catch in the fine print?

Fusion is not free. Memory layout is a design decision, and every decision carries a bill. Microsoft Research measured vLLM's paged kernels and found them up to 2.8x slower than FlashAttention-2 kernels (vAttention, ASPLOS 2025). The paged prefill kernel inside FA2 ran 37 percent slower, and FlashInfer's ran 42 percent slower, than the non-paged version in the same library.

Their fix kept virtual memory contiguous through a CUDA API, and gained up to 1.23x. The lesson for buyers is plain. Every published speedup belongs to a specific engine, version, and test setup.

Kernels also age. FlashAttention-2 reached only 25 to 40 percent of peak FLOPs/s on an A100, and better work partitioning lifted it to 50 to 73 percent. On the newer H100, FA2 sat at 35 percent utilization until the third version added warp specialization and FP8. FA3 then beat FA2 by 1.5 to 2.0x, at 740 TFLOPs/s in FP16 (Shah et al., 2024).

So a benchmark from 2023 can flatter or undersell the chip you buy in 2026. The date of the test matters as much as the number in it.

The insight most teams miss

Kernel fusion is not a hardware purchase. It is a software choice you can inspect before you spend money.

A model is a recipe. The engine is the kitchen. The second one decides how fast dinner reaches the table.

Two teams can rent the same GPU and run the same open-weight model, then see results an order of magnitude apart. The difference lives in the serving engine, the kernel library, and the version numbers. Vendors rarely publish those details, because a number without its setup proves very little.

In our view, most SMB GPU budgets are set at the wrong layer. Owners pick a model, then pay for whatever speed that choice happens to deliver. The cheaper move is to keep the model and shop for the engine.

One more thing deserves a plain sentence. None of this asks your team to write CUDA code. The open-source engines named here ship as ready packages, and you can test them on the hardware you already have.

Takeaway

Before you renew a GPU rental or buy a card, run one test. Take your own prompts. Run them on two engines on the same hardware, for example vLLM against llama.cpp or HuggingFace Transformers. Record tokens per second, cost per 1,000 tokens, and peak memory.

Then ask every vendor a single question: which engine and which version produced that number, on which chip, at which batch size? A supplier who answers with specifics is a supplier you can plan around.

Start with our note on compute as a new asset class. It frames the same decision in money, which is the language your accountant already speaks. If you want the open-source path, our infrastructure guide for SMBs covers what runs on what, and the GB10 benchmark writeup shows how we read local chip numbers before we quote them to a client.

Sources

• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness: fused attention kernel, memory table, IO complexity.
• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning: utilization figures and the matrix versus non-matrix ratio.
• FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision: H100 utilization and measurement conditions.
• Efficient Memory Management for Large Language Model Serving with PagedAttention: KV cache waste, paged blocks, serving throughput.
• SGLang: Efficient Execution of Structured Language Model Programs: radix tree prefix reuse on agent and JSON tasks.
• CUDA Graphs: kernel execution versus launch overhead.
• Orca: A Distributed Serving System for Transformer-Based Generative Models: iteration-level scheduling results.
• Continuous batching for LLM inference: batching gains and the memory-IO bound claim.
• vLLM launch post: serving comparison and shared memory results.
• Liger Kernel technical report and Liger Kernel repository: fused norm, RoPE, and loss kernels.
• vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention: cost of paged kernels and prefill penalties.
• SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression: 3 to 4 bit weight compression with runtime dequantization.
• AQLM: Extreme Compression of Large Language Models via Additive Quantization: 2-bit kernels matching FP16 on Llama 2.
• vLLM architecture overview: attention backends, CUDA Graphs, fused MoE kernel.
• TensorRT-LLM documentation: paged attention, in-flight batching, CUDA graph support, quant formats.
• llama.cpp build documentation: multiple kernel backends from one codebase.

Related reading

• Token Cheap as Electricity
• Local AI hardware in 2026
• Compute as an asset class
• Open-source AI infrastructure for SMBs
• GB10 benchmark writeup