# YaRN (RoPE scaling for long context)

> VI: https://5ac.vn/wiki/yarn/ · EN: https://5ac.vn/en/wiki/yarn/ · lang: en · spec_as_of 2026-10-22 (spec no formal spec) · cluster C17

How a model trained on short sequences reads far longer ones — RoPE done right, at a tenth of the training tokens earlier extension methods needed.

---
## Definition

YaRN (Yet another RoPE extensioN) is a method for extending the context window of a model that was only ever trained on short sequences. It does not retrain the model; it changes how the model reads token positions — RoPE (rotary position embeddings), the rotary position encoding nearly every current transformer architecture uses. The problem YaRN solves is narrow and real: RoPE learns positions only up to the length seen during training, and past that point the model does not generalize on its own. When the context window is what you need to size, the full definition lives in the [Context window](/en/wiki/context-window/) entry.

## Mechanism

The YaRN paper (arXiv:2309.00071, YaRN: Efficient Context Window Extension of Large Language Models, v3) offers a compute-efficient way to stretch that window: by the authors' account it needs one tenth the training tokens and 2.5× fewer training steps than previous methods, and it lets LLaMA models read and extrapolate well beyond their original pre-training length.

Inside 5ac's own measurements, YaRN mostly shows up as a limit. The DSpark build does not use YaRN, so context is capped near ~262K tokens, and turning on the YARN=1 flag is not a workaround but an error report: AttributeError: 'PreTrainedConfig' object has no attribute 'max_position_embeddings'. When a 1-million-token window is required, the DeepSeek V4 Flash recipe leans on NVFP4 + MTP instead of YaRN. And because the KV cache grows with length, every context extension drags memory along with it — that half belongs to the [KV cache](/en/wiki/kv-cache/) entry.

## Application context

That limit is a property of the build rather than of the model, so it is worth checking before you commit to a toolchain: any workflow that needs more than 262K tokens should either pick a build that ships YaRN or pick a different inference mechanism. YaRN is also not speculative decoding — the two are different and often read as one because they sit next to each other in a config file; the predict-then-verify-once mechanism is the [Speculative decoding](/en/wiki/speculative-decoding/) entry, and it is what actually decides fast or slow on the GB10.

## Sources fetched

- [DSpark vs MTP on DGX Spark — a practical benchmark for speculative decoding](/en/blog/dspark-vs-mtp-benchmark-gb10/) — no YaRN in the DSpark build, ~262K context, the YARN=1 error — accessed 2026-10-22
- [Deploy DeepSeek V4 Flash on 2× DGX Spark — the complete, measured recipe](/en/blog/deploy-deepseek-v4-flash-2x-dgx-spark/) — YaRN / context above 262K as a build limit, NVFP4 + MTP doing 1M — accessed 2026-10-22
- <https://arxiv.org/abs/2309.00071> — the YaRN paper on context window extension (v3) — verified via arXiv API 2026-10-22

## Related

- [Context window](/en/wiki/context-window/)
- [KV cache](/en/wiki/kv-cache/)
- [Speculative decoding](/en/wiki/speculative-decoding/)