# Mixture of Experts (MoE, sparse expert architectures)

> VI: https://5ac.vn/wiki/moe/ · EN: https://5ac.vn/en/wiki/moe/ · lang: en · spec_as_of 2026-10-22 (spec no formal spec) · cluster C17

Many experts per layer, a router waking only a few of them per token: MoE is heavy in stored weights, light in compute per token — the trade-off behind DeepSeek-class models.

---
## Definition

Mixture of Experts (MoE) is a neural-network architecture in which each processing layer is not one monolithic block but a set of same-purpose sub-networks — the experts — plus a small router that picks the few experts each token needs. The architecture is called sparse because only a fraction of parameters takes part in the computation of any given token: a model of hundreds of billions of total parameters computes per token like something orders of magnitude smaller. The bargain is explicit — MoE buys capability in memory for stored weights, not in arithmetic per token. 5ac's own recipe gives the working example: "DeepSeek V4 Flash is a Mixture of Experts model of roughly 284 billion parameters".

## Mechanism

The design was formalized in *Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer* (arXiv:1701.06538, Noam Shazeer et al., 2017): a layer stacks many experts side by side, a trained softmax gate routes each token into a couple of them, and the rest sit idle for that step. Three engineering consequences follow:

- every expert's weights must stay resident in memory even though only a few experts work per step;
- load across experts shifts with each batch, so serving systems must handle the balancing;
- moving tokens between experts — not raw arithmetic — shapes throughput on multi-machine setups.

How engines (vLLM, SGLang) actually serve MoE — weight loading, dispatch, the metrics they report — belongs to the [Inference engine](/en/wiki/inference-engine/) entry, and the model-by-model comparison for local servers lives in [Local LLM Deployment](/en/wiki/local-llm-deployment/). This entry defines the architecture and does not redefine those.

## Application context

For self-hosters, MoE is what lets a very large model enter open-source hardware budgets — while rewriting the hardware problem. 5ac's 2× DGX Spark recipe records that each node carries 128GB of unified memory and "CUDA sees roughly 121.7 GiB", and that running a 284-billion-parameter model on one node fails because "the combined weights and KV cache exceed memory". The answer the article gives — split the model across two machines and quantize the cache down to NVFP4 — has entries of its own; what matters here is that both moves exist because the model is an MoE.

## Sources fetched

- [Deploy DeepSeek V4 Flash on 2× DGX Spark — recipe](/en/blog/deploy-deepseek-v4-flash-2x-dgx-spark/) — a Mixture of Experts model of roughly 284 billion parameters, 128GB unified memory per node, ~121.7 GiB seen by CUDA, weights plus KV cache exceed one node — accessed 2026-10-22
- <https://arxiv.org/abs/1701.06538> — the original sparsely-gated MoE layer paper (v1) — verified via arXiv API 2026-10-22

## Related

- [Inference engine](/en/wiki/inference-engine/)
- [Local LLM Deployment](/en/wiki/local-llm-deployment/)