# Tensor parallelism (splitting model weights across devices)

> VI: https://5ac.vn/wiki/tensor-parallelism/ · EN: https://5ac.vn/en/wiki/tensor-parallelism/ · lang: en · spec_as_of 2026-10-22 (spec no formal spec) · cluster C17

Splitting a model's weights across machines so each node keeps a share and computes every step together: at 284 billion parameters with a 1M window, a precondition.

---
## Definition

Tensor parallelism (TP) splits one model's weights across machines: each node keeps a share of the weights and takes part in every inference step, instead of one node holding the whole model. It is a configuration detail, not the whole deployment problem — quantization, batching and KV-cache management are the serving runtime's job, covered in the [Inference engine](/en/wiki/inference-engine/) entry. At roughly 284 billion parameters with a 1-million-token window, a single 128 GB unified-memory node cannot hold weights plus KV cache together, so splitting stops being an optimization and becomes the condition for running the model at all.

## Mechanism

The Megatron-LM paper (arXiv:2104.04473, Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM, v5) states the bottleneck plainly: GPU memory capacity is limited enough that a large model does not fit even on a multi-GPU server, and using parallelism naively brings two problems of its own — expensive cross-node communication, or devices waiting on other devices to make progress. The paper therefore studies how tensor, pipeline and data parallelism compose, and what each trade-off costs.

On 2× DGX Spark the DeepSeek V4 Flash recipe builds tensor parallelism (TP=2) end to end, and puts it in one sentence: the model is split across the two machines, each holding part of the weights and participating in every inference step. Two nodes double the problem: synchronization, network fabric, NCCL, and boot order. The TP=2 network environment is WORKER_HOST, MASTER_ADDR, NCCL_IB_HCA, NCCL_SOCKET_IFNAME, VLLM_HOST_IP and HF_CACHE, and the link must be a QSFP56 RoCE 200G DAC cable rather than ordinary Ethernet. The ops scripts run worker-first, head-last — reverse that order and the NCCL rendezvous hangs. The spin-wait patch even drops SpinCondition.busy_loop_s from 1 to 0.002 on TP=2 so it stops holding a whole Grace P-core.

## Application context

The minimum viable configuration for DeepSeek V4 Flash is two boxes: the model needs TP=2 with 1M context and NVFP4, and a single node would require a lighter model that fits in 128 GB. The two-node TP=2 model table lives in the [Local LLM Deployment](/en/wiki/local-llm-deployment/) entry. Two ideas are worth keeping apart: splitting one model across machines is what TP does, while putting prefill and decode on two separate GPU clusters is a different architectural problem — see the [Prefill/decode disaggregation](/en/wiki/prefill-decode-disaggregation/) entry.

## Sources fetched

- [Deploy DeepSeek V4 Flash on 2× DGX Spark — the complete, measured recipe](/en/blog/deploy-deepseek-v4-flash-2x-dgx-spark/) — TP=2, NCCL environment, the QSFP56 RoCE 200G DAC cable, worker/head boot order — accessed 2026-10-22
- <https://arxiv.org/abs/2104.04473> — the Megatron-LM paper on tensor and pipeline parallelism (v5) — verified via arXiv API 2026-10-22

## Related

- [Local LLM Deployment](/en/wiki/local-llm-deployment/)
- [Inference engine](/en/wiki/inference-engine/)
- [Prefill/decode disaggregation](/en/wiki/prefill-decode-disaggregation/)