# Local LLM Deployment > VI: https://5ac.vn/wiki/local-llm-deployment/ · EN: https://5ac.vn/en/wiki/local-llm-deployment/ · lang: en · spec_as_of 2026-10-19 (spec no formal spec) · cluster C19 Running large language models on hardware the business owns instead of third-party APIs: VRAM and quantization math, inference engines, and the cost-and-data logic of on-prem AI. --- ## Definition Local LLM deployment (on-prem AI, self-hosted AI) means running large language models on hardware the business owns: a workstation, a private VPS, even a small cluster under a desk, instead of sending every request to a third-party API. 5ac's articles frame it as a complete stack, not "buy a GPU and you're set": hardware, inference engine and open-weight models, mature enough in 2026 that one affordable workstation replaces a swath of daily cloud calls. Three motivations recur across the cluster: long-run cost once token volume is high, data control so customer code and contracts never leave the perimeter, and independence from whatever an API vendor does with pricing or policy next quarter. ## Mechanism Capacity decides what fits, bandwidth decides how fast it runs, and confusing the two is the most common beginner mistake. The VRAM formula: GB ≈ parameters (billions) × (effective bits per weight ÷ 8) — about 2 GB per 1B at FP16, ~1 GB at FP8/INT8, ~0.5 GB at 4-bit. A 24 GB card (RTX 3090/4090) therefore holds the 35–40B class at 4-bit, enough for most office work, coding and agents, while the KV cache adds a 10–30% memory tax per run. The whole inference-optimization field, with roofline-model analysis, is surveyed in *LLM Inference Unveiled: Survey and Roofline Model Insights* (arXiv:2402.16363, Zhihang Yuan et al., 2024). The SMB-grade tier in 5ac's pieces centers on the NVIDIA DGX Spark: a Grace Blackwell GB10 machine with 128 GB of unified memory, 273 GB/s bandwidth, roughly 38 W during inference, around $4,000 per node, and nodes that link directly without a network switch. Mixture-of-Experts models make this class worthwhile because each token activates only a slice of the parameters: | Model | Size | Weight footprint | Measured (5ac articles) | | --- | --- | --- | --- | | Qwen3.8 Flash Next | 176B total, 6B active | 78 GB quant | 64 tok/s peak (vLLM NVFP4 recipe), ~43 tok/s on real coding, one node | | Ling-3.0-flash | 124B MoE, 5.1B active | ~72 GB INT4 | 141 tok/s at 8 concurrent streams, one node | | DeepSeek V4 Flash | 284B MoE, 13B active | ~149 GB FP8 | ~350 tok/s across two nodes with TP=2, true 1M context | At three to four nodes and up, a division of labor appears: the large model orchestrates and routes work while smaller models act as workers executing pieces of it. ## Application context 5ac sketches three practical hardware paths for a small business: one 24 GB card for trials and mid-size RAG; an Apple unified-memory machine when a very large model must fit in one quiet box; a DGX Spark cluster when the goal is a full AI shop on the desk. Pick the model by asking three questions: how much context you need, how many agents run in parallel and what the budget is. Never pick by the headline number on a spec sheet. ## Sources fetched - [DeepSeek V4 Flash vs Ling-3.0-flash for local deployment](/en/blog/local-llm-deploy-comparison/) — the three motivations, both models' measurements, the context-and-budget tradeoff — accessed 2026-10-19 - [Local AI 2026: From an RTX 3090 to Enterprise On-Prem](/en/blog/local-ai-2026-from-rtx3090-to-enterprise/) — VRAM formula, quantization tiers, bandwidth map, KV-cache tax — accessed 2026-10-19 - [Local AI: DGX Spark configs 2026](/en/blog/ai-local-dgx-spark-configs-2026/) — GB10, 128 GB unified memory, 38 W, one-to-six-node recipes, orchestrator/worker split — accessed 2026-10-19 - — LLM Inference Unveiled (v6) — verified via arXiv API 2026-10-19 ## Related - [What is an AI Agent?](/en/wiki/ai-agent/) - [Agentic RAG](/en/wiki/agentic-rag/) - [Agent governance](/en/wiki/agent-governance/)