The first thirty minutes are rarely about the model. They are about the install script that wanted a different CUDA, the OpenAI client pointed at the wrong port, and the server that answered one request and then went quiet.
Three tools come up every time a small team decides to run a model on its own machine: Ollama, vLLM and llama.cpp. All three load weights and answer questions. That is where the similarity stops. One behaves like an app you install, one behaves like a serving system you operate, and one behaves like a box of parts you assemble.
Choosing badly rarely fails loudly. It shows up three weeks later as an afternoon lost to out-of-memory errors, a context window that quietly dropped page twelve of the customer's contract, or a throughput ceiling nobody measured before launch.
The short answer: if your open question is which model to run, we covered that in our local deployment model comparison. This piece answers a different question: which software should serve it. Ollama gets you from download to a working endpoint fastest. vLLM is built for a box that answers many clients at once. llama.cpp is for the team that owns the target hardware and wants every knob within reach.
TL;DR
- All three expose an OpenAI-style API, so your application code can stay portable. What differs is how much of the stack each one hides from you.
- Ollama optimizes for the first hour. vLLM optimizes for the hundredth concurrent request. llama.cpp optimizes for the hardware already sitting under your desk.
- Match the runtime to the job in front of you this month. Switching later is a configuration change while your app speaks the standard API; unwinding a pile of runtime-specific behaviour is not.
Sources: ollama.com/blog/all-aboard-open-models + docs.vllm.ai + docs.ollama.com/context-length, all accessed 27 October 2026.
Why this decision landed on your desk in 2026
Two years ago, running a model locally was something engineers did for fun. It is now a purchasing decision, and three forces put it there.
The first is money. Once your monthly token volume passes what your API bill costs, owning the hardware starts to win, and the runtime decides how much of that hardware you actually use. The second is data. Client contracts, support transcripts and internal code stop leaving the building when inference happens on your own box. The third is continuity: an on-premise endpoint keeps working when a provider changes pricing, rate-limits you, or has a bad day.
The adoption signal is hard to miss. Ollama reported serving 8.9 million developers when it announced an $88 million round in July 2026 (Ollama blog, accessed 27 October 2026). A tool does not reach that scale on curiosity alone. It reaches it because "run it here" became the default expectation for a lot of teams.
What has not changed: hardware budgets stay fixed while ambitions grow. That gap is where the runtime choice earns its keep, because the same GPU can feel generous or stingy depending on what software is scheduling work onto it.
What each runtime is built to be
Ollama is a product layer. You install an app, run one command to pull a model by name, and a REST API is listening. Its local server lives on port 11434 and offers a subset of the OpenAI API at the familiar /v1 path; the OpenAI client needs an API key value, and Ollama ignores it (Ollama, OpenAI compatibility, accessed 27 October 2026). Under the hood it is no longer a single engine: since March 2026 Apple silicon machines run on Apple's MLX framework, and the June 2026 release added GGUF support through llama.cpp for everything else (Ollama blog, accessed 27 October 2026). In other words, Ollama packages other people's engines and hides the wiring.
vLLM is a serving system. It came out of the Sky Computing Lab at UC Berkeley and now lists more than 2,000 contributors across institutions and companies (vLLM documentation, accessed 27 October 2026). Its feature list reads like an operations backlog: continuous batching, chunked prefill, prefix caching, tensor and pipeline parallelism, and PagedAttention for managing attention memory. The docs claim support for over 200 model architectures from Hugging Face, an OpenAI-compatible API server alongside an Anthropic Messages API and gRPC, and quantization formats from FP8 through INT4 to GGUF. This is software written for a machine whose job is answering requests all day.
llama.cpp is a codebase. Plain C and C++, MIT licensed, with the stated goal of minimal setup and strong performance across a wide range of hardware (llama.cpp repository, accessed 27 October 2026). The repo shows 132.5k stars and over eleven thousand commits. Its backend list spans Metal, CUDA, HIP, Vulkan, SYCL, OpenCL and WebGPU, and it supports CPU plus GPU hybrid inference so a model larger than your video memory can still run. The quick start gives you a one-line installer, Docker images, pre-built binaries, or a source build, and llama serve starts an OpenAI-compatible server.
Notice the layering. Ollama now uses llama.cpp to run GGUF files. Many teams run llama.cpp behind a small gateway, and vLLM reads GGUF too. These are not three rival kingdoms; they are three points on a spectrum from "hide everything" to "expose everything."
Setup, API, memory, quantization: the four columns that decide it
Feature grids usually compare things nobody argues about. These four are the ones that change your week.
| Column | Ollama | vLLM | llama.cpp |
|---|---|---|---|
| Setup friction | App download or one install script, then ollama run pulls the weights and serves them |
Linux host, Python 3.11 to 3.14, a virtual environment, then uv pip install vllm plus vllm download-kernels |
One-line installer, Docker, pre-built release binaries, or build from source with the backend you need |
| OpenAI-compatible API | Yes, a subset: base URL http://localhost:11434/v1/, key value required by the client and ignored by the server |
Yes: OpenAI-compatible API server on port 8000, plus Anthropic Messages API and gRPC | Yes: llama serve (llama-server) exposes the chat completions endpoint |
| GPU memory management | Models load on demand and drop out after an idle period; ollama ps shows size, processor split and a countdown until unload |
PagedAttention splits the KV cache into blocks; the project reports waste under 4% where older systems wasted 60% to 80% | You choose the offload level per run, and CPU plus GPU hybrid inference covers models bigger than total VRAM |
| Quantization | Pick a tagged variant when you pull; the engine underneath is MLX on Apple silicon and llama.cpp elsewhere | FP8, MXFP8, MXFP4, NVFP4, INT8, INT4, GPTQ, AWQ, GGUF and compressed-tensors among others | Integer quantization from 1.5-bit up to 8-bit, as files you download and keep on disk |
| Hardware surface | macOS, Windows and Linux; a GPU helps but is not required to start | Linux first, with macOS available through vLLM-Metal; NVIDIA, AMD, Intel, TPU and more | Anything with a backend: Apple silicon, NVIDIA, AMD, Intel, Vulkan devices, and plain CPUs |
| Configuration surface | Environment variables such as host and model storage path, plus per-request options | Server flags, engine arguments and parallelism settings read from config | Command-line flags for context, layers, sampling and grammar, all documented in the repo |
Read the memory row twice, because it is where the three philosophies separate. Ollama treats memory as a cache that expires: convenient on a laptop, surprising when a background job fires after lunch and pays a reload. vLLM treats memory as a resource to be partitioned finely, because wasted kilobytes multiply across concurrent requests. llama.cpp hands you the dial and trusts you to turn it.
Pick by job: dev machine, production box, embedded target
Dev and test work: Ollama
If the task is evaluating prompts, wiring a prototype, or letting a non-engineer try a model this afternoon, take Ollama. The install is short, the command to serve is short, and the model library means you are not hunting for weights. Swapping models costs one line, which is exactly what an evaluation phase needs.
The trade shows up under load. One interactive session feels fine; a queue of background jobs does not, because the runtime was not designed around fair scheduling for many clients. Reach for something else the moment your test starts measuring concurrent behaviour.
Production throughput: vLLM
If a service answers requests from users or agents all day on a GPU box, vLLM is the intended tool. Batching, prefill scheduling and paged attention memory exist for one reason: keep the accelerator busy when requests arrive faster than a single stream can drain them. The official PagedAttention post reports up to 24x the throughput of Hugging Face Transformers on the tested setups, with the memory-waste reduction doing most of the work (vLLM blog, accessed 27 October 2026). Treat that as a reported ceiling from the project's own benchmark rather than a promise about your workload, and measure with your own traffic before you quote it to anyone.
The cost is operational. You inherit a Python environment, a Linux host, kernel builds, and an upgrade cadence. That is a reasonable bill for a team with an on-call rotation. It is a heavy bill for one developer who wanted a demo by Friday.
Embedded targets and maximum control: llama.cpp
If the target is an appliance, a colleague's laptop, an edge box with odd hardware, or a product where the binary footprint matters, llama.cpp is the answer. It runs on backends the other two do not cover, it builds with no dependency stack, and it lets you pin the exact quantization file shipped with your product.
This is also the choice that demands the most from you. Nobody decides the offload level, the context size or the sampling defaults for you. For a team that treats inference as part of the product, that control is the point. For a team that just wants a URL, it is extra surface area with no upside.
What actually breaks in the first hour
These are the four failures we see most often, each with the fix that belongs beside it.
Ollama's context window starts smaller than you expect. The default scales with video memory: 4k tokens on a modest GPU, 32k in the middle band and 256k at the top (Ollama, context length, accessed 27 October 2026). The same page recommends at least 64,000 tokens for web search, agents and coding tools, so the low default truncates exactly the workloads small teams care about. Raise it with the OLLAMA_CONTEXT_LENGTH environment variable, the /set parameter num_ctx command in an interactive session, or the num_ctx option on an API call (Ollama FAQ, accessed 27 October 2026), and set it deliberately before anyone judges output quality.
Loaded does not mean loaded forever. The same FAQ shows ollama ps reporting each model's size, its processor split between system memory and GPU, and a countdown until it unloads. A job that runs once an hour pays a cold load most runs. If your workload is bursty, either keep the model resident or accept the reload in your latency budget.
vLLM's first start is the slowest one. The quickstart requires a Linux host and Python between 3.11 and 3.14, and notes that kernels which are not pre-downloaded get compiled at startup, which can add several minutes on newer GPUs (vLLM quickstart, accessed 27 October 2026). Run vllm download-kernels during install and that surprise moves from your first deploy to your setup script. Budget a real environment for it instead of installing into whatever Python happens to be on the box.
"OpenAI-compatible" means a subset everywhere. Ollama says so outright, and the other two document their own gaps around less common endpoints. A client library feature you rely on can return a not-found error while everything else works. Test the exact calls your application makes against your chosen runtime before you promise a migration date.
Hardware comes after the runtime, not before
The tempting order is wrong. Teams pick a GPU first, then discover the serving software they chose wants a driver they do not have, and the hardware sits half-used for a quarter.
Work in the other direction. Decide the runtime from the job, confirm it runs on the hardware you own or are about to buy, then size the machine. Our local AI hardware ladder from the RTX 3090 up to enterprise gear covers what each tier actually buys you, and our DGX Spark configuration notes show what a worked example looks like when the box is already on the desk.
One reassurance for whoever signs the invoice: the runtime is the cheap part of this decision. Reinstalling a different serving stack costs an afternoon. Replacing a GPU that the wrong software could not drive costs the budget.
The 5ac take: a runtime is an operational commitment
You are not picking the best piece of software. You are picking who maintains it, what you are allowed to change, and how much of your week it eats when something breaks at 11 p.m.
Most comparison articles stop at benchmarks, and benchmarks are the easy part. The real difference appears on the bad days. With a packaged tool, an upgrade is a download and a restart, and you file issues upstream when it misbehaves. With a serving system, an upgrade means a dependency review, a config diff and a canary run. With a bare library, you own the whole stack, which means you can fix anything and you also have to fix everything.
There is a second layer most teams miss: the runtime shapes what your team learns. Ollama teaches nothing about memory, which is fine until capacity planning arrives. vLLM teaches scheduling and batching, which is what a production box runs on. llama.cpp teaches the memory hierarchy itself, because every flag touches it. Pick the tool whose lessons you want internalized, not only the one with the shortest install.
Our bias, stated plainly: we run open weights on hardware we control because owning the brain of the company matters more to us than renting it. Within that conviction the runtime rotates with the job, and we keep every application on the standard API so the rotation stays boring.
Takeaway
Three lines, and you can close this tab:
- Prototype or single user on your own machine: Ollama today. Set the context length on purpose before anyone judges the output.
- A service answering many clients on a GPU host: vLLM, with a Linux box and a real environment behind it.
- A shipped product, odd hardware, or a need to own every setting: llama.cpp, with the quantization file pinned.
Want the local stack chosen and running for you? 5ac deploys G-Company OS on hardware you own, runtime included, as part of the setup. See how the deployment tiers are priced.
Frequently asked questions
Can I switch runtimes later without rewriting my application?
Usually yes, as long as you stay on the standard chat-completions endpoint and avoid vendor-only extras. The portability lives in your client code, not in the weights, so keep the base URL in one configuration place and the swap becomes a config change.
Do I need a GPU before I start?
No. All three run on a CPU, and llama.cpp lists CPU support as a first-class target with backends for the common instruction sets. A GPU changes how many requests you can answer per minute and how large a model stays responsive, which is a different question from whether the setup works.
Is one runtime more accurate than the others?
The runtime does not change the weights, so two runtimes serving the same file at the same precision answer about the same. Perceived accuracy differences come from the quantization you chose, the context length you allowed, and sampling defaults, all of which you control independently of the runtime.
Which one should serve several users at once?
vLLM, because it was built for concurrent serving rather than for a single interactive session. Its batching and memory management target the case where requests arrive from many clients at the same time. For one or two developers on a laptop, that machinery buys you nothing.
References
- Ollama documentation: OpenAI compatibility : local base URL, the ignored API key, and which endpoints are covered. Accessed 27 October 2026.
- Ollama documentation: Context length : VRAM-based default context lengths and the recommended minimum for agent and coding work. Accessed 27 October 2026.
- Ollama documentation: FAQ : the
ollama psoutput fields, context overrides, and server environment variables. Accessed 27 October 2026. - Ollama blog: all aboard open models (9 July 2026) : developer count and funding announcement. Accessed 27 October 2026.
- Ollama blog: improved performance and GGUF support (5 June 2026) : the MLX and llama.cpp engine split. Accessed 27 October 2026.
- vLLM documentation : feature list, contributor count, supported architectures and quantization formats. Accessed 27 October 2026.
- vLLM documentation: Quickstart : operating system and Python prerequisites, install commands, and the kernel download step. Accessed 27 October 2026.
- vLLM blog: Easy, Fast, and Cheap LLM Serving with PagedAttention (20 June 2023) : the throughput comparison, KV cache size figures and the memory waste reduction. Accessed 27 October 2026.
- llama.cpp repository on GitHub : license, quick start, supported backends, quantization range and repository statistics. Accessed 27 October 2026.
— James Marcus, Content Lead 5ac.vn, October 2026. Every source above was opened on the access date shown; re-check the ones that move, especially repository statistics and release notes.