Guide
Ollama vs vLLM: which one to run an LLM on your own server, and when llama.cpp is the better pick
Ollama gets a model talking on a laptop with one command. vLLM is built to serve many requests from one GPU server. This guide shows which one fits your case, with numbers from the vLLM server we run ourselves.
Ollama and vLLM both run open-weight language models on hardware you control, and both answer OpenAI clients. Ollama is built for one person with a model on their own machine, vLLM for a GPU server that many people, apps or agents use at the same time.
Ollama, vLLM or llama.cpp: the decision table
| Your situation | Use | Why |
|---|---|---|
| Laptop or Mac, one person trying models | Ollama | One command per model, memory freed when a model sits idle |
| CPU only, or a model larger than your GPU memory | Ollama or llama.cpp | GGUF runs on the CPU or split across CPU and GPU |
| GPU server for a team, apps or agents | vLLM | Serves many requests at once |
| The model exists only as GGUF | Ollama or llama.cpp | vLLM handles GGUF only experimentally |
| The model comes as safetensors, AWQ, GPTQ or FP8 | vLLM | Its native formats |
| GGUF on a server, no desktop app | llama.cpp (llama-server) | Local only by default, API key built in |
What is Ollama?
Ollama is a free tool under the MIT license for macOS, Windows, Linux and Docker. One command, such as ollama run gemma4, downloads a model and opens a chat, and a server in the background answers on port 11434, also in the OpenAI format. Ollama runs models in GGUF, the compact single-file format of the llama.cpp project it builds on.
What is vLLM?
vLLM is a library for serving language models fast on GPUs. It keeps the working memory of each conversation, the KV cache, in small blocks, a method called PagedAttention, so almost nothing goes to waste. One command, vllm serve, starts an OpenAI-compatible server.
Ollama vs vLLM performance: one user against many
With one conversation, the speed of the GPU’s memory sets the pace, whatever the engine: every new token (a word or part of one) means reading almost the whole model. On our 20 GB server, vLLM already comes close to that limit, so for a single user another engine has little left to win.
The difference starts with the second request. By default, Ollama works on one request per model at a time while the others wait in a queue; a setting raises the limit. vLLM adds each new request to the running batch, and all of them share every pass over the model. On our server, one request runs at 28.8 tokens per second, two at once at 28.3 each.
With many users the gap grows. In a Red Hat test from August 2025 with up to 256 users at once on one A100 GPU, vLLM peaked at 793 tokens per second and Ollama at 41.
Chats also benefit from a cache: each message resends the whole conversation, and vLLM reuses what it has already computed. On our server, a repeated long prompt starts answering after 0.06 seconds instead of 1.09.
Memory: sized to the request, or reserved up front
Ollama loads a model when you ask for it and unloads it after five idle minutes, which frees the memory for other work. vLLM works the other way round: at start-up it claims a set share of the card, 92% by default, and turns everything beyond the model into one shared cache for all requests. It keeps that memory as long as it runs, so vLLM wants the GPU to itself. The VRAM calculator shows what your model needs.
Security defaults: open ports and API keys
On a server with a public IP address, the two start from opposite ends. Ollama listens only on the machine itself, and its local API has no login. Once you open it to the network, anyone who reaches the port can use, download and delete models. vLLM listens on the whole network by default, and its API key option does not cover every route.
The safe setup is the same for both: keep the model server local and put a reverse proxy or tunnel with a login in front. With real data, also switch off Ollama’s cloud models, because they run on Ollama’s servers. More on data protection: self-hosted LLM and GDPR.
Models and formats: GGUF or safetensors
Often the file format decides before speed does. Ollama and llama.cpp load GGUF; vLLM loads safetensors and the compressed (quantized) formats AWQ, GPTQ and FP8.
Uncensored and abliterated models, the versions with refusals removed, are often published as GGUF files, and Ollama loads them straight from Hugging Face. vLLM calls its own GGUF support highly experimental, so check for a safetensors, AWQ or GPTQ release before you plan the server. Which models fit which GPU: LLM VRAM requirements.
Ollama vs vLLM on a Mac, a CPU or an NVIDIA GPU
Ollama runs on most hardware people already own: Macs with Apple Silicon, NVIDIA and AMD graphics cards, and plain CPUs. vLLM is built for GPUs from NVIDIA, AMD and Intel and runs on a Mac only experimentally, so there Ollama or llama.cpp is the practical choice.
Why we run vLLM on our own GPU server
Our own model runs on a Hetzner GEX44 with a 20 GB graphics card: Qwen3-14B, compressed to 4 bits and served by vLLM. A chat produces exactly the parallel and repeated requests measured above. The server listens only locally and does not log requests. For one person on a Mac, Ollama is the better tool. The setup in detail: running an LLM on a Hetzner GPU server with vLLM.
If you want an open-weight model, standard or uncensored, on a dedicated GPU server in Germany without the setup work, that is what Private AI is for. We install, adapt and run the model for you, with a private chat interface and an API. Prompts and answers are not logged unless you decide otherwise, and every client gets a data processing agreement.
Common questions
Is Ollama or vLLM better?
It depends on the job. Ollama suits one person on a laptop, a Mac or a machine without a GPU: one command per model, and memory is freed when a model sits idle. vLLM suits a GPU server that several people, apps or agents use at once, because it serves their requests together. For GGUF models on a server, llama.cpp's llama-server is a third option.
Is vLLM faster than Ollama?
With many users at once, yes, by far. In Red Hat's test from August 2025, vLLM peaked at 793 tokens per second and Ollama at 41. With a single user, the GPU's memory speed limits both, so another engine has little to gain. On our 20 GB server, vLLM writes 28.8 tokens per second for one request and 56.4 combined for two.
Does Ollama have an API key?
Not on the local server. Its API on port 11434 has no login and only listens on the machine itself. If you open it to the network, anyone who reaches the port can use, download and delete models, so put a reverse proxy with a login in front. vLLM has an API key option, but it does not cover every route, so the same advice applies.
Can vLLM load GGUF models?
Only through a separate plugin, and vLLM's own documentation calls GGUF support highly experimental. For vLLM, look for the model as safetensors or in AWQ, GPTQ or FP8. GGUF is the native format of Ollama and llama.cpp.
What is the difference between Ollama and llama.cpp?
llama.cpp is the engine that runs GGUF models, and it ships its own server, llama-server, which you configure on the command line. Ollama builds on llama.cpp and adds a model library, one-command downloads, desktop apps for macOS and Windows and cloud models.
Sources
- Ollama on GitHub github.com
- Ollama docs: FAQ docs.ollama.com
- vLLM docs: security docs.vllm.ai
- llama.cpp: llama-server README github.com
- Red Hat Developer: Ollama vs. vLLM performance benchmark (August 2025) developers.redhat.com
vLLM, Ollama, Hugging Face, NVIDIA and Hetzner are trademarks of their respective owners. Naming them does not imply endorsement by, or a business relationship with, OCTOO AI.