Running an LLM on a Hetzner GPU server with vLLM: the setup we use

Since July 2026 we have been running an open-weight LLM, Qwen3-14B, with vLLM on a rented Hetzner GPU server in Germany. No model provider sits between a question and its answer, and the model has no public endpoint.

The server: a Hetzner GEX44 with 20 GB of VRAM

Our server is a Hetzner GEX44 with an NVIDIA RTX 4000 SFF Ada, a compact 70-watt card with 20 GB of graphics memory (VRAM). Around it sit an Intel Core i5-13500, 64 GB of RAM and two NVMe drives. For chat, one number matters more than any other: memory bandwidth. This card moves 280 GB/s, and that sets the speed. CPU and RAM have little to do.

The model: Qwen3-14B in 4-bit AWQ

We run Qwen’s own AWQ version of Qwen3-14B: 14.8 billion parameters, open weights under the Apache 2.0 licence. At 16 bits per weight the model alone would need about 29.6 GB, more than the card has. AWQ (activation-aware weight quantization) stores the weights in 4 bits and protects the few that matter most, so the model fits into about 10 GB. The rest of the card goes to the KV cache, the working memory for the conversations in progress.

The weights are stored on the server and pinned to one version, so a restart never pulls a newer upload. GGUF, the format of llama.cpp and Ollama, is still experimental in vLLM; Ollama vs vLLM explains the difference. Which other open models fit into 20, 24 or 96 GB: VRAM guide.

The vllm serve command

Path, model name and port are placeholders. The settings are the ones we run.

vllm serve /path/to/model \
  --served-model-name my-model \
  --host 127.0.0.1 --port 8000 \
  --max-model-len 8192 \
  --max-num-seqs 2 \
  --max-num-batched-tokens 2048 \
  --gpu-memory-utilization 0.75 \
  --enforce-eager \
  --no-enable-log-requests \
  --disable-uvicorn-access-log

What the settings do:

• Host 127.0.0.1, the server’s internal loopback address: nothing answers on the model port from the internet.
• 8,192 tokens per request, prompt and answer together. Without this limit, vLLM would plan for the model’s full 40,960; with it, two full conversations fit with room to spare.
• Two requests at once. A third waits in the queue. Long prompts are processed in chunks of up to 2,048 tokens.
• 75% of the GPU memory for vLLM instead of the default 92%, which leaves headroom on a 20 GB card.
• Eager mode turns off CUDA graphs. That saves memory and costs some speed.
• No request logs: vLLM writes no line per request, and its built-in web server writes no access line. Prompts stay out of the logs.

On our card the weights take almost half the memory and the KV cache about a quarter, enough for almost four full conversations of 8,192 tokens. For other models and cards, try the VRAM calculator.

Qwen3’s thinking mode counts too

Qwen3 starts in thinking mode. Its thinking comes out of the same 8,192 tokens and costs time. For plain chat, switch it off as the model card describes before you raise the limit.

vLLM as a systemd service, closed to the internet

vLLM runs as a systemd service that starts at boot and has memory limits. Our own systems reach it through a restricted tunnel; from the internet it cannot be reached at all. If you open a port instead, don’t rely on vLLM’s API key option alone. vLLM’s security guide lists routes the key does not protect and recommends a reverse proxy that lets through only the endpoints you need. Also worth switching off: the usage statistics vLLM collects by default.

Measured: tokens per second on the GEX44

With one request, the model writes 28.8 tokens per second, and the first token arrives after about 54 ms. Two requests at once run at 28.3 tokens per second each, 56.4 together, so the second one rides along almost for free.

The limit for a single answer is memory bandwidth: for every token the card reads almost all the weights once, and at 280 GB/s that allows about 33 tokens per second. Long prompts cost time only once: 1,701 tokens took 1.09 s to the first token, the same prompt again 63 ms, because vLLM’s prefix cache keeps what it has already read. That counts in chat, where every message sends the whole conversation again.

What a Hetzner GPU server costs

For a 14B model in 4 bits and a few users, 20 GB is enough, as our GEX44 shows. For larger work there is the GEX131 with an NVIDIA RTX PRO 6000 Blackwell: 96 GB of VRAM and 1,792 GB/s, for up to €1,199 a month plus €599 setup. That is the class for 30B models at 16 bits, large mixture-of-experts models, several models side by side, or fine-tuning. Prices exclude VAT.

Our server stands in Germany, so prompts are processed there. What that means for data protection: self-hosted LLM and GDPR.

If you want a model like this on a server of your own, without doing the server work yourself, that is what Private AI is for. We set up a dedicated GPU server at Hetzner in Germany, install and adapt an open-weight model, test it against your own questions and maintain the machine. You get a private chat interface and an API, and every client gets a data processing agreement. The first call is free.