VRAM calculator

Which LLM fits your GPU? A VRAM calculator that counts context and users

Pick a model, a context length and how many requests run at the same time. The calculator adds up weights, KV cache and runtime and shows which GPU sizes and Hetzner GPU servers have room for it. It runs in your browser. Nothing you enter is sent anywhere.

RunningCalibrated on our own GPU server: set up the same way, it gives the 31,920 tokens of KV cache that vLLM reported there.

Model and file Model Quantisation

Load

Context per request (tokens) Requests at the same time

1k = 1,024 tokens. Answers being written at once, not users in total.

Engine

vLLMreserves a share of the card llama.cpp or Ollamasized to your context

GPU memory utilisation

vLLM’s default is 0.92, our server runs 0.75.

KV cache precision

16-bit 8-bit

fp8 in vLLM, q8_0 in llama.cpp

Card and extras

Your GPU

Extra overhead (GiB)

Other processes, a desktop, CUDA graphs.

Result

Smallest card in the table that fits: 20 GB.

Weights: 9.3 GiB
KV cache: 2.5 GiB
Runtime: 850 MiB
Vision projector
Extra
Unused KV pool: 2.4 GiB
Free
Outside vLLM’s share: 5.0 GiB

Weights: 9.98 GB file on Hugging Face (Qwen/Qwen3-14B-AWQ) = 9.3 GiB
KV cache: 160 KiB per token × 8,192 tokens × 2 requests = 2.5 GiB
Runtime: 400 MiB + 450 MiB for activations, scaled by hidden size 5,120 = 850 MiB
vLLM: 0.75 × 20,475 MiB = 15.0 GiB reserved; pool 4.9 GiB = 31,920 tokens, 3.90× at 8,192.

Start command for this setup

vllm serve Qwen/Qwen3-14B-AWQ \
  --max-model-len 8192 \
  --max-num-seqs 2 \
  --max-num-batched-tokens 2048 \
  --gpu-memory-utilization 0.75 \
  --enforce-eager

Back to our server’s setup

Every GPU size at your settings

Card	At your settings	KV pool	Largest context, 2 at once	Requests of 8,192 tokens
8 GB	too big	4.1 GiB short	no room	0
12 GB	too big	1.1 GiB short	no room	0
16 GB	fewer at once	1.9 GiB	6k	1.50×
20 GBHetzner GEX44 · our server	fits	4.9 GiB	15k	3.90×
24 GBHetzner GEX45 · Helsinki	fits	7.9 GiB	25k	6.30×
32 GB	fits	13.9 GiB	40k (model max)	11.10×
48 GB	fits	25.9 GiB	40k (model max)	20.70×
80 GB	fits	49.9 GiB	40k (model max)	39.90×
96 GBHetzner GEX131 · Falkenstein	fits	61.6 GiB	40k (model max)	49.26×

Hetzner GPU servers: GEX44 (20 GB, our server), GEX45 (24 GB, Helsinki), GEX131 (96 GB, Falkenstein). How we set one up: vLLM on a Hetzner GPU server.

Want one of these models on a server of your own, sized and run for you? That is what Private AI is for.

Context length and the KV cache

While a request runs, the model keeps keys and values for every token in each attention layer. That is the KV cache: 160 KiB per token for Qwen3-14B, 5.0 GiB at 32,768 tokens. Newer models need far less. In Qwen3.8 27B only 16 of 64 layers keep a growing cache, and in Gemma 4 31B 50 of 60 layers only look at the last 1,024 tokens.

That is why Qwen3.8 27B at Q4_K_M with 32k of context fits a 24 GB card with about 4.6 GiB to spare, although a formula that counts every layer says it does not. An 8-bit cache needs about half the memory.

Several users at once

Every request running at the same time needs its own cache, so memory grows with context times requests. What counts is how many answers are being written at once. Ten people in a chat tool rarely send at the same moment, and requests above the limit wait in a queue. Our 20 GB server runs 2 requests of 8,192 tokens at once and has room for 3.9.

vLLM, llama.cpp and Ollama count differently

vLLM takes a fixed share of the card at startup and turns whatever the weights leave into KV cache, so nvidia-smi shows the reservation, not what the model needs. llama.cpp and Ollama size the cache to the context you set, so their reading is what the setup really uses. The calculator shows both.

MoE models: every expert stays in memory

A mixture-of-experts (MoE) model computes with a few experts per token but keeps all of them in memory, because any of them can be next. gpt-oss-120b uses 5.1B parameters per token and still needs its whole 65.3 GB file. Fewer active parameters make a model faster, not smaller.

Which LLM can I run on 8, 16, 24 or 96 GB?

The largest models that fit each card with their 4-bit file, 8,192 tokens of context and one user, with at least 5 % to spare. The figure is the total memory they need. DeepSeek V4 Flash is listed with its 2-bit file, because its 4-bit files are too big for one card. Each entry opens in the calculator.

8 GB

16 GB

24 GB

96 GB

The models we deploy, by GPU size: our Private AI catalogue.

How the estimate is calculated

weights  = file size on Hugging Face, or parameters × bits ÷ 8
KV cache = KV per token × context × requests (growing layers)
         + per request: sliding-window layers (window plus the tokens in flight), linear-attention state
runtime  = 400 MiB + 450 MiB × hidden size ÷ 5,120
           llama.cpp: at least 400 MiB + logits of 512 tokens (vocabulary × 512 × 4 bytes)
total    = weights + KV cache + runtime (+ vision projector, + extra)
vLLM     = share × card − weights − runtime = KV pool; pool ÷ cache per request = requests at once

Calibrated on our server

Value	Calculated	Measured
KV cache (tokens)	31,920	31,920
Requests of 8,192 tokens at once	3.90×	3.90×
Engine in nvidia-smi (MiB)	14,905	14,898

Measured with vLLM on an RTX 4000 SFF Ada running Qwen3-14B AWQ. The full setup: vLLM on a Hetzner GPU server.

File sizes and model layouts come from Hugging Face. Without a file for a quantisation, the size is an estimate and marked as one. Hugging Face lists files in decimal GB, the calculator works in GiB: a 24.0 GB file is 22.4 GiB. Model by model: which LLMs fit 20, 24 or 96 GB of VRAM.

What it does not cover

One GPU per model: no offload to system RAM or the CPU, no splitting across several GPUs.
Memory only, no speed estimate.
Macs with unified memory, and fine-tuning.

Questions

VRAM for LLMs, briefly answered.

How much VRAM do I need for an LLM?

Weights plus KV cache plus runtime. The weights are the file size: 9.98 GB for Qwen3-14B in 4-bit AWQ, 17.6 GB for Qwen3.8 27B at Q4_K_M, 65.3 GB for gpt-oss-120b. The cache grows with the context and with the requests running at once, and the runtime takes about 0.6 to 1.1 GiB. Rule of thumb: a 4-bit GGUF needs 0.6 to 0.7 GB per billion parameters, plus context.

Does context length affect VRAM usage?

Yes, linearly. Qwen3-14B needs 160 KiB per token, so 5 GiB at 32,768 tokens. Qwen3.8 27B needs 64 KiB and Qwen3.6 35B-A3B only 20 KiB, because most of their layers do not grow with the context. Every request running at once needs its own share, and an 8-bit cache roughly halves the figures.

How many users can one GPU serve at the same time?

Divide the memory left after weights and runtime by the cache of one request. Our 20 GB server with Qwen3-14B at 8,192 tokens has room for 3.9 full requests, and we allow two. Ten users rarely means ten requests at once, and extra requests wait in a queue.

Why does vLLM use so much VRAM?

It reserves the memory on purpose, 92 % of the card by default. On our 20 GB card we set 75 %, which is 15,356 MiB: about 9,515 for the weights, about 4,988 for a 31,920-token cache and the rest for runtime and activations. So nvidia-smi shows the reservation, not what the model needs.

Which GPU do I need to run an LLM locally?

8 GB is enough for 8B models at 4-bit, 16 GB for models up to 14B or gpt-oss-20b. 24 GB runs the 27B to 35B class at 4-bit. 96 GB runs gpt-oss-120b, 70B models at 8-bit and DeepSeek V4 Flash at 2-bit. In Germany, Hetzner rents a 96 GB GPU server, the GEX131, in Falkenstein.