How Much VRAM Do You Actually Need to Run Local LLMs?

Every “can I run this model?” thread ends up at the same argument, and almost none of them show the arithmetic. Sizing VRAM for local LLMs is not guesswork: it comes down to three numbers you can work out in about a minute, and one nasty cliff you fall off when you get it wrong.

The three numbers

Total memory needed is roughly: weights + KV cache + overhead. Get each one and add them up.

1. Weights: parameters x bytes per weight

A model’s weights occupy its parameter count multiplied by however many bytes each weight is stored in. That second number is set by the quantisation you choose:

• FP16 / BF16 — 2 bytes per weight
• 8-bit — about 1 byte
• 6-bit — about 0.75 bytes
• 4-bit — about 0.5 bytes in theory, nearer 0.55-0.6 in practice, because mixed schemes like Q4_K_M keep some tensors at higher precision

So an 8B model at 4-bit is roughly 8 x 0.55 = 4.4GB. A 32B at 4-bit is around 18GB. A 70B at 4-bit lands near 38-40GB. You do not have to trust those: the quickest sanity check is the file size on disk, which is very close to what the weights will occupy in memory.

ls -lh ~/.ollama/models/blobs/ | sort -k5 -h | tail -5
du -h model-q4_k_m.gguf

2. KV cache: the part everyone forgets

Every token you have in context keeps a key and value vector per layer, and that cache lives in VRAM alongside the weights. The formula is:

kv_bytes = 2 * n_layers * n_kv_heads * head_dim * bytes_per_element * n_tokens

The leading 2 is for keys and values. All those model shape values are in the model’s config, and they vary a lot between architectures — which is exactly why a single “add 2GB for context” rule of thumb is useless.

Take a typical 8B-class model with grouped-query attention: 32 layers, 8 key/value heads, head dimension 128, FP16 cache. That is 2 x 32 x 8 x 128 x 2 = 131,072 bytes per token, or about 128KB. At 8k context that is roughly 1GB. At 32k it is roughly 4GB — more than the jump from a 4-bit to a 5-bit quantisation of the whole model.

Older architectures without grouped-query attention use the full head count for keys and values, which can be four to eight times more cache for the same context. If a model seems to eat memory far faster than its size suggests, that is usually why.

3. Overhead

Compute buffers, activations, the CUDA or Metal context and whatever your desktop is already using. Budget roughly 1-2GB on a headless Linux box, more if the same GPU is driving your monitors.

A calculator you can paste

python3 - <<'EOF'
params_b      = 8      # billions of parameters
bytes_per_w   = 0.55   # 4-bit ~0.55, 8-bit ~1.0, fp16 = 2.0
layers        = 32
kv_heads      = 8
head_dim      = 128
kv_bytes      = 2      # fp16 cache; 1 if you quantise the cache to 8-bit
ctx           = 16384

weights = params_b * 1e9 * bytes_per_w / 1e9
kv      = 2 * layers * kv_heads * head_dim * kv_bytes * ctx / 1e9
print(f"weights {weights:.1f} GB + kv {kv:.1f} GB + ~1.5 GB overhead "
      f"= {weights + kv + 1.5:.1f} GB")
EOF

Change one variable at a time and the trade-offs become obvious: context scales linearly, and quantising the cache halves its cost.

What happens when it does not fit

This is the bit that matters, because nothing crashes. The runtime quietly keeps some layers in system RAM and runs them on the CPU, and your throughput collapses.

Token generation is memory-bandwidth-bound: to produce one token the machine reads essentially the whole model. So the ceiling is roughly memory bandwidth divided by model size. A 4GB model on a discrete GPU with around 900GB/s of bandwidth has a theoretical ceiling in the low hundreds of tokens per second, and you will see a decent fraction of it. The same model running from dual-channel DDR5 at roughly 80-100GB/s has a ceiling around twenty tokens per second, and partial offload is bounded by the slow half. Spilling even a handful of layers is not a gentle 10% tax — it is a cliff.

Check what actually happened rather than guessing:

ollama ps                 # shows size and the CPU/GPU split per loaded model
nvidia-smi --query-gpu=memory.used,memory.total --format=csv

If ollama ps says anything other than 100% GPU, you are paying for it. With llama.cpp directly, -ngl 99 pushes every layer to the GPU and the startup log tells you how many it managed.

Buying yourself headroom

Three levers, in the order I reach for them:

• Quantise the KV cache. 8-bit cache roughly halves the context cost for very little quality loss. In llama.cpp that needs flash attention enabled.
• Ask for less context. Most runtimes allocate the cache for the full context window up front, so a 32k window costs you whether you use it or not.
• Drop a quantisation level — but going below 4-bit is where quality degradation becomes obvious, particularly for code.

llama-server -m model-q4_k_m.gguf -ngl 99 -c 16384   --flash-attn --cache-type-k q8_0 --cache-type-v q8_0

# equivalent knobs for Ollama
export OLLAMA_FLASH_ATTENTION=1
export OLLAMA_KV_CACHE_TYPE=q8_0

What I’d aim for

Work out your requirement and then add 20%. If you are doing agentic coding work, where long contexts are the norm rather than the exception, the KV cache is the term that will bite you — the same 8B model at 4k and at 64k are very different purchases. My own line is that 16GB is the point where local models stop being a demo, 24GB is where a 32B-class model with real context becomes comfortable, and everything below 12GB is small-model territory. A well-chosen 7B is still genuinely useful, and if you want to see how far the low end goes, running models on a Raspberry Pi 5 with Ollama is an instructive exercise in exactly these constraints.

Once you know the number, the hardware question gets much simpler. If the plan is to point a coding agent at it, read the complete guide to using Claude Code with local LLM models first — the context lengths that setup wants will change your answer.