About this tool
Can this model fit on my GPU? The answer decides what you can self-host, and getting it wrong means a crashed loader or a surprise cloud bill. This calculator estimates inference VRAM from the three factors that matter: parameter count, quantization level, and context length. Weights are straightforward — a 7B model at FP16 needs about 14 GB, at Q4 roughly 4 GB — and the tool shows the bytes-per-parameter math openly. The part people forget is the KV cache, which grows with both context length and model size and can add several gigabytes at long contexts; a simplified per-parameter-per-token estimate is included so the total reflects real usage, not just the weight file. An activation and overhead buffer rounds out the estimate, since CUDA contexts and frameworks take their cut. Pick from common model presets or enter any parameter count, choose FP16, INT8, Q4 or Q2, set your context, and see whether that 24 GB card is enough. All math is local and the formulas are shown so you can adapt them.
How to use LLM VRAM Calculator
- Enter the model size in billions of parameters, or pick a preset.
- Choose the quantization level: FP16, INT8, Q4 or Q2.
- Set the context length you plan to run.
- Read the VRAM estimate broken into weights, KV cache and overhead.
Frequently asked questions
How accurate is the estimate?
Within roughly 10-20% for typical architectures. Real usage varies with attention implementation, framework overhead and batch size.
Why does context length matter so much?
The KV cache stores attention state per token. Long contexts multiply it — at 128K tokens it can rival the weights themselves.
Which quantization should I choose?
Q4 offers the best size-to-quality trade-off for most models; Q2 shows visible quality loss; FP16 is for when quality is paramount and VRAM is plentiful.
Does this cover training?
No. Training needs gradients and optimizer states — typically 4-8x inference memory. This estimates inference only.