How Much VRAM Does an LLM Need?

The short answer

Start with the memory required for model weights, then add KV cache, temporary tensors, and runtime overhead. Parameter count alone is not enough: two deployments of the same model can have very different memory requirements when precision, context length, batch size, or serving software changes.

For a first estimate, multiply parameter count by bytes per parameter. BF16 and FP16 use about two bytes per parameter, INT8 about one, and common 4-bit formats roughly half a byte before metadata and quantization overhead.

Why context length matters

During autoregressive generation, attention keys and values are retained for earlier tokens. This KV cache grows with sequence length, concurrent sequences, layer count, hidden dimensions, and cache precision. Long-context serving can therefore consume substantial memory even when a quantized model fits comfortably.

Batch size in a calculator should represent concurrent sequences, not merely how many requests arrive during a day. Continuous-batching servers change membership dynamically, so leave operating headroom rather than planning for a perfectly full device.

Plan for the runtime

Inference frameworks allocate memory for kernels, workspaces, activations, and memory pools. Quantization formats also carry scales and metadata. A practical estimate should add an overhead margin and then round up to an available GPU capacity.

Tensor parallelism can distribute weights across GPUs, but it does not make every byte divide perfectly. Communication buffers, uneven layers, and replicated state reduce usable capacity. Treat a multi-GPU estimate as a starting point for load testing.

A reliable workflow

Calculate weights and KV cache separately, add a conservative overhead percentage, select hardware with spare capacity, and test using the actual model file, serving framework, context distribution, and concurrency target. Record observed peak memory so later capacity decisions use production evidence rather than a generic rule.