LLM Quantization: FP16, INT8, and INT4 Explained

What quantization changes

Quantization represents model values with fewer bits. Reducing precision usually lowers storage and memory bandwidth requirements, making larger models practical on smaller or less expensive hardware. It can also increase throughput when the runtime and accelerator provide optimized kernels.

The headline calculation is simple, but real formats include grouping, scales, zero points, and metadata. A “4-bit model” therefore uses somewhat more than exactly half a byte per parameter.

Choosing a precision

FP16 or BF16 is a dependable baseline when quality and compatibility matter most. INT8 can reduce weight memory substantially with limited quality impact for many workloads. INT4 provides a larger reduction but quality and speed depend heavily on the quantization method, model, hardware, and kernel implementation.

Lower precision does not automatically make every workload faster. Dequantization, unsupported operators, CPU offload, or inefficient kernels can erase the theoretical advantage.

What remains in memory

Quantizing weights does not necessarily quantize KV cache, activations, or runtime workspaces. A long-context workload may still be dominated by a higher-precision KV cache. Always calculate these components independently instead of applying one precision setting to the entire process.

How to evaluate

Compare representative prompts and outputs, not only benchmark perplexity. Measure latency, throughput, peak memory, and task quality using the exact runtime and hardware intended for production. Keep a higher-precision reference so regressions are visible.