INT4 vs INT8 for LLM Inference

The memory tradeoff

INT8 weight storage is roughly half of FP16 before format overhead. INT4 can roughly halve weight memory again. Actual files and loaded memory include scales, grouping metadata, and sometimes higher-precision layers.

KV cache and runtime buffers may remain at a different precision, so total VRAM rarely falls by exactly the same ratio as weight bits.

Quality is workload-specific

INT8 is commonly a conservative choice. INT4 can work very well, but sensitivity differs by model, quantization method, and task. Small quality shifts may be irrelevant for classification yet visible in reasoning or structured generation.

Performance is not guaranteed

A smaller representation reduces memory traffic, but speed depends on optimized kernels and accelerator support. Unsupported formats may introduce conversion or fallback costs. Benchmark the exact artifact and runtime.

Choosing between them

Use INT8 when compatibility and lower regression risk dominate. Evaluate INT4 when memory capacity or cost is the binding constraint. Compare both against a higher-precision reference using real prompts and operational metrics.