Choosing a GPU for Local LLMs

Capacity comes first

A GPU that cannot hold the required weights, cache, and runtime workspace will require offload or multiple devices. Start with a memory estimate and preserve headroom for real runtime behavior.

Quantization may make a model fit, but fitting is only the first constraint. The resulting quality and supported kernels still matter.

Bandwidth and compute

Token generation is often sensitive to memory bandwidth because model weights must be read repeatedly. Prompt processing can use compute differently. Compare hardware using the phase and batch sizes that match your application, not a single generic score.

Software compatibility

Check framework support, driver requirements, quantization kernels, attention implementations, and multi-GPU behavior. Mature support can be more valuable than a theoretical hardware advantage.

Make a workload-based decision

Define model, precision, context distribution, concurrency, latency goal, and expected utilization. Benchmark a representative build, include power and hosting costs, and leave enough capacity for model or traffic growth.