Best GPUs for Running LLMs Locally

Compare 56 GPUs for local AI inference. Find the best GPU for your budget based on VRAM, memory bandwidth, and model compatibility.

Browse by VRAM Tier

Find the best models for your VRAM budget

Entry-level for LLMs (RTX 4060, RX 7600, Apple M-series base) — 7B models at Q4, small models at Q8

Mid-range (RTX 3060, RTX 4070, RTX 5070) — 7-13B models at Q4-Q6

Upper mid-range (RTX 4080, RTX 5070 Ti, Arc A770, Apple M4 16GB) — 13B models, some 30B at Q4

Enthusiast (RTX 3090, RTX 4090, RX 7900 XTX) — 30B+ models at Q4-Q6, 70B at aggressive quant

Professional / Apple Silicon (RTX 6000 Ada, L40S, MacBook Pro M4 Max 48GB) — 70B at Q4-Q5

Popular GPUs

View all 56 →

AMD Instinct MI210

AMD · CDNA 2

1638.4 GB/s6,656 SP300W TDP

AMD Instinct MI250X

AMD · CDNA 2

3276.8 GB/s14,080 SP560W TDP

AMD Instinct MI300X

AMD · CDNA 3

5300.0 GB/s19,456 SP750W TDP

AMD Radeon PRO W7800

AMD · RDNA 3

576.0 GB/s4,480 SP260W TDP$2,499

AMD Radeon PRO W7900

AMD · RDNA 3

864.0 GB/s6,144 SP295W TDP$3,999

AMD Radeon RX 6700 XT

AMD · RDNA 2

384.0 GB/s2,560 SP230W TDP$479

AMD Radeon RX 6800

AMD · RDNA 2

512.0 GB/s3,840 SP250W TDP$579

AMD Radeon RX 6800 XT

AMD · RDNA 2

512.0 GB/s4,608 SP300W TDP$649

AMD Radeon RX 6900 XT

AMD · RDNA 2

512.0 GB/s5,120 SP300W TDP$999

AMD Radeon RX 7600

AMD · RDNA 3

288.0 GB/s2,048 SP165W TDP$269

AMD Radeon RX 7700 XT

AMD · RDNA 3

432.0 GB/s3,456 SP245W TDP$449

AMD Radeon RX 7800 XT

AMD · RDNA 3

624.0 GB/s3,840 SP263W TDP$499

Which GPU Do You Need for AI?

The amount of VRAM is the most important specification for running LLMs locally. Most 7B parameter models require 4–8 GB of VRAM at common quantization levels, while 70B models need 24–48 GB. Memory bandwidth determines how fast the model generates tokens — faster bandwidth means faster responses.