llmrun

NVIDIA · Ada Lovelace

Best AI Models for NVIDIA RTX 4000 Ada Generation (20 GB)

Runs 14B models well; 27–31B fits but is slow (13 tok/s).

20 GB GDDR6360 GB/s107 TFLOPS FP16130 WAda Lovelace 2023

What LLMs can the RTX 4000 Ada Generation run?

#ModelSpeedllmrun ScoreQuant · VRAM
1
Gemma 4 26B A4B ITGoogleHow to run →
Q4_K_M · 16.5 GB · 74.7 t/s
74.7 tok/sQ4_K_M · 16.5 GB
2
Qwen3.5 9BAlibaba
Q4_K_M · 6.6 GB · 36.8 t/s
36.8 tok/sQ4_K_M · 6.6 GB
3
Qwen3.5 4BAlibaba
Q4_K_M · 3.4 GB · 71.8 t/s
71.8 tok/sQ4_K_M · 3.4 GB
4
GPT OSS 20BOpenAI
Q4_K_M · 13.1 GB · 80.7 t/s
80.7 tok/sQ4_K_M · 13.1 GB
5
LFM2.5 24B A2BLiquid AI
Q4_K_M · 15.0 GB · 115 t/s
115 tok/sQ4_K_M · 15.0 GB
See all 318 models on the RTX 4000 Ada Generation →Best quant that fits · speeds estimated on this card

About the RTX 4000 Ada Generation

The RTX 4000 Ada Generation runs 14B models well. Ada Lovelace generation, released 2023.

GenerationAda Lovelacereleased 2023

RTX 4000 Ada Generation strengths and weaknesses

Strengths

  • Runs 35B MoE models: Qwen3.6 35B A3B at 75.1 tok/s in 19.8 GB
  • Reads long prompts fast: A 4K-token prompt on Gemma 4 26B A4B IT starts answering in 749 ms
  • Low power draw: 130 W board power; a 300 W PSU is enough

Weaknesses

  • 70B models don't fit: Llama 3.3 70B Instruct needs 51.3 GB at Q4 vs 20 GB here
  • Slow on large dense models: Qwen3.8 27B at 13.4 tok/s

RTX 4000 Ada Generation tokens per second by model size

Output speed depends on memory bandwidth (360 GB/s). Grey rows need more than 20 GB.

  1. 9BQwen3.5 9B · Q4_K_M · 6.6 GB
    36.8tok/s
  2. 14BQwen3 14B · Q4_K_M · 11.9 GB
    24.6tok/s
  3. 27–31BQwen3.8 27B · Q4_K_M · 17.9 GB
    13.4tok/s
  4. 35B MoEQwen3.6 35B A3B · IQ4_XS · 19.8 GB tight · IQ4_XS
    75.1tok/s
  5. 70BLlama 3.3 70B Instruct · not at 4-bit
    needs 51.3 GB
  6. 120B MoEGPT OSS 120B · not at 4-bit
    needs 70.8 GB

Estimated speeds: Q4_K_M, 4K context, one user. How we estimate

All 1117 models on the RTX 4000 Ada Generation

Featured only

Best quant that fits, ordered by llmrun Score. Fit grades how well it runs here, S best. How ranking works

LLM models compatible with NVIDIA RTX 4000 Ada Generation — ranked by Score, recency and Fit
ModelVRAMScoreFit
LFM2 350M Extract
Q4_K_M·403.4 t/s
0.8 GBS100, Runs great
Llama 3.2 3B
Q4_K_M·110.4 t/s
4.1 GBS93, Runs great
Nemotron Content Safety Reasoning 4B
Q4_K_M·72.2 t/s
3.6 GBS86, Runs great
Nemotron 3 Nano Omni 30B A3B Reasoning BF16
IQ3_S·79.0 t/s
19.3 GBB62, Decent
Nemotron Terminal 32B
Q3_K_M·14.1 t/s
19.0 GBC30, Usable
MiniCPM O 2 6
Q4_K_M·41.6 t/s
6.4 GBA76, Runs well
Llama 3.1 Nemotron Safety Guard 8B v3
Q4_K_M·43.4 t/s
7.3 GBA77, Runs well
Llama 3.1 Nemotron Nano 8B V1
Q4_K_M·43.6 t/s
7.3 GBA77, Runs well
DeepSeek v2 Lite
Q4_K_M·112.9 t/s
10.2 GBS94, Runs great
Opt 6.7B
FP16·15.9 t/s
14.7 GBB56, Decent
Gemma 3 1B Pt
Q4_K_M·354.5 t/s
2.1 GBS100, Runs great
Llama Guard 3 1B
Q4_K_M·236.4 t/s
2.6 GBS100, Runs great
Qwen2.5 Coder 1.5B Instruct
Q4_K_M·182.8 t/s
1.7 GBS99, Runs great
Mellum2 12B A2.5B Thinking SFT
Q4_K_M·123.3 t/s
7.8 GBS95, Runs great
Llava V1.6 Mistral 7B HF
FP16·14.1 t/s
19.2 GBD29, Barely runs

Rent NVIDIA RTX 4000 Ada Generation in the cloud

Hourly rate on demand for one RTX 4000 Ada Generation.

  • Vast.aiMarketplace

    from$0.26/hr

    Deploy

Buying one ($1,250) costs as much as ~4,800 hours of rental, about 20 months at 8 hours a day.

Cost per million tokens

Output-token cost on an RTX 4000 Ada Generation rented at $0.26/hr, one request at a time. Batch several users on the same card and the cost per token drops several-fold.

ModelSpeedPer 1M output
LFM2.5 24B A2B115 tok/s$0.63$0.0074 input
NVIDIA Nemotron 3.5 Lightning 30B A3B BF16111 tok/s$0.66$0.0072 input
GPT OSS 20B81 tok/s$0.90$0.013 input
Gemma 4 26B A4B IT75 tok/s$0.98$0.013 input
Qwen3.8 27B13 tok/s$5.44$0.084 input

RTX 4000 Ada Generation specs, power and PSU

Specs

Architecture
Ada Lovelace
VRAM
20 GB GDDR6
Memory bandwidth
360.0 GB/s
FP16 (dense)
106.90 TFLOPS
CUDA cores
6,144
Tensor cores
192
Compute capability
8.9
TDP
130 W
Released
2023-08-09
MSRP
$1,250

RTX 4000 Ada Generation FAQ

What is the largest model the RTX 4000 Ada Generation can run?

Qwen3.6 35B A3B (36.0B) is the largest at 4-bit or better: at IQ4_XS it uses 19.8 GB of 20 GB, and runs at about 75.1 tok/s.

Can the RTX 4000 Ada Generation run Gemma 4 26B A4B IT?

Yes. Gemma 4 26B A4B IT fits at Q4_K_M, using 16.5 GB of 20 GB, and it is the best model that runs well on this card.

Can the RTX 4000 Ada Generation run 70B models?

No. Llama 3.3 70B Instruct needs 51.3 GB at Q4, more than the 20 GB on this card.

How many tokens per second does the RTX 4000 Ada Generation give?

About 74.7 tok/s on Gemma 4 26B A4B IT at Q4_K_M, with the first token after 0.7 s on a 4K-token prompt.

What power supply does the RTX 4000 Ada Generation need?

Plan a 300 W PSU (2× the 130 W TDP).

Is 20 GB enough for coding and agents?

Yes for coding: Gemma 4 26B A4B IT, the best of the 3 coding models that fit, runs at about 75 tok/s. For agents, not yet: no agent-scored model fits in less than 96 GB.

Related

Best coding LLMs16 GB VRAM24 GB VRAMGPUs with 20 GB or moreRTX 5090RTX 3090 Ti