NVIDIA · Ampere · 48 GB VRAM tier
Best AI Models for NVIDIA RTX A6000 (48 GB)
Runs 27–31B and 35B MoE models well; 70B barely fits and is slow (12 tok/s).
| # | Model | Speed | llmrun Score | Quant · VRAM |
|---|---|---|---|---|
| 1 | Qwen3.8 27BAlibabaHow to run → Q4_K_M · 17.9 GB · 28.7 t/s | 28.7 tok/s | Q4_K_M · 17.9 GB | |
| 2 | Qwen3.6 27BAlibaba Q4_K_M · 17.9 GB · 28.7 t/s | 28.7 tok/s | Q4_K_M · 17.9 GB | |
| 3 | Gemma 4 31B ITGoogle Q4_K_M · 21.7 GB · 24.5 t/s | 24.5 tok/s | Q4_K_M · 21.7 GB | |
| 4 | Gemma 4 26B A4B ITGoogle Q4_K_M · 16.5 GB · 140 t/s | 140 tok/s | Q4_K_M · 16.5 GB | |
| 5 | Q4_K_M · 19.3 GB · 174 t/s | 174 tok/s | Q4_K_M · 19.3 GB |
| # | Model | Speed | Coding Score | Quant · VRAM |
|---|---|---|---|---|
| 1 | Qwen3.8 27BAlibabaHow to run → Q4_K_M · 17.9 GB · 28.7 t/s | 28.7 tok/s | Q4_K_M · 17.9 GB | |
| 2 | Gemma 4 31B ITGoogle Q4_K_M · 21.7 GB · 24.5 t/s | 24.5 tok/s | Q4_K_M · 21.7 GB | |
| 3 | Qwen3.6 35B A3BAlibaba Q4_K_M · 22.0 GB · 125 t/s | 125 tok/s | Q4_K_M · 22.0 GB | |
| 4 | Gemma 4 26B A4B ITGoogle Q4_K_M · 16.5 GB · 140 t/s | 140 tok/s | Q4_K_M · 16.5 GB | |
| 5 | Q4_K_M · 19.3 GB · 174 t/s | 174 tok/s | Q4_K_M · 19.3 GB |
| # | Model | Speed | Reasoning Score | Quant · VRAM |
|---|---|---|---|---|
| 1 | Qwen3.8 27BAlibabaHow to run → Q4_K_M · 17.9 GB · 28.7 t/s | 28.7 tok/s | Q4_K_M · 17.9 GB | |
| 2 | Gemma 4 31B ITGoogle Q4_K_M · 21.7 GB · 24.5 t/s | 24.5 tok/s | Q4_K_M · 21.7 GB | |
| 3 | Qwen3.5 27BAlibaba Q4_K_M · 17.9 GB · 28.7 t/s | 28.7 tok/s | Q4_K_M · 17.9 GB | |
| 4 | Gemma 4 26B A4B ITGoogle Q4_K_M · 16.5 GB · 140 t/s | 140 tok/s | Q4_K_M · 16.5 GB | |
| 5 | Q4_K_M · 19.3 GB · 174 t/s | 174 tok/s | Q4_K_M · 19.3 GB |
| # | Model | Speed | Math Score | Quant · VRAM |
|---|---|---|---|---|
| 1 | Qwen3.8 27BAlibabaHow to run → Q4_K_M · 17.9 GB · 28.7 t/s | 28.7 tok/s | Q4_K_M · 17.9 GB | |
| 2 | Qwen3.6 27BAlibaba Q4_K_M · 17.9 GB · 28.7 t/s | 28.7 tok/s | Q4_K_M · 17.9 GB | |
| 3 | Qwen3.6 35B A3BAlibaba Q4_K_M · 22.0 GB · 125 t/s | 125 tok/s | Q4_K_M · 22.0 GB | |
| 4 | Gemma 4 31B ITGoogle Q4_K_M · 21.7 GB · 24.5 t/s | 24.5 tok/s | overall | Q4_K_M · 21.7 GB |
| 5 | Q4_K_M · 19.3 GB · 174 t/s | 174 tok/s | Q4_K_M · 19.3 GB |
| # | Model | Speed ↓ | llmrun Score | Quant · VRAM |
|---|---|---|---|---|
| 1 | GPT OSS 20BOpenAIHow to run → Q4_K_M · 13.1 GB · 154 t/s | 154 tok/s | Q4_K_M · 13.1 GB | |
| 2 | Q4_K_M · 19.3 GB · 174 t/s | 174 tok/s | Q4_K_M · 19.3 GB | |
| 3 | Qwen3.5 4BAlibaba Q4_K_M · 3.4 GB · 153 t/s | 153 tok/s | Q4_K_M · 3.4 GB | |
| 4 | Qwen3 30B A3B Thinking 2507Alibaba Q4_K_M · 19.4 GB · 148 t/s | 148 tok/s | Q4_K_M · 19.4 GB | |
| 5 | Gemma 4 26B A4B ITGoogle Q4_K_M · 16.5 GB · 140 t/s | 140 tok/s | Q4_K_M · 16.5 GB |
| # | Model | Speed | Vision Score | Quant · VRAM |
|---|---|---|---|---|
| 1 | Qwen3.8 27BAlibabaHow to run → Q4_K_M · 17.9 GB · 28.7 t/s | 28.7 tok/s | Q4_K_M · 17.9 GB | |
| 2 | Qwen3.6 27BAlibaba Q4_K_M · 17.9 GB · 28.7 t/s | 28.7 tok/s | Q4_K_M · 17.9 GB | |
| 3 | Gemma 4 31B ITGoogle Q4_K_M · 21.7 GB · 24.5 t/s | 24.5 tok/s | Q4_K_M · 21.7 GB | |
| 4 | Gemma 4 26B A4B ITGoogle Q4_K_M · 16.5 GB · 140 t/s | 140 tok/s | Q4_K_M · 16.5 GB | |
| 5 | Muse Glimmer 30BMeta Q4_K_M · 19.6 GB · 27.2 t/s | 27.2 tok/s | Q4_K_M · 19.6 GB |
The RTX A6000 runs 27–31B and 35B MoE models well. Ampere generation, released 2020.
RTX A6000 strengths and weaknesses
Strengths
- Runs 35B MoE models: Qwen3.6 35B A3B at 125 tok/s in 22 GB
- Room for long context: Qwen3.8 27B at Q4 uses 17.9 GB, leaving 30.1 GB free
Weaknesses
- 120B MoE models don't fit: GPT OSS 120B needs 70.8 GB at Q4 vs 48 GB here
- No FP8 or NVFP4: Ampere predates both; use GGUF, AWQ or GPTQ builds
Output speed depends on memory bandwidth (768 GB/s). Grey rows need more than 48 GB.
- 9BQwen3.5 9B · Q4_K_M · 6.6 GB78.5tok/s
- 14BQwen3 14B · Q4_K_M · 11.9 GB52.5tok/s
- 27–31BQwen3.8 27B · Q4_K_M · 17.9 GB28.7tok/s
- 35B MoEQwen3.6 35B A3B · Q4_K_M · 22.0 GB125tok/s
- 12.0tok/s
- 120B MoEGPT OSS 120B · not at 4-bitneeds 70.8 GB
Estimated speeds: Q4_K_M, 4K context, one user. How we estimate
All 1231 models on the RTX A6000
Featured onlyBest quant that fits, ordered by llmrun Score. Fit grades how well it runs here, S best. How ranking works
| Model | Quant | VRAM | Speed | Context | Score | Fit |
|---|---|---|---|---|---|---|
| Q4_K_M | 12.1 GB | 207.7 t/s | 33K | S100, Runs great | ||
Mistral 7B v0.3Chat | Q4_K_M | 6.8 GB | 101.5 t/s | 33K | S92, Runs great | |
| Q4_K_M | 14.2 GB | 199.9 t/s | 131K | S100, Runs great | ||
Glm 4 9B ChatChat | Q4_K_M | 8.9 GB | 80.5 t/s | 131K | S88, Runs great | |
| Q4_K_M | 47.7 GB | 11.6 t/s | 66K | D20, Barely runs | ||
Phi 3 Mini 4k InstructChatCode | Q4_K_M | 4.2 GB | 146.8 t/s | 4K | S97, Runs great | |
Granite 3.3 8B InstructChatFunctions | Q4_K_M | 7.9 GB | 90.1 t/s | 131K | S90, Runs great | |
XiYanSQL QwenCoder 32B 2504ChatCode | Q4_K_M | 23.8 GB | 24.9 t/s | 33K | A66, Runs well | |
| Q4_K_M | 35.7 GB | 53.0 t/s | 33K | A81, Runs well | ||
NVIDIA Nemotron Labs 3 Puzzle 75B A9B BF16ChatFunctions | Q3_K_M | 45.2 GB | 81.2 t/s | 262K | B62, Decent | |
Gemma 3 1B PtChat | Q4_K_M | 2.1 GB | 756.4 t/s | — | S100, Runs great | |
Qwen3 8B BaseChat | Q4_K_M | 7.6 GB | 90.4 t/s | 33K | S90, Runs great | |
Hermes 4 70BChatReasoningRoleplay | Q4_K_M | 48 GB | 11.5 t/s | 131K | D18, Barely runs | |
Qwen2.5 Coder 1.5B InstructChatCodeFunctions | Q4_K_M | 1.7 GB | 390.0 t/s | 33K | S100, Runs great | |
Phi 3 Small 8k InstructChatCode | BF16 | 16.2 GB | 32.5 t/s | 8K | A72, Runs well |
Rent NVIDIA RTX A6000 in the cloud
Hourly rate on demand for one RTX A6000.
RunPod$5+ bonus creditLowest price
from$0.33/hr
Vast.aiMarketplace
from$0.48/hr
Buying one ($4,649) costs as much as ~14,100 hours of rental, about 59 months at 8 hours a day.
Cost per million tokens
Output-token cost on an RTX A6000 rented at $0.33/hr, one request at a time. Batch several users on the same card and the cost per token drops several-fold.
| Model | Quant | Speed | Per 1M output |
|---|---|---|---|
| NVIDIA Nemotron 3.5 Lightning 30B A3B BF16 | 174 tok/s | $0.53$0.0062 input | |
| Gemma 4 26B A4B IT | 140 tok/s | $0.66$0.012 input | |
| Qwen3.6 35B A3B | 125 tok/s | $0.73$0.012 input | |
| Qwen3.8 27B | 29 tok/s | $3.19$0.073 input | |
| Muse Glimmer 30B | 27 tok/s | $3.37$0.078 input |
Specs
- Architecture
- Ampere
- VRAM
- 48 GB GDDR6
- Memory bandwidth
- 768.0 GB/s
- FP16 (dense)
- 154.80 TFLOPS
- CUDA cores
- 10,752
- Tensor cores
- 336
- Compute capability
- 8.6
- TDP
- 300 W
- Released
- 2020-10-05
- MSRP
- $4,649
RTX A6000 FAQ
What is the largest model the RTX A6000 can run?
Qwen2.5 72B Instruct (72.7B) is the largest at 4-bit or better: at Q4_K_S it uses 46.6 GB of 48 GB, and runs at about 11.9 tok/s.
Can the RTX A6000 run Qwen3.8 27B?
Yes. Qwen3.8 27B fits at Q4_K_M, using 17.9 GB of 48 GB, and it is the best model that runs well on this card.
Can the RTX A6000 run 120B MoE models?
No. GPT OSS 120B needs 70.8 GB at Q4, more than the 48 GB on this card.
How many tokens per second does the RTX A6000 give?
About 28.7 tok/s on Qwen3.8 27B at Q4_K_M, with the first token after 3.3 s on a 4K-token prompt.
What power supply does the RTX A6000 need?
Plan a 600 W PSU (2× the 300 W TDP).
Is 48 GB enough for coding and agents?
Yes for coding: Qwen3.8 27B, the best of the 10 coding models that fit, runs at about 29 tok/s. For agents, not yet: no agent-scored model fits in less than 96 GB.