NVIDIA · 8x A100 SXM4 · Server

Best AI Models for NVIDIA DGX A100 640GB

Runs 70–120B models well.

640 GB unified16312 GB/s2496 TFLOPS FP166500 W2020

What LLMs can the NVIDIA DGX A100 640GB run?

#	Model	Speed	Quant · Memory
1	DeepSeek V4.1 Flash 1017DeepSeekHow to run → Q4_K_M · 458.4 GB · 322 t/s	322 tok/s	Q4_K_M · 458.4 GB
2	MiMo V2.6 Flash RLXiaomi Q4_K_M · 186.5 GB · 287 t/s	287 tok/s	Q4_K_M · 186.5 GB
3	GLM 5.3Z.ai Q4_K_M · 453.8 GB · 150 t/s	150 tok/s	Q4_K_M · 453.8 GB
4	DeepSeek V4.1 FlashDeepSeek Q4_K_M · 458.4 GB · 322 t/s	322 tok/s	Q4_K_M · 458.4 GB
5	Hy3Tencent Q4_K_M · 182.3 GB · 179 t/s	179 tok/s	Q4_K_M · 182.3 GB

#	Model	Speed	Quant · Memory
1	DeepSeek V4.1 Flash 1017DeepSeekHow to run → Q4_K_M · 458.4 GB · 322 t/s	322 tok/s	Q4_K_M · 458.4 GB
2	GLM 5.3Z.ai Q4_K_M · 453.8 GB · 150 t/s	150 tok/s	Q4_K_M · 453.8 GB
3	Muse Spark Open 816B A42BMeta Q4_K_M · 551.9 GB · 190 t/s	190 tok/s	Q4_K_M · 551.9 GB
4	MiMo V2.6 Flash RLXiaomi Q4_K_M · 186.5 GB · 287 t/s	287 tok/s	Q4_K_M · 186.5 GB
5	Hy3Tencent Q4_K_M · 182.3 GB · 179 t/s	179 tok/s	Q4_K_M · 182.3 GB

#	Model	Speed	Quant · Memory
1	DeepSeek V4.1 Flash 1017DeepSeekHow to run → Q4_K_M · 458.4 GB · 322 t/s	322 tok/s	Q4_K_M · 458.4 GB
2	DeepSeek V4.1 FlashDeepSeek Q4_K_M · 458.4 GB · 322 t/s	322 tok/s	Q4_K_M · 458.4 GB
3	GLM 5.3Z.ai Q4_K_M · 453.8 GB · 150 t/s	150 tok/s	Q4_K_M · 453.8 GB
4	Muse Spark Open 816B A42BMeta Q4_K_M · 551.9 GB · 190 t/s	190 tok/s	Q4_K_M · 551.9 GB
5	Hy3Tencent Q4_K_M · 182.3 GB · 179 t/s	179 tok/s	Q4_K_M · 182.3 GB

#	Model	Speed ↓	Quant · Memory
1	Qwen3.5 4BAlibabaHow to run → Q4_K_M · 3.4 GB · 3252 t/s	3252 tok/s	Q4_K_M · 3.4 GB
2	Qwen3.5 9BAlibaba Q4_K_M · 6.6 GB · 1667 t/s	1667 tok/s	Q4_K_M · 6.6 GB
3	GPT OSS 20BOpenAI Q4_K_M · 13.1 GB · 639 t/s	639 tok/s	Q4_K_M · 13.1 GB
4	Gemma 4 26B A4B ITGoogle Q4_K_M · 16.5 GB · 524 t/s	524 tok/s	Q4_K_M · 16.5 GB
5	Hy3Tencent Q4_K_M · 182.3 GB · 179 t/s	179 tok/s	Q4_K_M · 182.3 GB

#	Model	Speed	Quant · Memory
1	DeepSeek V4.1 Flash 1017DeepSeekHow to run → Q4_K_M · 458.4 GB · 322 t/s	322 tok/s	Q4_K_M · 458.4 GB
2	DeepSeek V4.1 FlashDeepSeek Q4_K_M · 458.4 GB · 322 t/s	322 tok/s	Q4_K_M · 458.4 GB
3	Muse Spark Open 816B A42BMeta Q4_K_M · 551.9 GB · 190 t/s	190 tok/s	Q4_K_M · 551.9 GB
4	GLM 5.3 FlashZ.ai Q4_K_M · 205.2 GB · 284 t/s	284 tok/s	Q4_K_M · 205.2 GB
5	Qwen3.8 Flash NextAlibaba Q4_K_M · 109.0 GB · 163 t/s	163 tok/s	Q4_K_M · 109.0 GB

#	Model	Speed	Quant · Memory
1	DeepSeek V4.1 Flash 1017DeepSeekHow to run → Q4_K_M · 458.4 GB · 322 t/s	322 tok/s	Q4_K_M · 458.4 GB
2	GLM 5.3Z.ai Q4_K_M · 453.8 GB · 150 t/s	150 tok/s	Q4_K_M · 453.8 GB
3	GLM 5.2Z.ai Q4_K_M · 453.8 GB · 150 t/s	150 tok/s	Q4_K_M · 453.8 GB
4	DeepSeek V4.1 FlashDeepSeek Q4_K_M · 458.4 GB · 322 t/s	322 tok/s	Q4_K_M · 458.4 GB
5	Hy3Tencent Q4_K_M · 182.3 GB · 179 t/s	179 tok/s	Q4_K_M · 182.3 GB

See all 440 models on the NVIDIA DGX A100 640GB →Bars = traction for models without a Score yet

About the NVIDIA DGX A100 640GB

The NVIDIA DGX A100 640GB runs 70–120B models well. Prompt reading is its weak spot: a 4K-token prompt waits 120 ms before the first word on DeepSeek V4.1 Flash 1017. At its $199,000 MSRP it gives 2.0 tok/s per $1,000, 6th of the 6 machines compared here. 8x A100 SXM4, released 2020.

For models640 GBall of it

Chip8x A100 SXM4released 2020

NVIDIA DGX A100 640GB strengths and weaknesses

Strengths

Runs 120B MoE models: GPT OSS 120B at 433 tok/s in 70.8 GB
Room for long context: Llama 3.3 70B Instruct at Q4 uses 51.3 GB, leaving 588.7 GB free
Reads long prompts fast: A 32K-token prompt on DeepSeek V4.1 Flash 1017 starts answering in 930 ms

Weaknesses

No FP8 or NVFP4: Ampere predates both; use GGUF, AWQ or GPTQ builds

Memory for models

640 GB of 640 GB

120B MoE70.8 GB1020B614.1 GB

For models 640 GBThe system keeps 0 GBModel size at Q4

Everything on the size ladder fits. MiMo V2.6 Flash RL and DeepSeek V4 Flash 0731 together use 369.5 GB and leave 270.5 GB free.

Speed per dollar and per watt

NewUsed

On Qwen3.6 35B A3B (35B params, 3B active) at Q4_K_M. New means a shop price, either a street quote or the maker's own price; used is compared with used. The last two columns are the speed you get per $1,000 of price and per 100 W of power; higher is better. The NVIDIA DGX A100 640GB has no current new price, so it is shown at its MSRP.

Machine	New price	tok/s	tok/s per $1,000	tok/s per 100 W
RTX 509032 GB	about $5,376	213	39.7	37.1
Mac Studio M4 Max128 GB	about $3,499	64.2	18.3	44.3
Mac Studio M2 Ultra192 GB	about $5,599	75.9	13.6	25.7
MacBook Pro 16" M5 Max128 GB	about $5,399	67.8	12.6	–
Mac Studio M3 Ultra512 GB	about $9,499	76.6	8.1	28.4
DGX A100 640GB640 GB	about $199,000	404	2.0	6.2

Machine	Used price	tok/s	tok/s per $1,000	tok/s per 100 W
RTX 409024 GB	about $3,000	151	50.3	33.6
RTX 509032 GB	about $6,400	213	33.3	37.1

Cost per token: the NVIDIA DGX A100 640GB against 24–640 GB machines and GPUs

On Qwen3.6 35B A3B (35B params, 3B active, Q4_K_M, 4K context). Speed, cost per token and power per token depend on the model.

24–36 GB96–128 GB192+ GBthis serverat MSRP

Cost per 1M output tokenson Qwen3.6 35B A3B · card over 3 years + power

$31.89

Machine, MSRP: $31.22
Power: $0.67

3-year cost of ownership$203,271

Include card price

Power$/kWhUseh/dayKeepyears

Each dot is a machine or GPU; hover or tap it for its numbers, click to open it. Machines without a price for the chosen basis are left out. Machines without a published power figure are left out of Per watt and costed without power. How we calculate cost per token

Similar devices and alternatives

The same chip from other makers first, then machines and GPUs people weigh against it, each with the one reason it is worth a look. Speeds on Qwen3.6 35B A3B.

Machine, 81% slowerMac Studio M3 Ultra (512 GB)Apple · 508.5 GB for models · M3 Ultra

▼ 81% speed▼ 131.5 GB$9,499

Step down to 188.5 GBMac Studio M2 Ultra (192 GB)Apple · 188.5 GB for models · M2 Ultra

▼ 81% speed▼ 451.5 GB$5,599

Step down to 124.5 GBMac Studio M4 Max (128 GB)Apple · 124.5 GB for models · M4 Max

▼ 84% speed▼ 515.5 GB$3,499

Step down to 124.5 GBMacBook Pro 16" M5 Max (128 GB)Apple · 124.5 GB for models · M5 Max

▼ 83% speed▼ 515.5 GB$5,399

Step down to 24 GBRTX 4090NVIDIA · 24 GB VRAM · GPU

▼ 63% speed▼ 616 GB$1,599rent $0.34/hr

Step down to 32 GBRTX 5090NVIDIA · 32 GB VRAM · GPU

▼ 47% speed▼ 608 GB$5,376rent $0.50/hr

NVIDIA DGX A100 640GB tokens per second by model size

Output speed depends on memory bandwidth (16312 GB/s). Grey rows need more than the 640 GB available.

9BQwen3.5 9B · Q4_K_M · 6.6 GB
1667tok/s
14BQwen3 14B · Q4_K_M · 11.9 GB
1116tok/s
27–31BQwen3.8 27B · Q4_K_M · 17.9 GB
609tok/s
35B MoEQwen3.6 35B A3B · Q4_K_M · 22.0 GB
404tok/s
70BLlama 3.3 70B Instruct · Q4_K_M · 51.3 GB
228tok/s
120B MoEGPT OSS 120B · Q4_K_M · 70.8 GB
433tok/s

Estimated speeds: Q4_K_M, 4K context, one user. How we estimate

Unified-memory platforms: writing vs compute

Writing, tok/sCompute, TFLOPS

M3 Ultra512 GB · 819 GB/s
76.6
66
8x A100 SXM4640 GB · 16312 GB/s
404
2496

Machines with about the same memory bandwidth write at about the same speed; compute decides how fast prompts are read. Speeds on Qwen3.6 35B A3B.

All 1355 models on the NVIDIA DGX A100 640GB

Featured only

Best quant that fits in 640 GB, ordered by llmrun Score. Fit grades how well it runs here, S best. How ranking works

LLM models compatible with NVIDIA DGX A100 640GB — ranked by Score, recency and Fit
Model	Quant	VRAM	Speed	Context	Fit
Starcoder15.8BChatCode Q4_K_M·1015.6 t/s	Q4_K_M	13.6 GB2%	1015.6 t/s	—	S100, Runs great
Qwen3 1.7B Base1.7BChat Q4_K_M·6753.4 t/s	Q4_K_M	3.2 GB1%	6753.4 t/s	33K	S100, Runs great
Phi 3.5 Mini Instruct3.8BChatCode Q4_K_M·3118.5 t/s	Q4_K_M	9.0 GB1%	3118.5 t/s	131K	S100, Runs great
Qwen3 0.6B Base596MChat Q4_K_M·13593.3 t/s	Q4_K_M	1.6 GB0%	13593.3 t/s	33K	S100, Runs great
Wildguard7.2BChat Q4_K_M·2218.2 t/s	Q4_K_M	7.3 GB1%	2218.2 t/s	—	S100, Runs great
OLMo 2 1124 7B Instruct7.3BChat Q4_K_M·1844.0 t/s	Q4_K_M	6.8 GB1%	1844.0 t/s	4K	S100, Runs great
Gemma 7B8.5BChat Q4_K_M·1883.3 t/s	Q4_K_M	8.2 GB1%	1883.3 t/s	—	S100, Runs great
OLMoE 1B 7B 09246.9BChat Q4_K_M·1039.0 t/s	Q4_K_M	5.0 GB1%	1039.0 t/s	4K	S100, Runs great
Meta Llama 3.1 8B8.0BChat Q4_K_M·1967.1 t/s	Q4_K_M	7.3 GB1%	1967.1 t/s	131K	S100, Runs great
Llama Guard 3 8B8.0BChat BF16·600.0 t/s	BF16	20.2 GB3%	600.0 t/s	—	S100, Runs great
Qwen3 4B Base4.0BChat Q4_K_M·3656.1 t/s	Q4_K_M	4.2 GB1%	3656.1 t/s	33K	S100, Runs great
Qwen 7B7.7BChat Q4_K_M·2079.0 t/s	Q4_K_M	7.6 GB1%	2079.0 t/s	33K	S100, Runs great
Llama 2 7B Chat HF6.7BChat Q4_K_M·2382.7 t/s	Q4_K_M	6.9 GB1%	2382.7 t/s	—	S100, Runs great
Mellum2 12B A2.5B Instruct SFT12.1BChat Q4_K_M·593.5 t/s	Q4_K_M	7.8 GB1%	593.5 t/s	131K	S100, Runs great
Meta Llama 3 8B Instruct8.0BChat Q4_K_M·2000.5 t/s	Q4_K_M	7.9 GB1%	2000.5 t/s	—	S100, Runs great

NVIDIA DGX A100 640GB specs and power

Specs

Chip: 8x A100 SXM4
CPU cores: 128
GPU cores: 55,296
Unified memory: 640 GB
For models: 640 GB
Memory bandwidth: 16312 GB/s
FP16 (dense): 2496 TFLOPS
Power: 6500 W max
Released: 2020-05-14
MSRP: $199,000

NVIDIA DGX A100 640GB FAQ

Is the NVIDIA DGX A100 640GB worth it for local LLMs in 2026?

At its $199,000 MSRP it gives 2.0 tok/s per $1,000 on Qwen3.6 35B A3B, 6th of the 6 machines compared here. It runs 70–120B models well.

How much memory is available for AI on the NVIDIA DGX A100 640GB?

All 640 GB can hold model weights.

What is the largest model the NVIDIA DGX A100 640GB can run?

MiMo V2.6 Pro RL (1020B) is the largest at 4-bit or better: at Q4_K_M it uses 614.1 GB, 96% of the available memory, and runs at about 161 tok/s.

Can the NVIDIA DGX A100 640GB run DeepSeek V4.1 Flash 1017?

Yes. DeepSeek V4.1 Flash 1017 fits at Q4_K_M, using 458.4 GB of the 640 GB available, and it is the best model that runs well on this machine.

How long does the NVIDIA DGX A100 640GB take to read a long prompt?

On DeepSeek V4.1 Flash 1017, about 120 ms for a 4K-token prompt and 930 ms for a 32K-token document before the first word appears.

48 GB GPUs Servers DGX H100 Best coding LLMs All hardware