NVIDIA · 8x H100 SXM5 · Server

Best AI Models for NVIDIA DGX H100 (640 GB)

Runs 70–120B models well.

640 GB unified26800 GB/s7915 TFLOPS FP1610200 W2022

What LLMs can the NVIDIA DGX H100 run?

#	Model	Speed	Quant · Memory
1	DeepSeek V4.1 Flash 1017DeepSeekHow to run → Q4_K_M · 458.4 GB · 363 t/s	363 tok/s	Q4_K_M · 458.4 GB
2	MiMo V2.6 Flash RLXiaomi Q4_K_M · 186.5 GB · 317 t/s	317 tok/s	Q4_K_M · 186.5 GB
3	GLM 5.3Z.ai Q4_K_M · 453.8 GB · 174 t/s	174 tok/s	Q4_K_M · 453.8 GB
4	DeepSeek V4.1 FlashDeepSeek Q4_K_M · 458.4 GB · 363 t/s	363 tok/s	Q4_K_M · 458.4 GB
5	Hy3Tencent Q4_K_M · 182.3 GB · 195 t/s	195 tok/s	Q4_K_M · 182.3 GB

#	Model	Speed	Quant · Memory
1	DeepSeek V4.1 Flash 1017DeepSeekHow to run → Q4_K_M · 458.4 GB · 363 t/s	363 tok/s	Q4_K_M · 458.4 GB
2	GLM 5.3Z.ai Q4_K_M · 453.8 GB · 174 t/s	174 tok/s	Q4_K_M · 453.8 GB
3	Muse Spark Open 816B A42BMeta Q4_K_M · 551.9 GB · 236 t/s	236 tok/s	Q4_K_M · 551.9 GB
4	MiMo V2.6 Flash RLXiaomi Q4_K_M · 186.5 GB · 317 t/s	317 tok/s	Q4_K_M · 186.5 GB
5	Hy3Tencent Q4_K_M · 182.3 GB · 195 t/s	195 tok/s	Q4_K_M · 182.3 GB

#	Model	Speed	Quant · Memory
1	DeepSeek V4.1 Flash 1017DeepSeekHow to run → Q4_K_M · 458.4 GB · 363 t/s	363 tok/s	Q4_K_M · 458.4 GB
2	DeepSeek V4.1 FlashDeepSeek Q4_K_M · 458.4 GB · 363 t/s	363 tok/s	Q4_K_M · 458.4 GB
3	GLM 5.3Z.ai Q4_K_M · 453.8 GB · 174 t/s	174 tok/s	Q4_K_M · 453.8 GB
4	Muse Spark Open 816B A42BMeta Q4_K_M · 551.9 GB · 236 t/s	236 tok/s	Q4_K_M · 551.9 GB
5	Hy3Tencent Q4_K_M · 182.3 GB · 195 t/s	195 tok/s	Q4_K_M · 182.3 GB

#	Model	Speed ↓	Quant · Memory
1	Qwen3.5 4BAlibabaHow to run → Q4_K_M · 3.4 GB · 5344 t/s	5344 tok/s	Q4_K_M · 3.4 GB
2	Qwen3.5 9BAlibaba Q4_K_M · 6.6 GB · 2739 t/s	2739 tok/s	Q4_K_M · 6.6 GB
3	Gemma 4 31B ITGoogle Q4_K_M · 21.7 GB · 854 t/s	854 tok/s	Q4_K_M · 21.7 GB
4	DeepSeek R1 Distill Qwen 32BDeepSeek Q4_K_M · 24.3 GB · 850 t/s	850 tok/s	Q4_K_M · 24.3 GB
5	Hy3Tencent Q4_K_M · 182.3 GB · 195 t/s	195 tok/s	Q4_K_M · 182.3 GB

#	Model	Speed	Quant · Memory
1	DeepSeek V4.1 Flash 1017DeepSeekHow to run → Q4_K_M · 458.4 GB · 363 t/s	363 tok/s	Q4_K_M · 458.4 GB
2	DeepSeek V4.1 FlashDeepSeek Q4_K_M · 458.4 GB · 363 t/s	363 tok/s	Q4_K_M · 458.4 GB
3	Muse Spark Open 816B A42BMeta Q4_K_M · 551.9 GB · 236 t/s	236 tok/s	Q4_K_M · 551.9 GB
4	GLM 5.3 FlashZ.ai Q4_K_M · 205.2 GB · 321 t/s	321 tok/s	Q4_K_M · 205.2 GB
5	Qwen3.8 Flash NextAlibaba Q4_K_M · 109.0 GB · 210 t/s	210 tok/s	Q4_K_M · 109.0 GB

#	Model	Speed	Quant · Memory
1	DeepSeek V4.1 Flash 1017DeepSeekHow to run → Q4_K_M · 458.4 GB · 363 t/s	363 tok/s	Q4_K_M · 458.4 GB
2	GLM 5.3Z.ai Q4_K_M · 453.8 GB · 174 t/s	174 tok/s	Q4_K_M · 453.8 GB
3	GLM 5.2Z.ai Q4_K_M · 453.8 GB · 174 t/s	174 tok/s	Q4_K_M · 453.8 GB
4	DeepSeek V4.1 FlashDeepSeek Q4_K_M · 458.4 GB · 363 t/s	363 tok/s	Q4_K_M · 458.4 GB
5	Hy3Tencent Q4_K_M · 182.3 GB · 195 t/s	195 tok/s	Q4_K_M · 182.3 GB

See all 440 models on the NVIDIA DGX H100 →Bars = traction for models without a Score yet

About the NVIDIA DGX H100

The NVIDIA DGX H100 runs 70–120B models well. Prompt reading is its weak spot: a 4K-token prompt waits 80 ms before the first word on DeepSeek V4.1 Flash 1017. At its $482,000 MSRP it gives 0.9 tok/s per $1,000, 6th of the 6 machines compared here. 8x H100 SXM5, released 2022.

For models640 GBall of it

Chip8x H100 SXM5released 2022

NVIDIA DGX H100 strengths and weaknesses

Strengths

Runs 120B MoE models: GPT OSS 120B at 459 tok/s in 70.8 GB
Room for long context: Llama 3.3 70B Instruct at Q4 uses 51.3 GB, leaving 588.7 GB free
Reads long prompts fast: A 32K-token prompt on DeepSeek V4.1 Flash 1017 starts answering in 660 ms

Memory for models

640 GB of 640 GB

120B MoE70.8 GB1020B614.1 GB

For models 640 GBThe system keeps 0 GBModel size at Q4

Everything on the size ladder fits. MiMo V2.6 Flash RL and DeepSeek V4 Flash 0731 together use 369.5 GB and leave 270.5 GB free.

Speed per dollar and per watt

NewUsed

On Qwen3.6 35B A3B (35B params, 3B active) at Q4_K_M. New means a shop price, either a street quote or the maker's own price; used is compared with used. The last two columns are the speed you get per $1,000 of price and per 100 W of power; higher is better. The NVIDIA DGX H100 has no current new price, so it is shown at its MSRP.

Machine	New price	tok/s	tok/s per $1,000	tok/s per 100 W
RTX 509032 GB	about $5,376	213	39.7	37.1
Mac Studio M4 Max128 GB	about $3,499	64.2	18.3	44.3
Mac Studio M2 Ultra192 GB	about $5,599	75.9	13.6	25.7
MacBook Pro 16" M5 Max128 GB	about $5,399	67.8	12.6	–
Mac Studio M3 Ultra512 GB	about $9,499	76.6	8.1	28.4
DGX H100640 GB	about $482,000	423	0.9	4.1

Machine	Used price	tok/s	tok/s per $1,000	tok/s per 100 W
RTX 409024 GB	about $3,000	151	50.3	33.6
RTX 509032 GB	about $6,400	213	33.3	37.1

Cost per token: the NVIDIA DGX H100 against 24–640 GB machines and GPUs

On Qwen3.6 35B A3B (35B params, 3B active, Q4_K_M, 4K context). Speed, cost per token and power per token depend on the model.

24–36 GB96–128 GB192+ GBthis serverat MSRP

Cost per 1M output tokenson Qwen3.6 35B A3B · card over 3 years + power

$73.34

Machine, MSRP: $72.33
Power: $1.01

3-year cost of ownership$488,701

Include card price

Power$/kWhUseh/dayKeepyears

Each dot is a machine or GPU; hover or tap it for its numbers, click to open it. Machines without a price for the chosen basis are left out. Machines without a published power figure are left out of Per watt and costed without power. How we calculate cost per token

Similar devices and alternatives

The same chip from other makers first, then machines and GPUs people weigh against it, each with the one reason it is worth a look. Speeds on Qwen3.6 35B A3B.

Machine, 82% slowerMac Studio M3 Ultra (512 GB)Apple · 508.5 GB for models · M3 Ultra

▼ 82% speed▼ 131.5 GB$9,499

Step down to 188.5 GBMac Studio M2 Ultra (192 GB)Apple · 188.5 GB for models · M2 Ultra

▼ 82% speed▼ 451.5 GB$5,599

Step down to 124.5 GBMac Studio M4 Max (128 GB)Apple · 124.5 GB for models · M4 Max

▼ 85% speed▼ 515.5 GB$3,499

Step down to 124.5 GBMacBook Pro 16" M5 Max (128 GB)Apple · 124.5 GB for models · M5 Max

▼ 84% speed▼ 515.5 GB$5,399

Step down to 24 GBRTX 4090NVIDIA · 24 GB VRAM · GPU

▼ 64% speed▼ 616 GB$1,599rent $0.43/hr

Step down to 32 GBRTX 5090NVIDIA · 32 GB VRAM · GPU

▼ 49% speed▼ 608 GB$5,376rent $0.50/hr

NVIDIA DGX H100 tokens per second by model size

Output speed depends on memory bandwidth (26800 GB/s). Grey rows need more than the 640 GB available.

9BQwen3.5 9B · Q4_K_M · 6.6 GB
2739tok/s
14BQwen3 14B · Q4_K_M · 11.9 GB
1834tok/s
27–31BQwen3.8 27B · Q4_K_M · 17.9 GB
1000tok/s
35B MoEQwen3.6 35B A3B · Q4_K_M · 22.0 GB
423tok/s
70BLlama 3.3 70B Instruct · Q4_K_M · 51.3 GB
374tok/s
120B MoEGPT OSS 120B · Q4_K_M · 70.8 GB
459tok/s

Estimated speeds: Q4_K_M, 4K context, one user. How we estimate

Unified-memory platforms: writing vs compute

Writing, tok/sCompute, TFLOPS

M3 Ultra512 GB · 819 GB/s
76.6
66
8x H100 SXM5640 GB · 26800 GB/s
423
7915

Machines with about the same memory bandwidth write at about the same speed; compute decides how fast prompts are read. Speeds on Qwen3.6 35B A3B.

All 1355 models on the NVIDIA DGX H100

Featured only

Best quant that fits in 640 GB, ordered by llmrun Score. Fit grades how well it runs here, S best. How ranking works

LLM models compatible with NVIDIA DGX H100 — ranked by Score, recency and Fit
Model	Quant	VRAM	Speed	Context	Fit
Step 3.5 Flash Base197.8BChat Q4_K_M·358.0 t/s	Q4_K_M	136.8 GB21%	358.0 t/s	33K	S100, Runs great
Qwen2.5 32B Instruct32.8BChatFunctions Q4_K_M·849.8 t/s	Q4_K_M	24.3 GB4%	849.8 t/s	33K	S100, Runs great
Nidum Gemma 2B Uncensored2.5BChat Q4_K_M·9467.4 t/s	Q4_K_M	1.9 GB0%	9467.4 t/s	8K	S100, Runs great
Llama3 8B Chinese Chat8.0BChat Q4_K_M·3231.9 t/s	Q4_K_M	6.2 GB1%	3231.9 t/s	8K	S100, Runs great
GPT OSS Safeguard 20B21.5BChatFunctions Q4_K_M·671.2 t/s	Q4_K_M	13.5 GB2%	671.2 t/s	131K	S100, Runs great
Qwen3.6 27B MTPLX Optimized26.9BChat Q4_K_M·1032.0 t/s	Q4_K_M	17.3 GB3%	1032.0 t/s	262K	S100, Runs great
Finance Llama3 8B8.0BChat Q4_K_M·3231.9 t/s	Q4_K_M	6.2 GB1%	3231.9 t/s	8K	S100, Runs great
Vicuna 13B V1.313BChat Q4_K_M·2030.3 t/s	Q4_K_M	8.6 GB1%	2030.3 t/s	2K	S100, Runs great
Mellum2 12B A2.5B Thinking12.1BChatFunctions Q4_K_M·614.1 t/s	Q4_K_M	7.8 GB1%	614.1 t/s	131K	S100, Runs great
Phi 4 Mini Instruct3.8BChatCodeFunctions Q4_K_M·6069.7 t/s	Q4_K_M	4.8 GB1%	6069.7 t/s	131K	S100, Runs great
TinyStories 1M1MChat BF16·1742000.0 t/s	BF16	0.0 GB0%	1742000.0 t/s	2K	S100, Runs great
MobileLLaMA 1.4B Chat1.4BChat Q4_K_M·11311.7 t/s	Q4_K_M	1.5 GB0%	11311.7 t/s	2K	S100, Runs great
StableBeluga270BChat Q4_K_M·405.4 t/s	Q4_K_M	43.6 GB7%	405.4 t/s	4K	S100, Runs great
HarmBench Llama 2 13B Cls13.0BChat Q4_K_M·1779.4 t/s	Q4_K_M	9.8 GB2%	1779.4 t/s	2K	S100, Runs great
Vicuna 33B V1.333BChat Q4_K_M·799.8 t/s	Q4_K_M	21.8 GB3%	799.8 t/s	2K	S100, Runs great

NVIDIA DGX H100 specs and power

Specs

Chip: 8x H100 SXM5
CPU cores: 112
GPU cores: 135,168
Unified memory: 640 GB
For models: 640 GB
Memory bandwidth: 26800 GB/s
FP16 (dense): 7915.2 TFLOPS
Power: 10200 W max
Released: 2022-09-01
MSRP: $482,000

NVIDIA DGX H100 FAQ

Is the NVIDIA DGX H100 worth it for local LLMs in 2026?

At its $482,000 MSRP it gives 0.9 tok/s per $1,000 on Qwen3.6 35B A3B, 6th of the 6 machines compared here. It runs 70–120B models well.

How much memory is available for AI on the NVIDIA DGX H100?

All 640 GB can hold model weights.

What is the largest model the NVIDIA DGX H100 can run?

MiMo V2.6 Pro RL (1020B) is the largest at 4-bit or better: at Q4_K_M it uses 614.1 GB, 96% of the available memory, and runs at about 189 tok/s.

Can the NVIDIA DGX H100 run DeepSeek V4.1 Flash 1017?

Yes. DeepSeek V4.1 Flash 1017 fits at Q4_K_M, using 458.4 GB of the 640 GB available, and it is the best model that runs well on this machine.

How long does the NVIDIA DGX H100 take to read a long prompt?

On DeepSeek V4.1 Flash 1017, about 80 ms for a 4K-token prompt and 660 ms for a 32K-token document before the first word appears.

48 GB GPUs Servers DGX A100 640GB Best coding LLMs All hardware