NVIDIA · 8x H200 SXM5 · Server
Best AI Models for NVIDIA DGX H200 (1.1 TB)
Runs models up to 1.7T parameters, like DeepSeek V4 Pro 0813.
1.1 TB HBM38.4 TB/s7.9 PFLOPS FP1610.2 kW2024
| # | Model | Speed | llmrun Score | Quant · Memory |
|---|---|---|---|---|
| 1 | Inkling 2 SmallThinking MachinesHow to run → Q4_K_M · 359.1 GB · 289 t/s | 289 tok/s | Q4_K_M · 359.1 GB | |
| 2 | DeepSeek V4.1 Flash 1017DeepSeek Q4_K_M · 458.4 GB · 337 t/s | 337 tok/s | Q4_K_M · 458.4 GB | |
| 3 | MiMo V2.6 Pro RLXiaomi Q4_K_M · 614.1 GB · 206 t/s | 206 tok/s | Q4_K_M · 614.1 GB | |
| 4 | MiMo V2.6 Flash RLXiaomi Q4_K_M · 186.5 GB · 245 t/s | 245 tok/s | Q4_K_M · 186.5 GB | |
| 5 | Hy3Tencent Q4_K_M · 182.3 GB · 156 t/s | 156 tok/s | Q4_K_M · 182.3 GB |
| # | Model | Speed | Coding Score | Quant · Memory |
|---|---|---|---|---|
| 1 | Q4_K_M · 458.4 GB · 337 t/s | 337 tok/s | Q4_K_M · 458.4 GB | |
| 2 | GLM 5.3Z.ai Q4_K_M · 453.8 GB · 158 t/s | 158 tok/s | Q4_K_M · 453.8 GB | |
| 3 | Q4_K_M · 551.9 GB · 206 t/s | 206 tok/s | Q4_K_M · 551.9 GB | |
| 4 | DeepSeek V4 Pro 0813DeepSeek Q4_K_M · 990.8 GB · 172 t/s | 172 tok/s | Q4_K_M · 990.8 GB | |
| 5 | Hy3Tencent Q4_K_M · 182.3 GB · 156 t/s | 156 tok/s | Q4_K_M · 182.3 GB |
| # | Model | Speed | Reasoning Score | Quant · Memory |
|---|---|---|---|---|
| 1 | Q4_K_M · 458.4 GB · 337 t/s | 337 tok/s | Q4_K_M · 458.4 GB | |
| 2 | Inkling 2 SmallThinking Machines Q4_K_M · 359.1 GB · 289 t/s | 289 tok/s | Q4_K_M · 359.1 GB | |
| 3 | DeepSeek V4.1 FlashDeepSeek Q4_K_M · 458.4 GB · 337 t/s | 337 tok/s | Q4_K_M · 458.4 GB | |
| 4 | GLM 5.3Z.ai Q4_K_M · 453.8 GB · 158 t/s | 158 tok/s | Q4_K_M · 453.8 GB | |
| 5 | Hy3Tencent Q4_K_M · 182.3 GB · 156 t/s | 156 tok/s | Q4_K_M · 182.3 GB |
| # | Model | Speed | Math Score | Quant · Memory |
|---|---|---|---|---|
| 1 | Q4_K_M · 458.4 GB · 337 t/s | 337 tok/s | Q4_K_M · 458.4 GB | |
| 2 | DeepSeek V4.1 FlashDeepSeek Q4_K_M · 458.4 GB · 337 t/s | 337 tok/s | Q4_K_M · 458.4 GB | |
| 3 | GLM 5.3Z.ai Q4_K_M · 453.8 GB · 158 t/s | 158 tok/s | Q4_K_M · 453.8 GB | |
| 4 | Q4_K_M · 551.9 GB · 206 t/s | 206 tok/s | Q4_K_M · 551.9 GB | |
| 5 | Hy3Tencent Q4_K_M · 182.3 GB · 156 t/s | 156 tok/s | Q4_K_M · 182.3 GB |
| # | Model | Speed ↓ | llmrun Score | Quant · Memory |
|---|---|---|---|---|
| 1 | Qwen3.5 4BAlibabaHow to run → Q4_K_M · 3.4 GB · 957 t/s | 957 tok/s | Q4_K_M · 3.4 GB | |
| 2 | Qwen3.5 9BAlibaba Q4_K_M · 6.6 GB · 491 t/s | 491 tok/s | Q4_K_M · 6.6 GB | |
| 3 | GPT OSS 20BOpenAI Q4_K_M · 13.1 GB · 465 t/s | 465 tok/s | Q4_K_M · 13.1 GB | |
| 4 | Inkling SmallThinking Machines Q4_K_M · 162.7 GB · 404 t/s | 404 tok/s | Q4_K_M · 162.7 GB | |
| 5 | Hy3Tencent Q4_K_M · 182.3 GB · 156 t/s | 156 tok/s | Q4_K_M · 182.3 GB |
| # | Model | Speed | Vision Score | Quant · Memory |
|---|---|---|---|---|
| 1 | Inkling 2 SmallThinking MachinesHow to run → Q4_K_M · 359.1 GB · 289 t/s | 289 tok/s | Q4_K_M · 359.1 GB | |
| 2 | DeepSeek V4.1 Flash 1017DeepSeek Q4_K_M · 458.4 GB · 337 t/s | 337 tok/s | Q4_K_M · 458.4 GB | |
| 3 | DeepSeek V4.1 FlashDeepSeek Q4_K_M · 458.4 GB · 337 t/s | 337 tok/s | Q4_K_M · 458.4 GB | |
| 4 | Q4_K_M · 551.9 GB · 206 t/s | 206 tok/s | Q4_K_M · 551.9 GB | |
| 5 | Muse Glimmer 30BMeta Q4_K_M · 19.6 GB · 170 t/s | 170 tok/s | Q4_K_M · 19.6 GB |
| # | Model | Speed | Agents Score | Quant · Memory |
|---|---|---|---|---|
| 1 | Q4_K_M · 458.4 GB · 337 t/s | 337 tok/s | Q4_K_M · 458.4 GB | |
| 2 | GLM 5.3Z.ai Q4_K_M · 453.8 GB · 158 t/s | 158 tok/s | Q4_K_M · 453.8 GB | |
| 3 | MiMo V2.6 Pro RLXiaomi Q4_K_M · 614.1 GB · 206 t/s | 206 tok/s | Q4_K_M · 614.1 GB | |
| 4 | GLM 5.2Z.ai Q4_K_M · 453.8 GB · 158 t/s | 158 tok/s | Q4_K_M · 453.8 GB | |
| 5 | Hy3Tencent Q4_K_M · 182.3 GB · 156 t/s | 156 tok/s | Q4_K_M · 182.3 GB |
See all 452 models on the NVIDIA DGX H200 →Bars = traction for models without a Score yet
The NVIDIA DGX H200 runs models up to 1.7T parameters, like DeepSeek V4 Pro 0813. A 4K-token prompt waits 80 ms before the first word on Inkling 2 Small. 8x H200 SXM5 (Hopper), released 2024.
For models1.1 TBall of it
Chip8x H200 SXM5released 2024
NVIDIA DGX H200 strengths and weaknesses
Strengths
- Runs 120B MoE models: GPT OSS 120B at 323 tok/s in 70.8 GB
- Room for long context: Llama 3.3 70B Instruct at Q4 uses 51.3 GB, leaving 1.1 TB free
- Reads long prompts fast: A 32K-token prompt on Inkling 2 Small starts answering in 660 ms
Memory for models
1.1 TB of 1.1 TB120B MoE70.8 GB1650B990.8 GB
120B MoE70.8 GB1650B990.8 GB
For models 1.1 TBModel size at Q4
Everything on the size ladder fits. Inkling 2 Small and DeepSeek V4.1 Flash 1017 together use 817.5 GB and leave 310.5 GB free.
The same chip from other makers first, then machines and GPUs people weigh against it, each with the one reason it is worth a look. Speeds on Qwen3.6 35B A3B.
Step down to 508.5 GBMac Studio M5 Ultra (512 GB)Apple · 508.5 GB for models · M5 Ultra
▼ 72% speed▼ 619.5 GBabout $22,299
Step down to 508.5 GBMac Studio M3 Ultra (512 GB)Apple · 508.5 GB for models · M3 Ultra▼ 76% speed▼ 619.5 GBabout $9,499
Step down to 188.5 GBMac Studio M2 Ultra (192 GB)Apple · 188.5 GB for models · M2 Ultra▼ 76% speed▼ 939.5 GBabout $5,599
Step down to 124.5 GBMac Studio M4 Max (128 GB)Apple · 124.5 GB for models · M4 Max▼ 80% speed▼ 1 TBabout $3,499
Step down to 24 GBRTX 4090NVIDIA · 24 GB VRAM · GPU▼ 53% speed▼ 1.1 TBabout $1,599rent $0.34/hr
Step down to 32 GBRTX 5090NVIDIA · 32 GB VRAM · GPU▼ 33% speed▼ 1.1 TBabout $5,502rent $0.50/hr
Output speed depends on memory bandwidth: each model runs on as few of the 8 GPUs as hold it, at 4.8 TB/s per GPU. Grey rows need more than the 1.1 TB available.
- 9BQwen3.5 9B · Q4_K_M · 6.6 GB491tok/s
- 14BQwen3 14B · Q4_K_M · 11.9 GB328tok/s
- 27–31BQwen3.8 27B · Q4_K_M · 17.9 GB179tok/s
- 35B MoEQwen3.6 35B A3B · Q4_K_M · 22.0 GB320tok/s
- 70BLlama 3.3 70B Instruct · Q4_K_M · 51.3 GB67.0tok/s
- 120B MoEGPT OSS 120B · Q4_K_M · 70.8 GB323tok/s
Estimated speeds: Q4_K_M, 4K context, one user. How we estimate
All 452 models on the NVIDIA DGX H200
Featured onlyBest quant that fits in 1128 GB, ordered by llmrun Score. Fit grades how well it runs here, S best. How ranking works
| Model | Quant | VRAM | Speed | Context | Score | Fit |
|---|---|---|---|---|---|---|
Qwen AgentWorld 35B A3BChatFunctions | Q4_K_M | 21.3 GB | 347.6 t/s | 262K | S100, Runs great | |
Qwen3 30B A3B Instruct 2507ChatFunctions | Q4_K_M | 19.4 GB | 303.1 t/s | 262K | S100, Runs great | |
LFM2.5 230MChatFunctions | Q4_K_M | 0.6 GB | 6240.0 t/s | 128K | S100, Runs great | |
| Q4_K_M | 8.4 GB | 541.7 t/s | — | S100, Runs great | ||
Qwen3 0.6BChatFunctions | Q4_K_M | 1.7 GB | 3586.2 t/s | 41K | S100, Runs great | |
Unlimited OCR 2Vision | Q4_K_M | 3.3 GB | 1078.1 t/s | 131K | S100, Runs great | |
MiniCPM V 4Vision | Q4_K_M | 3.1 GB | 1122.3 t/s | 33K | S100, Runs great | |
InternVL3 5 1BVision | Q4_K_M | 2.2 GB | 4457.1 t/s | — | S100, Runs great | |
Olmo 3 32B ThinkChat | Q4_K_M | 21.5 GB | 154.6 t/s | 66K | S97, Runs great | |
Cogito 671B V2.1ChatFunctions | Q4_K_M | 404.1 GB | 194.1 t/s | 164K | S99, Runs great | |
ChandraVisionFunctions | Q4_K_M | 8.0 GB | 532.4 t/s | 262K | S100, Runs great | |
Ling 3.0 Flash VLVisionFunctions | Q4_K_M | 76 GB | 288.2 t/s | 131K | S100, Runs great | |
Qwen3 235B A22BChatFunctions | Q4_K_M | 142.9 GB | 136.8 t/s | 41K | S96, Runs great | |
DeepSeek OCRVision | Q4_K_M | 2.8 GB | 1079.5 t/s | 8K | S100, Runs great | |
ERNIE 4.5 VL 28B A3B PTVision | BF16 | 60.0 GB | 287.6 t/s | 131K | S100, Runs great |
Specs
- Chip
- 8x H200 SXM5 · NVIDIA H200
- CPU cores
- 112
- GPU cores
- 135,168
- Memory
- 1.1 TB
- For models
- 1.1 TB
- Memory bandwidth
- 38.4 TB/s
- FP16 (dense)
- 7.9 PFLOPS
- Power
- 10.2 kW max
- Size
- 8U
- Released
- 2024-04-24
NVIDIA DGX H200 FAQ
How much memory is available for AI on the NVIDIA DGX H200?
All 1.1 TB can hold model weights.
What is the largest model the NVIDIA DGX H200 can run?
DeepSeek V4 Pro 0813 (1650.5B) is the largest at 4-bit or better: at Q4_K_M it uses 990.8 GB, 88% of the available memory, and runs at about 172 tok/s.
Can the NVIDIA DGX H200 run Inkling 2 Small?
Yes. Inkling 2 Small fits at Q4_K_M, using 359.1 GB of the 1.1 TB available, and it is the best model that runs well on this machine.
How long does the NVIDIA DGX H200 take to read a long prompt?
On Inkling 2 Small, about 80 ms for a 4K-token prompt and 660 ms for a 32K-token document before the first word appears.