llmrun

NVIDIA · Hopper

Best AI Models for NVIDIA H100 SXM (80 GB)

Runs everything up to 120B MoE well.

80 GB HBM33352 GB/s989 TFLOPS FP16700 WHopper 2022

What LLMs can the H100 SXM run?

#ModelSpeedllmrun ScoreQuant · VRAM
1
Qwen3.8 27BAlibabaHow to run →
Q4_K_M · 17.9 GB · 125 t/s
125 tok/sQ4_K_M · 17.9 GB
2
Qwen3.6 27BAlibaba
Q4_K_M · 17.9 GB · 125 t/s
125 tok/sQ4_K_M · 17.9 GB
3
Gemma 4 31B ITGoogle
Q4_K_M · 21.7 GB · 107 t/s
107 tok/sQ4_K_M · 21.7 GB
4
Gemma 4 26B A4B ITGoogle
Q4_K_M · 16.5 GB · 344 t/s
344 tok/sQ4_K_M · 16.5 GB
5
NVIDIA Nemotron 3.5 Lightning 30B A3B BF16NVIDIA
Q4_K_M · 19.3 GB · 284 t/s
284 tok/sQ4_K_M · 19.3 GB
See all 383 models on the H100 SXM →Best quant that fits · speeds estimated on this card

About the H100 SXM

The H100 SXM runs everything up to 120B MoE well. Hopper generation, released 2022.

GenerationHopperreleased 2022

H100 SXM strengths and weaknesses

Strengths

  • Runs 120B MoE models: GPT OSS 120B at 280 tok/s in 70.8 GB
  • Room for long context: Llama 3.3 70B Instruct at Q4 uses 51.3 GB, leaving 28.7 GB free
  • Reads long prompts fast: A 4K-token prompt on Qwen3.8 27B starts answering in 511 ms

Weaknesses

  • 700 W under load: Plan a 1,400 W PSU; ≈$153 a year at 4 h a day

H100 SXM tokens per second by model size

Output speed depends on memory bandwidth (3352 GB/s).

  1. 9BQwen3.5 9B · Q4_K_M · 6.6 GB
    343tok/s
  2. 14BQwen3 14B · Q4_K_M · 11.9 GB
    229tok/s
  3. 27–31BQwen3.8 27B · Q4_K_M · 17.9 GB
    125tok/s
  4. 35B MoEQwen3.6 35B A3B · Q4_K_M · 22.0 GB
    283tok/s
  5. 70BLlama 3.3 70B Instruct · Q4_K_M · 51.3 GB
    46.8tok/s
  6. 120B MoEGPT OSS 120B · Q4_K_M · 70.8 GB
    280tok/s

Estimated speeds: Q4_K_M, 4K context, one user. How we estimate

All 1258 models on the H100 SXM

Featured only

Best quant that fits, ordered by llmrun Score. Fit grades how well it runs here, S best. How ranking works

LLM models compatible with NVIDIA H100 SXM — ranked by Score, recency and Fit
ModelVRAMScoreFit
Medgemma 27B Text IT
BF16·36.7 t/s
63.0 GBA73, Runs well
Qwen 7B Chat
Q4_K_M·427.2 t/s
7.6 GBS100, Runs great
Qwen2 1.5B
Q4_K_M·1702.2 t/s
1.7 GBS100, Runs great
Falcon 7B Instruct
Q4_K_M·457.7 t/s
7.3 GBS100, Runs great
Qwen2.5 Coder 32B
Q4_K_M·106.3 t/s
24.3 GBS92, Runs great
Granite Guardian 3.3 8B
Q4_K_M·393.3 t/s
7.9 GBS100, Runs great
Internlm Chat 7B
FP16·141.5 t/s
15.4 GBS96, Runs great
Gemma 2 9B
Q4_K_M·357.2 t/s
8.8 GBS100, Runs great
Deepseek Coder 6.7B Instruct
Q4_K_M·402.0 t/s
12.9 GBS100, Runs great
Qwen2.5 Coder 7B
Q4_K_M·436.6 t/s
5.8 GBS100, Runs great
Llama 2 13B HF
Q4_K_M·253.6 t/s
11.5 GBS100, Runs great
Llama 2 7B HF
Q4_K_M·489.6 t/s
6.9 GBS100, Runs great
Granite Guardian 3.2 8B Factuality Detection
Q4_K_M·393.3 t/s
7.9 GBS100, Runs great
Llama 3.1 70B
Q4_K_M·46.8 t/s
51.3 GBA78, Runs well
Qwen1.5 MoE A2.7B
Q4_K_M·471.5 t/s
10.5 GBS100, Runs great

H100 SXM specs, power and PSU

Specs

Architecture
Hopper
VRAM
80 GB HBM3
Memory bandwidth
3352.0 GB/s
FP16 (dense)
989.40 TFLOPS
CUDA cores
16,896
Tensor cores
528
Compute capability
9.0
TDP
700 W
Released
2022-09-01

H100 SXM FAQ

What is the largest model the H100 SXM can run?

Mistral Medium 3.5 128B (127.7B) is the largest at 4-bit or better: at Q4_K_S it uses 78.0 GB of 80 GB, and runs at about 29.9 tok/s.

Can the H100 SXM run Qwen3.8 27B?

Yes. Qwen3.8 27B fits at Q4_K_M, using 17.9 GB of 80 GB, and it is the best model that runs well on this card.

How many tokens per second does the H100 SXM give?

About 125 tok/s on Qwen3.8 27B at Q4_K_M, with the first token after 0.5 s on a 4K-token prompt.

What power supply does the H100 SXM need?

Plan a 1400 W PSU (2× the 700 W TDP).

Is 80 GB enough for coding and agents?

Yes for coding: Qwen3.8 27B, the best of the 12 coding models that fit, runs at about 125 tok/s. For agents, not yet: no agent-scored model fits in less than 96 GB.

Related

Best coding LLMs48 GB VRAMGPUs with 80 GB or moreRTX 5090