llmrun

NVIDIA · Hopper

Best AI Models for NVIDIA H100 PCIe (80 GB)

Runs everything up to 120B MoE well.

80 GB HBM2e2039 GB/s757 TFLOPS FP16350 WHopper 2022

What LLMs can the H100 PCIe run?

#ModelSpeedllmrun ScoreQuant · VRAM
1
Qwen3.8 27BAlibabaHow to run →
Q4_K_M · 17.9 GB · 76.1 t/s
76.1 tok/sQ4_K_M · 17.9 GB
2
Qwen3.6 27BAlibaba
Q4_K_M · 17.9 GB · 76.1 t/s
76.1 tok/sQ4_K_M · 17.9 GB
3
Gemma 4 31B ITGoogle
Q4_K_M · 21.7 GB · 65.0 t/s
65.0 tok/sQ4_K_M · 21.7 GB
4
Gemma 4 26B A4B ITGoogle
Q4_K_M · 16.5 GB · 269 t/s
269 tok/sQ4_K_M · 16.5 GB
5
NVIDIA Nemotron 3.5 Lightning 30B A3B BF16NVIDIA
Q4_K_M · 19.3 GB · 253 t/s
253 tok/sQ4_K_M · 19.3 GB
See all 383 models on the H100 PCIe →Best quant that fits · speeds estimated on this card

About the H100 PCIe

The H100 PCIe runs everything up to 120B MoE well. Hopper generation, released 2022.

GenerationHopperreleased 2022

H100 PCIe strengths and weaknesses

Strengths

  • Runs 120B MoE models: GPT OSS 120B at 218 tok/s in 70.8 GB
  • Room for long context: Llama 3.3 70B Instruct at Q4 uses 51.3 GB, leaving 28.7 GB free
  • Reads long prompts fast: A 4K-token prompt on Qwen3.8 27B starts answering in 669 ms

H100 PCIe tokens per second by model size

Output speed depends on memory bandwidth (2039 GB/s).

  1. 9BQwen3.5 9B · Q4_K_M · 6.6 GB
    208tok/s
  2. 14BQwen3 14B · Q4_K_M · 11.9 GB
    140tok/s
  3. 27–31BQwen3.8 27B · Q4_K_M · 17.9 GB
    76.1tok/s
  4. 35B MoEQwen3.6 35B A3B · Q4_K_M · 22.0 GB
    228tok/s
  5. 70BLlama 3.3 70B Instruct · Q4_K_M · 51.3 GB
    28.5tok/s
  6. 120B MoEGPT OSS 120B · Q4_K_M · 70.8 GB
    218tok/s

Estimated speeds: Q4_K_M, 4K context, one user. How we estimate

All 1258 models on the H100 PCIe

Featured only

Best quant that fits, ordered by llmrun Score. Fit grades how well it runs here, S best. How ranking works

LLM models compatible with NVIDIA H100 PCIe — ranked by Score, recency and Fit
ModelVRAMScoreFit
Openchat 3.5 0106
Q4_K_M·269.9 t/s
5.7 GBS100, Runs great
Llama 3.3 70B Instruct Abliterated
Q4_K_M·30.6 t/s
48 GBA71, Runs well
Phi 4 Quantized.w8a8
Q4_K_M·139.2 t/s
12.5 GBS96, Runs great
Qwen3.6 27B DFlash
Q4_K_M·80.1 t/s
16.9 GBS88, Runs great
Nemotron Labs Diffusion 3B
Q4_K_M·480.2 t/s
3.9 GBS100, Runs great
Chatglm3 6B
Q4_K_M·321.7 t/s
5.2 GBS100, Runs great
Baichuan2 7B Chat
Q4_K_M·286.9 t/s
5.0 GBS100, Runs great
OpenELM 1 1B Instruct
Q4_K_M·1866.7 t/s
2.2 GBS100, Runs great
Yi Coder 9B Chat
Q4_K_M·228.5 t/s
7.2 GBS100, Runs great
Gemma 4 26B A4B
Q4_K_M·268.6 t/s
16.5 GBS100, Runs great
Mistral 7B Instruct v0.3
Q4_K_M·269.4 t/s
6.8 GBS100, Runs great
Salamandra 2B Instruct
Q4_K_M·646.5 t/s
3.3 GBS100, Runs great
C4ai Command A 03 2025
Q4_K_M·18.1 t/s
78.7 GBC30, Usable
Typhoon Ocr 3B
Q4_K_M·503.9 t/s
3.2 GBS100, Runs great
Llama 3.1 8B Lexi Uncensored v2
Q4_K_M·245.9 t/s
7.3 GBS100, Runs great

Rent NVIDIA H100 PCIe in the cloud

Hourly rate on demand for one H100 PCIe.

  • Vast.aiMarketplace

    from$2.82/hr

    Deploy

Cost per million tokens

Output-token cost on an H100 PCIe rented at $2.82/hr, one request at a time. Batch several users on the same card and the cost per token drops several-fold.

ModelSpeedPer 1M output
NVIDIA Nemotron 3.5 Lightning 30B A3B BF16253 tok/s$3.10$0.016 input
GLM 4.7 Flash209 tok/s$3.76$0.021 input
Qwen3.8 27B76 tok/s$10.30$0.13 input
Muse Glimmer 30B72 tok/s$10.84$0.14 input
Gemma 4 31B IT65 tok/s$12.06$0.14 input

H100 PCIe specs, power and PSU

Specs

Architecture
Hopper
VRAM
80 GB HBM2e
Memory bandwidth
2039.0 GB/s
FP16 (dense)
756.50 TFLOPS
CUDA cores
14,592
Tensor cores
456
Compute capability
9.0
TDP
350 W
Released
2022-09-01

H100 PCIe FAQ

What is the largest model the H100 PCIe can run?

Mistral Medium 3.5 128B (127.7B) is the largest at 4-bit or better: at Q4_K_S it uses 78.0 GB of 80 GB, and runs at about 18.2 tok/s.

Can the H100 PCIe run Qwen3.8 27B?

Yes. Qwen3.8 27B fits at Q4_K_M, using 17.9 GB of 80 GB, and it is the best model that runs well on this card.

How many tokens per second does the H100 PCIe give?

About 76.1 tok/s on Qwen3.8 27B at Q4_K_M, with the first token after 0.7 s on a 4K-token prompt.

What power supply does the H100 PCIe need?

Plan a 700 W PSU (2× the 350 W TDP).

Is 80 GB enough for coding and agents?

Yes for coding: Qwen3.8 27B, the best of the 12 coding models that fit, runs at about 76 tok/s. For agents, not yet: no agent-scored model fits in less than 96 GB.

Related

Best coding LLMs48 GB VRAMGPUs with 80 GB or more