NVIDIA · Orin · AI box
Best AI Models for NVIDIA Jetson AGX Orin 32GB
Runs 9B and 35B MoE models well; 14B and 27–31B fit but are slow (14 and 8 tok/s), and 70B doesn't fit in 32 GB.
| # | Model | Speed | llmrun Score | Quant · Memory |
|---|---|---|---|---|
| 1 | Qwen3.6 35B A3BAlibabaHow to run → Q4_K_M · 22.0 GB · 41.7 t/s | 41.7 tok/s | Q4_K_M · 22.0 GB | |
| 2 | Gemma 4 26B A4B ITGoogle Q4_K_M · 16.5 GB · 44.9 t/s | 44.9 tok/s | Q4_K_M · 16.5 GB | |
| 3 | Qwen3 30B A3B Thinking 2507Alibaba Q4_K_M · 19.4 GB · 55.3 t/s | 55.3 tok/s | Q4_K_M · 19.4 GB | |
| 4 | GPT OSS 20BOpenAI Q4_K_M · 13.1 GB · 48.1 t/s | 48.1 tok/s | Q4_K_M · 13.1 GB | |
| 5 | Q4_K_M · 19.3 GB · 72.9 t/s | 72.9 tok/s | Q4_K_M · 19.3 GB |
| # | Model | Speed | Coding Score | Quant · Memory |
|---|---|---|---|---|
| 1 | Qwen3.6 35B A3BAlibabaHow to run → Q4_K_M · 22.0 GB · 41.7 t/s | 41.7 tok/s | Q4_K_M · 22.0 GB | |
| 2 | Gemma 4 26B A4B ITGoogle Q4_K_M · 16.5 GB · 44.9 t/s | 44.9 tok/s | Q4_K_M · 16.5 GB | |
| 3 | GPT OSS 20BOpenAI Q4_K_M · 13.1 GB · 48.1 t/s | 48.1 tok/s | Q4_K_M · 13.1 GB | |
| 4 | Q4_K_M · 7.9 GB · 25.1 t/s | 25.1 tok/s | Q4_K_M · 7.9 GB | |
| 5 | Q4_K_M · 19.3 GB · 72.9 t/s | 72.9 tok/s | Q4_K_M · 19.3 GB |
| # | Model | Speed | Reasoning Score | Quant · Memory |
|---|---|---|---|---|
| 1 | Qwen3.6 35B A3BAlibabaHow to run → Q4_K_M · 22.0 GB · 41.7 t/s | 41.7 tok/s | Q4_K_M · 22.0 GB | |
| 2 | Gemma 4 26B A4B ITGoogle Q4_K_M · 16.5 GB · 44.9 t/s | 44.9 tok/s | Q4_K_M · 16.5 GB | |
| 3 | Qwen3.5 9BAlibaba Q4_K_M · 6.6 GB · 20.9 t/s | 20.9 tok/s | Q4_K_M · 6.6 GB | |
| 4 | GPT OSS 20BOpenAI Q4_K_M · 13.1 GB · 48.1 t/s | 48.1 tok/s | Q4_K_M · 13.1 GB | |
| 5 | Q4_K_M · 19.3 GB · 72.9 t/s | 72.9 tok/s | Q4_K_M · 19.3 GB |
| # | Model | Speed ↓ | llmrun Score | Quant · Memory |
|---|---|---|---|---|
| 1 | Q4_K_M · 19.4 GB · 55.3 t/s | 55.3 tok/s | Q4_K_M · 19.4 GB | |
| 2 | Q4_K_M · 19.3 GB · 72.9 t/s | 72.9 tok/s | Q4_K_M · 19.3 GB | |
| 3 | Qwen3 30B A3B Instruct 2507Alibaba Q4_K_M · 19.4 GB · 55.3 t/s | 55.3 tok/s | Q4_K_M · 19.4 GB | |
| 4 | GPT OSS 20BOpenAI Q4_K_M · 13.1 GB · 48.1 t/s | 48.1 tok/s | Q4_K_M · 13.1 GB | |
| 5 | Gemma 4 26B A4B ITGoogle Q4_K_M · 16.5 GB · 44.9 t/s | 44.9 tok/s | Q4_K_M · 16.5 GB |
The NVIDIA Jetson AGX Orin 32GB is one of the fastest 32 GB machines: it runs 9B and 35B MoE models well. Among 32 GB machines, only the M1 Max and M2 Max write faster. With 27 TFLOPS it reads long prompts faster than most 32 GB machines: a 4K-token prompt takes 3.2 s. Orin (Ampere), released 2022.
NVIDIA Jetson AGX Orin 32GB strengths and weaknesses
Strengths
- Runs 35B MoE models: Qwen3.6 35B A3B at 41.7 tok/s in 22 GB
- Low power draw: 40 W at full load for the whole machine
Weaknesses
- 70B models don't fit: Llama 3.3 70B Instruct needs 51.3 GB at Q4 vs 32 GB usable here
- Slow on large dense models: Qwen3.8 27B at 7.6 tok/s
- No FP8 or NVFP4: Ampere predates both; use GGUF, AWQ or GPTQ builds
Memory for models
32 GB of 32 GB70B models need 51.3 GB at Q4, more than the 32 GB available.Or rent an A100 (80 GB) from $1.24/hr →
The same chip from other makers first, then machines and GPUs people weigh against it, each with the one reason it is worth a look. Speeds on Qwen3.6 35B A3B.
Output speed depends on memory bandwidth (205 GB/s). Grey rows need more than the 32 GB available.
- 9BQwen3.5 9B · Q4_K_M · 6.6 GB20.9tok/s
- 14BQwen3 14B · Q4_K_M · 11.9 GB14.0tok/s
- 7.6tok/s
- 35B MoEQwen3.6 35B A3B · Q4_K_M · 22.0 GB41.7tok/s
- 70BLlama 3.3 70B Instruct · not at 4-bitneeds 51.3 GB
- 120B MoEGPT OSS 120B · not at 4-bitneeds 70.8 GB
Estimated speeds: Q4_K_M, 4K context, one user. How we estimate
All 1176 models on the NVIDIA Jetson AGX Orin 32GB
Featured onlyBest quant that fits in 32 GB, ordered by llmrun Score. Fit grades how well it runs here, S best. How ranking works
| Model | Quant | VRAM | Speed | Context | Score | Fit |
|---|---|---|---|---|---|---|
DeepSeek R1 Distill Qwen 14B Abliterated v2ChatReasoning | Q4_K_M | 12.4 GB | 13.9 t/s | 131K | B53, Decent | |
Tiny Qwen3ForCausalLMChatFunctions | Q4_K_M | 0.6 GB | 443.7 t/s | 41K | S100, Runs great | |
Nerdsking Python Coder 7B IChatCode | Q4_K_M | 7.1 GB | 28.8 t/s | — | A69, Runs well | |
Gpt2 XlChat | Q4_K_M | 1.1 GB | 125.6 t/s | 1K | S95, Runs great | |
| BF16 | 3 GB | 44.4 t/s | 2K | A77, Runs well | ||
Pythia 1.4BChat | Q4_K_M | 1 GB | 133.1 t/s | 2K | S95, Runs great | |
Chatglm3 6BChat | Q4_K_M | 5.2 GB | 32.3 t/s | 8K | A72, Runs well | |
Mellum2 12B A2.5B ThinkingChatFunctions | Q4_K_M | 7.8 GB | 76.4 t/s | 131K | S87, Runs great | |
VLM2Vec FullChat | BF16 | 15.0 GB | 14.2 t/s | 131K | B53, Decent | |
Yi 1.5 34B ChatChat | Q4_K_M | 21.9 GB | 6.2 t/s | 4K | C43, Usable | |
| BF16 | 8.6 GB | 20.2 t/s | — | B60, Decent | ||
NVIDIA Nemotron 3.5 Bolt 74B A9B BF16VisionFunctions | IQ2_XS | 29.5 GB | 40.1 t/s | — | B58, Decent | |
PLLuM 12B ChatChat | Q4_K_M | 11.0 GB | 16.5 t/s | 131K | B56, Decent | |
Codegen 350M MonoChatCode | FP16 | 0.8 GB | 172.9 t/s | 2K | S98, Runs great | |
| Q4_K_M | 11.0 GB | 16.5 t/s | 1024K | B56, Decent |
Specs
- Chip
- Orin
- CPU cores
- 8
- GPU cores
- 1,792
- Unified memory
- 32 GB LPDDR5
- For models
- 32 GB
- Memory bandwidth
- 204.8 GB/s
- FP16 (dense)
- 26.9 TFLOPS
- Power
- 40 W max
- Released
- 2022-03-22
NVIDIA Jetson AGX Orin 32GB FAQ
How much memory is available for AI on the NVIDIA Jetson AGX Orin 32GB?
All 32 GB can hold model weights.
What is the largest model the NVIDIA Jetson AGX Orin 32GB can run?
Qwen3.6 35B A3B (36.0B) is the largest at 4-bit or better: at Q4_K_M it uses 22.0 GB, 69% of the available memory, and runs at about 41.7 tok/s.
Can the NVIDIA Jetson AGX Orin 32GB run 70B models?
No. Llama 3.3 70B Instruct needs 51.3 GB at Q4, more than the 32 GB available.
How long does the NVIDIA Jetson AGX Orin 32GB take to read a long prompt?
On Qwen3.6 35B A3B, about 3.2 s for a 4K-token prompt and 26 s for a 32K-token document before the first word appears.
Is renting a GPU cheaper than buying the NVIDIA Jetson AGX Orin 32GB?
An A100 (40 GB) rents from $0.40/hr.