NVIDIA · Orin · AI box

Best AI Models for NVIDIA Jetson AGX Orin 64GB

Runs 9B and 35B MoE models well and fits 120B MoE; 14B, 27–31B and 70B fit but are slow (14, 8 and 3 tok/s).

64 GB unified204.8 GB/s43 TFLOPS FP1660 W2022

Orin from

Jetson AGX Orin 32GBJetson AGX Orin 64GB

What LLMs can the NVIDIA Jetson AGX Orin 64GB run?

#	Model	Speed	Quant · Memory
1	Qwen3.6 35B A3BAlibabaHow to run → Q4_K_M · 22.0 GB · 41.7 t/s	41.7 tok/s	Q4_K_M · 22.0 GB
2	Gemma 4 26B A4B ITGoogle Q4_K_M · 16.5 GB · 44.9 t/s	44.9 tok/s	Q4_K_M · 16.5 GB
3	Qwen3 30B A3B Thinking 2507Alibaba Q4_K_M · 19.4 GB · 55.3 t/s	55.3 tok/s	Q4_K_M · 19.4 GB
4	GPT OSS 20BOpenAI Q4_K_M · 13.1 GB · 48.1 t/s	48.1 tok/s	Q4_K_M · 13.1 GB
5	NVIDIA Nemotron 3.5 Lightning 30B A3B BF16NVIDIA Q4_K_M · 19.3 GB · 72.9 t/s	72.9 tok/s	Q4_K_M · 19.3 GB

#	Model	Speed	Quant · Memory
1	Qwen3.6 35B A3BAlibabaHow to run → Q4_K_M · 22.0 GB · 41.7 t/s	41.7 tok/s	Q4_K_M · 22.0 GB
2	Gemma 4 26B A4B ITGoogle Q4_K_M · 16.5 GB · 44.9 t/s	44.9 tok/s	Q4_K_M · 16.5 GB
3	GPT OSS 20BOpenAI Q4_K_M · 13.1 GB · 48.1 t/s	48.1 tok/s	Q4_K_M · 13.1 GB
4	Llama 3.1 8B InstructMeta Q4_K_M · 7.9 GB · 25.1 t/s	25.1 tok/s	Q4_K_M · 7.9 GB
5	NVIDIA Nemotron 3.5 Lightning 30B A3B BF16NVIDIA Q4_K_M · 19.3 GB · 72.9 t/s	72.9 tok/s	Q4_K_M · 19.3 GB

#	Model	Speed	Quant · Memory
1	Qwen3.6 35B A3BAlibabaHow to run → Q4_K_M · 22.0 GB · 41.7 t/s	41.7 tok/s	Q4_K_M · 22.0 GB
2	Gemma 4 26B A4B ITGoogle Q4_K_M · 16.5 GB · 44.9 t/s	44.9 tok/s	Q4_K_M · 16.5 GB
3	Qwen3.5 9BAlibaba Q4_K_M · 6.6 GB · 20.9 t/s	20.9 tok/s	Q4_K_M · 6.6 GB
4	GPT OSS 20BOpenAI Q4_K_M · 13.1 GB · 48.1 t/s	48.1 tok/s	Q4_K_M · 13.1 GB
5	NVIDIA Nemotron 3.5 Lightning 30B A3B BF16NVIDIA Q4_K_M · 19.3 GB · 72.9 t/s	72.9 tok/s	Q4_K_M · 19.3 GB

#	Model	Speed ↓	Quant · Memory
1	Qwen3 30B A3B Thinking 2507AlibabaHow to run → Q4_K_M · 19.4 GB · 55.3 t/s	55.3 tok/s	Q4_K_M · 19.4 GB
2	NVIDIA Nemotron 3.5 Lightning 30B A3B BF16NVIDIA Q4_K_M · 19.3 GB · 72.9 t/s	72.9 tok/s	Q4_K_M · 19.3 GB
3	Qwen3 30B A3B Instruct 2507Alibaba Q4_K_M · 19.4 GB · 55.3 t/s	55.3 tok/s	Q4_K_M · 19.4 GB
4	GPT OSS 20BOpenAI Q4_K_M · 13.1 GB · 48.1 t/s	48.1 tok/s	Q4_K_M · 13.1 GB
5	Gemma 4 26B A4B ITGoogle Q4_K_M · 16.5 GB · 44.9 t/s	44.9 tok/s	Q4_K_M · 16.5 GB

See all 373 models on the NVIDIA Jetson AGX Orin 64GB →Bars = traction for models without a Score yet

About the NVIDIA Jetson AGX Orin 64GB

The NVIDIA Jetson AGX Orin 64GB is a 64 GB AI box with more memory than speed: it runs 9B and 35B MoE models well and fits 120B MoE. With 43 TFLOPS it reads long prompts faster than most 64 GB machines: a 4K-token prompt takes 2.0 s. Orin (Ampere), released 2022.

Speed in 64 GB#10205 GB/s

Prompt reading#243 TFLOPS

For models64 GBall of it

ChipOrinreleased 2022

NVIDIA Jetson AGX Orin 64GB strengths and weaknesses

Strengths

Runs 120B MoE models: GPT OSS 120B at 39.4 tok/s in 63.5 GB
Reads long prompts fast: A 32K-token prompt on Qwen3.6 35B A3B starts answering in 16.0 s
Low power draw: 60 W at full load for the whole machine

Weaknesses

Slow on large dense models: Llama 3.3 70B Instruct at 2.9 tok/s, Qwen3.8 27B at 7.6 tok/s
No FP8 or NVFP4: Ampere predates both; use GGUF, AWQ or GPTQ builds

Memory for models

64 GB of 64 GB

9B6.6 GB14B11.9 GB35B MoE22 GB70B51.3 GB120B MoE63.5 GB

35B MoE22 GB70B51.3 GB120B MoE63.5 GB

For models 64 GBThe system keeps 0 GBModel size at Q4

Everything on the size ladder fits. Qwen3.6 35B A3B and Gemma 4 26B A4B IT together use 38.5 GB and leave 25.5 GB free.

Similar devices and alternatives

The same chip from other makers first, then machines and GPUs people weigh against it, each with the one reason it is worth a look. Speeds on Qwen3.6 35B A3B.

Same chip, 32 GB less memoryJetson AGX Orin 32GBNVIDIA · 32 GB for models · Orin

same speed▼ 32 GBno price

Same family 54% faster, 3.5 GB less memoryMac Studio M4 Max (64 GB)Apple · 60.5 GB for models · M4 Max

▲ 54% speed▼ 3.5 GB$2,699

12% faster, 3.5 GB less memoryMacBook Pro 14 M5 ProApple · 60.5 GB for models · M5 Pro

▲ 12% speed▼ 3.5 GB$2,299

84% faster, 28.5 GB more memoryMac Studio M3 Ultra (96 GB)Apple · 92.5 GB for models · M3 Ultra

▲ 84% speed▲ 28.5 GB$3,999

3.6× faster if your models fit in 24 GBRTX 4090NVIDIA · 24 GB VRAM · GPU

▲ 262% speed▼ 40 GB$1,599rent $0.34/hr

5.1× faster if your models fit in 32 GBRTX 5090NVIDIA · 32 GB VRAM · GPU

▲ 412% speed▼ 32 GB$5,376rent $0.50/hr

NVIDIA Jetson AGX Orin 64GB tokens per second by model size

Output speed depends on memory bandwidth (205 GB/s). Grey rows need more than the 64 GB available.

9BQwen3.5 9B · Q4_K_M · 6.6 GB
20.9tok/s
14BQwen3 14B · Q4_K_M · 11.9 GB
14.0tok/s
27–31BQwen3.8 27B · Q4_K_M · 17.9 GB slow
7.6tok/s
35B MoEQwen3.6 35B A3B · Q4_K_M · 22.0 GB
41.7tok/s
70BLlama 3.3 70B Instruct · Q4_K_M · 51.3 GB slow
2.9tok/s
120B MoEGPT OSS 120B · IQ4_XS · 63.5 GB tight · IQ4_XS
39.4tok/s

Estimated speeds: Q4_K_M, 4K context, one user. How we estimate

Unified-memory platforms: writing vs compute

Writing, tok/sCompute, TFLOPS

M3 Pro36 GB · 150 GB/s
28.1
13
Orin64 GB · 204.8 GB/s
41.7
43
Snapdragon X2 Elite Extreme (X2E-96-100)48 GB · 228 GB/s
42.5
15
Ryzen AI Max+ 395128 GB · 256 GB/s
46.7
30
GB10128 GB · 273 GB/s
54.0
125

Machines with about the same memory bandwidth write at about the same speed; compute decides how fast prompts are read. Speeds on Qwen3.6 35B A3B.

All 1235 models on the NVIDIA Jetson AGX Orin 64GB

Featured only

Best quant that fits in 64 GB, ordered by llmrun Score. Fit grades how well it runs here, S best. How ranking works

LLM models compatible with NVIDIA Jetson AGX Orin 64GB — ranked by Score, recency and Fit
Model	Quant	VRAM	Speed	Context	Fit
Gemma 7B IT8.5BChat BF16·7.1 t/s	BF16	21.4 GB33%	7.1 t/s	—	C44, Usable
Qwen2.5 32B32.8BChat Q4_K_M·6.5 t/s	Q4_K_M	24.3 GB38%	6.5 t/s	131K	C43, Usable
Granite Guardian 3.2 8B Factuality Detection8.2BChat Q4_K_M·24.0 t/s	Q4_K_M	7.9 GB12%	24.0 t/s	131K	B64, Decent
Qwen2.5 Coder 32B32.8BChatCodeFunctions Q4_K_M·6.5 t/s	Q4_K_M	24.3 GB38%	6.5 t/s	33K	C43, Usable
Deepseek Coder 33B Base33.3BChatCode Q4_K_M·6.4 t/s	Q4_K_M	24.5 GB38%	6.4 t/s	16K	C43, Usable
Llama 2 13B HF13.0BChat Q4_K_M·15.5 t/s	Q4_K_M	11.5 GB18%	15.5 t/s	—	B55, Decent
Internlm Chat 7B7BChat FP16·8.6 t/s	FP16	15.4 GB24%	8.6 t/s	2K	B46, Decent
Mixtral 8x22B Instruct v0.1140.6BChatFunctions Q2_K·7.7 t/s	Q2_K	63.8 GB100%	7.7 t/s	66K	F13, Won't fit
Qwen2.5 Coder 14B14.8BChatCode Q4_K_M·13.9 t/s	Q4_K_M	12.4 GB19%	13.9 t/s	33K	B53, Decent
Qwen1.5 110B111.2BChat Q3_K_M·2.4 t/s	Q3_K_M	59.9 GB94%	2.4 t/s	33K	D17, Barely runs
Deepseek Coder 1.3B Base1.3BChatCode Q4_K_M·89.9 t/s	Q4_K_M	4.3 GB7%	89.9 t/s	16K	S89, Runs great
Medgemma 27B Text IT27.0BChat BF16·2.2 t/s	BF16	63.0 GB98%	2.2 t/s	—	F10, Won't fit
WizardLM 2 8x22B140.6BChat Q2_K·7.7 t/s	Q2_K	63.8 GB100%	7.7 t/s	66K	F13, Won't fit
Deepseek Coder 6.7B Base6.7BChatCode Q4_K_M·24.6 t/s	Q4_K_M	12.9 GB20%	24.6 t/s	16K	A65, Runs well
Meta Llama Guard 2 8B8.0BChat BF16·7.5 t/s	BF16	20.2 GB32%	7.5 t/s	—	B45, Decent

NVIDIA Jetson AGX Orin 64GB specs and power

Specs

Chip: Orin
CPU cores: 12
GPU cores: 2,048
Unified memory: 64 GB LPDDR5
For models: 64 GB
Memory bandwidth: 204.8 GB/s
FP16 (dense): 42.5 TFLOPS
Power: 60 W max
Released: 2022-03-22

NVIDIA Jetson AGX Orin 64GB FAQ

How much memory is available for AI on the NVIDIA Jetson AGX Orin 64GB?

All 64 GB can hold model weights.

What is the largest model the NVIDIA Jetson AGX Orin 64GB can run?

GPT OSS 120B (116.8B) is the largest at 4-bit or better: at IQ4_XS it uses 63.5 GB, 99% of the available memory, and runs at about 39.4 tok/s.

Can the NVIDIA Jetson AGX Orin 64GB run Qwen3.6 35B A3B?

Yes. Qwen3.6 35B A3B fits at Q4_K_M, using 22.0 GB of the 64 GB available, and it is the best model that runs well on this machine.

Can the NVIDIA Jetson AGX Orin 64GB run 27–31B models?

Yes, but slowly: Qwen3.8 27B at Q4_K_M uses 17.9 GB and writes about 7.6 tok/s.

How long does the NVIDIA Jetson AGX Orin 64GB take to read a long prompt?

On Qwen3.6 35B A3B, about 2.0 s for a 4K-token prompt and 16 s for a 32K-token document before the first word appears.

Is renting a GPU cheaper than buying the NVIDIA Jetson AGX Orin 64GB?

An A100 (80 GB) rents from $1.26/hr.

48 GB GPUs AI boxes Mac Pro M2 Ultra Mac Studio M1 Ultra Mac Studio M2 Ultra Mac Studio M5 Max MacBook Pro 14" M5 Max MacBook Pro 16" M5 Max DGX Spark Best coding LLMs All hardware