NVIDIA · Orin · AI box

Best AI Models for NVIDIA Jetson AGX Orin 64GB

Runs 9B and 35B MoE models well and fits 120B MoE; 14B, 27–31B and 70B fit but are slow (14, 8 and 3 tok/s).

64 GB unified204.8 GB/s43 TFLOPS FP1660 W2022

Orin from

Jetson AGX Orin 32GBJetson AGX Orin 64GB

What LLMs can the NVIDIA Jetson AGX Orin 64GB run?

#	Model	Speed	Quant · Memory
1	Qwen3.6 35B A3BAlibabaHow to run → Q4_K_M · 22.0 GB · 41.7 t/s	41.7 tok/s	Q4_K_M · 22.0 GB
2	Gemma 4 26B A4B ITGoogle Q4_K_M · 16.5 GB · 44.9 t/s	44.9 tok/s	Q4_K_M · 16.5 GB
3	Qwen3 30B A3B Thinking 2507Alibaba Q4_K_M · 19.4 GB · 55.3 t/s	55.3 tok/s	Q4_K_M · 19.4 GB
4	GPT OSS 20BOpenAI Q4_K_M · 13.1 GB · 48.1 t/s	48.1 tok/s	Q4_K_M · 13.1 GB
5	NVIDIA Nemotron 3.5 Lightning 30B A3B BF16NVIDIA Q4_K_M · 19.3 GB · 72.9 t/s	72.9 tok/s	Q4_K_M · 19.3 GB

#	Model	Speed	Quant · Memory
1	Qwen3.6 35B A3BAlibabaHow to run → Q4_K_M · 22.0 GB · 41.7 t/s	41.7 tok/s	Q4_K_M · 22.0 GB
2	Gemma 4 26B A4B ITGoogle Q4_K_M · 16.5 GB · 44.9 t/s	44.9 tok/s	Q4_K_M · 16.5 GB
3	GPT OSS 20BOpenAI Q4_K_M · 13.1 GB · 48.1 t/s	48.1 tok/s	Q4_K_M · 13.1 GB
4	Llama 3.1 8B InstructMeta Q4_K_M · 7.9 GB · 25.1 t/s	25.1 tok/s	Q4_K_M · 7.9 GB
5	NVIDIA Nemotron 3.5 Lightning 30B A3B BF16NVIDIA Q4_K_M · 19.3 GB · 72.9 t/s	72.9 tok/s	Q4_K_M · 19.3 GB

#	Model	Speed	Quant · Memory
1	Qwen3.6 35B A3BAlibabaHow to run → Q4_K_M · 22.0 GB · 41.7 t/s	41.7 tok/s	Q4_K_M · 22.0 GB
2	Gemma 4 26B A4B ITGoogle Q4_K_M · 16.5 GB · 44.9 t/s	44.9 tok/s	Q4_K_M · 16.5 GB
3	Qwen3.5 9BAlibaba Q4_K_M · 6.6 GB · 20.9 t/s	20.9 tok/s	Q4_K_M · 6.6 GB
4	GPT OSS 20BOpenAI Q4_K_M · 13.1 GB · 48.1 t/s	48.1 tok/s	Q4_K_M · 13.1 GB
5	NVIDIA Nemotron 3.5 Lightning 30B A3B BF16NVIDIA Q4_K_M · 19.3 GB · 72.9 t/s	72.9 tok/s	Q4_K_M · 19.3 GB

#	Model	Speed ↓	Quant · Memory
1	Qwen3 30B A3B Thinking 2507AlibabaHow to run → Q4_K_M · 19.4 GB · 55.3 t/s	55.3 tok/s	Q4_K_M · 19.4 GB
2	NVIDIA Nemotron 3.5 Lightning 30B A3B BF16NVIDIA Q4_K_M · 19.3 GB · 72.9 t/s	72.9 tok/s	Q4_K_M · 19.3 GB
3	Qwen3 30B A3B Instruct 2507Alibaba Q4_K_M · 19.4 GB · 55.3 t/s	55.3 tok/s	Q4_K_M · 19.4 GB
4	GPT OSS 20BOpenAI Q4_K_M · 13.1 GB · 48.1 t/s	48.1 tok/s	Q4_K_M · 13.1 GB
5	Gemma 4 26B A4B ITGoogle Q4_K_M · 16.5 GB · 44.9 t/s	44.9 tok/s	Q4_K_M · 16.5 GB

See all 373 models on the NVIDIA Jetson AGX Orin 64GB →Bars = traction for models without a Score yet

About the NVIDIA Jetson AGX Orin 64GB

The NVIDIA Jetson AGX Orin 64GB is a 64 GB AI box with more memory than speed: it runs 9B and 35B MoE models well and fits 120B MoE. With 43 TFLOPS it reads long prompts faster than most 64 GB machines: a 4K-token prompt takes 2.0 s. Orin (Ampere), released 2022.

Speed in 64 GB#10205 GB/s

Prompt reading#243 TFLOPS

For models64 GBall of it

ChipOrinreleased 2022

NVIDIA Jetson AGX Orin 64GB strengths and weaknesses

Strengths

Runs 120B MoE models: GPT OSS 120B at 39.4 tok/s in 63.5 GB
Reads long prompts fast: A 32K-token prompt on Qwen3.6 35B A3B starts answering in 16.0 s
Low power draw: 60 W at full load for the whole machine

Weaknesses

Slow on large dense models: Llama 3.3 70B Instruct at 2.9 tok/s, Qwen3.8 27B at 7.6 tok/s
No FP8 or NVFP4: Ampere predates both; use GGUF, AWQ or GPTQ builds

Memory for models

64 GB of 64 GB

9B6.6 GB14B11.9 GB35B MoE22 GB70B51.3 GB120B MoE63.5 GB

35B MoE22 GB70B51.3 GB120B MoE63.5 GB

For models 64 GBThe system keeps 0 GBModel size at Q4

Everything on the size ladder fits. Qwen3.6 35B A3B and Gemma 4 26B A4B IT together use 38.5 GB and leave 25.5 GB free.

Similar devices and alternatives

The same chip from other makers first, then machines and GPUs people weigh against it, each with the one reason it is worth a look. Speeds on Qwen3.6 35B A3B.

Same chip, 32 GB less memoryJetson AGX Orin 32GBNVIDIA · 32 GB for models · Orin

same speed▼ 32 GBno price

Same family 54% faster, 3.5 GB less memoryMac Studio M4 Max (64 GB)Apple · 60.5 GB for models · M4 Max

▲ 54% speed▼ 3.5 GB$2,699

12% faster, 3.5 GB less memoryMacBook Pro 14 M5 ProApple · 60.5 GB for models · M5 Pro

▲ 12% speed▼ 3.5 GB$2,299

84% faster, 28.5 GB more memoryMac Studio M3 Ultra (96 GB)Apple · 92.5 GB for models · M3 Ultra

▲ 84% speed▲ 28.5 GB$3,999

3.6× faster if your models fit in 24 GBRTX 4090NVIDIA · 24 GB VRAM · GPU

▲ 262% speed▼ 40 GB$1,599rent $0.34/hr

5.1× faster if your models fit in 32 GBRTX 5090NVIDIA · 32 GB VRAM · GPU

▲ 412% speed▼ 32 GB$5,376rent $0.50/hr

NVIDIA Jetson AGX Orin 64GB tokens per second by model size

Output speed depends on memory bandwidth (205 GB/s). Grey rows need more than the 64 GB available.

9BQwen3.5 9B · Q4_K_M · 6.6 GB
20.9tok/s
14BQwen3 14B · Q4_K_M · 11.9 GB
14.0tok/s
27–31BQwen3.8 27B · Q4_K_M · 17.9 GB slow
7.6tok/s
35B MoEQwen3.6 35B A3B · Q4_K_M · 22.0 GB
41.7tok/s
70BLlama 3.3 70B Instruct · Q4_K_M · 51.3 GB slow
2.9tok/s
120B MoEGPT OSS 120B · IQ4_XS · 63.5 GB tight · IQ4_XS
39.4tok/s

Estimated speeds: Q4_K_M, 4K context, one user. How we estimate

Unified-memory platforms: writing vs compute

Writing, tok/sCompute, TFLOPS

M3 Pro36 GB · 150 GB/s
28.1
13
Orin64 GB · 204.8 GB/s
41.7
43
Snapdragon X2 Elite Extreme (X2E-96-100)48 GB · 228 GB/s
42.5
15
Ryzen AI Max+ 395128 GB · 256 GB/s
46.7
30
GB10128 GB · 273 GB/s
54.0
125

Machines with about the same memory bandwidth write at about the same speed; compute decides how fast prompts are read. Speeds on Qwen3.6 35B A3B.

All 1235 models on the NVIDIA Jetson AGX Orin 64GB

Featured only

Best quant that fits in 64 GB, ordered by llmrun Score. Fit grades how well it runs here, S best. How ranking works

LLM models compatible with NVIDIA Jetson AGX Orin 64GB — ranked by Score, recency and Fit
Model	Quant	VRAM	Speed	Context	Fit
CyberPal2.0 20B20.9BChatReasoning Q4_K_M·48.1 t/s	Q4_K_M	13.1 GB21%	48.1 t/s	131K	A79, Runs well
Chandra8.8BVisionFunctions Q4_K_M·22.7 t/s	Q4_K_M	8.0 GB12%	22.7 t/s	262K	B63, Decent
Deepseek Vl2 Tiny3.4BVision BF16·45.3 t/s	BF16	9.4 GB15%	45.3 t/s	—	A77, Runs well
GLM 4.7 Flash Heretic 1.2.029.9BChat Q4_K_M·54.0 t/s	Q4_K_M	19.1 GB30%	54.0 t/s	203K	A81, Runs well
Jina Ocr V13.4BVision Q4_K_M·141.2 t/s	Q4_K_M	3.3 GB5%	141.2 t/s	33K	S96, Runs great
DeepSeek R1 Distill Qwen 1.5B1.8BChatReasoningFunctions Q4_K_M·93.7 t/s	Q4_K_M	1.8 GB3%	93.7 t/s	131K	S90, Runs great
Gemma 4 31B IT Speculator.eagle331BChat Q4_K_M·6.5 t/s	Q4_K_M	24.2 GB38%	6.5 t/s	—	C43, Usable
HunyuanOCR1.1BVision Q4_K_M·124.4 t/s	Q4_K_M	1.8 GB3%	124.4 t/s	131K	S95, Runs great
GPT OSS 20B Heretic Ara v321.5BChat Q4_K_M·42.5 t/s	Q4_K_M	13.5 GB21%	42.5 t/s	131K	A76, Runs well
Sarvam 30B Uncensored32.2BChat Q4_K_M·45.9 t/s	Q4_K_M	19.9 GB31%	45.9 t/s	131K	A78, Runs well
Dolphin 2.9.1 Yi 1.5 34B34.4BChat Q4_K_M·6.2 t/s	Q4_K_M	22.9 GB36%	6.2 t/s	8K	C43, Usable
Param2 17B A2.4B Thinking17.2BChat BF16·25.0 t/s	BF16	34.8 GB54%	25.0 t/s	4K	A66, Runs well
Qwen3.8 27B EfficientThink Uncensored K3 Opus5 Grok4.6 GPT5.6Sol SFT SimPO DFlash227BChatReasoningFunctions Q4_K_M·7.5 t/s	Q4_K_M	21.4 GB33%	7.5 t/s	—	B45, Decent
AliceAI Foundation 80B A3B Base81.3BChat Q4_K_M·36.3 t/s	Q4_K_M	49.9 GB78%	36.3 t/s	262K	A73, Runs well
Apriel 1.6 15B Thinker14.9BVisionFunctions Q4_K_M·13.7 t/s	Q4_K_M	13.2 GB21%	13.7 t/s	262K	B53, Decent

NVIDIA Jetson AGX Orin 64GB specs and power

Specs

Chip: Orin
CPU cores: 12
GPU cores: 2,048
Unified memory: 64 GB LPDDR5
For models: 64 GB
Memory bandwidth: 204.8 GB/s
FP16 (dense): 42.5 TFLOPS
Power: 60 W max
Released: 2022-03-22

NVIDIA Jetson AGX Orin 64GB FAQ

How much memory is available for AI on the NVIDIA Jetson AGX Orin 64GB?

All 64 GB can hold model weights.

What is the largest model the NVIDIA Jetson AGX Orin 64GB can run?

GPT OSS 120B (116.8B) is the largest at 4-bit or better: at IQ4_XS it uses 63.5 GB, 99% of the available memory, and runs at about 39.4 tok/s.

Can the NVIDIA Jetson AGX Orin 64GB run Qwen3.6 35B A3B?

Yes. Qwen3.6 35B A3B fits at Q4_K_M, using 22.0 GB of the 64 GB available, and it is the best model that runs well on this machine.

Can the NVIDIA Jetson AGX Orin 64GB run 27–31B models?

Yes, but slowly: Qwen3.8 27B at Q4_K_M uses 17.9 GB and writes about 7.6 tok/s.

How long does the NVIDIA Jetson AGX Orin 64GB take to read a long prompt?

On Qwen3.6 35B A3B, about 2.0 s for a 4K-token prompt and 16 s for a 32K-token document before the first word appears.

Is renting a GPU cheaper than buying the NVIDIA Jetson AGX Orin 64GB?

An A100 (80 GB) rents from $1.12/hr.

48 GB GPUs AI boxes Mac Pro M2 Ultra Mac Studio M1 Ultra Mac Studio M2 Ultra Mac Studio M5 Max MacBook Pro 14" M5 Max MacBook Pro 16" M5 Max DGX Spark Best coding LLMs All hardware