NVIDIA · Orin NX · AI box

Best AI Models for NVIDIA Jetson Orin NX 16GB

Runs 14B models at 7 tok/s; 9B fits but is slow at 11 tok/s, and 27–31B only just fit in 13 GB.

16 GB unified13 GB for models102.4 GB/s15 TFLOPS FP1640 W2023

What LLMs can the NVIDIA Jetson Orin NX 16GB run?

#	Model	Speed	Quant · Memory
1	Qwen3.5 9BAlibabaHow to run → Q4_K_M · 6.6 GB · 10.5 t/s	10.5 tok/s	Q4_K_M · 6.6 GB
2	Qwen3.5 4BAlibaba Q4_K_M · 3.4 GB · 20.4 t/s	20.4 tok/s	Q4_K_M · 3.4 GB
3	Gemma 3 4B ITGoogle Q4_K_M · 5.0 GB · 23.4 t/s	23.4 tok/s	Q4_K_M · 5.0 GB
4	Ling 3.0 TinyInclusion AI Q4_K_M · 5.5 GB · 65.1 t/s	65.1 tok/s	Q4_K_M · 5.5 GB
5	Gemma 4 E2B ITGoogle Q4_K_M · 3.5 GB · 19.4 t/s	19.4 tok/s	Q4_K_M · 3.5 GB

#	Model	Speed	Reasoning Score	Quant · Memory
1	Gemma 3 4B ITGoogleHow to run → Q4_K_M · 5.0 GB · 23.4 t/s	23.4 tok/s		Q4_K_M · 5.0 GB
2	Qwen3.5 9BAlibaba Q4_K_M · 6.6 GB · 10.5 t/s	10.5 tok/s		Q4_K_M · 6.6 GB
3	Qwen3.5 4BAlibaba Q4_K_M · 3.4 GB · 20.4 t/s	20.4 tok/s	overall	Q4_K_M · 3.4 GB
4	Ling 3.0 TinyInclusion AI Q4_K_M · 5.5 GB · 65.1 t/s	65.1 tok/s		Q4_K_M · 5.5 GB
5	Gemma 4 E2B ITGoogle Q4_K_M · 3.5 GB · 19.4 t/s	19.4 tok/s		Q4_K_M · 3.5 GB

#	Model	Speed	Quant · Memory
1	Qwen3.5 4BAlibabaHow to run → Q4_K_M · 3.4 GB · 20.4 t/s	20.4 tok/s	Q4_K_M · 3.4 GB
2	Ling 3.0 TinyInclusion AI Q4_K_M · 5.5 GB · 65.1 t/s	65.1 tok/s	Q4_K_M · 5.5 GB
3	Gemma 4 E2B ITGoogle Q4_K_M · 3.5 GB · 19.4 t/s	19.4 tok/s	Q4_K_M · 3.5 GB
4	Qwen3.5 2BAlibaba Q4_K_M · 1.9 GB · 37.6 t/s	37.6 tok/s	Q4_K_M · 1.9 GB
5	MiniCPM5 2BOpenBMB Q4_K_M · 2.5 GB · 35.0 t/s	35.0 tok/s	Q4_K_M · 2.5 GB

#	Model	Speed ↓	Quant · Memory
1	Qwen3.5 4BAlibabaHow to run → Q4_K_M · 3.4 GB · 20.4 t/s	20.4 tok/s	Q4_K_M · 3.4 GB
2	Ling 3.0 TinyInclusion AI Q4_K_M · 5.5 GB · 65.1 t/s	65.1 tok/s	Q4_K_M · 5.5 GB
3	Qwen3.5 2BAlibaba Q4_K_M · 1.9 GB · 37.6 t/s	37.6 tok/s	Q4_K_M · 1.9 GB
4	MiniCPM5 2BOpenBMB Q4_K_M · 2.5 GB · 35.0 t/s	35.0 tok/s	Q4_K_M · 2.5 GB
5	Gemma 4 E2B ITGoogle Q4_K_M · 3.5 GB · 19.4 t/s	19.4 tok/s	Q4_K_M · 3.5 GB

#	Model	Speed	Quant · Memory
1	Qwen3.5 4BAlibabaHow to run → Q4_K_M · 3.4 GB · 20.4 t/s	20.4 tok/s	Q4_K_M · 3.4 GB
2	Gemma 3 4B ITGoogle Q4_K_M · 5.0 GB · 23.4 t/s	23.4 tok/s	Q4_K_M · 5.0 GB
3	Qwen3.5 9BAlibaba Q4_K_M · 6.6 GB · 10.5 t/s	10.5 tok/s	Q4_K_M · 6.6 GB
4	MiniCPM V 4.7 1BOpenBMB Q4_K_M · 1.6 GB · 52.0 t/s	52.0 tok/s	Q4_K_M · 1.6 GB
5	LFM3 Omni 2BLiquid AI Q4_K_M · 2.4 GB · 39.2 t/s	39.2 tok/s	Q4_K_M · 2.4 GB

See all 285 models on the NVIDIA Jetson Orin NX 16GB →Bars = traction for models without a Score yet

About the NVIDIA Jetson Orin NX 16GB

The NVIDIA Jetson Orin NX 16GB is a 16 GB AI box with more memory than speed: it runs 14B models at 7 tok/s. The system keeps about 3 GB, leaving 13 GB for models. With 15 TFLOPS it reads long prompts faster than most 16 GB machines: a 4K-token prompt takes 12 s. At its $599 MSRP it gives 17.5 tok/s per $1,000, 4th of the 6 machines compared here. Orin NX (Ampere), released 2023.

Speed in 16 GB#5102 GB/s

Prompt reading#215 TFLOPS

For models13 GBThe system keeps 3 GB

ChipOrin NXreleased 2023

NVIDIA Jetson Orin NX 16GB strengths and weaknesses

Strengths

Low power draw: 40 W at full load for the whole machine

Weaknesses

35B MoE models don't fit: Qwen3.6 35B A3B needs 22 GB at Q4 vs 13 GB usable here
Slow on large dense models: Qwen3.8 27B at 5.3 tok/s
Slow prompt reading: A 32K-token prompt on Qwen3.5 9B waits 1.6 min for the first token
No FP8 or NVFP4: Ampere predates both; use GGUF, AWQ or GPTQ builds

Memory for models

13 GB of 16 GB

9B6.6 GB27–31B13 GB

For models 13 GBThe system keeps 3 GBModel size at Q4

35B MoE models need 22 GB at Q4, more than the 13 GB available.Or rent an RTX 5090 (32 GB) from $0.50/hr →

Speed per dollar and per watt

NewUsed

On Qwen3.5 9B (9B params) at Q4_K_M. New means a shop price, either a street quote or the maker's own price; used is compared with used. The last two columns are the speed you get per $1,000 of price and per 100 W of power; higher is better. The NVIDIA Jetson Orin NX 16GB has no current new price, so it is shown at its MSRP.

Machine	New price	tok/s	tok/s per $1,000	tok/s per 100 W
RTX 509032 GB	about $5,376	183	34.1	31.8
Mac Mini M4 Pro24 GB	about $1,599	30.0	18.8	21.4
Mac Studio M4 Max36 GB	about $2,499	45.1	18.0	31.1
Jetson Orin NX 16GB16 GB	about $599	10.5	17.5	26.3
Mac Mini M416 GB	about $799	13.2	16.5	20.3
MacBook Pro 14 M532 GB	about $1,839	16.9	9.2	–

Machine	Used price	tok/s	tok/s per $1,000	tok/s per 100 W
RTX 409024 GBBest used value	about $3,000	103	34.3	22.9
RTX 509032 GB	about $6,400	183	28.6	31.8
Mac Studio M4 Max36 GB	about $2,000	45.1	22.6	31.1

Cost per token: the NVIDIA Jetson Orin NX 16GB against 16–36 GB machines and GPUs

On Qwen3.5 9B (9B params, Q4_K_M, 4K context). Speed, cost per token and power per token depend on the model.

≤16 GB24–36 GBthis AI boxat MSRP

Cost per 1M output tokenson Qwen3.5 9B · card over 3 years + power

$3.78

Machine, MSRP: $3.62
Power: $0.16

3-year cost of ownership$625

Or rent an RTX 3090 in the cloud$0.12/hr

Renting costs as much as buying after ≈5,200 hours (≈3.5 years at 4 h a day).

Include card price

Power$/kWhUseh/dayKeepyears

Each dot is a machine or GPU; hover or tap it for its numbers, click to open it. Machines without a price for the chosen basis are left out. Machines without a published power figure are left out of Per watt and costed without power. How we calculate cost per token

Similar devices and alternatives

The same chip from other makers first, then machines and GPUs people weigh against it, each with the one reason it is worth a look. Speeds on Qwen3.5 9B.

26% faster, same memoryMac Mini M4 (16 GB)Apple · 12.5 GB for models · M4

▲ 26% speedsame memory$799

186% faster, 7.5 GB more memoryMac Mini M4 Pro (24 GB)Apple · 20.5 GB for models · M4 Pro

▲ 186% speed▲ 7.5 GB$1,599

61% faster, 15.5 GB more memoryMacBook Pro 14 M5Apple · 28.5 GB for models · M5

▲ 61% speed▲ 15.5 GB$1,839

330% faster, 19.5 GB more memoryMac Studio M4 Max (36 GB)Apple · 32.5 GB for models · M4 Max

▲ 330% speed▲ 19.5 GB$2,499

Reads prompts 11× faster, 24 GB for modelsRTX 4090NVIDIA · 24 GB VRAM · GPU

▲ 11× prompts▲ 881% speed▲ 11 GB$1,599rent $0.34/hr

Reads prompts 14× faster, 32 GB for modelsRTX 5090NVIDIA · 32 GB VRAM · GPU

▲ 14× prompts▲ 1644% speed▲ 19 GB$5,376rent $0.50/hr

NVIDIA Jetson Orin NX 16GB tokens per second by model size

Output speed depends on memory bandwidth (102 GB/s). Grey rows need more than the 13 GB available.

9BQwen3.5 9B · Q4_K_M · 6.6 GB
10.5tok/s
14BQwen3 14B · Q4_K_M · 11.9 GB slow
7.0tok/s
27–31BQwen3.8 27B · IQ3_S · 13.0 GB tight · IQ3_S
5.3tok/s
35B MoEQwen3.6 35B A3B · not at 4-bit
needs 22.0 GB
70BLlama 3.3 70B Instruct · not at 4-bit
needs 51.3 GB
120B MoEGPT OSS 120B · not at 4-bit
needs 70.8 GB

Estimated speeds: Q4_K_M, 4K context, one user. How we estimate

Unified-memory platforms: writing vs compute

Writing, tok/sCompute, TFLOPS

M116 GB · 68.3 GB/s
7.5
5
M216 GB · 100 GB/s
11.0
7
Orin Nano8 GB · 102 GB/s
10.4
17
Orin NX16 GB · 102.4 GB/s
10.5
15
M316 GB · 102.4 GB/s
11.3
7
Ryzen AI 9 HX 37032 GB · 120 GB/s
11.3
12

Machines with about the same memory bandwidth write at about the same speed; compute decides how fast prompts are read. Speeds on Qwen3.5 9B.

All 985 models on the NVIDIA Jetson Orin NX 16GB

Featured only

Best quant that fits in 13 GB, ordered by llmrun Score. Fit grades how well it runs here, S best. How ranking works

LLM models compatible with NVIDIA Jetson Orin NX 16GB — ranked by Score, recency and Fit
Model	Quant	VRAM	Speed	Context	Fit
Gemma 4 12B12.0BChat Q4_K_M·8.1 t/s	Q4_K_M	9.0 GB69%	8.1 t/s	262K	B45, Decent
Pythia 160M213MChat Q4_K_M·475.4 t/s	Q4_K_M	0.1 GB1%	475.4 t/s	2K	S100, Runs great
Protocol Phantom 12B12.2BChat Q4_K_M·8.2 t/s	Q4_K_M	11.0 GB85%	8.2 t/s	131K	C39, Usable
MobileLLM R1.5 950M950MChatReasoning BF16·31.8 t/s	BF16	3.5 GB27%	31.8 t/s	—	A71, Runs well
Ornith 1.0 9B9.4BChatFunctions Q4_K_M·10.7 t/s	Q4_K_M	6.5 GB50%	10.7 t/s	262K	B49, Decent
Kappa 20B 131k Mxfp420.9BChat Q3_K_M·30.2 t/s	Q3_K_M	10.8 GB83%	30.2 t/s	131K	A67, Runs well
Granite 4.0 H Tiny6.9BChatFunctions Q4_K_M·59.2 t/s	Q4_K_M	5.8 GB45%	59.2 t/s	131K	A83, Runs well
T5gemma B B Ul2 IT591MChat BF16·51.2 t/s	BF16	2.5 GB20%	51.2 t/s	—	A80, Runs well
Qliphoth 12B V1.212.2BChat Q4_K_M·8.2 t/s	Q4_K_M	11.0 GB85%	8.2 t/s	1024K	C39, Usable
Gemma3 1B IT1BChat BF16·30.3 t/s	BF16	3.6 GB28%	30.3 t/s	—	A71, Runs well
ERNIE 4.5 0.3B Paddle361MChat BF16·64.0 t/s	BF16	1.2 GB9%	64.0 t/s	131K	A84, Runs well
Gemma 4 E2B5.1BChat Q4_K_M·19.4 t/s	Q4_K_M	3.5 GB27%	19.4 t/s	131K	B59, Decent
Tiny Random Phi33MChat Q4_K_M·221.9 t/s	Q4_K_M	0.3 GB2%	221.9 t/s	4K	S100, Runs great
Jan Code 4B4.4BChatFunctionsCode Q4_K_M·21.2 t/s	Q4_K_M	4.5 GB34%	21.2 t/s	262K	B61, Decent
MedScholar 1.5B1.5BChat Q4_K_M·52.0 t/s	Q4_K_M	1.7 GB13%	52.0 t/s	33K	A81, Runs well

NVIDIA Jetson Orin NX 16GB specs and power

Specs

Chip: Orin NX · Ampere GPU (Orin NX)
CPU cores: 8
GPU cores: 1,024
Unified memory: 16 GB LPDDR5
For models: 13 GB (The system keeps 3 GB)
Memory bandwidth: 102.4 GB/s
FP16 (dense): 15 TFLOPS
Power: 40 W max
Released: 2023-02-01
MSRP: $599

NVIDIA Jetson Orin NX 16GB FAQ

Is the NVIDIA Jetson Orin NX 16GB worth it for local LLMs in 2026?

At its $599 MSRP it gives 17.5 tok/s per $1,000 on Qwen3.5 9B, 4th of the 6 machines compared here. It runs 14B models at 7 tok/s; 9B fits but is slow at 11 tok/s, and 27–31B only just fit in 13 GB.

How much memory is available for AI on the NVIDIA Jetson Orin NX 16GB?

About 13 GB of the 16 GB. The system keeps roughly 3 GB for itself and other apps.

What is the largest model the NVIDIA Jetson Orin NX 16GB can run?

GPT OSS 20B (20.9B) is the largest at 4-bit or better: at Q4_K_S it uses 12.4 GB, 95% of the available memory, and runs at about 26.4 tok/s.

Can the NVIDIA Jetson Orin NX 16GB run Qwen3.5 9B?

Yes. Qwen3.5 9B fits at Q4_K_M, using 6.6 GB of the 13 GB available, and it is the best model that runs well on this machine.

Can the NVIDIA Jetson Orin NX 16GB run 35B MoE models?

No. Qwen3.6 35B A3B needs 22.0 GB at Q4, more than the 13 GB available.

How long does the NVIDIA Jetson Orin NX 16GB take to read a long prompt?

On Qwen3.5 9B, about 12 s for a 4K-token prompt and 1.6 min for a 32K-token document before the first word appears.

Is renting a GPU cheaper than buying the NVIDIA Jetson Orin NX 16GB?

An RTX 3090 (24 GB) rents from $0.12/hr. Renting costs as much as buying after ≈5,200 hours (≈3.5 years at 4 h a day).

12 GB GPUs AI boxes Mac Mini M2 Pro MacBook Pro 14" M1 Pro MacBook Pro 14" M2 Pro MacBook Pro 16" M1 Pro MacBook Pro 16" M2 Pro MacBook Air 13" M5 Jetson AGX Orin 32GB Best coding LLMs All hardware