llmrun

NVIDIA · Orin NX · AI box

Best AI Models for NVIDIA Jetson Orin NX 16GB

Runs 14B models at 7 tok/s; 9B fits but is slow at 11 tok/s, and 27–31B only just fit in 13 GB.

16 GB unified13 GB for models102.4 GB/s15 TFLOPS FP1640 W2023
Buyabout $599

What LLMs can the NVIDIA Jetson Orin NX 16GB run?

#ModelSpeedllmrun ScoreQuant · Memory
1
Qwen3.5 9BAlibabaHow to run →
Q4_K_M · 6.6 GB · 10.5 t/s
10.5 tok/sQ4_K_M · 6.6 GB
2
Qwen3.5 4BAlibaba
Q4_K_M · 3.4 GB · 20.4 t/s
20.4 tok/sQ4_K_M · 3.4 GB
3
Gemma 3 4B ITGoogle
Q4_K_M · 5.0 GB · 23.4 t/s
23.4 tok/sQ4_K_M · 5.0 GB
4
Ling 3.0 TinyInclusion AI
Q4_K_M · 5.5 GB · 65.1 t/s
65.1 tok/sQ4_K_M · 5.5 GB
5
Gemma 4 E2B ITGoogle
Q4_K_M · 3.5 GB · 19.4 t/s
19.4 tok/sQ4_K_M · 3.5 GB
See all 285 models on the NVIDIA Jetson Orin NX 16GB →Bars = traction for models without a Score yet

About the NVIDIA Jetson Orin NX 16GB

The NVIDIA Jetson Orin NX 16GB is a 16 GB AI box with more memory than speed: it runs 14B models at 7 tok/s. The system keeps about 3 GB, leaving 13 GB for models. With 15 TFLOPS it reads long prompts faster than most 16 GB machines: a 4K-token prompt takes 12 s. At its $599 MSRP it gives 17.5 tok/s per $1,000, 4th of the 6 machines compared here. Orin NX (Ampere), released 2023.

Speed in 16 GB#5102 GB/s
Prompt reading#215 TFLOPS
For models13 GBThe system keeps 3 GB
ChipOrin NXreleased 2023

NVIDIA Jetson Orin NX 16GB strengths and weaknesses

Strengths

  • Low power draw: 40 W at full load for the whole machine

Weaknesses

  • 35B MoE models don't fit: Qwen3.6 35B A3B needs 22 GB at Q4 vs 13 GB usable here
  • Slow on large dense models: Qwen3.8 27B at 5.3 tok/s
  • Slow prompt reading: A 32K-token prompt on Qwen3.5 9B waits 1.6 min for the first token
  • No FP8 or NVFP4: Ampere predates both; use GGUF, AWQ or GPTQ builds

Memory for models

13 GB of 16 GB
9B6.6 GB27–31B13 GB
9B6.6 GB27–31B13 GB
For models 13 GBThe system keeps 3 GBModel size at Q4

35B MoE models need 22 GB at Q4, more than the 13 GB available.Or rent an RTX 5090 (32 GB) from $0.50/hr →

Speed per dollar and per watt

On Qwen3.5 9B (9B params) at Q4_K_M. New means a shop price, either a street quote or the maker's own price; used is compared with used. The last two columns are the speed you get per $1,000 of price and per 100 W of power; higher is better. The NVIDIA Jetson Orin NX 16GB has no current new price, so it is shown at its MSRP.

MachineNew pricetok/stok/s per $1,000tok/s per 100 W
RTX 509032 GB
about $5,376183
34.1
31.8
Mac Mini M4 Pro24 GB
about $1,59930.0
18.8
21.4
Mac Studio M4 Max36 GB
about $2,49945.1
18.0
31.1
Jetson Orin NX 16GB16 GB
about $59910.5
17.5
26.3
Mac Mini M416 GB
about $79913.2
16.5
20.3
MacBook Pro 14 M532 GB
about $1,83916.9
9.2
–
MachineUsed pricetok/stok/s per $1,000tok/s per 100 W
RTX 409024 GBBest used value
about $3,000103
34.3
22.9
RTX 509032 GB
about $6,400183
28.6
31.8
Mac Studio M4 Max36 GB
about $2,00045.1
22.6
31.1

Cost per token: the NVIDIA Jetson Orin NX 16GB against 16–36 GB machines and GPUs

On Qwen3.5 9B (9B params, Q4_K_M, 4K context). Speed, cost per token and power per token depend on the model.

≤16 GB24–36 GBthis AI boxat MSRP
$0.00$1.00$2.00$3.00$4.00$5.00$6.00$7.00$8.00$0$2,000$4,000$6,000$ per 1M output tokens · lower is betternew price →Mac Mini M4 · 16 GB · $799 MSRP · $4.04 per 1M tokensMac Mini M4 Pro · 24 GB · $1,599 MSRP · $3.57 per 1M tokensMacBook Pro 14 M5 · 32 GB · ≈$1,839 · $6.90 per 1M tokensMac Studio M4 Max · 36 GB · $2,499 MSRP · $3.65 per 1M tokensRTX 5090 · 32 GB · ≈$5,376 · $1.99 per 1M tokensJetson Orin NX 16GB · 16 GB · $599 MSRP · $3.78 per 1M tokensJetson Orin NX 16GB 16 GBRTX 5090Mac Studio M4 Max 36 GBMac Mini M4 Pro 24 GBMacBook Pro 14 M5 32 GBMac Mini M4 16 GB
$0.00$1.00$2.00$3.00$4.00$5.00$6.00$7.00$8.00$0$2k$4k$6k$ per 1M output tokens · lower is betternew price →Mac Mini M4 · 16 GB · $799 MSRP · $4.04 per 1M tokensMac Mini M4 Pro · 24 GB · $1,599 MSRP · $3.57 per 1M tokensMacBook Pro 14 M5 · 32 GB · ≈$1,839 · $6.90 per 1M tokensMac Studio M4 Max · 36 GB · $2,499 MSRP · $3.65 per 1M tokensRTX 5090 · 32 GB · ≈$5,376 · $1.99 per 1M tokensJetson Orin NX 16GB · 16 GB · $599 MSRP · $3.78 per 1M tokensJetson Orin NX 16GB 16 GBRTX 5090Mac Studio M4 Max 36 GB
Cost per 1M output tokenson Qwen3.5 9B · card over 3 years + power
$3.78
Machine, MSRP
$3.62
Power
$0.16
3-year cost of ownership$625
Or rent an RTX 3090 in the cloud$0.18/hr

Renting costs as much as buying after ≈3,400 hours (≈2.3 years at 4 h a day).

Include card price

Each dot is a machine or GPU; hover or tap it for its numbers, click to open it. Machines without a price for the chosen basis are left out. Machines without a published power figure are left out of Per watt and costed without power. How we calculate cost per token

Similar devices and alternatives

The same chip from other makers first, then machines and GPUs people weigh against it, each with the one reason it is worth a look. Speeds on Qwen3.5 9B.

26% faster, same memoryMac Mini M4 (16 GB)Apple · 12.5 GB for models · M4
▲ 26% speedsame memoryabout $799
186% faster, 7.5 GB more memoryMac Mini M4 Pro (24 GB)Apple · 20.5 GB for models · M4 Pro
▲ 186% speed▲ 7.5 GBabout $1,599
61% faster, 15.5 GB more memoryMacBook Pro 14 M5Apple · 28.5 GB for models · M5
▲ 61% speed▲ 15.5 GBabout $1,839
330% faster, 19.5 GB more memoryMac Studio M4 Max (36 GB)Apple · 32.5 GB for models · M4 Max
▲ 330% speed▲ 19.5 GBabout $2,499
Reads prompts 11× faster, 24 GB for modelsRTX 4090NVIDIA · 24 GB VRAM · GPU
▲ 11× prompts▲ 881% speed▲ 11 GBabout $1,599rent $0.37/hr
Reads prompts 14× faster, 32 GB for modelsRTX 5090NVIDIA · 32 GB VRAM · GPU
▲ 14× prompts▲ 1644% speed▲ 19 GBabout $5,376rent $0.50/hr

NVIDIA Jetson Orin NX 16GB tokens per second by model size

Output speed depends on memory bandwidth (102 GB/s). Grey rows need more than the 13 GB available.

  1. 9BQwen3.5 9B · Q4_K_M · 6.6 GB
    10.5tok/s
  2. 14BQwen3 14B · Q4_K_M · 11.9 GB slow
    7.0tok/s
  3. 27–31BQwen3.8 27B · IQ3_S · 13.0 GB tight · IQ3_S
    5.3tok/s
  4. 35B MoEQwen3.6 35B A3B · not at 4-bit
    needs 22.0 GB
  5. 70BLlama 3.3 70B Instruct · not at 4-bit
    needs 51.3 GB
  6. 120B MoEGPT OSS 120B · not at 4-bit
    needs 70.8 GB

Estimated speeds: Q4_K_M, 4K context, one user. How we estimate

Unified-memory platforms: writing vs compute

Writing, tok/sCompute, TFLOPS
  1. M116 GB · 68.3 GB/s
    7.5
    5
  2. M216 GB · 100 GB/s
    11.0
    7
  3. Orin Nano8 GB · 102 GB/s
    10.4
    17
  4. Orin NX16 GB · 102.4 GB/s
    10.5
    15
  5. M316 GB · 102.4 GB/s
    11.3
    7
  6. Ryzen AI 9 HX 37032 GB · 120 GB/s
    11.3
    12

Machines with about the same memory bandwidth write at about the same speed; compute decides how fast prompts are read. Speeds on Qwen3.5 9B.

All 985 models on the NVIDIA Jetson Orin NX 16GB

Featured only

Best quant that fits in 13 GB, ordered by llmrun Score. Fit grades how well it runs here, S best. How ranking works

LLM models compatible with NVIDIA Jetson Orin NX 16GB — ranked by Score, recency and Fit
ModelVRAMScoreFit
Vicuna 7B V1.5
Q4_K_M·11.9 t/s
6.6 GBB51, Decent
Datarus R1 14B Preview
Q4_K_M·7.0 t/s
12.4 GBD21, Barely runs
Tiny CohereForCausalLM
Q4_K_M·221.9 t/s
0.4 GBS100, Runs great
LFM2 1.2B RAG
Q4_K_M·62.2 t/s
1.2 GBA84, Runs well
Llama 7B
Q4_K_M·15.0 t/s
4.5 GBB55, Decent
LFM2 1.2B Extract
Q4_K_M·62.2 t/s
1.2 GBA84, Runs well
MiMo 7B Base
Q4_K_M·12.6 t/s
7.4 GBB51, Decent
Llama3 OpenBioLLM 8B
Q4_K_M·12.4 t/s
6.2 GBB51, Decent
Dolphin Mistral 24B Venice Edition
IQ3_XXS·6.6 t/s
13.0 GBF12, Won't fit
Nemotron Orchestrator 8B
Q4_K_M·12.1 t/s
7.6 GBB51, Decent
Olmo 3 1025 7B
Q4_K_M·11.6 t/s
8.4 GBB50, Decent
SOLAR 10.7B Instruct v1.0
Q4_K_M·9.3 t/s
7.5 GBB47, Decent
Granite 3.1 2B Instruct
Q4_K_M·33.4 t/s
3.2 GBA72, Runs well
NVIDIA Nemotron Nano 9B v2
Q4_K_M·11.0 t/s
6.0 GBB49, Decent
Llama 3.1 8B
Q4_K_M·12.6 t/s
7.9 GBB51, Decent

NVIDIA Jetson Orin NX 16GB specs and power

Specs

Chip
Orin NX · Ampere GPU (Orin NX)
CPU cores
8
GPU cores
1,024
Unified memory
16 GB LPDDR5
For models
13 GB (The system keeps 3 GB)
Memory bandwidth
102.4 GB/s
FP16 (dense)
15 TFLOPS
Power
40 W max
Released
2023-02-01
MSRP
$599

NVIDIA Jetson Orin NX 16GB FAQ

Is the NVIDIA Jetson Orin NX 16GB worth it for local LLMs in 2026?

At its $599 MSRP it gives 17.5 tok/s per $1,000 on Qwen3.5 9B, 4th of the 6 machines compared here. It runs 14B models at 7 tok/s; 9B fits but is slow at 11 tok/s, and 27–31B only just fit in 13 GB.

How much memory is available for AI on the NVIDIA Jetson Orin NX 16GB?

About 13 GB of the 16 GB. The system keeps roughly 3 GB for itself and other apps.

What is the largest model the NVIDIA Jetson Orin NX 16GB can run?

GPT OSS 20B (20.9B) is the largest at 4-bit or better: at Q4_K_S it uses 12.4 GB, 95% of the available memory, and runs at about 26.4 tok/s.

Can the NVIDIA Jetson Orin NX 16GB run Qwen3.5 9B?

Yes. Qwen3.5 9B fits at Q4_K_M, using 6.6 GB of the 13 GB available, and it is the best model that runs well on this machine.

Can the NVIDIA Jetson Orin NX 16GB run 35B MoE models?

No. Qwen3.6 35B A3B needs 22.0 GB at Q4, more than the 13 GB available.

How long does the NVIDIA Jetson Orin NX 16GB take to read a long prompt?

On Qwen3.5 9B, about 12 s for a 4K-token prompt and 1.6 min for a 32K-token document before the first word appears.

Is renting a GPU cheaper than buying the NVIDIA Jetson Orin NX 16GB?

An RTX 3090 (24 GB) rents from $0.18/hr. Renting costs as much as buying after ≈3,400 hours (≈2.3 years at 4 h a day).

Related

12 GB GPUsAI boxesMac Mini M2 ProMacBook Pro 14" M1 ProMacBook Pro 14" M2 ProMacBook Pro 16" M1 ProMacBook Pro 16" M2 ProMacBook Air 13" M5Jetson AGX Orin 32GBBest coding LLMsAll hardware