llmrun

NVIDIA · Orin Nano · AI box

Best AI Models for NVIDIA Jetson Orin Nano 8GB (Super)

Runs 9B models at 10 tok/s; 14B doesn't fit in 8 GB.

8 GB unified102 GB/s17 TFLOPS FP1625 W2023

What LLMs can the NVIDIA Jetson Orin Nano 8GB (Super) run?

#ModelSpeedllmrun ScoreQuant · Memory
1
Qwen3.5 9BAlibabaHow to run →
Q4_K_M · 6.6 GB · 10.4 t/s
10.4 tok/sQ4_K_M · 6.6 GB
2
Qwen3.5 4BAlibaba
Q4_K_M · 3.4 GB · 20.3 t/s
20.3 tok/sQ4_K_M · 3.4 GB
3
Gemma 3 4B ITGoogle
Q4_K_M · 5.0 GB · 23.3 t/s
23.3 tok/sQ4_K_M · 5.0 GB
4
Ling 3.0 TinyInclusion AI
Q4_K_M · 5.5 GB · 64.9 t/s
64.9 tok/sQ4_K_M · 5.5 GB
5
Gemma 4 E2B ITGoogle
Q4_K_M · 3.5 GB · 19.3 t/s
19.3 tok/sQ4_K_M · 3.5 GB
See all 216 models on the NVIDIA Jetson Orin Nano 8GB (Super) →Bars = traction for models without a Score yet

About the NVIDIA Jetson Orin Nano 8GB (Super)

The NVIDIA Jetson Orin Nano 8GB (Super) is the fastest 8 GB machine: it runs 9B models at 10 tok/s. With 17 TFLOPS it reads long prompts faster than most 8 GB machines: a 4K-token prompt takes 10 s. Orin Nano (Ampere), released 2023.

Speed in 8 GB#1102 GB/s
Prompt reading#117 TFLOPS
For models8 GBall of it
ChipOrin Nanoreleased 2023

NVIDIA Jetson Orin Nano 8GB (Super) strengths and weaknesses

Strengths

  • Low power draw: 25 W at full load for the whole machine

Weaknesses

  • 14B models don't fit: Qwen3 14B needs 11.9 GB at Q4 vs 8 GB usable here
  • Little room for context: Qwen3.5 9B uses 6.6 of 8 GB usable, 1.4 GB left
  • Slow prompt reading: A 32K-token prompt on Qwen3.5 9B waits 1.4 min for the first token
  • No FP8 or NVFP4: Ampere predates both; use GGUF, AWQ or GPTQ builds

Memory for models

8 GB of 8 GB
9B6.6 GB
9B6.6 GB
For models 8 GBThe system keeps 0 GBModel size at Q4

14B models need 11.9 GB at Q4, more than the 8 GB available.Or rent an RTX 3090 (24 GB) from $0.19/hr →

Similar devices and alternatives

The same chip from other makers first, then machines and GPUs people weigh against it, each with the one reason it is worth a look. Speeds on Qwen3.5 9B.

27% faster, 4.5 GB more memoryMac Mini M4 (16 GB)Apple · 12.5 GB for models · M4
▲ 27% speed▲ 4.5 GBabout $799
189% faster, 12.5 GB more memoryMac Mini M4 Pro (24 GB)Apple · 20.5 GB for models · M4 Pro
▲ 189% speed▲ 12.5 GBabout $1,599
63% faster, 20.5 GB more memoryMacBook Pro 14 M5Apple · 28.5 GB for models · M5
▲ 63% speed▲ 20.5 GBabout $1,839
334% faster, 24.5 GB more memoryMac Studio M4 Max (36 GB)Apple · 32.5 GB for models · M4 Max
▲ 334% speed▲ 24.5 GBabout $2,499
Reads prompts 10× faster, 24 GB for modelsRTX 4090NVIDIA · 24 GB VRAM · GPU
▲ 10× prompts▲ 890% speed▲ 16 GBabout $1,599rent $0.34/hr
Reads prompts 13× faster, 32 GB for modelsRTX 5090NVIDIA · 32 GB VRAM · GPU
▲ 13× prompts▲ 1661% speed▲ 24 GBabout $5,376rent $0.50/hr

NVIDIA Jetson Orin Nano 8GB (Super) tokens per second by model size

Output speed depends on memory bandwidth (102 GB/s). Grey rows need more than the 8 GB available.

  1. 9BQwen3.5 9B · Q4_K_M · 6.6 GB
    10.4tok/s
  2. 14BQwen3 14B · not at 4-bit
    needs 11.9 GB
  3. 27–31BQwen3.8 27B · not at 4-bit
    needs 17.9 GB
  4. 35B MoEQwen3.6 35B A3B · not at 4-bit
    needs 22.0 GB
  5. 70BLlama 3.3 70B Instruct · not at 4-bit
    needs 51.3 GB
  6. 120B MoEGPT OSS 120B · not at 4-bit
    needs 70.8 GB

Estimated speeds: Q4_K_M, 4K context, one user. How we estimate

Unified-memory platforms: writing vs compute

Writing, tok/sCompute, TFLOPS
  1. M116 GB · 68.3 GB/s
    7.5
    5
  2. M216 GB · 100 GB/s
    11.0
    7
  3. Orin Nano8 GB · 102 GB/s
    10.4
    17
  4. Orin NX16 GB · 102.4 GB/s
    10.5
    15
  5. M316 GB · 102.4 GB/s
    11.3
    7
  6. M416 GB · 120 GB/s
    13.2
    9

Machines with about the same memory bandwidth write at about the same speed; compute decides how fast prompts are read. Speeds on Qwen3.5 9B.

All 822 models on the NVIDIA Jetson Orin Nano 8GB (Super)

Featured only

Best quant that fits in 8 GB, ordered by llmrun Score. Fit grades how well it runs here, S best. How ranking works

LLM models compatible with NVIDIA Jetson Orin Nano 8GB (Super) — ranked by Score, recency and Fit
ModelVRAMScoreFit
Olmo 3 7B Instruct
IQ4_XS·12.5 t/s
8.0 GBD20, Barely runs
Qwen2.5 Coder 7B Instruct
Q4_K_M·13.3 t/s
5.8 GBB53, Decent
H2o Danube3 500M Chat
Q4_K_M·93.4 t/s
1.0 GBS90, Runs great
Qianfan OCR
Q4_K_M·19.9 t/s
4.7 GBB60, Decent
Dream V0 Instruct 7B
Q4_K_M·13.3 t/s
5.8 GBB53, Decent
Gemma3 1B IT
BF16·30.1 t/s
3.6 GBA71, Runs well
Gemma 4 E2B
Q4_K_M·19.3 t/s
3.5 GBB59, Decent
Llava 1.5 7B HF
Q4_K_M·14.2 t/s
5.0 GBB53, Decent
T5gemma B B Ul2 IT
BF16·51.0 t/s
2.5 GBA80, Runs well
Granite Docling 258M
Q4_K_M·132.6 t/s
0.6 GBS95, Runs great
Gemma 3 270M IT
Q4_K_M·368.3 t/s
1.2 GBS100, Runs great
UserLM 8B
Q4_K_M·12.3 t/s
6.2 GBB51, Decent
Opt 125M
FP16·236.8 t/s
0.3 GBS100, Runs great
Ornith 1.0 9B
Q4_K_M·10.7 t/s
6.5 GBB47, Decent
Gemma 2 9B IT
Q4_K_M·10.9 t/s
7.3 GBC34, Usable

NVIDIA Jetson Orin Nano 8GB (Super) specs and power

Specs

Chip
Orin Nano
CPU cores
6
GPU cores
1,024
Unified memory
8 GB LPDDR5
For models
8 GB
Memory bandwidth
102 GB/s
FP16 (dense)
16.8 TFLOPS
Power
25 W max
Released
2023-03-08

NVIDIA Jetson Orin Nano 8GB (Super) FAQ

How much memory is available for AI on the NVIDIA Jetson Orin Nano 8GB (Super)?

All 8 GB can hold model weights.

What is the largest model the NVIDIA Jetson Orin Nano 8GB (Super) can run?

Qwen3.5 9B (9.7B) is the largest at 4-bit or better: at Q4_K_M it uses 6.6 GB, 83% of the available memory, and runs at about 10.4 tok/s.

Can the NVIDIA Jetson Orin Nano 8GB (Super) run 14B models?

No. Qwen3 14B needs 11.9 GB at Q4, more than the 8 GB available.

How long does the NVIDIA Jetson Orin Nano 8GB (Super) take to read a long prompt?

On Qwen3.5 9B, about 10 s for a 4K-token prompt and 84 s for a 32K-token document before the first word appears.

Is renting a GPU cheaper than buying the NVIDIA Jetson Orin Nano 8GB (Super)?

An RTX 3080 (10 GB) rents from $0.11/hr.

Related

8 GB GPUsAI boxesiMac 24" M3MacBook Air 13" M3MacBook Air 15" M3MacBook Pro 14" M3Mac Mini M2MacBook Air 13" M2Jetson Orin NX 16GBBest coding LLMsAll hardware