Best small local LLMs

Find open models that run on just 8 GB of memory or less, perfect for laptops, older GPUs, and modest hardware. We've ranked them using the llmrun Composite Score to highlight our top pick for both 8 and 12 GB systems.

Best for small models on

8 GB 12 GB

Top pick

Qwen3.5 9B

Alibaba · Q4_K_M · 6.6 GB · up to 128K context · on RTX 5070

Score
Speed: 68.7 t/s
Grade: S

How to run →

All 140 small models for 12 GB ↓

Also worth a look

Gemma 2 9B IT

Q4_K_M · 7.3 GB · 71.6 t/s · 8K ctx

Qwen3 8B

Q4_K_M · 7.6 GB · 79.1 t/s · 40K ctx

Also worth a look

Gemma 2 9B IT

Q4_K_M · 7.3 GB · 71.6 t/s · 8K ctx

Best small model by memory

Highest llmrun Composite Score that fits each size with at least 8K context and 10 t/s, at a Q3 or better quant. Speed is decode tokens per second on that size's reference GPU. Full ranking

8 GB · RTX 5060
Qwen3.5 9B45.8 t/s · 32K ctx
Q4_K_M · 6.6 GB
Runner-up: Gemma 2 9B IT
12 GB · RTX 5070
Qwen3.5 9B68.7 t/s · 128K ctx
Q4_K_M · 6.6 GB
Runner-up: Gemma 2 9B IT

Best small model for each memory size
Memory	Best pick	Score	Speed	Context	Runner-up
8 GBRTX 5060	Qwen3.5 9BQ4_K_M · 6.6 GB		45.8 t/s	32K	Gemma 2 9B IT68.0 · 47.7 t/s
12 GBRTX 5070	Qwen3.5 9BQ4_K_M · 6.6 GB		68.7 t/s	128K	Gemma 2 9B IT68.0 · 71.6 t/s

Small models that run on 12 GB140

Ranked by Composite Score, models without one after. Best quant that fits with a 16K context; speed is decode tokens per second on the reference GPU.

140 models · 133 run great · 7 run well

Featured only

Score: llmrun Score, 100 = average model. Fit: how well it runs on this hardware, S best to F won't fit. Ordered by Score, recency and Fit. How ranking works

Showing compatibility for NVIDIA GeForce RTX 5070

LLM models compatible with NVIDIA GeForce RTX 5070 — ranked by Score, recency and Fit
Model	Quant	VRAM	Speed	Context	Fit
LFM2.5 8B A1B8.5BChatFunctions Q4_K_M·262.0 t/s	Q4_K_M	5.6 GB47%	262.0 t/s	128K	S100, Runs great
MiniCPM V 4.7 1B1.6BVisionFunctions Q4_K_M·341.3 t/s	Q4_K_M	1.6 GB14%	341.3 t/s	262K	S100, Runs great
MinerU2.5 Pro 2605 1.2B1.2BVision Q4_K_M·428.2 t/s	Q4_K_M	1.1 GB9%	428.2 t/s	8K	S100, Runs great
D1 3B3.1BVisionFunctions Q4_K_M·189.9 t/s	Q4_K_M	2.6 GB22%	189.9 t/s	33K	S99, Runs great
Gemma 4 E2B IT5.1BChatFunctions Q4_K_M·127.3 t/s	Q4_K_M	3.5 GB29%	127.3 t/s	131K	S95, Runs great
FrogNano 4B 26094.7BChatFunctions Q4_K_M·134.0 t/s	Q4_K_M	3.4 GB29%	134.0 t/s	262K	S95, Runs great
Granite Vision 4.1 4B4.0BVisionFunctions Q4_K_M·152.2 t/s	Q4_K_M	4.0 GB34%	152.2 t/s	131K	S97, Runs great
MiniCPM V 4.61.3BVisionFunctions Q4_K_M·386.5 t/s	Q4_K_M	1.5 GB12%	386.5 t/s	262K	S100, Runs great
North Micro Vision Instruct2.5BVisionChat BF16·79.4 t/s	BF16	7.1 GB60%	79.4 t/s	500K	S87, Runs great
MiniCPM V 4.6 Thinking1.3BVisionFunctions Q4_K_M·386.5 t/s	Q4_K_M	1.5 GB12%	386.5 t/s	262K	S100, Runs great
D1 Omni 600M587MVision Q4_K_M·606.7 t/s	Q4_K_M	1.2 GB10%	606.7 t/s	128K	S100, Runs great
Ornith 1.0 9B9.4BChatFunctions Q4_K_M·70.3 t/s	Q4_K_M	6.5 GB54%	70.3 t/s	262K	S86, Runs great
NVIDIA Nemotron Labs Dory1.0 4B BF164.0BChatFunctions Q4_K_M·142.7 t/s	Q4_K_M	5.7 GB48%	142.7 t/s	262K	S96, Runs great
DeepSeek R1 0528 Qwen3 8B8.2BChatReasoningFunctions Q4_K_M·79.1 t/s	Q4_K_M	7.6 GB64%	79.1 t/s	131K	S87, Runs great
MinerU2.5 Pro 2604 1.2B1.2BVision Q4_K_M·428.2 t/s	Q4_K_M	1.1 GB9%	428.2 t/s	8K	S100, Runs great

How much memory do you need for a small model?

On 8 GB the best small model is Qwen3.5 9B (score 99). Here "small" describes models fitting into 8 GB when using 4-bit quantization. When running on a laptop, remember to factor in the OS and background apps for memory usage, and expect slower speeds compared to a desktop with GPU acceleration.

Frequently Asked Questions

What is the best small local LLM?

On 8 GB, Qwen3.5 9B is the top pick with a score of 99, running at about 46 t/s on RTX 5060. It fits up to 32K tokens of context there. Open a model to see which GPUs and Macs run it.

Can I run an LLM on a laptop without a GPU?

Yes. Smaller models can run on the CPU or an integrated GPU, where speed depends on memory bandwidth. You might get a few up to several dozen tokens per second, varying by both the model and your hardware.

Are small models good enough for daily use?

Chat, summaries, basic code, and rewrites work well; they lag behind bigger models when it comes to tough reasoning or lengthy documents. Using a 4-bit quantization mostly preserves the quality.