Best small local LLMs

Q: What is the best small local LLM?

On 8 GB, Qwen3.5 9B is the top pick with a score of 99, running at about 46 t/s on RTX 5060. It fits up to 32K tokens of context there. Open a model to see which GPUs and Macs run it.

Q: Can I run an LLM on a laptop without a GPU?

Yes. Smaller models can run on the CPU or an integrated GPU, where speed depends on memory bandwidth. You might get a few up to several dozen tokens per second, varying by both the model and your hardware.

Q: Are small models good enough for daily use?

Chat, summaries, basic code, and rewrites work well; they lag behind bigger models when it comes to tough reasoning or lengthy documents. Using a 4-bit quantization mostly preserves the quality.

Find open models that run on just 8 GB of memory or less, perfect for laptops, older GPUs, and modest hardware. We've ranked them using the llmrun Composite Score to highlight our top pick for both 8 and 12 GB systems.

Best for small models on

8 GB 12 GB

Top pick

Qwen3.5 9B

Alibaba · Q4_K_M · 6.6 GB · up to 32K context · on RTX 5060

Score
Speed: 45.8 t/s
Grade: A

How to run →

All 140 small models for 8 GB ↓

Also worth a look

Gemma 2 9B IT

Q4_K_M · 7.3 GB · 47.7 t/s · 8K ctx

Qwen3 8B

Q4_K_M · 7.6 GB · 52.8 t/s · 16K ctx

Also worth a look

Gemma 2 9B IT

Q4_K_M · 7.3 GB · 47.7 t/s · 8K ctx

Best small model by memory

Highest llmrun Composite Score that fits each size with at least 8K context and 10 t/s, at a Q3 or better quant. Speed is decode tokens per second on that size's reference GPU. Full ranking

8 GB · RTX 5060
Qwen3.5 9B45.8 t/s · 32K ctx
Q4_K_M · 6.6 GB
Runner-up: Gemma 2 9B IT
12 GB · RTX 5070
Qwen3.5 9B68.7 t/s · 128K ctx
Q4_K_M · 6.6 GB
Runner-up: Gemma 2 9B IT

Best small model for each memory size
Memory	Best pick	Score	Speed	Context	Runner-up
8 GBRTX 5060	Qwen3.5 9BQ4_K_M · 6.6 GB		45.8 t/s	32K	Gemma 2 9B IT68.0 · 47.7 t/s
12 GBRTX 5070	Qwen3.5 9BQ4_K_M · 6.6 GB		68.7 t/s	128K	Gemma 2 9B IT68.0 · 71.6 t/s

Small models that run on 8 GB140

Ranked by Composite Score, models without one after. Best quant that fits with a 16K context; speed is decode tokens per second on the reference GPU.

140 models · 76 run great · 33 run well

Featured only

Score: llmrun Score, 100 = average model. Fit: how well it runs on this hardware, S best to F won't fit. Ordered by Score, recency and Fit. How ranking works

Showing compatibility for NVIDIA GeForce RTX 5060

LLM models compatible with NVIDIA GeForce RTX 5060 — ranked by Score, recency and Fit
Model	Quant	VRAM	Speed	Context	Fit
Llama 3.2 1B Instruct1.2BChatFunctions Q4_K_M·355.1 t/s	Q4_K_M	2.3 GB29%	355.1 t/s	131K	S100, Runs great
Translategemma 4B IT4.3BVision Q4_K_M·102.5 t/s	Q4_K_M	5 GB63%	102.5 t/s	—	S92, Runs great
Mathstral 7B v0.17.2BMath Q4_K_M·59.2 t/s	Q4_K_M	6.8 GB85%	59.2 t/s	33K	A76, Runs well
Llama 3.1 Nemotron Nano VL 8B V18.7BVision Q3_K_M·62.2 t/s	Q3_K_M	7.3 GB91%	62.2 t/s	—	A69, Runs well
InternVL3 5 8B8.5BVision Q4_K_S·55.2 t/s	Q4_K_S	7.9 GB99%	55.2 t/s	—	C44, Usable
Yi 1.5 9B Chat8.8BChat < 8K context fits Q4_K_M·50.2 t/s	Q4_K_M	6 GB75%	50.2 t/s	4K	A79, Runs well
Qwen3 VL 8B Instruct8.8BVisionFunctions Q4_K_M·49.7 t/s	Q4_K_M	8.0 GB100%	49.7 t/s	262K	C44, Usable
Qwen2.5 VL 3B Instruct3.8BVision Q4_K_M·110.7 t/s	Q4_K_M	3.2 GB40%	110.7 t/s	128K	S93, Runs great
ZDTaichu5.0 9B9.8BVisionFunctions Q3_K_M·55.5 t/s	Q3_K_M	8.0 GB100%	55.5 t/s	—	C44, Usable
Qwen3 VL 2B Thinking2.1BVisionFunctions Q4_K_M·160.9 t/s	Q4_K_M	3.5 GB43%	160.9 t/s	262K	S97, Runs great
Qwen3.5 4B Base4.7BVision Q4_K_M·89.3 t/s	Q4_K_M	3.4 GB43%	89.3 t/s	262K	S89, Runs great
Qwen2 VL 7B Instruct8.3BVision Q4_K_M·54.0 t/s	Q4_K_M	6.2 GB78%	54.0 t/s	33K	A81, Runs well
Llama 3.1 Tulu 3 8B8.0BChat Q4_K_M·54.0 t/s	Q4_K_M	7.3 GB91%	54.0 t/s	131K	A65, Runs well
Starling LM 7B Beta7.2BChat Q4_K_M·59.3 t/s	Q4_K_M	5.7 GB72%	59.3 t/s	8K	A84, Runs well
Qwen3.5 0.8B Base873MVision Q4_K_M·334.7 t/s	Q4_K_M	0.9 GB12%	334.7 t/s	262K	S100, Runs great

How much memory do you need for a small model?

On 8 GB the best small model is Qwen3.5 9B (score 99). Here "small" describes models fitting into 8 GB when using 4-bit quantization. When running on a laptop, remember to factor in the OS and background apps for memory usage, and expect slower speeds compared to a desktop with GPU acceleration.

Frequently Asked Questions

What is the best small local LLM?

On 8 GB, Qwen3.5 9B is the top pick with a score of 99, running at about 46 t/s on RTX 5060. It fits up to 32K tokens of context there. Open a model to see which GPUs and Macs run it.

Can I run an LLM on a laptop without a GPU?

Yes. Smaller models can run on the CPU or an integrated GPU, where speed depends on memory bandwidth. You might get a few up to several dozen tokens per second, varying by both the model and your hardware.

Are small models good enough for daily use?

Chat, summaries, basic code, and rewrites work well; they lag behind bigger models when it comes to tough reasoning or lengthy documents. Using a 4-bit quantization mostly preserves the quality.