Best small local LLMs

Q: What is the best small local LLM?

On 8 GB, Qwen3.5 9B is the top pick with a score of 99, running at about 46 t/s on RTX 5060. It fits up to 32K tokens of context there. Open a model to see which GPUs and Macs run it.

Q: Can I run an LLM on a laptop without a GPU?

Yes. Smaller models can run on the CPU or an integrated GPU, where speed depends on memory bandwidth. You might get a few up to several dozen tokens per second, varying by both the model and your hardware.

Q: Are small models good enough for daily use?

Chat, summaries, basic code, and rewrites work well; they lag behind bigger models when it comes to tough reasoning or lengthy documents. Using a 4-bit quantization mostly preserves the quality.

Find open models that run on just 8 GB of memory or less, perfect for laptops, older GPUs, and modest hardware. We've ranked them using the llmrun Composite Score to highlight our top pick for both 8 and 12 GB systems.

Best for small models on

8 GB 12 GB

Top pick

Qwen3.5 9B

Alibaba · Q4_K_M · 6.6 GB · up to 32K context · on RTX 5060

Score
Speed: 45.8 t/s
Grade: A

How to run →

All 399 small models for 8 GB ↓

Also worth a look

Gemma 2 9B IT

Q4_K_M · 7.3 GB · 47.7 t/s · 8K ctx

Qwen3 8B

Q4_K_M · 7.6 GB · 52.8 t/s · 16K ctx

Also worth a look

Gemma 2 9B IT

Q4_K_M · 7.3 GB · 47.7 t/s · 8K ctx

Best small model by memory

Highest llmrun Composite Score that fits each size with at least 8K context and 10 t/s, at a Q3 or better quant. Speed is decode tokens per second on that size's reference GPU. Full ranking

8 GB · RTX 5060
Qwen3.5 9B45.8 t/s · 32K ctx
Q4_K_M · 6.6 GB
Runner-up: Gemma 2 9B IT
12 GB · RTX 5070
Qwen3.5 9B68.7 t/s · 128K ctx
Q4_K_M · 6.6 GB
Runner-up: Gemma 2 9B IT

Best small model for each memory size
Memory	Best pick	Score	Speed	Context	Runner-up
8 GBRTX 5060	Qwen3.5 9BQ4_K_M · 6.6 GB		45.8 t/s	32K	Gemma 2 9B IT68.0 · 47.7 t/s
12 GBRTX 5070	Qwen3.5 9BQ4_K_M · 6.6 GB		68.7 t/s	128K	Gemma 2 9B IT68.0 · 71.6 t/s

Small models that run on 8 GB399

Ranked by Composite Score, models without one after. Best quant that fits with a 16K context; speed is decode tokens per second on the reference GPU.

399 models · 252 run great · 85 run well

Featured only

Score: llmrun Score, 100 = average model. Fit: how well it runs on this hardware, S best to F won't fit. Ordered by Score, recency and Fit. How ranking works

Showing compatibility for NVIDIA GeForce RTX 5060

LLM models compatible with NVIDIA GeForce RTX 5060 — ranked by Score, recency and Fit
Model	Quant	VRAM	Speed	Context	Fit
ALoDLM 8B8.2BChatCodeFunctions Q4_K_M·52.8 t/s	Q4_K_M	7.6 GB95%	52.8 t/s	41K	B58, Decent
Ouro 1.4B1.4BChatReasoning BF16·81.6 t/s	BF16	6.4 GB80%	81.6 t/s	66K	S88, Runs great
Llm Jp 4.1 8B Thinking8.6BChatFunctions Q4_K_M·50.9 t/s	Q4_K_M	7.6 GB95%	50.9 t/s	66K	B57, Decent
SmolVLM2 256M Video Instruct256MVision Q4_K_M·582.4 t/s	Q4_K_M	0.6 GB8%	582.4 t/s	8K	S100, Runs great
DeepSeek R1 Distill Qwen 1.5B1.8BChatReasoningFunctions Q4_K_M·205.1 t/s	Q4_K_M	1.8 GB23%	205.1 t/s	131K	S100, Runs great
Functiongemma 270M IT268MChat BF16·493.6 t/s	BF16	1.6 GB20%	493.6 t/s	—	S100, Runs great
AI21 Jamba Reasoning 3B3.2BChatReasoningFunctions Q4_K_M·129.4 t/s	Q4_K_M	2.3 GB28%	129.4 t/s	262K	S95, Runs great
SmolLM 135M Instruct FP32135MChat < 8K context fits Q4_K_M·677.2 t/s	Q4_K_M	0.4 GB5%	677.2 t/s	2K	S100, Runs great
GLM 4.6V Flash10.3BVisionFunctions Q4_K_M·44.4 t/s	Q4_K_M	7.2 GB89%	44.4 t/s	131K	A65, Runs well
SmolVLM Instruct2.2BVision Q4_K_M·142.0 t/s	Q4_K_M	4.9 GB61%	142.0 t/s	16K	S96, Runs great
Phi Tiny MoE Instruct3.8BChat < 8K context fits Q4_K_M·196.7 t/s	Q4_K_M	3.1 GB39%	196.7 t/s	4K	S100, Runs great
InternVL2 1B938MVision Q4_K_M·469.7 t/s	Q4_K_M	2.0 GB25%	469.7 t/s	—	S100, Runs great
InternVL2 2B2.2BVision Q4_K_M·199.5 t/s	Q4_K_M	3.3 GB41%	199.5 t/s	—	S100, Runs great
Qwen3 4B Instruct 25074.0BChatFunctions Q4_K_M·100.4 t/s	Q4_K_M	4.2 GB53%	100.4 t/s	262K	S91, Runs great
A.X 4.0 Light7.3BChatFunctions Q4_K_M·61.0 t/s	Q4_K_M	5.6 GB70%	61.0 t/s	16K	A84, Runs well

How much memory do you need for a small model?

On 8 GB the best small model is Qwen3.5 9B (score 99). Here "small" describes models fitting into 8 GB when using 4-bit quantization. When running on a laptop, remember to factor in the OS and background apps for memory usage, and expect slower speeds compared to a desktop with GPU acceleration.

Frequently Asked Questions

What is the best small local LLM?

On 8 GB, Qwen3.5 9B is the top pick with a score of 99, running at about 46 t/s on RTX 5060. It fits up to 32K tokens of context there. Open a model to see which GPUs and Macs run it.

Can I run an LLM on a laptop without a GPU?

Yes. Smaller models can run on the CPU or an integrated GPU, where speed depends on memory bandwidth. You might get a few up to several dozen tokens per second, varying by both the model and your hardware.

Are small models good enough for daily use?

Chat, summaries, basic code, and rewrites work well; they lag behind bigger models when it comes to tough reasoning or lengthy documents. Using a 4-bit quantization mostly preserves the quality.