Best local LLMs for tool use

Q: What is the best local LLM for tool use?

On 24 GB, Qwen3.8 27B is the top pick with a score of 122, running at about 38 t/s on RTX 4090. It fits up to 64K tokens of context there. On 8 GB, the best fit is Qwen3.5 9B. With no memory limit, the best is MiMo V2.6 Pro RL. Open a model to see which GPUs and Macs run it.

Q: Can I run a tool-use model on a laptop or an 8 GB GPU?

Yes. On 8 GB the best pick is Qwen3.5 9B (Q4_K_M, 6.6 GB), which runs at about 46 t/s on RTX 5060. Laptops with 8 to 16 GB of memory can run smaller models in the same way.

Q: What is tool calling?

Instead of plain text, the model outputs a structured call like a function name plus its arguments. Your code executes that function, sends back the result, and lets the model continue from there.

Q: Which runtime supports tool calling for local models?

Ollama, LM Studio, llama.cpp, and vLLM all offer an OpenAI-compatible tools interface for models built to use it. Remember that the model's chat template needs to support these tools.

Run open models capable of function calling, designed for agents and apps that use tools and APIs. We rank them using the llmrun Composite Score, offering the top choice for every memory capacity.

Best for tool use on

8 GB 12 GB 16 GB 24 GB 32 GB 48 GB+

Top pick

Qwen3.5 9B

Alibaba · Q4_K_M · 6.6 GB · up to 32K context · on RTX 5060

Score
Speed: 45.8 t/s
Grade: A

How to run →

All 129 tool-use models for 8 GB ↓

Also worth a look

Gemma 4 E4B IT

Q4_K_M · 5.6 GB · 54.7 t/s · 64K ctx

Qwen2.5 7B Instruct

Q4_K_M · 5.8 GB · 58.4 t/s · 32K ctx

Also worth a look

Gemma 4 E4B IT

Q4_K_M · 5.6 GB · 54.7 t/s · 64K ctx

Best tool-use model by memory

Highest llmrun Composite Score that fits each size with at least 32K context and 15 t/s, at a Q3 or better quant. Speed is decode tokens per second on that size's reference GPU. The last row has no memory limit: the best model whatever it needs, on the GPU plan shown. Full ranking

8 GB · RTX 5060
Qwen3.5 9B45.8 t/s · 32K ctx
Q4_K_M · 6.6 GB
Runner-up: Gemma 4 E4B IT
12 GB · RTX 5070
Qwen3.5 9B68.7 t/s · 128K ctx
Q4_K_M · 6.6 GB
Runner-up: GPT OSS 20B
16 GB · RTX 5080
Qwen3.8 27B42.6 t/s · 32K ctx
Q4_0 · 15.1 GB
Runner-up: Gemma 4 26B A4B IT
24 GB · RTX 4090
Qwen3.8 27B37.6 t/s · 64K ctx
Q4_K_M · 17.9 GB
Runner-up: Gemma 4 31B IT
32 GB · RTX 5090
Qwen3.8 27B66.9 t/s · 256K ctx
Q4_K_M · 17.9 GB
Runner-up: Gemma 4 31B IT
48 GB+ · no limit · about 5× H200 SXM 141 GB
MiMo V2.6 Pro RL1M ctx
Q4_K_M · needs 614 GB
Runner-up: GLM 5.3

Best tool-use model for each memory size
Memory	Best pick	Speed	Context	Runner-up
8 GBRTX 5060	Qwen3.5 9BQ4_K_M · 6.6 GB	45.8 t/s	32K	Gemma 4 E4B IT54.7 t/s
12 GBRTX 5070	Qwen3.5 9BQ4_K_M · 6.6 GB	68.7 t/s	128K	GPT OSS 20B96.4 · 156.0 t/s
16 GBRTX 5080	Qwen3.8 27BQ4_0 · 15.1 GB	42.6 t/s	32K	Gemma 4 26B A4B IT105.8 · 178.1 t/s
24 GBRTX 4090	Qwen3.8 27BQ4_K_M · 17.9 GB	37.6 t/s	64K	Gemma 4 31B IT106.6 · 32.1 t/s
32 GBRTX 5090	Qwen3.8 27BQ4_K_M · 17.9 GB	66.9 t/s	256K	Gemma 4 31B IT106.6 · 57.1 t/s
48 GB+ · no limitabout 5× H200 SXM 141 GB	MiMo V2.6 Pro RLQ4_K_M · needs 614 GB	n/a	1M	GLM 5.3130.1

Tool-use models that run on 8 GB129

Ranked by Composite Score, models without one after. Best quant that fits with a 16K context; speed is decode tokens per second on the reference GPU.

129 models · 73 run great · 28 run well

Featured only

Score: llmrun Score, 100 = average model. Fit: how well it runs on this hardware, S best to F won't fit. Ordered by Score, recency and Fit. How ranking works

Showing compatibility for NVIDIA GeForce RTX 5060

LLM models compatible with NVIDIA GeForce RTX 5060 — ranked by Score, recency and Fit
Model	Quant	VRAM	Speed	Context	Fit
K2 Horizon 7B9.0BChatFunctions < 32K context fits Q4_K_S·51.4 t/s	Q4_K_S	7.8 GB97%	51.4 t/s	524K	B54, Decent
Chandra8.8BVisionFunctions < 32K context fits Q4_K_M·49.7 t/s	Q4_K_M	8.0 GB100%	49.7 t/s	262K	C44, Usable
Granite 4.0 350M352MChatFunctions Q4_K_M·510.9 t/s	Q4_K_M	1.0 GB12%	510.9 t/s	33K	S100, Runs great
Olmo 3 7B Instruct7.3BChatFunctions < 32K context fits IQ4_XS·54.9 t/s	IQ4_XS	8.0 GB100%	54.9 t/s	66K	C44, Usable
LFM2.5 350M354MChatFunctions Q4_K_M·502.1 t/s	Q4_K_M	0.8 GB9%	502.1 t/s	128K	S100, Runs great
Dream V0 Instruct 7B7.6BChatFunctions Q4_K_M·58.4 t/s	Q4_K_M	5.8 GB73%	58.4 t/s	131K	A83, Runs well
Sarashina2.2 0.5B Instruct v0.1793MChatFunctions < 32K context fits Q4_K_M·323.6 t/s	Q4_K_M	1.3 GB16%	323.6 t/s	8K	S100, Runs great
NVIDIA Nemotron 3 Nano 4B BF164.0BChatFunctions Q4_K_M·100.4 t/s	Q4_K_M	2.9 GB37%	100.4 t/s	262K	S91, Runs great
Granite Switch 4.1 8B8.8BChatFunctions < 32K context fits Q3_K_L·56.5 t/s	Q3_K_L	7.6 GB95%	56.5 t/s	131K	B60, Decent
Granite 4.0 H Tiny6.9BChatFunctions Q4_K_M·179.9 t/s	Q4_K_M	5.8 GB73%	179.9 t/s	131K	S99, Runs great
Llama 3.2 1B Instruct1.2BChatFunctions Q4_K_M·355.1 t/s	Q4_K_M	2.3 GB29%	355.1 t/s	131K	S100, Runs great
Qwen2.5 3B Instruct3.1BChatFunctions Q4_K_M·130.6 t/s	Q4_K_M	2.8 GB35%	130.6 t/s	33K	S95, Runs great
Qwen2.5 1.5B Instruct1.5BChatFunctions Q4_K_M·227.5 t/s	Q4_K_M	1.7 GB21%	227.5 t/s	33K	S100, Runs great
Openthaigpt1.5 7B Instruct7.6BChatFunctions Q4_K_M·58.4 t/s	Q4_K_M	5.8 GB73%	58.4 t/s	33K	A83, Runs well
Qwen3 VL 8B Instruct8.8BVisionFunctions < 32K context fits Q4_K_M·49.7 t/s	Q4_K_M	8.0 GB100%	49.7 t/s	262K	C44, Usable

How much memory do you need for a tool-use model?

From 8 to 12 GB the best tool-use model is Qwen3.5 9B (score 99). From 16 to 32 GB it is Qwen3.8 27B (score 122). With no memory limit, the best is MiMo V2.6 Pro RL (score 133). Since agents pass lengthy prompts packed with tool definitions and outcomes, both context window size and processing speed matter. Choose a model that retains a generous amount of context within its memory.

Frequently Asked Questions

What is the best local LLM for tool use?

On 24 GB, Qwen3.8 27B is the top pick with a score of 122, running at about 38 t/s on RTX 4090. It fits up to 64K tokens of context there. On 8 GB, the best fit is Qwen3.5 9B. With no memory limit, the best is MiMo V2.6 Pro RL. Open a model to see which GPUs and Macs run it.

Can I run a tool-use model on a laptop or an 8 GB GPU?

Yes. On 8 GB the best pick is Qwen3.5 9B (Q4_K_M, 6.6 GB), which runs at about 46 t/s on RTX 5060. Laptops with 8 to 16 GB of memory can run smaller models in the same way.

What is tool calling?

Instead of plain text, the model outputs a structured call like a function name plus its arguments. Your code executes that function, sends back the result, and lets the model continue from there.

Which runtime supports tool calling for local models?

Ollama, LM Studio, llama.cpp, and vLLM all offer an OpenAI-compatible tools interface for models built to use it. Remember that the model's chat template needs to support these tools.