Best local LLMs for long context

Q: What is the best local LLM for long context?

On 24 GB, Qwen3.8 27B is the top pick with a score of 122, running at about 40 t/s on RTX 4090. It fits up to 128K tokens of context there. On 8 GB, the best fit is Gemma 4 E4B IT. With no memory limit, the best is DeepSeek V4.1 Flash 1017. Open a model to see which GPUs and Macs run it.

Q: Does a long context window use more memory?

The key-value cache scales linearly as token count rises, reaching sizes that can match the model's weights around 128K tokens. Techniques like cache quantization or using grouped-query attention can shrink this footprint.

Q: Is a bigger context window always better?

Quality for info buried deep in a long prompt often slips, plus handling a lengthy request eats up time. Stick to the length you actually need instead of maxing it out.

Run open models featuring a context window of at least 128K tokens to handle large documents and full codebases easily. We rank them using the llmrun Composite Score, highlighting our top recommendation for every memory capacity.

Best for long context on

8 GB 12 GB 16 GB 24 GB 32 GB 48 GB+

Top pick

Qwen3.8 27B

Alibaba · Q4_K_M · 17.9 GB · up to 256K context · on RTX 5090Alibaba · Q4_K_M · 17.9 GB · on RTX 5090

Score
Speed: 66.9 t/s
Grade: S
Context: up to 256K context

How to run →

All 96 long-context models for 32 GB ↓

Also worth a look

Gemma 4 31B IT

IQ3_M · 17 GB · 74.2 t/s · 128K ctx

Qwen3.6 27B

Q4_K_M · 17.9 GB · 66.9 t/s · 256K ctx

Also worth a look

Gemma 4 31B IT

IQ3_M · 17 GB · 74.2 t/s · 128K ctx

Best long-context model by memory

Highest llmrun Composite Score that fits each size with at least 128K context and 10 t/s, at a Q3 or better quant. Speed is decode tokens per second on that size's reference GPU. The last row has no memory limit: the best model whatever it needs, on the GPU plan shown. Full ranking

8 GB · RTX 5060
Gemma 4 E4B IT58.0 t/s · 128K ctx
Q4_K_S · 5.3 GB
Runner-up: Ling 3.0 Tiny
12 GB · RTX 5070
Qwen3.5 9B68.7 t/s · 128K ctx
Q4_K_M · 6.6 GB
Runner-up: GPT OSS 20B
16 GB · RTX 5080
Gemma 4 26B A4B IT207.5 t/s · 128K ctx
IQ3_S · 12 GB
Runner-up: Qwen3.5 9B
24 GB · RTX 4090
Qwen3.8 27B40.0 t/s · 128K ctx
Q4_K_S · 16.8 GB
Runner-up: Gemma 4 26B A4B IT
32 GB · RTX 5090
Qwen3.8 27B66.9 t/s · 256K ctx
Q4_K_M · 17.9 GB
Runner-up: Gemma 4 31B IT
48 GB+ · no limit · about 7× H100 SXM 80 GB
DeepSeek V4.1 Flash 10171M ctx
Q4_K_M · needs 458 GB
Runner-up: MiMo V2.6 Pro RL

Best long-context model for each memory size
Memory	Best pick	Score	Speed	Context (fits)	Runner-up
8 GBRTX 5060	Gemma 4 E4B ITQ4_K_S · 5.3 GB	n/a	58.0 t/s	128K	Ling 3.0 Tiny243.0 t/s
12 GBRTX 5070	Qwen3.5 9BQ4_K_M · 6.6 GB		68.7 t/s	128K	GPT OSS 20B96.4 · 172.1 t/s
16 GBRTX 5080	Gemma 4 26B A4B ITIQ3_S · 12 GB		207.5 t/s	128K	Qwen3.5 9B99.4 · 98.1 t/s
24 GBRTX 4090	Qwen3.8 27BQ4_K_S · 16.8 GB		40.0 t/s	128K	Gemma 4 26B A4B IT105.8 · 171.1 t/s
32 GBRTX 5090	Qwen3.8 27BQ4_K_M · 17.9 GB		66.9 t/s	256K	Gemma 4 31B IT106.6 · 74.2 t/s
48 GB+ · no limitabout 7× H100 SXM 80 GB	DeepSeek V4.1 Flash 1017Q4_K_M · needs 458 GB		n/a	1M	MiMo V2.6 Pro RL133.2

Long-context models that run on 32 GB96

Ranked by Composite Score, models without one after. Best quant that fits with a 16K context; speed is decode tokens per second on the reference GPU.

96 models · 86 run great · 10 run well

Featured only

Score: llmrun Score, 100 = average model. Fit: how well it runs on this hardware, S best to F won't fit. Ordered by Score, recency and Fit. How ranking works

Showing compatibility for NVIDIA GeForce RTX 5090

LLM models compatible with NVIDIA GeForce RTX 5090 — ranked by Score, recency and Fit
Model	Quant	VRAM	Speed	Context	Fit
Mistral Nemo Instruct 240712.2BChatFunctions < 128K context fits Q4_K_M·144.3 t/s	Q4_K_M	11.0 GB34%	144.3 t/s	131K	S96, Runs great
Llama 3.1 8B Instruct8.0BChatFunctions Q4_K_M·219.8 t/s	Q4_K_M	7.9 GB25%	219.8 t/s	131K	S100, Runs great
Muse Glimmer 30B29.8BVision Q4_K_M·63.5 t/s	Q4_K_M	19.6 GB61%	63.5 t/s	131K	A84, Runs well
NVIDIA Nemotron 3.5 Lightning 30B A3B BF1631.6BChatFunctions Q4_K_M·243.8 t/s	Q4_K_M	19.3 GB60%	243.8 t/s	262K	S100, Runs great
GLM 4.7 Flash31.2BChatFunctions Q4_K_M·196.2 t/s	Q4_K_M	19.9 GB62%	196.2 t/s	203K	S100, Runs great
MiniCPM V 4.7 35B A3B35.2BVisionFunctions Q4_K_M·232.5 t/s	Q4_K_M	22.1 GB69%	232.5 t/s	262K	S100, Runs great
LFM2.5 24B A2B24.0BChatFunctions Q4_K_M·285.3 t/s	Q4_K_M	15.0 GB47%	285.3 t/s	131K	S100, Runs great
Ornith 1.5 35B A3B36.0BChatFunctions Q4_K_M·213.3 t/s	Q4_K_M	22.0 GB69%	213.3 t/s	262K	S100, Runs great
Flint39.8BChatFunctions Q4_K_M·249.7 t/s	Q4_K_M	25.0 GB78%	249.7 t/s	262K	S100, Runs great
Ornith 1.0 35B35.1BChatFunctions Q4_K_M·235.5 t/s	Q4_K_M	21.5 GB67%	235.5 t/s	262K	S100, Runs great
Ling 3.0 Tiny7.9BChatFunctions Q4_K_M·471.1 t/s	Q4_K_M	5.5 GB17%	471.1 t/s	131K	S100, Runs great
MiniCPM5 2B2.5BChatFunctions Q4_K_M·613.1 t/s	Q4_K_M	2.5 GB8%	613.1 t/s	131K	S100, Runs great
Hy Memory 30B A3B30.1BChatFunctions Q4_K_M·237.0 t/s	Q4_K_M	19.1 GB60%	237.0 t/s	262K	S100, Runs great
Gemma 4 12B IT12.0BChatFunctions Q4_K_M·141.5 t/s	Q4_K_M	9.0 GB28%	141.5 t/s	262K	S96, Runs great
Qwen3.5 4B4.7BVisionFunctions Q4_K_M·357.3 t/s	Q4_K_M	3.4 GB11%	357.3 t/s	262K	S100, Runs great

How much memory do you need for long context?

On 8 GB the best long-context model is Gemma 4 E4B IT. At 12 GB it is Qwen3.5 9B (score 99). At 16 GB it is Gemma 4 26B A4B IT (score 106). From 24 to 32 GB it is Qwen3.8 27B (score 122). With no memory limit, the best is DeepSeek V4.1 Flash 1017 (score 133). Weights cover just part of the cost. As tokens fill the cache in your window, a complete 128K prompt may require several gigabytes beyond the model itself. The listed sizes allow space for up to a 16K context.

Frequently Asked Questions

What is the best local LLM for long context?

On 24 GB, Qwen3.8 27B is the top pick with a score of 122, running at about 40 t/s on RTX 4090. It fits up to 128K tokens of context there. On 8 GB, the best fit is Gemma 4 E4B IT. With no memory limit, the best is DeepSeek V4.1 Flash 1017. Open a model to see which GPUs and Macs run it.

Does a long context window use more memory?

The key-value cache scales linearly as token count rises, reaching sizes that can match the model's weights around 128K tokens. Techniques like cache quantization or using grouped-query attention can shrink this footprint.

Is a bigger context window always better?

Quality for info buried deep in a long prompt often slips, plus handling a lengthy request eats up time. Stick to the length you actually need instead of maxing it out.