llmrun

Best local LLMs for long context

Run open models featuring a context window of at least 128K tokens to handle large documents and full codebases easily. We rank them using the llmrun Composite Score, highlighting our top recommendation for every memory capacity.

Best for long context on

8 GB12 GB16 GB24 GB32 GB48 GB+

Gemma 4 E4B IT

Google · Q4_K_S · 5.3 GB · up to 128K context · on RTX 5060

58.0 t/s
A
How to run →

All 104 long-context models for 8 GB ↓

Also worth a look

Ling 3.0 Tiny

Q3_K_L · 4.8 GB · 243.0 t/s · 128K ctx

Best long-context model by memory

Highest llmrun Composite Score that fits each size with at least 128K context and 10 t/s, at a Q3 or better quant. Speed is decode tokens per second on that size's reference GPU. The last row has no memory limit: the best model whatever it needs, on the GPU plan shown. Full ranking

Long-context models that run on 8 GB104

Ranked by Composite Score, models without one after. Best quant that fits with a 16K context; speed is decode tokens per second on the reference GPU.

104 models · 62 run great · 24 run well

Featured only

Score: llmrun Score, 100 = average model. Fit: how well it runs on this hardware, S best to F won't fit. Ordered by Score, recency and Fit. How ranking works

Showing compatibility for NVIDIA GeForce RTX 5060

LLM models compatible with NVIDIA GeForce RTX 5060 — ranked by Score, recency and Fit
ModelVRAMScoreFit
AI21 Jamba Reasoning 3B
Q4_K_M·129.4 t/s
2.3 GBS95, Runs great
GLM 4.6V Flash
< 128K context fits
Q4_K_M·44.4 t/s
7.2 GBA65, Runs well
Qwen3 4B Instruct 2507
< 128K context fits
Q4_K_M·100.4 t/s
4.2 GBS91, Runs great
Ilama 3.2 1B
BF16·102.5 t/s
3.3 GBS92, Runs great
ERNIE 4.5 0.3B PT
Q4_K_M·539.3 t/s
0.7 GBS100, Runs great
Qwen3.5 9B Base
< 128K context fits
Q4_K_M·45.8 t/s
6.6 GBA74, Runs well
Qwen3 VL 2B Instruct
< 128K context fits
Q4_K_M·160.9 t/s
3.5 GBS97, Runs great
Granite 4.0 1B
< 128K context fits
Q4_K_M·200.8 t/s
2.6 GBS100, Runs great
K2 Horizon 7B
< 128K context fits
Q4_K_S·51.4 t/s
7.8 GBB54, Decent
Chandra
< 128K context fits
Q4_K_M·49.7 t/s
8.0 GBC44, Usable
Dream V0 Instruct 7B
< 128K context fits
Q4_K_M·58.4 t/s
5.8 GBA83, Runs well
Phi 4 Mini Flash Reasoning
< 128K context fits
Q4_K_M·98.7 t/s
5.3 GBS91, Runs great
NVIDIA Nemotron 3 Nano 4B BF16
Q4_K_M·100.4 t/s
2.9 GBS91, Runs great
ReaderLM v2
Q4_K_M·227.5 t/s
1.7 GBS100, Runs great
Granite Switch 4.1 8B
< 128K context fits
Q3_K_L·56.5 t/s
7.6 GBB60, Decent

How much memory do you need for long context?

On 8 GB the best long-context model is Gemma 4 E4B IT. At 12 GB it is Qwen3.5 9B (score 99). At 16 GB it is Gemma 4 26B A4B IT (score 106). From 24 to 32 GB it is Qwen3.8 27B (score 122). With no memory limit, the best is DeepSeek V4.1 Flash 1017 (score 133). Weights cover just part of the cost. As tokens fill the cache in your window, a complete 128K prompt may require several gigabytes beyond the model itself. The listed sizes allow space for up to a 16K context.

Frequently Asked Questions

What is the best local LLM for long context?

On 24 GB, Qwen3.8 27B is the top pick with a score of 122, running at about 40 t/s on RTX 4090. It fits up to 128K tokens of context there. On 8 GB, the best fit is Gemma 4 E4B IT. With no memory limit, the best is DeepSeek V4.1 Flash 1017. Open a model to see which GPUs and Macs run it.

Does a long context window use more memory?

The key-value cache scales linearly as token count rises, reaching sizes that can match the model's weights around 128K tokens. Techniques like cache quantization or using grouped-query attention can shrink this footprint.

Is a bigger context window always better?

Quality for info buried deep in a long prompt often slips, plus handling a lengthy request eats up time. Stick to the length you actually need instead of maxing it out.