llmrun
Coding

Aider Polyglot Leaderboard

The Aider Polyglot benchmark measures real-world coding across several programming languages: the model edits code to solve Exercism exercises, and is scored on whether the final solution actually runs and passes the tests.

Source: epoch18 open models ranked+51 proprietaryData through Dec 2025

Open modelsAll models

All models ranked on Aider Polyglot

Proprietary / closed models are shown dimmed — you can't run them locally, but they show where the open field stands.

#ModelScoreVRAM @ Q4
1GPT 5 (Aug 07, 2025, high) · proprietary
88.0%
API only
2GPT 5 (Aug 07, 2025, medium) · proprietary
86.7%
API only
3O3 Pro (Jun 10, 2025, high) · proprietary
84.9%
API only
4Gemini 2.5 Pro Preview (Jun 05, 32K) · proprietary
83.1%
API only
5GPT 5 (Aug 07, 2025, low) · proprietary
81.3%
API only
6O3 (Apr 16, 2025, high) · proprietary
81.3%
API only
7Grok 4 (Jul 09, high) · proprietary
79.6%
API only
8Grok 4 (Jul 09) · proprietary
79.6%
API only
9Gemini 2.5 Pro Preview (Jun 05) · proprietary
79.1%
API only
10Gemini 2.5 Pro Preview (May 06) · proprietary
76.9%
API only
11O3 (Apr 16, 2025, medium) · proprietary
76.9%
API only
12O3 (Apr 16, 2025, unspecified) · proprietary
76.9%
API only
13DeepSeek Reasoner · proprietary
74.2%
API only
14DeepSeek V3.2 Exp · 685.4B
74.2%
~415 GB
15Gemini 2.5 Pro Exp (Mar 25) · proprietary
72.9%
API only
16Gemini 2.5 Pro Preview (Mar 25) · proprietary
72.9%
API only
17Claude Opus 4 (May 14, 2025, 32K) · proprietary
72.0%
API only
18O4 Mini (Apr 16, 2025, high) · proprietary
72.0%
API only
19DeepSeek R1 0528 · 684.5B
71.4%
~415 GB
20Claude Opus 4 (May 14, 2025) · proprietary
70.7%
API only
21DeepSeek Chat · proprietary
70.2%
API only
22Claude 3.7 Sonnet (Feb 19, 2025, 32K) · proprietary
64.9%
API only
23O1 (Dec 17, 2024, high) · proprietary
61.7%
API only
24Claude Sonnet 4 (May 14, 2025, 32K) · proprietary
61.3%
API only
25Claude 3.7 Sonnet (Feb 19, 2025) · proprietary
60.4%
API only
26O3 Mini (Jan 31, 2025, high) · proprietary
60.4%
API only
27Qwen3 235B A22B · 235.1B
59.6%
~142 GB
28Qwen3 235B A22B Instruct 2507 · 235.1B
59.6%
~142 GB
29Kimi K2 Instruct · 1026.4B
59.1%
~620 GB
30Kimi K2 Instruct 0905 · 1026.5B
59.1%
~620 GB
31DeepSeek R1 · 684.5B
56.9%
~415 GB
32Claude Sonnet 4 (May 14, 2025) · proprietary
56.4%
API only
33DeepSeek v3 0324 · 684.5B
55.1%
~415 GB
34Gemini 2.5 Flash Preview (May 20, 23K) · proprietary
55.1%
API only
35O3 Mini (Jan 31, 2025, medium) · proprietary
53.8%
API only
36Grok 3 Beta · proprietary
53.3%
API only
37GPT 4.1 (Apr 14, 2025) · proprietary
52.4%
API only
38Claude 3.5 Sonnet (Oct 22, 2024) · proprietary
51.6%
API only
39Grok 3 Mini Beta (high) · proprietary
49.3%
API only
40DeepSeek v3 · 684.5B
48.4%
~415 GB
41Gemini 2.5 Flash Preview (Apr 17) · proprietary
47.1%
API only
42ChatGPT 4o (Mar 27, 2025) · proprietary
45.3%
API only
43GPT 4.5 Preview (Feb 27, 2025) · proprietary
44.9%
API only
44Gemini 2.5 Flash Preview (May 20) · proprietary
44.0%
API only
45GPT OSS 120B · 116.8B
41.8%
~71 GB
46Qwen3 32B · 32.8B
40.0%
~20 GB
47Gemini Exp (Dec 06) · proprietary
38.2%
API only
48Gemini 2.0 Pro Exp (Feb 05) · proprietary
35.6%
API only
49Grok 3 Mini Beta (low) · proprietary
34.7%
API only
50O1 Mini (Sep 12, 2024, unspecified) · proprietary
32.9%
API only
51GPT 4.1 Mini (Apr 14, 2025) · proprietary
32.4%
API only
52Claude 3.5 Haiku (Oct 22, 2024) · proprietary
28.0%
API only
53ChatGPT 4o (Jan 29, 2025) · proprietary
27.1%
API only
54GPT 4o (Aug 06, 2024) · proprietary
23.1%
API only
55Gemini 2.0 Flash Exp · proprietary
22.2%
API only
56Qwen Max (Jan 25, 2025) · proprietary
21.8%
API only
57QwQ 32B · 32.8B
20.9%
~21 GB
58Gemini 2.0 Flash Thinking Exp (Jan 21) · proprietary
18.2%
API only
59GPT 4o (Nov 20, 2024) · proprietary
18.2%
API only
60DeepSeek V2.5 · 235.7B
17.8%
~144 GB
61Qwen2.5 Coder 32B Instruct · 32.8B
16.4%
~21 GB
62Llama 4 Maverick 17B 128E Instruct · 401.6B
15.6%
~265 GB
63Yi Lightning · proprietary
12.9%
API only
64C4ai Command A 03 2025 · 111.1B
12.0%
~73 GB
65Codestral 2501 · proprietary
11.1%
API only
66Openhands Lm 32B v0.1 · 32.8B
10.2%
~21 GB
67GPT 4.1 Nano (Apr 14, 2025) · proprietary
8.9%
API only
68Gemma 3 27B IT · 27.4B
4.9%
~18 GB
69GPT 4o Mini (Jul 18, 2024) · proprietary
3.6%
API only

Score vs model size

Which models give the most quality for their size — the ones worth running locally.

100B1Tmodel size (log scale) →74.2%4.9%Qwen3 235B A22B Instruct 2507 · 235B · 59.6%Kimi K2 Instruct · 1T · 59.1%Kimi K2 Instruct 0905 · 1T · 59.1%DeepSeek R1 · 684B · 56.9%DeepSeek v3 0324 · 685B · 55.1%DeepSeek v3 · 685B · 48.4%QwQ 32B · 33B · 20.9%DeepSeek V2.5 · 236B · 17.8%Qwen2.5 Coder 32B Instruct · 33B · 16.4%Llama 4 Maverick 17B 128E Instruct · 402B · 15.6%C4ai Command A 03 2025 · 111B · 12.0%Openhands Lm 32B v0.1 · 33B · 10.2%Gemma 3 27B IT · 27B · 4.9%Gemma 3 27B ITQwen3 32B · 33B · 40.0%Qwen3 32BGPT OSS 120B · 117B · 41.8%GPT OSS 120BQwen3 235B A22B · 235B · 59.6%Qwen3 235B A22BDeepSeek R1 0528 · 685B · 71.4%DeepSeek R1 0528DeepSeek V3.2 Exp · 685B · 74.2%DeepSeek V3.2 Exp
Each dot is a model. Up = higher score, left = smaller (easier to run locally). The dashed line marks the efficiency frontier — the best score you can get at each size or smaller.
  • Gemma 3 27B IT, 27B, score 4.9% — on the efficiency frontier (best score at its size or smaller).
  • Qwen3 32B, 33B, score 40.0% — on the efficiency frontier (best score at its size or smaller).
  • GPT OSS 120B, 117B, score 41.8% — on the efficiency frontier (best score at its size or smaller).
  • Qwen3 235B A22B, 235B, score 59.6% — on the efficiency frontier (best score at its size or smaller).
  • DeepSeek R1 0528, 685B, score 71.4% — on the efficiency frontier (best score at its size or smaller).
  • DeepSeek V3.2 Exp, 685B, score 74.2% — on the efficiency frontier (best score at its size or smaller).

Aider Polyglot: frequently asked questions

What is the best open LLM on Aider Polyglot?
DeepSeek V3.2 Exp is the top open model on Aider Polyglot, scoring 74.2%. Among all models tested — including proprietary ones — it ranks #13. The top model overall is GPT 5 (Aug 07, 2025, high) (OpenAI) at 88.0%.
What's the best Aider Polyglot model you can run on a 24 GB GPU?
Qwen3 32B is the highest-scoring open model that fits in 24 GB at 4-bit quantization (~20 GB), scoring 40.0% on Aider Polyglot.
Can open models match proprietary models on Aider Polyglot?
Not quite on Aider Polyglot: the strongest proprietary model (GPT 5 (Aug 07, 2025, high)) scores 88.0%, ahead of the best open model (DeepSeek V3.2 Exp) at 74.2% — but you can run the open one yourself.

Scores aggregated from epoch. llmrun does not run this benchmark — see the source for methodology, or the about benchmarks for what it measures.