llmrun
Coding

ALE-Bench Leaderboard

ALE-Bench, from Sakana AI and AtCoder, tests long-horizon algorithmic optimisation using problems drawn from AtCoder Heuristic Contests, where there is no single correct answer and solutions are scored on quality. Results are reported as an AtCoder-style performance rating, where higher is better.

Source: epoch51 open models ranked+87 proprietaryData through Nov 2026

All models ranked on ALE-Bench

Proprietary / closed models are shown dimmed — you can't run them locally, but they show where the open field stands.

#ModelScoreVRAM @ Q4
1GPT 6 Astra Max · proprietary
2951.3
API only
2GPT 6.1 Sol Max · proprietary
2743.8
API only
3GPT 6.1 Terra Max · proprietary
2608.8
API only
4Claude Fable 5.2 (high) · proprietary
2509.8
API only
5GPT 6 Sol Max · proprietary
2462.0
API only
6GPT 5.6 Sol Max · proprietary
2176.9
API only
7Claude Opus 5 (high) · proprietary
2164.6
API only
8Claude Opus 5.5 (high) · proprietary
2147.4
API only
9Claude Fable 5.1 (high) · proprietary
2143.2
API only
10Gemini 4 Argon (high) · proprietary
2100.1
API only
11Claude Fable 5 (high) · proprietary
2041.3
API only
12GPT 5.6 Terra Max · proprietary
1951.4
API only
13GPT 5.5 (xhigh) · proprietary
1943.0
API only
14Kimi K3.1 Pro · 2780.1B
1932.8
~1674 GB
15Claude Sonnet 5.5 (high) · proprietary
1819.1
API only
16Gemini 4 Argon (low) · proprietary
1705.4
API only
17Muse Spark 1.3 (xhigh) · proprietary
1688.3
API only
18Qwen3.8 Max (Sep 02, xhigh) · proprietary
1679.1
API only
19GPT 5.6 Luna Max · proprietary
1667.4
API only
20GPT 5.3 Codex (xhigh) · proprietary
1655.2
API only
21DeepSeek V4.1 Flash 1017 · 763.2B
1622.0
~458 GB
22GPT 5.4 (Mar 05, 2026, high) · proprietary
1607.0
API only
23Muse Spark Open 816B A42B · 822.0B
1590.1
~543 GB
24GPT 5.5 (medium) · proprietary
1589.4
API only
25Claude Haiku 5.5 (high) · proprietary
1587.0
API only
26GPT 6 Luna (xhigh) · proprietary
1576.9
API only
27Claude Opus 4.8 (high) · proprietary
1563.8
API only
28Grok 4.7 (xhigh) · proprietary
1545.8
API only
29Kimi K3 · 2779.9B
1524.5
~1674 GB
30GPT 5.4 (Mar 05, 2026, medium) · proprietary
1520.7
API only
31Grok 4.6 (xhigh) · proprietary
1508.1
API only
32Claude Sonnet 5 (high) · proprietary
1463.1
API only
33Claude Opus 4.8 None · proprietary
1411.8
API only
34DeepSeek V4 Pro 0813 · 1650.5B
1403.2
~991 GB
35Gemini 3 Flash Preview · proprietary
1367.2
API only
36Claude Sonnet 4.6 (medium) · proprietary
1327.3
API only
37Claude Opus 4.7 · proprietary
1323.0
API only
38GLM 5.3 · 753.3B
1317.4
~456 GB
39Grok 4.5 (high) · proprietary
1309.3
API only
40DeepSeek V4 Flash 0731 · 304.2B
1306.1
~183 GB
41GPT 5.2 Codex · proprietary
1299.9
API only
42GPT 5.2 (Dec 11, 2025, high) · proprietary
1293.5
API only
43Gemini 3.8 Flash (high) · proprietary
1269.7
API only
44GPT 5.2 (Dec 11, 2025, medium) · proprietary
1249.8
API only
45GPT 5.1 Codex · proprietary
1244.9
API only
46MiMo V2.6 Flash RL · 309B
1244.2
~186 GB
47GPT 5.1 Codex Max (xhigh) · proprietary
1228.3
API only
48Mistral Large 4 1T A52B · 1000B
1213.2
~604 GB
49GPT 5.1 Codex Max · proprietary
1208.8
API only
50GPT 5.1 (Nov 13, 2025, high) · proprietary
1192.2
API only
51Qwen3.7 Max · proprietary
1189.4
API only
52GPT 5.4 Mini (Mar 17, 2026, high) · proprietary
1188.6
API only
53Gemini 3 Pro Preview · proprietary
1176.8
API only
54GPT 5 (Aug 07, 2025, high) · proprietary
1162.5
API only
55Gemini 3.1 Pro Preview · proprietary
1160.6
API only
56MiMo V2.6 Pro RL · 1020B
1158.1
~613 GB
57Grok 4.20 · proprietary
1150.3
API only
58GPT 5.5 None · proprietary
1127.6
API only
59Beam 1.0 501B A23B · 501.3B
1097.5
~301 GB
60Kimi K2.6 · 1026.9B
1092.7
~620 GB
61DeepSeek V4.1 Flash · 763.2B
1092.3
~458 GB
62GPT 5.4 2026 03.05 None · proprietary
1086.0
API only
63Gemini 3.1 Pro Preview (low) · proprietary
1054.8
API only
64GLM 5.2 · 753.3B
1047.0
~456 GB
65Claude Opus 4.5 (Nov 01, 2025, 16K) · proprietary
1025.4
API only
66DeepSeek V4 Pro · 1598.8B
1006.1
~960 GB
67GPT 5.4 Nano (Mar 17, 2026, high) · proprietary
1004.5
API only
68Claude Opus 4.6 · proprietary
996.5
API only
69Gemini 3 Pro Preview (low) · proprietary
988.2
API only
70Inkling · 952.4B
946.0
~572 GB
71Grok 4.3 (unspecified) · proprietary
944.2
API only
72O3 (Apr 16, 2025, high) · proprietary
933.5
API only
73Gemma 4 26B A4B IT · 25.8B
927.2
~16 GB
74Gemma 4 31B IT · 31.3B
925.5
~20 GB
75Gemini 3.5 Flash (high) · proprietary
911.0
API only
76Gemini 3.7 Flash (high) · proprietary
904.3
API only
77MiMo V2.5 Pro · 1023.2B
899.8
~614 GB
78GLM 5.1 · 753.9B
887.1
~457 GB
79Kimi K2.7 Code · 1026.9B
886.2
~620 GB
80O4 Mini (Apr 16, 2025, high) · proprietary
826.2
API only
81Kimi K2.5 · 1026.9B
821.6
~620 GB
82GPT 5 (Aug 07, 2025, minimal) · proprietary
807.6
API only
83DeepSeek R1 0528 · 684.5B
804.1
~415 GB
84GPT 5 Mini (Aug 07, 2025, high) · proprietary
799.8
API only
85Gemini 3.1 Flash Lite · proprietary
797.7
API only
86Claude Sonnet 4.5 (Sep 29, 2025, 32K) · proprietary
796.1
API only
87Mercury 2 · proprietary
785.6
API only
88Gemini 2.5 Pro (32K) · proprietary
785.5
API only
89Mimo v2 Pro · proprietary
785.2
API only
90GLM 5 · 753.9B
765.6
~457 GB
91Gemini 3.5 Flash Lite (high) · proprietary
765.3
API only
92Mistral Medium 3.5 128B · 127.7B
764.0
~78 GB
93DeepSeek V3.1 Terminus · 684.5B
745.2
~415 GB
94MiMo v2 Flash · 309.8B
738.0
~186 GB
95GPT 5 Nano (Aug 07, 2025, high) · proprietary
718.7
API only
96Gemini 3.6 Flash (high) · proprietary
715.5
API only
97Step 3.7 Flash · 201.4B
694.1
~133 GB
98DeepSeek V4 Flash · 290.9B
678.2
~175 GB
99Claude Opus 4.1 (Aug 05, 2025, 16K) · proprietary
674.8
API only
100Qwen3.6 Plus · proprietary
670.1
API only
101Gemini 2.5 Flash · proprietary
661.9
API only
102Claude Sonnet 4 (May 14, 2025, 32K) · proprietary
655.4
API only
103Claude Haiku 4.5 (Oct 01, 2025, 32K) · proprietary
653.5
API only
104MiniMax M3 · 427.0B
640.0
~257 GB
105MiniMax M2.1 · 228.7B
623.8
~138 GB
106Qwen3.5 397B A17B · 403.4B
621.9
~242 GB
107MiniMax M2.5 · 228.7B
618.2
~138 GB
108Qwen3.5 Plus · proprietary
608.9
API only
109MiniMax M2.7 · 228.7B
599.3
~138 GB
110Kimi K2 Thinking · 1026.4B
597.5
~620 GB
111Grok Code Fast 1 · proprietary
587.7
API only
112GPT OSS 120B · 116.8B
575.6
~71 GB
113GPT OSS 20B · 20.9B
566.0
~13 GB
114GPT 4.1 (Apr 14, 2025) · proprietary
558.1
API only
115MiMo V2.5 · 310.8B
514.0
~187 GB
116Mistral Small 4 119B 2603 · 119.4B
497.6
~73 GB
117Qwen3 Coder 480B A35B Instruct · 480.2B
461.4
~289 GB
118Qwen3 Coder Plus · proprietary
456.5
API only
119Ring 2.6 1T · 1025.4B
432.6
~2057 GB
120GLM 4.7 · 358.3B
399.5
~216 GB
121Grok 4.1 Fast Reasoning · proprietary
394.9
API only
122Qwen3 Max (Sep 23, 2025) · proprietary
370.4
API only
123Qwen3.5 27B · 27.8B
349.4
~17 GB
124GLM 4.5 · 358.3B
344.8
~216 GB
125GLM 4.6 · 356.8B
340.8
~215 GB
126Qwen3.6 Flash · proprietary
326.4
API only
127Gemini 2.5 Flash Lite Preview Thinking (Jun 17) · proprietary
325.9
API only
128GLM 5.3 Flash · 321.3B
303.6
~195 GB
129Mercury 2.5 (high) · proprietary
301.6
API only
130Kimi K2 Instruct 0905 · 1026.4B
267.1
~620 GB
131Qwen3.5 Flash · proprietary
265.9
API only
132Mistral Large 3 675B Instruct 2512 · 675B
264.7
~446 GB
133Amazon.nova Lite v1:0 · proprietary
236.3
API only
134Qwen3.5 35B A3B · 36.0B
221.8
~22 GB
135NVIDIA Nemotron 3 Super 120B A12B BF16 · 123.6B
213.9
~75 GB
136Mistral Medium 2508 · proprietary
210.2
API only
137Llama 4 Maverick 17B 128E Instruct · 401.6B
173.0
~265 GB
138Codestral 2508 · proprietary
137.8
API only

Score vs model size

Which models give the most quality for their size — the ones worth running locally.

100B1Tmodel size (log scale) →1932.8173.0Muse Spark Open 816B A42B · 822B · 1590.1Kimi K3 · 2.8T · 1524.5DeepSeek V4 Pro 0813 · 1.7T · 1403.2MiMo V2.6 Flash RL · 309B · 1244.2Mistral Large 4 1T A52B · 1T · 1213.2MiMo V2.6 Pro RL · 1T · 1158.1Beam 1.0 501B A23B · 501B · 1097.5Kimi K2.6 · 1T · 1092.7DeepSeek V4.1 Flash · 763B · 1092.3GLM 5.2 · 753B · 1047.0DeepSeek V4 Pro · 1.6T · 1006.1Inkling · 952B · 946.0Gemma 4 31B IT · 31B · 925.5MiMo V2.5 Pro · 1T · 899.8GLM 5.1 · 754B · 887.1Kimi K2.7 Code · 1T · 886.2Kimi K2.5 · 1T · 821.6DeepSeek R1 0528 · 685B · 804.1GLM 5 · 754B · 765.6Mistral Medium 3.5 128B · 128B · 764.0DeepSeek V3.1 Terminus · 685B · 745.2MiMo v2 Flash · 310B · 738.0Step 3.7 Flash · 201B · 694.1DeepSeek V4 Flash · 291B · 678.2MiniMax M3 · 427B · 640.0MiniMax M2.1 · 229B · 623.8Qwen3.5 397B A17B · 403B · 621.9MiniMax M2.5 · 229B · 618.2MiniMax M2.7 · 229B · 599.3Kimi K2 Thinking · 1T · 597.5GPT OSS 120B · 117B · 575.6MiMo V2.5 · 311B · 514.0Mistral Small 4 119B 2603 · 119B · 497.6Qwen3 Coder 480B A35B Instruct · 480B · 461.4Ring 2.6 1T · 1T · 432.6GLM 4.7 · 358B · 399.5Qwen3.5 27B · 28B · 349.4GLM 4.5 · 358B · 344.8GLM 4.6 · 357B · 340.8GLM 5.3 Flash · 321B · 303.6Kimi K2 Instruct 0905 · 1T · 267.1Mistral Large 3 675B Instruct 2512 · 675B · 264.7Qwen3.5 35B A3B · 36B · 221.8NVIDIA Nemotron 3 Super 120B A12B BF16 · 124B · 213.9Llama 4 Maverick 17B 128E Instruct · 402B · 173.0GPT OSS 20B · 21B · 566.0GPT OSS 20BGemma 4 26B A4B IT · 26B · 927.2Gemma 4 26B A4B ITDeepSeek V4 Flash 0731 · 304B · 1306.1DeepSeek V4 Flash 0731GLM 5.3 · 753B · 1317.4GLM 5.3DeepSeek V4.1 Flash 1017 · 763B · 1622.0DeepSeek V4.1 Flash 1…Kimi K3.1 Pro · 2.8T · 1932.8Kimi K3.1 Pro
Each dot is a model. Up = higher score, left = smaller (easier to run locally). The dashed line marks the efficiency frontier — the best score you can get at each size or smaller.
  • GPT OSS 20B, 21B, score 566.0 — on the efficiency frontier (best score at its size or smaller).
  • Gemma 4 26B A4B IT, 26B, score 927.2 — on the efficiency frontier (best score at its size or smaller).
  • DeepSeek V4 Flash 0731, 304B, score 1306.1 — on the efficiency frontier (best score at its size or smaller).
  • GLM 5.3, 753B, score 1317.4 — on the efficiency frontier (best score at its size or smaller).
  • DeepSeek V4.1 Flash 1017, 763B, score 1622.0 — on the efficiency frontier (best score at its size or smaller).
  • Kimi K3.1 Pro, 2.8T, score 1932.8 — on the efficiency frontier (best score at its size or smaller).

ALE-Bench: frequently asked questions

What is the best open LLM on ALE-Bench?
Kimi K3.1 Pro is the top open model on ALE-Bench, scoring 1932.8. Among all models tested — including proprietary ones — it ranks #14. The top model overall is GPT 6 Astra Max (OpenAI) at 2951.3.
What's the best ALE-Bench model you can run on a 24 GB GPU?
Gemma 4 26B A4B IT is the highest-scoring open model that fits in 24 GB at 4-bit quantization (~16 GB), scoring 927.2 on ALE-Bench.
Can open models match proprietary models on ALE-Bench?
Not quite on ALE-Bench: the strongest proprietary model (GPT 6 Astra Max) scores 2951.3, ahead of the best open model (Kimi K3.1 Pro) at 1932.8 — but you can run the open one yourself.

Scores aggregated from epoch. llmrun does not run this benchmark — see the source for methodology, or the about benchmarks for what it measures.