llmrun
Coding

ALE-Bench Leaderboard

ALE-Bench, from Sakana AI and AtCoder, tests long-horizon algorithmic optimisation using problems drawn from AtCoder Heuristic Contests, where there is no single correct answer and solutions are scored on quality. Results are reported as an AtCoder-style performance rating, where higher is better.

Source: epoch41 open models ranked+75 proprietaryData through Sep 2026

Open modelsAll models

All models ranked on ALE-Bench

Proprietary / closed models are shown dimmed — you can't run them locally, but they show where the open field stands.

#ModelScore
1GPT 6 Astra Max · proprietary
2951.3
2GPT 6 Sol Max · proprietary
2462.0
3GPT 5.6 Sol Max · proprietary
2176.9
4Claude Opus 5 (high) · proprietary
2164.6
5Claude Fable 5.1 (high) · proprietary
2143.2
6Claude Fable 5 (high) · proprietary
2041.3
7GPT 5.6 Terra Max · proprietary
1951.4
8GPT 5.5 (xhigh) · proprietary
1943.0
9Claude Sonnet 5.5 (high) · proprietary
1819.1
10GPT 5.6 Luna Max · proprietary
1667.4
11GPT 5.3 Codex (xhigh) · proprietary
1655.2
12GPT 5.4 (Mar 05, 2026, high) · proprietary
1607.0
13GPT 5.5 (medium) · proprietary
1589.4
14GPT 6 Luna (xhigh) · proprietary
1576.9
15Claude Opus 4.8 (high) · proprietary
1563.8
16Kimi K3 · 2779.9B
1524.5
17GPT 5.4 (Mar 05, 2026, medium) · proprietary
1520.7
18Grok 4.6 (xhigh) · proprietary
1508.1
19Claude Sonnet 5 (high) · proprietary
1463.1
20Claude Opus 4.8 None · proprietary
1411.8
21DeepSeek V4 Pro 0813 · 1650.5B
1403.2
22Gemini 3 Flash Preview · proprietary
1367.2
23Claude Sonnet 4.6 (medium) · proprietary
1327.3
24Claude Opus 4.7 · proprietary
1323.0
25GLM 5.3 · 753.3B
1317.4
26Grok 4.5 (high) · proprietary
1309.3
27DeepSeek V4 Flash 0731 · 304.2B
1306.1
28GPT 5.2 Codex · proprietary
1299.9
29GPT 5.2 (Dec 11, 2025, high) · proprietary
1293.5
30Gemini 3.8 Flash (high) · proprietary
1269.7
31GPT 5.2 (Dec 11, 2025, medium) · proprietary
1249.8
32GPT 5.1 Codex · proprietary
1244.9
33GPT 5.1 Codex Max · proprietary
1208.8
34GPT 5.1 (Nov 13, 2025, high) · proprietary
1192.2
35Qwen3.7 Max · proprietary
1189.4
36GPT 5.4 Mini (Mar 17, 2026, high) · proprietary
1188.6
37Gemini 3 Pro Preview · proprietary
1176.8
38GPT 5 (Aug 07, 2025, high) · proprietary
1162.5
39Gemini 3.1 Pro Preview · proprietary
1160.6
40Grok 4.20 · proprietary
1150.3
41GPT 5.5 None · proprietary
1127.6
42Kimi K2.6 · 1026.9B
1092.7
43DeepSeek V4.1 Flash · 763.2B
1092.3
44GPT 5.4 2026 03.05 None · proprietary
1086.0
45GLM 5.2 · 753.3B
1047.0
46Claude Opus 4.5 (Nov 01, 2025, 16K) · proprietary
1025.4
47DeepSeek V4 Pro · 1598.8B
1006.1
48GPT 5.4 Nano (Mar 17, 2026, high) · proprietary
1004.5
49Claude Opus 4.6 · proprietary
996.5
50Inkling · 952.4B
946.0
51Grok 4.3 (unspecified) · proprietary
944.2
52O3 (Apr 16, 2025, high) · proprietary
933.5
53Gemma 4 26B A4B IT · 25.8B
927.2
54Gemma 4 31B IT · 31.3B
925.5
55Gemini 3.5 Flash (high) · proprietary
911.0
56Gemini 3.7 Flash (high) · proprietary
904.3
57MiMo V2.5 Pro · 1023.2B
899.8
58GLM 5.1 · 753.9B
887.1
59Kimi K2.7 Code · 1026.9B
886.2
60O4 Mini (Apr 16, 2025, high) · proprietary
826.2
61Kimi K2.5 · 1026.9B
821.6
62GPT 5 (Aug 07, 2025, minimal) · proprietary
807.6
63DeepSeek R1 0528 · 684.5B
804.1
64GPT 5 Mini (Aug 07, 2025, high) · proprietary
799.8
65Gemini 3.1 Flash Lite · proprietary
797.7
66Claude Sonnet 4.5 (Sep 29, 2025, 32K) · proprietary
796.1
67Mercury 2 · proprietary
785.6
68Gemini 2.5 Pro (32K) · proprietary
785.5
69Mimo v2 Pro · proprietary
785.2
70GLM 5 · 753.9B
765.6
71Gemini 3.5 Flash Lite (high) · proprietary
765.3
72Mistral Medium 2604 · proprietary
764.0
73DeepSeek V3.1 Terminus · 684.5B
745.2
74MiMo v2 Flash · 309.8B
738.0
75GPT 5 Nano (Aug 07, 2025, high) · proprietary
718.7
76Gemini 3.6 Flash (high) · proprietary
715.5
77Step 3.7 Flash · 201.4B
694.1
78DeepSeek V4 Flash · 290.9B
678.2
79Claude Opus 4.1 (Aug 05, 2025, 16K) · proprietary
674.8
80Qwen3.6 Plus · proprietary
670.1
81Gemini 2.5 Flash · proprietary
661.9
82Claude Sonnet 4 (May 14, 2025, 32K) · proprietary
655.4
83Claude Haiku 4.5 (Oct 01, 2025, 32K) · proprietary
653.5
84MiniMax M3 · 427.0B
640.0
85MiniMax M2.1 · 228.7B
623.8
86Qwen3.5 Plus · proprietary
621.9
87MiniMax M2.5 · 228.7B
618.2
88MiniMax M2.7 · 228.7B
599.3
89Kimi K2 Thinking · 1026.4B
597.5
90Grok Code Fast 1 · proprietary
587.7
91GPT OSS 120B · 116.8B
575.6
92GPT OSS 20B · 20.9B
566.0
93GPT 4.1 (Apr 14, 2025) · proprietary
558.1
94MiMo V2.5 · 310.8B
514.0
95Mistral Small 4 119B 2603 · 119.4B
497.6
96Qwen3 Coder 480B A35B Instruct · 480.2B
461.4
97Qwen3 Coder Plus · proprietary
456.5
98Ring 2.6 1T · 1025.4B
432.6
99GLM 4.7 · 358.3B
399.5
100Grok 4.1 Fast Reasoning · proprietary
394.9
101Qwen3 Max (Sep 23, 2025) · proprietary
370.4
102Qwen3.5 27B · 27.8B
349.4
103GLM 4.5 · 358.3B
344.8
104GLM 4.6 · 356.8B
340.8
105Qwen3.6 Flash · proprietary
326.4
106Gemini 2.5 Flash Lite Preview Thinking (Jun 17) · proprietary
325.9
107GLM 5.3 Flash · 321.3B
303.6
108Mercury 2.5 (high) · proprietary
301.6
109Kimi K2 Instruct 0905 · 1026.5B
267.1
110Mistral Large 3 675B Instruct 2512 · 675B
264.7
111Amazon.nova Lite v1:0 · proprietary
236.3
112Qwen3.5 Flash · proprietary
221.8
113NVIDIA Nemotron 3 Super 120B A12B BF16 · 123.6B
213.9
114Mistral Medium 2508 · proprietary
210.2
115Llama 4 Maverick 17B 128E Instruct · 401.6B
173.0
116Codestral 2508 · proprietary
137.8

Score vs model size

Which models give the most quality for their size — the ones worth running locally.

100B1Tmodel size (log scale) →1524.5173.0Kimi K2.6 · 1T · 1092.7DeepSeek V4.1 Flash · 763B · 1092.3GLM 5.2 · 753B · 1047.0DeepSeek V4 Pro · 1.6T · 1006.1Inkling · 952B · 946.0Gemma 4 31B IT · 31B · 925.5MiMo V2.5 Pro · 1T · 899.8GLM 5.1 · 754B · 887.1Kimi K2.7 Code · 1T · 886.2Kimi K2.5 · 1T · 821.6DeepSeek R1 0528 · 685B · 804.1GLM 5 · 754B · 765.6DeepSeek V3.1 Terminus · 685B · 745.2MiMo v2 Flash · 310B · 738.0Step 3.7 Flash · 201B · 694.1DeepSeek V4 Flash · 291B · 678.2MiniMax M3 · 427B · 640.0MiniMax M2.1 · 229B · 623.8MiniMax M2.5 · 229B · 618.2MiniMax M2.7 · 229B · 599.3Kimi K2 Thinking · 1T · 597.5GPT OSS 120B · 117B · 575.6MiMo V2.5 · 311B · 514.0Mistral Small 4 119B 2603 · 119B · 497.6Qwen3 Coder 480B A35B Instruct · 480B · 461.4Ring 2.6 1T · 1T · 432.6GLM 4.7 · 358B · 399.5Qwen3.5 27B · 28B · 349.4GLM 4.5 · 358B · 344.8GLM 4.6 · 357B · 340.8GLM 5.3 Flash · 321B · 303.6Kimi K2 Instruct 0905 · 1T · 267.1Mistral Large 3 675B Instruct 2512 · 675B · 264.7NVIDIA Nemotron 3 Super 120B A12B BF16 · 124B · 213.9Llama 4 Maverick 17B 128E Instruct · 402B · 173.0GPT OSS 20B · 21B · 566.0GPT OSS 20BGemma 4 26B A4B IT · 26B · 927.2Gemma 4 26B A4B ITDeepSeek V4 Flash 0731 · 304B · 1306.1DeepSeek V4 Flash 0731GLM 5.3 · 753B · 1317.4GLM 5.3DeepSeek V4 Pro 0813 · 1.7T · 1403.2DeepSeek V4 Pro 0813Kimi K3 · 2.8T · 1524.5Kimi K3
Each dot is a model. Up = higher score, left = smaller (easier to run locally). The dashed line marks the efficiency frontier — the best score you can get at each size or smaller.
  • GPT OSS 20B, 21B, score 566.0 — on the efficiency frontier (best score at its size or smaller).
  • Gemma 4 26B A4B IT, 26B, score 927.2 — on the efficiency frontier (best score at its size or smaller).
  • DeepSeek V4 Flash 0731, 304B, score 1306.1 — on the efficiency frontier (best score at its size or smaller).
  • GLM 5.3, 753B, score 1317.4 — on the efficiency frontier (best score at its size or smaller).
  • DeepSeek V4 Pro 0813, 1.7T, score 1403.2 — on the efficiency frontier (best score at its size or smaller).
  • Kimi K3, 2.8T, score 1524.5 — on the efficiency frontier (best score at its size or smaller).

ALE-Bench: frequently asked questions

What is the best open LLM on ALE-Bench?
Kimi K3 is the top open model on ALE-Bench, scoring 1524.5. Among all models tested — including proprietary ones — it ranks #16. The top model overall is GPT 6 Astra Max (OpenAI) at 2951.3.
What's the best ALE-Bench model you can run on a 24 GB GPU?
Gemma 4 26B A4B IT is the highest-scoring open model that fits in 24 GB at 4-bit quantization (about 14 GB), scoring 927.2 on ALE-Bench.
What's the best ALE-Bench model you can run on a 12 GB GPU?
GPT OSS 20B is the highest-scoring open model that fits in 12 GB at 4-bit quantization (about 12 GB), scoring 566.0 on ALE-Bench.
Can open models match proprietary models on ALE-Bench?
Not quite on ALE-Bench: the strongest proprietary model (GPT 6 Astra Max) scores 2951.3, ahead of the best open model (Kimi K3) at 1524.5 — but you can run the open one yourself.

Scores aggregated from epoch. llmrun does not run this benchmark — see the source for methodology, or the about benchmarks for what it measures.