llmrun
Reasoning

ARC-AGI-2 Leaderboard

ARC-AGI-2 is the second-generation ARC Prize puzzle set, rebuilt so that tasks solved by brute-force search or memorization were removed and every problem still requires inferring a novel rule from a handful of examples. It is markedly harder than the original ARC-AGI, and most models — open and closed alike — score near zero.

Source: epoch22 open models ranked+214 proprietaryData through Oct 2026

Open modelsAll models

All models ranked on ARC-AGI-2

Proprietary / closed models are shown dimmed — you can't run them locally, but they show where the open field stands.

#ModelScoreVRAM @ Q4
1GPT 6 Astra Max · proprietary
95.0%
API only
2GPT 6.1 Sol Max · proprietary
94.2%
API only
3Claude Opus 5.5 (high) · proprietary
93.3%
API only
4GPT 6 Astra (xhigh) · proprietary
93.3%
API only
5Claude Opus 5.5 (xhigh) · proprietary
92.5%
API only
6GPT 5.6 Sol Max · proprietary
92.5%
API only
7GPT 6 Astra (high) · proprietary
92.1%
API only
8GPT 6 Astra (medium) · proprietary
92.1%
API only
9Claude Opus 5.5 Max · proprietary
91.7%
API only
10GPT 6.1 Sol (high) · proprietary
91.7%
API only
11GPT 6.1 Sol (xhigh) · proprietary
91.7%
API only
12Gemini 4 Argon (high) · proprietary
91.3%
API only
13Claude Opus 5 Max · proprietary
90.4%
API only
14Claude Fable 5.1 (xhigh) · proprietary
90.0%
API only
15Claude Fable 5.1 Max · proprietary
90.0%
API only
16GPT 5.6 Sol (xhigh) · proprietary
90.0%
API only
17GPT 6 Sol Max · proprietary
89.6%
API only
18Claude Fable 5 Max · proprietary
89.2%
API only
19Gemini 3.8 Flash (high) · proprietary
89.2%
API only
20Claude Fable 5.1 (high) · proprietary
88.8%
API only
21Claude Fable 5 (xhigh) · proprietary
88.3%
API only
22Claude Opus 5 (high) · proprietary
88.3%
API only
23Claude Fable 5 (high) · proprietary
87.5%
API only
24Claude Opus 5.5 (medium) · proprietary
87.5%
API only
25Gemini 4 Argon (medium) · proprietary
87.1%
API only
26GPT 6.1 Sol (medium) · proprietary
86.7%
API only
27Claude Fable 5.1 (medium) · proprietary
86.3%
API only
28GPT 5.6 Sol (high) · proprietary
85.4%
API only
29GPT 6 Astra (low) · proprietary
85.4%
API only
30GPT 5.5 (xhigh) · proprietary
85.0%
API only
31Gemini 3 Deep Think Preview · proprietary
84.6%
API only
32Gemini 3.7 Flash (high) · proprietary
84.6%
API only
33GPT 5.5 Pro (high) · proprietary
84.6%
API only
34GPT 5.5 Pro (xhigh) · proprietary
84.2%
API only
35GPT 5.6 Terra Max · proprietary
83.9%
API only
36Claude Sonnet 5.5 (xhigh) · proprietary
83.9%
API only
37GPT 5.4 Pro (Mar 05, 2026, xhigh) · proprietary
83.3%
API only
38GPT 5.5 (high) · proprietary
83.3%
API only
39Gemini 3.8 Flash (medium) · proprietary
82.9%
API only
40Claude Fable 5 (medium) · proprietary
82.5%
API only
41Claude Fable 5.1 (low) · proprietary
78.3%
API only
42DeepSeek V4.1 Flash 1017 · 763.2B
78.1%
~458 GB
43GPT 6 Sol (xhigh) · proprietary
78.1%
API only
44Gemini 3.8 Flash (low) · proprietary
77.5%
API only
45Gemini 3.1 Pro Preview · proprietary
77.1%
API only
46Claude Fable 5 (low) · proprietary
76.8%
API only
47GPT 6.1 Sol (low) · proprietary
76.7%
API only
48Claude Opus 4.7 Max · proprietary
75.8%
API only
49GPT 5.6 Terra (xhigh) · proprietary
74.2%
API only
50GPT 5.4 (Mar 05, 2026, xhigh) · proprietary
74.0%
API only
51DeepSeek V4.1 Flash · 763.2B
72.9%
~458 GB
52Claude Opus 4.8 (high) · proprietary
72.1%
API only
53Gemini 3.5 Flash (high) · proprietary
72.1%
API only
54Claude Opus 4.8 (medium) · proprietary
71.7%
API only
55GPT 5.5 (medium) · proprietary
70.4%
API only
56Claude Opus 5.5 (low) · proprietary
70.1%
API only
57Claude Opus 4.6 (120K) · proprietary
69.2%
API only
58GPT 6 Sol (high) · proprietary
68.9%
API only
59Claude Opus 4.7 (high) · proprietary
68.3%
API only
60Claude Opus 4.7 (medium) · proprietary
67.5%
API only
61GPT 5.4 (Mar 05, 2026, high) · proprietary
67.5%
API only
62Claude Sonnet 5.5 (high) · proprietary
67.4%
API only
63GPT 5.6 Sol (medium) · proprietary
67.1%
API only
64GPT 5.6 Terra (high) · proprietary
67.1%
API only
65Grok 4.6 (xhigh) · proprietary
67.1%
API only
66GLM 5.3 Flash · 321.3B
65.8%
~195 GB
67Grok 4.20 · proprietary
65.1%
API only
68Grok 4.6 (high) · proprietary
65.1%
API only
69Gemini 3.7 Flash (medium) · proprietary
63.7%
API only
70Claude Opus 4.8 (low) · proprietary
62.2%
API only
71Claude Opus 4.7 (low) · proprietary
62.1%
API only
72DeepSeek V4 Flash 0731 · 304.2B
61.4%
~183 GB
73Grok 4.7 (xhigh) · proprietary
61.4%
API only
74Claude Haiku 5.5 (xhigh) · proprietary
61.3%
API only
75DeepSeek V4 Pro 0813 · 1650.5B
61.3%
~991 GB
76Grok 4.6 (medium) · proprietary
61.3%
API only
77Claude Sonnet 4.6 (high) · proprietary
60.4%
API only
78Gemini 3.6 Flash (high) · proprietary
60.4%
API only
79Kimi K3 · 2779.9B
60.4%
~1674 GB
80GPT 6 Astra None · proprietary
59.6%
API only
81GPT 5.6 Luna Max · proprietary
59.5%
API only
82GPT 6 Luna Max · proprietary
59.3%
API only
83Grok 4.7 (medium) · proprietary
58.8%
API only
84Claude Sonnet 4.6 Max · proprietary
58.3%
API only
85Grok 4.7 (high) · proprietary
58.3%
API only
86GPT 6 Sol (medium) · proprietary
57.8%
API only
87Muse Spark Open 816B A42B · 822.0B
57.5%
~543 GB
88GPT 5.4 (Mar 05, 2026, medium) · proprietary
55.4%
API only
89GPT 5.2 Pro (Dec 11, 2025, high) · proprietary
54.2%
API only
90Gemini 3.7 Flash (low) · proprietary
52.9%
API only
91GPT 5.2 (Dec 11, 2025, xhigh) · proprietary
52.9%
API only
92Grok 4.5 (high) · proprietary
52.6%
API only
93Grok 4.5 (medium) · proprietary
52.6%
API only
94Gemini 3.6 Flash (medium) · proprietary
50.4%
API only
95GPT 5.6 Luna (xhigh) · proprietary
47.6%
API only
96GPT 5.2 (Dec 11, 2025, high) · proprietary
43.3%
API only
97GPT 5.6 Sol (low) · proprietary
42.5%
API only
98Qwen3.8 27B · 27.8B
42.4%
~17 GB
99Gemini 4 Argon (low) · proprietary
42.1%
API only
100GPT 6 Luna (xhigh) · proprietary
41.9%
API only
101Inkling Small · 266.0B
40.1%
~160 GB
102GPT 5.2 Pro (Dec 11, 2025, medium) · proprietary
38.5%
API only
103Claude Opus 4.5 (Nov 01, 2025, 64K) · proprietary
37.6%
API only
104GPT 5.6 Terra (medium) · proprietary
37.5%
API only
105Inkling · 952.4B
36.5%
~572 GB
106Gemini 3 Flash Preview · proprietary
33.6%
API only
107GPT 5.5 (low) · proprietary
33.3%
API only
108Grok 4.5 (low) · proprietary
33.1%
API only
109Claude Sonnet 5.5 (medium) · proprietary
32.8%
API only
110GPT 6 Sol (low) · proprietary
31.5%
API only
111GPT 6 Luna (high) · proprietary
31.4%
API only
112Gemini 3 Pro Preview · proprietary
31.1%
API only
113Claude Haiku 5.5 (high) · proprietary
30.7%
API only
114Gemini 3.6 Flash (low) · proprietary
30.4%
API only
115GPT 5.6 Luna (high) · proprietary
29.3%
API only
116GPT 5.4 (Mar 05, 2026, low) · proprietary
29.2%
API only
117Grok 4.6 (low) · proprietary
27.6%
API only
118GPT 5.2 (Dec 11, 2025, medium) · proprietary
26.7%
API only
119Claude Sonnet 5.5 (low) · proprietary
24.0%
API only
120Beam 1.0 501B A23B · 501.3B
23.5%
~301 GB
121Claude Opus 4.5 (Nov 01, 2025, 16K) · proprietary
22.8%
API only
122GLM 5.2 · 753.3B
22.8%
~456 GB
123GPT 5.4 Mini (Mar 17, 2026, xhigh) · proprietary
18.9%
API only
124GPT 5.6 Terra (low) · proprietary
18.8%
API only
125GPT 5 Pro (Oct 06, 2025, unspecified) · proprietary
18.3%
API only
126GPT 6 Luna (medium) · proprietary
18.1%
API only
127GPT 5.1 (Nov 13, 2025, high) · proprietary
17.6%
API only
128Grok 4.7 (low) · proprietary
16.7%
API only
129Grok 4 (Jul 09) · proprietary
16.0%
API only
130Claude Opus 4.5 (Nov 01, 2025, 8K) · proprietary
13.9%
API only
131Claude Sonnet 4.5 (Sep 29, 2025, 32K) · proprietary
13.6%
API only
132GPT 5.4 Mini (Mar 17, 2026, high) · proprietary
13.2%
API only
133Kimi K2.5 · 1026.9B
11.8%
~620 GB
134Gemini 3.5 Flash Lite (high) · proprietary
10.3%
API only
135GPT 5 (Aug 07, 2025, high) · proprietary
9.9%
API only
136GPT 5.2 (Dec 11, 2025, low) · proprietary
9.7%
API only
137Claude Haiku 5.5 (medium) · proprietary
9.4%
API only
138Gemini 3.5 Flash (minimal) · proprietary
8.9%
API only
139Claude Opus 4 (May 14, 2025, 16K) · proprietary
8.6%
API only
140Claude Opus 4.5 (Nov 01, 2025) · proprietary
7.8%
API only
141GPT 5 (Aug 07, 2025, medium) · proprietary
7.5%
API only
142GPT 5.6 Luna (medium) · proprietary
7.4%
API only
143Claude Sonnet 4.5 (Sep 29, 2025, 16K) · proprietary
6.9%
API only
144Claude Sonnet 4.5 (Sep 29, 2025, 8K) · proprietary
6.9%
API only
145GPT 5.1 (Nov 13, 2025, medium) · proprietary
6.5%
API only
146O3 (Apr 16, 2025, high) · proprietary
6.5%
API only
147Tiny Recursion Model · proprietary
6.3%
API only
148O4 Mini (Apr 16, 2025, high) · proprietary
6.1%
API only
149Claude Sonnet 4 (May 14, 2025, 16K) · proprietary
5.9%
API only
150Claude Sonnet 4.5 (Sep 29, 2025, 1K) · proprietary
5.8%
API only
151GPT 5.4 Nano (Mar 17, 2026, xhigh) · proprietary
5.7%
API only
152Gemini 3.5 Flash Lite (medium) · proprietary
5.3%
API only
153Grok 4 Fast · proprietary
5.3%
API only
154GPT 5.6 Luna (low) · proprietary
5.1%
API only
155Gemini 2.5 Pro (32K) · proprietary
4.9%
API only
156GLM 5 · 753.9B
4.9%
~457 GB
157MiniMax M2.5 · 228.7B
4.9%
~138 GB
158O3 Pro (Jun 10, 2025, high) · proprietary
4.9%
API only
159GPT 6 Luna (low) · proprietary
4.6%
API only
160Claude Opus 4 (May 14, 2025, 8K) · proprietary
4.5%
API only
161GPT 5 Mini (Aug 07, 2025, high) · proprietary
4.4%
API only
162GPT 5.4 Mini (Mar 17, 2026, medium) · proprietary
4.4%
API only
163Claude Haiku 4.5 (Oct 01, 2025, 32K) · proprietary
4.0%
API only
164DeepSeek V3.2 · 685.4B
4.0%
~415 GB
165Gemini 2.5 Pro (16K) · proprietary
4.0%
API only
166GPT 5 Mini (Aug 07, 2025, medium) · proprietary
4.0%
API only
167Claude Sonnet 4.5 (Sep 29, 2025) · proprietary
3.8%
API only
168GPT 5.4 Nano (Mar 17, 2026, high) · proprietary
3.6%
API only
169O3 Mini (Jan 31, 2025, high) · proprietary
3.0%
API only
170O3 (Apr 16, 2025, medium) · proprietary
3.0%
API only
171Gemini 2.5 Pro (8K) · proprietary
2.9%
API only
172Claude Haiku 4.5 (Oct 01, 2025, 16K) · proprietary
2.8%
API only
173Claude Haiku 5.5 (low) · proprietary
2.8%
API only
174Gemini 3.6 Flash (minimal) · proprietary
2.6%
API only
175GPT 5 Nano (Aug 07, 2025, high) · proprietary
2.6%
API only
176Gemini 2.5 Flash Preview (May 20, 23K) · proprietary
2.5%
API only
177O4 Mini (Apr 16, 2025, medium) · proprietary
2.4%
API only
178Gemini 2.5 Flash Preview (May 20, 1K) · proprietary
2.2%
API only
179Claude Sonnet 4 (May 14, 2025, 8K) · proprietary
2.1%
API only
180Gemini 2.5 Flash Preview (May 20, 8K) · proprietary
2.1%
API only
181O3 Mini (Jan 31, 2025, medium) · proprietary
2.1%
API only
182O3 Pro (Jun 10, 2025, low) · proprietary
2.1%
API only
183O3 (Apr 16, 2025, low) · proprietary
2.0%
API only
184Gemini 2.5 Flash Preview (May 20, 16K) · proprietary
2.0%
API only
185GPT 5 (Aug 07, 2025, low) · proprietary
1.9%
API only
186GPT 5.1 (Nov 13, 2025, low) · proprietary
1.9%
API only
187GPT 5.4 Nano (Mar 17, 2026, medium) · proprietary
1.9%
API only
188O3 Pro (Jun 10, 2025, medium) · proprietary
1.9%
API only
189Gemini 2.5 Flash Preview (May 20) · proprietary
1.7%
API only
190Claude Haiku 4.5 (Oct 01, 2025, 8K) · proprietary
1.7%
API only
191GPT 5 Mini (Aug 07, 2025, minimal) · proprietary
1.7%
API only
192GPT 6 Sol None · proprietary
1.7%
API only
193O4 Mini (Apr 16, 2025, low) · proprietary
1.7%
API only
194Gemini 3.5 Flash Lite (low) · proprietary
1.5%
API only
195GPT 5.4 Nano (Mar 17, 2026, low) · proprietary
1.5%
API only
196DeepSeek R1 · 684.5B
1.3%
~415 GB
197Gemini 2.0 Flash 001 · proprietary
1.3%
API only
198Claude Opus 4 (May 14, 2025) · proprietary
1.3%
API only
199Claude Sonnet 4 (May 14, 2025) · proprietary
1.3%
API only
200Codex Mini (May 16, 2025) · proprietary
1.3%
API only
201Claude Haiku 4.5 (Oct 01, 2025, 1K) · proprietary
1.3%
API only
202Claude Haiku 4.5 (Oct 01, 2025) · proprietary
1.3%
API only
203Qwen3 235B A22B Instruct 2507 · 235.1B
1.3%
~142 GB
204DeepSeek R1 0528 · 684.5B
1.1%
~415 GB
205GPT 5.4 Mini (Mar 17, 2026, low) · proprietary
1.1%
API only
206Claude 3.7 Sonnet (Feb 19, 2025, 8K) · proprietary
0.9%
API only
207GPT 5 Nano (Aug 07, 2025, medium) · proprietary
0.9%
API only
208Claude Sonnet 4 (May 14, 2025, 1K) · proprietary
0.9%
API only
209Gemini 3.5 Flash Lite (minimal) · proprietary
0.8%
API only
210GPT 5 Mini (Aug 07, 2025, low) · proprietary
0.8%
API only
211GPT 5.2 (Dec 11, 2025, unspecified) · proprietary
0.8%
API only
212O1 Mini (Sep 12, 2024, unspecified) · proprietary
0.8%
API only
213Gemini 1.5 Pro 002 · proprietary
0.8%
API only
214GPT 4.5 Preview (Feb 27, 2025) · proprietary
0.8%
API only
215Claude 3.7 Sonnet (Feb 19, 2025, 16K) · proprietary
0.7%
API only
216GPT 4.1 (Apr 14, 2025) · proprietary
0.4%
API only
217GPT 5.1 2025 11.13 None · proprietary
0.4%
API only
218Grok 3 Mini (low) · proprietary
0.4%
API only
219Claude 3.7 Sonnet (Feb 19, 2025, 1K) · proprietary
0.4%
API only
220Claude 3.7 Sonnet (Feb 19, 2025) · proprietary
0.0%
API only
221Claude Opus 4 (May 14, 2025, 1K) · proprietary
0.0%
API only
222Gemini 2.5 Pro Preview (Jun 05, 1K) · proprietary
0.0%
API only
223GPT 4.1 Mini (Apr 14, 2025) · proprietary
0.0%
API only
224GPT 4.1 Nano (Apr 14, 2025) · proprietary
0.0%
API only
225GPT 4o (Nov 20, 2024) · proprietary
0.0%
API only
226GPT 4o Mini (Jul 18, 2024) · proprietary
0.0%
API only
227GPT 5 (Aug 07, 2025, minimal) · proprietary
0.0%
API only
228GPT 5 Nano (Aug 07, 2025, low) · proprietary
0.0%
API only
229GPT 5 Nano (Aug 07, 2025, minimal) · proprietary
0.0%
API only
230GPT 6 Luna None · proprietary
0.0%
API only
231Grok 3 · proprietary
0.0%
API only
232Llama 4 Maverick 17B 128E Instruct · 401.6B
0.0%
~265 GB
233Llama 4 Scout 17B 16E Instruct · 108.6B
0.0%
~72 GB
234Magistral Medium 2506 · proprietary
0.0%
API only
235Magistral Small 2506 · 23.6B
0.0%
~15 GB
236O3 Mini (Jan 31, 2025, low) · proprietary
0.0%
API only

Score vs model size

Which models give the most quality for their size — the ones worth running locally.

100B1Tmodel size (log scale) →78.1%0.0%DeepSeek V4.1 Flash · 763B · 72.9%DeepSeek V4 Pro 0813 · 1.7T · 61.3%Kimi K3 · 2.8T · 60.4%Muse Spark Open 816B A42B · 822B · 57.5%Inkling Small · 266B · 40.1%Inkling · 952B · 36.5%Beam 1.0 501B A23B · 501B · 23.5%GLM 5.2 · 753B · 22.8%Kimi K2.5 · 1T · 11.8%MiniMax M2.5 · 229B · 4.9%GLM 5 · 754B · 4.9%DeepSeek V3.2 · 685B · 4.0%DeepSeek R1 · 684B · 1.3%Qwen3 235B A22B Instruct 2507 · 235B · 1.3%DeepSeek R1 0528 · 685B · 1.1%Llama 4 Maverick 17B 128E Instruct · 402B · 0.0%Llama 4 Scout 17B 16E Instruct · 109B · 0.0%Magistral Small 2506 · 24B · 0.0%Magistral Small 2506Qwen3.8 27B · 28B · 42.4%Qwen3.8 27BDeepSeek V4 Flash 0731 · 304B · 61.4%DeepSeek V4 Flash 0731GLM 5.3 Flash · 321B · 65.8%GLM 5.3 FlashDeepSeek V4.1 Flash 1017 · 763B · 78.1%DeepSeek V4.1 Flash 1…
Each dot is a model. Up = higher score, left = smaller (easier to run locally). The dashed line marks the efficiency frontier — the best score you can get at each size or smaller.
  • Magistral Small 2506, 24B, score 0.0% — on the efficiency frontier (best score at its size or smaller).
  • Qwen3.8 27B, 28B, score 42.4% — on the efficiency frontier (best score at its size or smaller).
  • DeepSeek V4 Flash 0731, 304B, score 61.4% — on the efficiency frontier (best score at its size or smaller).
  • GLM 5.3 Flash, 321B, score 65.8% — on the efficiency frontier (best score at its size or smaller).
  • DeepSeek V4.1 Flash 1017, 763B, score 78.1% — on the efficiency frontier (best score at its size or smaller).

ARC-AGI-2: frequently asked questions

What is the best open LLM on ARC-AGI-2?
DeepSeek V4.1 Flash 1017 is the top open model on ARC-AGI-2, scoring 78.1%. Among all models tested — including proprietary ones — it ranks #42. The top model overall is GPT 6 Astra Max (OpenAI) at 95.0%.
What's the best ARC-AGI-2 model you can run on a 24 GB GPU?
Qwen3.8 27B is the highest-scoring open model that fits in 24 GB at 4-bit quantization (~17 GB), scoring 42.4% on ARC-AGI-2.
Can open models match proprietary models on ARC-AGI-2?
Not quite on ARC-AGI-2: the strongest proprietary model (GPT 6 Astra Max) scores 95.0%, ahead of the best open model (DeepSeek V4.1 Flash 1017) at 78.1% — but you can run the open one yourself.

Scores aggregated from epoch. llmrun does not run this benchmark — see the source for methodology, or the about benchmarks for what it measures.