Knowledge
ARC Challenge Leaderboard
ARC Challenge is a set of grade-school science questions, curated by the Allen Institute for AI (AI2) to exclude ones that simple word-matching or retrieval can answer. It measures science knowledge paired with the reasoning needed to apply it.
Source: epoch56 open models ranked+20 proprietaryData through Dec 2024
All models ranked on ARC Challenge
Proprietary / closed models are shown dimmed — you can't run them locally, but they show where the open field stands.
| # | Model | Score | VRAM @ Q4 |
|---|---|---|---|
| 1 | DeepSeek v3 · 684.5B | 95.3% | ~415 GB |
| 2 | Llama 3.1 405B · 405.9B | 95.3% | ~268 GB |
| 3 | Qwen2.5 72B · 72.7B | 94.5% | ~45 GB |
| 4 | DeepSeek v2 · 235.7B | 92.2% | ~144 GB |
| 5 | Phi 3 Medium 128k Instruct · 14.0B | 91.6% | ~9.1 GB |
| 6 | Phi 3 Small 8k Instruct · 7.4B | 90.7% | ~15 GB |
| 7 | GPT 3.5 Turbo (Nov 06) · proprietary | 87.4% | API only |
| 8 | Mixtral 8x7B v0.1 · 46.7B | 87.3% | ~29 GB |
| 9 | Claude Instant 1.2 · proprietary | 86.3% | API only |
| 10 | StableBeluga2 · 70B | 86.1% | ~43 GB |
| 11 | Claude Instant 1.1 · proprietary | 85.7% | API only |
| 12 | PaLM 540B · proprietary | 85.2% | API only |
| 13 | Text Davinci 002 · proprietary | 85.2% | API only |
| 14 | Phi 3 Mini 4k Instruct · 3.8B | 84.9% | ~3.4 GB |
| 15 | Qwen 14B · 14.2B | 84.4% | ~9.3 GB |
| 16 | Meta Llama 3 8B Instruct · 8.0B | 82.8% | ~5.3 GB |
| 17 | Internlm 20B · 20B | 81.7% | ~43 GB |
| 18 | Mistral 7B v0.1 · 7.2B | 78.6% | ~4.9 GB |
| 19 | Gemma 7B · 8.5B | 78.3% | ~19 GB |
| 20 | Llama 2 70B HF · 69.0B | 78.3% | ~152 GB |
| 21 | Phi 2 · 2.8B | 75.9% | ~2.6 GB |
| 22 | Qwen 7B · 7.7B | 75.3% | ~5.1 GB |
| 23 | Qwen2.5 Coder 32B · 32.8B | 70.5% | ~21 GB |
| 24 | Internlm 7B · 7B | 69.5% | ~15 GB |
| 25 | Llama 65B · proprietary | 69.5% | API only |
| 26 | PaLM 2 L · proprietary | 69.2% | API only |
| 27 | Falcon 180B · 180B | 67.8% | ~396 GB |
| 28 | Llama 33B · proprietary | 67.5% | API only |
| 29 | Qwen2.5 Coder 14B · 14.8B | 66.0% | ~9.6 GB |
| 30 | PaLM 2 M · proprietary | 64.9% | API only |
| 31 | DeepSeek Coder V2 Base · proprietary | 64.3% | API only |
| 32 | Falcon 40B · 41.8B | 61.9% | ~28 GB |
| 33 | Chatglm2 6B · 6B | 61.0% | ~4.0 GB |
| 34 | Qwen2.5 Coder 7B · 7.6B | 60.9% | ~5.0 GB |
| 35 | Llama 2 13B HF · 13.0B | 60.3% | ~8.6 GB |
| 36 | PaLM 2 S · proprietary | 59.6% | API only |
| 37 | DeepSeek Coder v2 Lite Base · 15.7B | 57.3% | ~10 GB |
| 38 | Yi 9B · 8.8B | 55.6% | ~5.8 GB |
| 39 | Nemotron 4 15B · proprietary | 55.5% | API only |
| 40 | INTELLECT 1 Instruct · 10.2B | 54.5% | ~6.8 GB |
| 41 | Llama 2 34B · proprietary | 54.5% | API only |
| 42 | Qwen 1 8B · 1.8B | 53.2% | ~1.2 GB |
| 43 | Text Davinci 001 · proprietary | 53.2% | API only |
| 44 | Qwen2.5 Coder 3B · 3.1B | 52.9% | ~2.2 GB |
| 45 | Llama 13B · proprietary | 52.7% | API only |
| 46 | PaLM 62B · proprietary | 52.5% | API only |
| 47 | Mpt 30B · proprietary | 50.6% | API only |
| 48 | Yi 6B · 6.1B | 50.3% | ~4.1 GB |
| 49 | Falcon 7B · 7.2B | 47.9% | ~4.8 GB |
| 50 | Llama 7B · 6.7B | 47.6% | ~4.5 GB |
| 51 | Starcoder2 15B · 16.0B | 47.2% | ~10 GB |
| 52 | Llama 2 7B HF · 6.7B | 45.9% | ~4.5 GB |
| 53 | Qwen2.5 Coder 1.5B · 1.5B | 45.2% | ~1.3 GB |
| 54 | Phi 1 5 · 1.4B | 44.4% | ~0.9 GB |
| 55 | Vicuna 13B v1.1 · proprietary | 43.2% | API only |
| 56 | Mpt 7B · proprietary | 42.6% | API only |
| 57 | Deepseek Coder 33B Base · 33.3B | 42.2% | ~21 GB |
| 58 | Gemma 2B · 2.5B | 42.1% | ~1.6 GB |
| 59 | Xgen 7B 8k Base · 7B | 41.2% | ~4.6 GB |
| 60 | GPT Neox 20B · 20.7B | 41.1% | ~14 GB |
| 61 | Dolly v2 12B · proprietary | 39.6% | API only |
| 62 | RedPajama INCITE 7B Base · 7B | 39.1% | ~4.6 GB |
| 63 | Open Llama 7B · 7B | 38.7% | ~4.6 GB |
| 64 | Starcoder2 7B · 7.2B | 38.7% | ~4.7 GB |
| 65 | Baichuan2 13B Base · 13B | 38.0% | ~8.6 GB |
| 66 | Deepseek Coder 6.7B Base · 6.7B | 36.4% | ~5.4 GB |
| 67 | GPT J 6B · 6B | 36.3% | ~4.0 GB |
| 68 | Opt 1.3B · 1.3B | 35.8% | ~2.9 GB |
| 69 | CodeQwen1.5 7B · 7.3B | 35.7% | ~4.8 GB |
| 70 | Qwen2.5 Coder 0.5B · 494M | 34.4% | ~0.6 GB |
| 71 | Starcoder2 3B · 3.0B | 34.2% | ~2.2 GB |
| 72 | Baichuan2 7B Base · 7B | 32.5% | ~4.6 GB |
| 73 | Cerebras GPT 13B · 13B | 32.4% | ~29 GB |
| 74 | Stablelm Tuned Alpha 7B · 7B | 27.0% | ~4.6 GB |
| 75 | Deepseek Coder 1.3B Base · 1.3B | 25.4% | ~1.5 GB |
| 76 | Gpt2 Xl · 1.6B | 25.0% | ~1.1 GB |
Score vs model size
Which models give the most quality for their size — the ones worth running locally.
- Qwen2.5 Coder 0.5B, 494M, score 34.4% — on the efficiency frontier (best score at its size or smaller).
- Opt 1.3B, 1B, score 35.8% — on the efficiency frontier (best score at its size or smaller).
- Phi 1 5, 1B, score 44.4% — on the efficiency frontier (best score at its size or smaller).
- Qwen2.5 Coder 1.5B, 2B, score 45.2% — on the efficiency frontier (best score at its size or smaller).
- Qwen 1 8B, 2B, score 53.2% — on the efficiency frontier (best score at its size or smaller).
- Phi 2, 3B, score 75.9% — on the efficiency frontier (best score at its size or smaller).
- Phi 3 Mini 4k Instruct, 4B, score 84.9% — on the efficiency frontier (best score at its size or smaller).
- Phi 3 Small 8k Instruct, 7B, score 90.7% — on the efficiency frontier (best score at its size or smaller).
- Phi 3 Medium 128k Instruct, 14B, score 91.6% — on the efficiency frontier (best score at its size or smaller).
- Qwen2.5 72B, 73B, score 94.5% — on the efficiency frontier (best score at its size or smaller).
- Llama 3.1 405B, 406B, score 95.3% — on the efficiency frontier (best score at its size or smaller).
ARC Challenge: frequently asked questions
- What is the best open LLM on ARC Challenge?
- DeepSeek v3 is the top open model on ARC Challenge, scoring 95.3%. Among all models tested — including proprietary ones — it ranks #1. That puts it ahead of every proprietary model we track, including GPT 3.5 Turbo (Nov 06) (OpenAI) at 87.4%.
- What's the best ARC Challenge model you can run on a 24 GB GPU?
- Phi 3 Medium 128k Instruct is the highest-scoring open model that fits in 24 GB at 4-bit quantization (~9.1 GB), scoring 91.6% on ARC Challenge.
- What's the best ARC Challenge model you can run on a 12 GB GPU?
- Phi 3 Small 8k Instruct is the highest-scoring open model that fits in 12 GB at 4-bit quantization (~15 GB), scoring 90.7% on ARC Challenge.
- Can open models match proprietary models on ARC Challenge?
- Yes — the best open model (DeepSeek v3, 95.3%) matches or beats every proprietary model we track on ARC Challenge.
Scores aggregated from epoch. llmrun does not run this benchmark — see the source for methodology, or the about benchmarks for what it measures.