Knowledge
MMLU Leaderboard
MMLU (Massive Multitask Language Understanding) spans 57 subjects from history to law to medicine as multiple-choice questions. It is the long-standing default for broad knowledge and remains the most widely-reported general benchmark.
Source: epoch85 open models ranked+51 proprietaryData through Dec 2024
All models ranked on MMLU
Proprietary / closed models are shown dimmed — you can't run them locally, but they show where the open field stands.
| # | Model | Score | VRAM @ Q4 |
|---|---|---|---|
| 1 | GPT 4o (Nov 20, 2024) · proprietary | 88.1% | API only |
| 2 | Claude 3.5 Sonnet (Oct 22, 2024) · proprietary | 87.3% | API only |
| 3 | DeepSeek v3 · 684.5B | 87.2% | ~415 GB |
| 4 | Gemini 1.5 Pro 002 · proprietary | 86.9% | API only |
| 5 | Claude 3.5 Sonnet (Jun 20, 2024) · proprietary | 86.5% | API only |
| 6 | GPT 4 (Mar 14) · proprietary | 86.4% | API only |
| 7 | Llama 3.3 70B Instruct · 70.6B | 86.3% | ~47 GB |
| 8 | Gemini 1.5 Pro 001 · proprietary | 85.9% | API only |
| 9 | Qwen2.5 72B Instruct · 72.7B | 85.3% | ~45 GB |
| 10 | Qwen2.5 72B · 72.7B | 85.0% | ~45 GB |
| 11 | Phi 4 · 14.7B | 84.8% | ~9.5 GB |
| 12 | Claude 3 Opus (Feb 29, 2024) · proprietary | 84.6% | API only |
| 13 | Llama 3.1 405B Instruct · 405.9B | 84.5% | ~268 GB |
| 14 | Llama 3.1 405B · 405.9B | 84.4% | ~268 GB |
| 15 | GPT 4o (Aug 06, 2024) · proprietary | 84.3% | API only |
| 16 | GPT 4o (May 13, 2024) · proprietary | 84.2% | API only |
| 17 | Gemini 1.5 Pro 001 Feb24 · proprietary | 82.7% | API only |
| 18 | GPT 4 (Jun 13) · proprietary | 82.4% | API only |
| 19 | Qwen2 72B Instruct · 72.7B | 82.4% | ~45 GB |
| 20 | Amazon.nova Pro v1:0 · proprietary | 82.0% | API only |
| 21 | GPT 4o Mini (Jul 18, 2024) · proprietary | 81.8% | API only |
| 22 | GPT 4 Turbo (Apr 09, 2024) · proprietary | 81.3% | API only |
| 23 | Llama 3.2 90B Vision Instruct · 88.6B | 80.3% | ~195 GB |
| 24 | Llama 3.1 70B Instruct · 70.6B | 80.1% | ~47 GB |
| 25 | Mistral Large Instruct 2407 · 122.6B | 80.0% | ~81 GB |
| 26 | Qwen2.5 14B Instruct · 14.8B | 79.9% | ~9.6 GB |
| 27 | Gemini 2.0 Flash Exp · proprietary | 79.7% | API only |
| 28 | GPT 4 Turbo · proprietary | 79.6% | API only |
| 29 | Meta Llama 3 70B Instruct · 70.6B | 79.3% | ~47 GB |
| 30 | Yi Large · proprietary | 79.3% | API only |
| 31 | Qwen2.5 Coder 32B · 32.8B | 79.1% | ~21 GB |
| 32 | Claude 2.0 · proprietary | 78.5% | API only |
| 33 | DeepSeek v2 · 235.7B | 78.4% | ~144 GB |
| 34 | Phi 3 Medium 128k Instruct · 14.0B | 78.0% | ~9.1 GB |
| 35 | Gemini 1.5 Flash 001 · proprietary | 77.9% | API only |
| 36 | Gemini 1.5 Flash (May 14) · proprietary | 77.8% | API only |
| 37 | Mixtral 8x22B v0.1 · 140.6B | 77.8% | ~85 GB |
| 38 | Amazon.nova Lite v1:0 · proprietary | 77.0% | API only |
| 39 | Claude 1.3 · proprietary | 77.0% | API only |
| 40 | Yi 34B · 34.4B | 76.3% | ~21 GB |
| 41 | Claude 3 Sonnet (Feb 29, 2024) · proprietary | 75.9% | API only |
| 42 | Gemma 2 27B IT · 27.2B | 75.7% | ~18 GB |
| 43 | Phi 3 Small 8k Instruct · 7.4B | 75.7% | ~15 GB |
| 44 | Qwen2.5 Coder 14B · 14.8B | 75.2% | ~9.6 GB |
| 45 | Qwen1.5 32B · 32.5B | 74.4% | ~20 GB |
| 46 | Claude 3.5 Haiku (Oct 22, 2024) · proprietary | 74.3% | API only |
| 47 | Gemini 1.5 Flash 002 · proprietary | 73.9% | API only |
| 48 | Claude 3 Haiku (Mar 07, 2024) · proprietary | 73.8% | API only |
| 49 | Claude 2.1 · proprietary | 73.5% | API only |
| 50 | Yi 34B Chat · 34.4B | 73.5% | ~21 GB |
| 51 | Claude Instant 1.1 · proprietary | 73.4% | API only |
| 52 | Claude Instant 1.2 · proprietary | 73.2% | API only |
| 53 | Qwen2.5 7B Instruct · 7.6B | 72.9% | ~5.0 GB |
| 54 | Inflection 1 · proprietary | 72.7% | API only |
| 55 | Gemma 2 9B IT · 9.2B | 72.1% | ~6.1 GB |
| 56 | GPT 3.5 Turbo (Nov 06) · proprietary | 71.4% | API only |
| 57 | Amazon.nova Micro v1:0 · proprietary | 70.8% | API only |
| 58 | Falcon 180B · 180B | 70.6% | ~396 GB |
| 59 | Mixtral 8x7B v0.1 · 46.7B | 70.6% | ~29 GB |
| 60 | Gemini 1.0 Pro 001 · proprietary | 70.0% | API only |
| 61 | Text Davinci 002 · proprietary | 70.0% | API only |
| 62 | Llama 2 70B HF · 69.0B | 69.9% | ~152 GB |
| 63 | C4ai Command R Plus 08 2024 · 103.8B | 69.4% | ~69 GB |
| 64 | PaLM 540B · proprietary | 69.3% | API only |
| 65 | GPT 3.5 Turbo (Jun 13) · proprietary | 68.9% | API only |
| 66 | Meta Llama 3 8B Instruct · 8.0B | 68.8% | ~5.3 GB |
| 67 | Mistral Large 2402 · proprietary | 68.8% | API only |
| 68 | Phi 3 Mini 4k Instruct · 3.8B | 68.8% | ~3.4 GB |
| 69 | Mistral Small 2402 · proprietary | 68.7% | API only |
| 70 | Qwen1.5 14B · 14.2B | 68.6% | ~10 GB |
| 71 | StableBeluga2 · 70B | 68.6% | ~43 GB |
| 72 | Yi 9B · 8.8B | 68.4% | ~5.8 GB |
| 73 | Qwen2.5 Coder 7B · 7.6B | 68.0% | ~5.0 GB |
| 74 | Chinchilla (70B) · proprietary | 67.5% | API only |
| 75 | GPT 3.5 Turbo (Jan 25) · proprietary | 67.3% | API only |
| 76 | Qwen 14B · 14.2B | 66.3% | ~9.3 GB |
| 77 | Gemma 7B · 8.5B | 66.1% | ~19 GB |
| 78 | C4ai Command R 08 2024 · 32.3B | 65.2% | ~21 GB |
| 79 | Qwen 14B Chat · 14.2B | 65.0% | ~9.3 GB |
| 80 | Starcoder2 15B · 16.0B | 64.1% | ~10 GB |
| 81 | Yi 6B · 6.1B | 64.0% | ~4.1 GB |
| 82 | Llama 65B · proprietary | 63.4% | API only |
| 83 | Llama 2 34B · proprietary | 62.6% | API only |
| 84 | Qwen1.5 7B · 7.7B | 62.6% | ~6.0 GB |
| 85 | Mistral 7B Instruct v0.2 · 7.2B | 62.5% | ~4.9 GB |
| 86 | Mistral 7B v0.1 · 7.2B | 62.5% | ~4.9 GB |
| 87 | Yi 6B Chat · 6.1B | 61.0% | ~4.1 GB |
| 88 | DeepSeek Coder v2 Lite Base · 15.7B | 60.5% | ~10 GB |
| 89 | Gopher (280B) · proprietary | 60.0% | API only |
| 90 | Llama 2 70B Chat HF · 69.0B | 59.9% | ~152 GB |
| 91 | Mistral 7B Instruct v0.3 · 7.2B | 59.9% | ~4.9 GB |
| 92 | Baichuan2 13B Base · 13B | 59.2% | ~8.6 GB |
| 93 | Llama 33B · proprietary | 58.7% | API only |
| 94 | Nemotron 4 15B · proprietary | 58.7% | API only |
| 95 | Falcon 11B · 11.1B | 58.4% | ~7.3 GB |
| 96 | Phi 2 · 2.8B | 58.4% | ~2.6 GB |
| 97 | Internlm Chat 20B · 20B | 57.4% | ~15 GB |
| 98 | Falcon 40B · 41.8B | 56.9% | ~28 GB |
| 99 | Llama 3.2 11B Vision Instruct · 10.7B | 56.5% | ~7.0 GB |
| 100 | Llama 3.1 8B Instruct · 8.0B | 56.1% | ~5.3 GB |
| 101 | Llama 2 13B HF · 13.0B | 55.6% | ~8.6 GB |
| 102 | Baichuan2 13B Chat · 13B | 55.1% | ~8.6 GB |
| 103 | Baichuan2 7B Base · 7B | 54.2% | ~4.6 GB |
| 104 | PaLM 62B · proprietary | 53.7% | API only |
| 105 | Qwen2.5 Coder 1.5B · 1.5B | 53.6% | ~1.3 GB |
| 106 | Baichuan 13B Base · 13B | 51.6% | ~8.6 GB |
| 107 | Internlm 7B · 7B | 51.0% | ~15 GB |
| 108 | Llama 2 13B Chat HF · 13.0B | 50.9% | ~8.6 GB |
| 109 | INTELLECT 1 Instruct · 10.2B | 49.9% | ~6.8 GB |
| 110 | Chatglm2 6B · 6B | 47.9% | ~4.0 GB |
| 111 | Mpt 30B · proprietary | 47.9% | API only |
| 112 | Llama 13B · proprietary | 47.7% | API only |
| 113 | Llama 2 7B HF · 6.7B | 45.8% | ~4.5 GB |
| 114 | Qwen 7B · 7.7B | 45.0% | ~5.1 GB |
| 115 | Text Davinci 001 · proprietary | 43.9% | API only |
| 116 | Baichuan 7B · 7B | 42.3% | ~4.6 GB |
| 117 | Gemma 2B · 2.5B | 42.3% | ~1.6 GB |
| 118 | Qwen2.5 Coder 0.5B · 494M | 42.0% | ~0.6 GB |
| 119 | CodeQwen1.5 7B · 7.3B | 40.5% | ~4.8 GB |
| 120 | Deepseek Coder 33B Base · 33.3B | 39.4% | ~21 GB |
| 121 | Starcoder2 7B · 7.2B | 38.8% | ~4.7 GB |
| 122 | Phi 1 5 · 1.4B | 37.6% | ~0.9 GB |
| 123 | Starcoder2 3B · 3.0B | 36.6% | ~2.2 GB |
| 124 | Deepseek Coder 6.7B Base · 6.7B | 36.4% | ~5.4 GB |
| 125 | Xgen 7B 8k Base · 7B | 36.3% | ~4.6 GB |
| 126 | Llama 7B · 6.7B | 35.6% | ~4.5 GB |
| 127 | Falcon 7B · 7.2B | 35.0% | ~4.8 GB |
| 128 | Mpt 7B · proprietary | 30.8% | API only |
| 129 | Open Llama 7B · 7B | 29.9% | ~4.6 GB |
| 130 | Qwen 1 8B · 1.8B | 28.2% | ~1.2 GB |
| 131 | RedPajama INCITE 7B Base · 7B | 26.3% | ~4.6 GB |
| 132 | Cerebras GPT 13B · 13B | 26.2% | ~29 GB |
| 133 | Dolly v2 12B · proprietary | 26.2% | API only |
| 134 | Deepseek Coder 1.3B Base · 1.3B | 25.8% | ~1.5 GB |
| 135 | GPT J 6B · 6B | 25.7% | ~4.0 GB |
| 136 | Opt 1.3B · 1.3B | 25.1% | ~2.9 GB |
Score vs model size
Which models give the most quality for their size — the ones worth running locally.
- Qwen2.5 Coder 0.5B, 494M, score 42.0% — on the efficiency frontier (best score at its size or smaller).
- Qwen2.5 Coder 1.5B, 2B, score 53.6% — on the efficiency frontier (best score at its size or smaller).
- Phi 2, 3B, score 58.4% — on the efficiency frontier (best score at its size or smaller).
- Phi 3 Mini 4k Instruct, 4B, score 68.8% — on the efficiency frontier (best score at its size or smaller).
- Phi 3 Small 8k Instruct, 7B, score 75.7% — on the efficiency frontier (best score at its size or smaller).
- Phi 3 Medium 128k Instruct, 14B, score 78.0% — on the efficiency frontier (best score at its size or smaller).
- Phi 4, 15B, score 84.8% — on the efficiency frontier (best score at its size or smaller).
- Llama 3.3 70B Instruct, 71B, score 86.3% — on the efficiency frontier (best score at its size or smaller).
- DeepSeek v3, 685B, score 87.2% — on the efficiency frontier (best score at its size or smaller).
MMLU: frequently asked questions
- What is the best open LLM on MMLU?
- DeepSeek v3 is the top open model on MMLU, scoring 87.2%. Among all models tested — including proprietary ones — it ranks #3. The top model overall is GPT 4o (Nov 20, 2024) (OpenAI) at 88.1%.
- What's the best MMLU model you can run on a 24 GB GPU?
- Phi 4 is the highest-scoring open model that fits in 24 GB at 4-bit quantization (~9.5 GB), scoring 84.8% on MMLU.
- What's the best MMLU model you can run on a 12 GB GPU?
- Phi 3 Small 8k Instruct is the highest-scoring open model that fits in 12 GB at 4-bit quantization (~15 GB), scoring 75.7% on MMLU.
- Can open models match proprietary models on MMLU?
- Not quite on MMLU: the strongest proprietary model (GPT 4o (Nov 20, 2024)) scores 88.1%, ahead of the best open model (DeepSeek v3) at 87.2% — but you can run the open one yourself.
Scores aggregated from epoch. llmrun does not run this benchmark — see the source for methodology, or the about benchmarks for what it measures.