Knowledge
HellaSwag Leaderboard
HellaSwag is a commonsense benchmark that asks a model to pick the most plausible continuation of an everyday situation. The wrong options are adversarially chosen to fool models, testing grounded commonsense understanding.
Source: epoch47 open models ranked+28 proprietaryData through Dec 2024
Open models ranked on HellaSwag
# shows rank among open models / rank overall (including proprietary).
| # | Model | Score | VRAM @ Q4 |
|---|---|---|---|
| 1 / 3 | Llama 3.1 405B · 405.9B | 89.2% | ~268 GB |
| 2 / 4 | Falcon 180B · 180B | 89.0% | ~396 GB |
| 3 / 5 | DeepSeek v3 · 684.5B | 88.9% | ~415 GB |
| 4 / 6 | DeepSeek v2 · 235.7B | 87.1% | ~144 GB |
| 5 / 8 | Mixtral 8x7B v0.1 · 46.7B | 86.7% | ~29 GB |
| 6 / 10 | Llama 2 70B HF · 69.0B | 85.3% | ~152 GB |
| 7 / 11 | Falcon 40B · 41.8B | 85.3% | ~28 GB |
| 8 / 12 | Qwen2.5 72B · 72.7B | 84.8% | ~45 GB |
| 9 / 14 | StableBeluga2 · 70B | 84.1% | ~43 GB |
| 10 / 17 | Qwen2.5 Coder 32B · 32.8B | 83.0% | ~21 GB |
| 11 / 18 | Falcon 11B · 11.1B | 82.9% | ~7.3 GB |
| 12 / 22 | Phi 3 Medium 128k Instruct · 14.0B | 82.4% | ~9.1 GB |
| 13 / 23 | Gemma 7B · 8.5B | 82.2% | ~19 GB |
| 14 / 26 | Mistral 7B v0.1 · 7.2B | 81.0% | ~4.9 GB |
| 15 / 27 | Llama 2 13B HF · 13.0B | 80.7% | ~8.6 GB |
| 16 / 28 | Qwen2.5 Coder 14B · 14.8B | 80.2% | ~9.6 GB |
| 17 / 33 | Falcon 7B · 7.2B | 78.1% | ~4.8 GB |
| 18 / 34 | Internlm 20B · 20B | 78.1% | ~43 GB |
| 19 / 37 | Llama 2 7B HF · 6.7B | 77.2% | ~4.5 GB |
| 20 / 38 | Phi 3 Small 8k Instruct · 7.4B | 77.0% | ~15 GB |
| 21 / 39 | Qwen2.5 Coder 7B · 7.6B | 76.8% | ~5.0 GB |
| 22 / 40 | Phi 3 Mini 4k Instruct · 3.8B | 76.7% | ~3.4 GB |
| 23 / 42 | Yi 9B · 8.8B | 76.4% | ~5.8 GB |
| 24 / 43 | Llama 7B · 6.7B | 76.2% | ~4.5 GB |
| 25 / 45 | Bloom · 176.2B | 74.4% | ~116 GB |
| 26 / 46 | Yi 6B · 6.1B | 74.4% | ~4.1 GB |
| 27 / 47 | Xgen 7B 8k Base · 7B | 74.2% | ~4.6 GB |
| 28 / 48 | Open Llama 7B · 7B | 71.8% | ~4.6 GB |
| 29 / 49 | INTELLECT 1 Instruct · 10.2B | 71.4% | ~6.8 GB |
| 30 / 50 | Gemma 2B · 2.5B | 71.4% | ~1.6 GB |
| 31 / 51 | Qwen2.5 Coder 3B · 3.1B | 70.9% | ~2.2 GB |
| 32 / 52 | Baichuan2 13B Base · 13B | 70.8% | ~8.6 GB |
| 33 / 54 | Internlm 7B · 7B | 70.6% | ~15 GB |
| 34 / 55 | GPT Neox 20B · 20.7B | 70.5% | ~14 GB |
| 35 / 56 | RedPajama INCITE 7B Base · 7B | 70.3% | ~4.6 GB |
| 36 / 57 | Opt 1.3B · 1.3B | 69.9% | ~2.9 GB |
| 37 / 59 | Baichuan2 7B Base · 7B | 68.0% | ~4.6 GB |
| 38 / 61 | GPT J 6B · 6B | 66.2% | ~4.0 GB |
| 39 / 62 | Qwen2.5 Coder 1.5B · 1.5B | 61.8% | ~1.3 GB |
| 40 / 63 | Cerebras GPT 13B · 13B | 59.4% | ~29 GB |
| 41 / 65 | Chatglm2 6B · 6B | 57.0% | ~4.0 GB |
| 42 / 68 | Phi 2 · 2.8B | 53.6% | ~2.6 GB |
| 43 / 69 | Qwen2.5 Coder 0.5B · 494M | 48.4% | ~0.6 GB |
| 44 / 70 | Phi 1 5 · 1.4B | 47.6% | ~0.9 GB |
| 45 / 73 | GPT Neo 2.7B · 2.7B | 42.7% | ~6.0 GB |
| 46 / 74 | Stablelm Tuned Alpha 7B · 7B | 40.7% | ~4.6 GB |
| 47 / 75 | Gpt2 Xl · 1.6B | 40.0% | ~1.1 GB |
Score vs model size
Which models give the most quality for their size — the ones worth running locally.
- Qwen2.5 Coder 0.5B, 494M, score 48.4% — on the efficiency frontier (best score at its size or smaller).
- Opt 1.3B, 1B, score 69.9% — on the efficiency frontier (best score at its size or smaller).
- Gemma 2B, 3B, score 71.4% — on the efficiency frontier (best score at its size or smaller).
- Phi 3 Mini 4k Instruct, 4B, score 76.7% — on the efficiency frontier (best score at its size or smaller).
- Llama 2 7B HF, 7B, score 77.2% — on the efficiency frontier (best score at its size or smaller).
- Falcon 7B, 7B, score 78.1% — on the efficiency frontier (best score at its size or smaller).
- Mistral 7B v0.1, 7B, score 81.0% — on the efficiency frontier (best score at its size or smaller).
- Gemma 7B, 9B, score 82.2% — on the efficiency frontier (best score at its size or smaller).
- Falcon 11B, 11B, score 82.9% — on the efficiency frontier (best score at its size or smaller).
- Qwen2.5 Coder 32B, 33B, score 83.0% — on the efficiency frontier (best score at its size or smaller).
- Falcon 40B, 42B, score 85.3% — on the efficiency frontier (best score at its size or smaller).
- Mixtral 8x7B v0.1, 47B, score 86.7% — on the efficiency frontier (best score at its size or smaller).
- Falcon 180B, 180B, score 89.0% — on the efficiency frontier (best score at its size or smaller).
- Llama 3.1 405B, 406B, score 89.2% — on the efficiency frontier (best score at its size or smaller).
HellaSwag: frequently asked questions
- What is the best open LLM on HellaSwag?
- Llama 3.1 405B is the top open model on HellaSwag, scoring 89.2%. Among all models tested — including proprietary ones — it ranks #3. The top model overall is GPT 4 (Mar 14) (OpenAI) at 95.3%.
- What's the best HellaSwag model you can run on a 24 GB GPU?
- Falcon 11B is the highest-scoring open model that fits in 24 GB at 4-bit quantization (~7.3 GB), scoring 82.9% on HellaSwag.
- What's the best HellaSwag model you can run on a 12 GB GPU?
- Falcon 11B is the highest-scoring open model that fits in 12 GB at 4-bit quantization (~7.3 GB), scoring 82.9% on HellaSwag.
- Can open models match proprietary models on HellaSwag?
- Not quite on HellaSwag: the strongest proprietary model (GPT 4 (Mar 14)) scores 95.3%, ahead of the best open model (Llama 3.1 405B) at 89.2% — but you can run the open one yourself.
Scores aggregated from epoch. llmrun does not run this benchmark — see the source for methodology, or the about benchmarks for what it measures.