Knowledge
OpenBookQA Leaderboard
OpenBookQA is a science-question benchmark modeled on open-book exams: the model is given a small set of core science facts and must combine them with broader common knowledge to answer multi-step questions. It targets reasoning over recall.
Source: epoch26 open models ranked+15 proprietaryData through Apr 2024
Open models ranked on OpenBookQA
# shows rank among open models / rank overall (including proprietary).
| # | Model | Score | VRAM @ Q4 |
|---|---|---|---|
| 1 / 1 | Phi 3 Mini 4k Instruct · 3.8B | 88.0% | ~3.4 GB |
| 2 / 2 | Phi 3 Small 8k Instruct · 7.4B | 88.0% | ~15 GB |
| 3 / 3 | Phi 3 Medium 128k Instruct · 14.0B | 87.4% | ~9.1 GB |
| 4 / 5 | Mixtral 8x7B v0.1 · 46.7B | 85.8% | ~29 GB |
| 5 / 6 | Meta Llama 3 8B Instruct · 8.0B | 82.6% | ~5.3 GB |
| 6 / 7 | Mistral 7B v0.1 · 7.2B | 79.8% | ~4.9 GB |
| 7 / 8 | Gemma 7B · 8.5B | 78.6% | ~19 GB |
| 8 / 9 | Phi 2 · 2.8B | 73.6% | ~2.6 GB |
| 9 / 12 | Falcon 180B · 180B | 64.2% | ~396 GB |
| 10 / 14 | Llama 2 70B HF · 69.0B | 60.2% | ~152 GB |
| 11 / 16 | Llama 2 7B HF · 6.7B | 58.6% | ~4.5 GB |
| 12 / 20 | Llama 7B · 6.7B | 57.2% | ~4.5 GB |
| 13 / 21 | Llama 2 13B HF · 13.0B | 57.0% | ~8.6 GB |
| 14 / 22 | Falcon 40B · 41.8B | 56.6% | ~28 GB |
| 15 / 26 | Falcon 7B · 7.2B | 51.6% | ~4.8 GB |
| 16 / 29 | Xgen 7B 8k Base · 7B | 40.2% | ~4.6 GB |
| 17 / 30 | RedPajama INCITE 7B Base · 7B | 40.0% | ~4.6 GB |
| 18 / 31 | Opt 1.3B · 1.3B | 39.8% | ~2.9 GB |
| 19 / 33 | Open Llama 7B · 7B | 39.0% | ~4.6 GB |
| 20 / 34 | GPT Neox 20B · 20.7B | 38.8% | ~14 GB |
| 21 / 35 | GPT J 6B · 6B | 38.2% | ~4.0 GB |
| 22 / 36 | Phi 1 5 · 1.4B | 37.2% | ~0.9 GB |
| 23 / 37 | Cerebras GPT 13B · 13B | 35.8% | ~29 GB |
| 24 / 39 | Stablelm Tuned Alpha 7B · 7B | 32.4% | ~4.6 GB |
| 25 / 40 | GPT Neo 2.7B · 2.7B | 23.2% | ~6.0 GB |
| 26 / 41 | Gpt2 Xl · 1.6B | 22.4% | ~1.1 GB |
Score vs model size
Which models give the most quality for their size — the ones worth running locally.
- Opt 1.3B, 1B, score 39.8% — on the efficiency frontier (best score at its size or smaller).
- Phi 2, 3B, score 73.6% — on the efficiency frontier (best score at its size or smaller).
- Phi 3 Mini 4k Instruct, 4B, score 88.0% — on the efficiency frontier (best score at its size or smaller).
OpenBookQA: frequently asked questions
- What is the best open LLM on OpenBookQA?
- Phi 3 Mini 4k Instruct is the top open model on OpenBookQA, scoring 88.0%. Among all models tested — including proprietary ones — it ranks #1. That puts it ahead of every proprietary model we track, including GPT 3.5 Turbo (Nov 06) (OpenAI) at 86.0%.
- What's the best OpenBookQA model you can run on a 24 GB GPU?
- Phi 3 Mini 4k Instruct is the highest-scoring open model that fits in 24 GB at 4-bit quantization (~3.4 GB), scoring 88.0% on OpenBookQA.
- What's the best OpenBookQA model you can run on a 12 GB GPU?
- Phi 3 Mini 4k Instruct is the highest-scoring open model that fits in 12 GB at 4-bit quantization (~3.4 GB), scoring 88.0% on OpenBookQA.
- Can open models match proprietary models on OpenBookQA?
- Yes — the best open model (Phi 3 Mini 4k Instruct, 88.0%) matches or beats every proprietary model we track on OpenBookQA.
Scores aggregated from epoch. llmrun does not run this benchmark — see the source for methodology, or the about benchmarks for what it measures.