Knowledge
OpenBookQA Leaderboard
OpenBookQA is a science-question benchmark modeled on open-book exams: the model is given a small set of core science facts and must combine them with broader common knowledge to answer multi-step questions. It targets reasoning over recall.
Source: epoch21 open models ranked+21 proprietaryData through Apr 2024
All models ranked on OpenBookQA
Proprietary / closed models are shown dimmed — you can't run them locally, but they show where the open field stands.
| # | Model | Score |
|---|---|---|
| 1 | Phi 3 Mini 4k Instruct · 3.8B | 88.0% |
| 2 | Phi 3 Small 8k Instruct · 7.4B | 88.0% |
| 3 | Phi 3 Medium 128K Instruct · proprietary | 87.4% |
| 4 | GPT 3.5 Turbo (Nov 06) · proprietary | 86.0% |
| 5 | Mixtral 8x7B v0.1 · 46.7B | 85.8% |
| 6 | Meta Llama 3 8B Instruct · 8.0B | 82.6% |
| 7 | Mistral 7B v0.1 · 7B | 79.8% |
| 8 | Gemma 7B · 8.5B | 78.6% |
| 9 | Phi 2 · 2.8B | 73.6% |
| 10 | PaLM 540B · proprietary | 68.0% |
| 11 | Text Davinci 001 · proprietary | 65.4% |
| 12 | Falcon 180B · 180B | 64.2% |
| 13 | GLaM (MoE) · proprietary | 63.0% |
| 14 | Llama 2 70B HF · 69.0B | 60.2% |
| 15 | Llama 65B · proprietary | 60.2% |
| 16 | Llama 2 7B HF · 6.7B | 58.6% |
| 17 | Llama 33B · proprietary | 58.6% |
| 18 | Llama 2 34B · proprietary | 58.2% |
| 19 | PaLM 2 M · proprietary | 57.4% |
| 20 | Llama 7B · 6.7B | 57.2% |
| 21 | Llama 2 13B HF · 13.0B | 57.0% |
| 22 | Falcon 40B · 41.8B | 56.6% |
| 23 | Llama 13B · proprietary | 56.4% |
| 24 | PaLM 2 S · proprietary | 56.2% |
| 25 | Mpt 30B · proprietary | 52.0% |
| 26 | Falcon 7B · 7.2B | 51.6% |
| 27 | Mpt 7B · proprietary | 51.4% |
| 28 | PaLM 62B · proprietary | 50.4% |
| 29 | Xgen 7B 8k Base · 7B | 40.2% |
| 30 | RedPajama INCITE 7B Base · proprietary | 40.0% |
| 31 | Opt 13B · proprietary | 39.8% |
| 32 | Dolly v2 12B · proprietary | 39.2% |
| 33 | Open Llama 7B · proprietary | 39.0% |
| 34 | GPT Neox 20B · 20.7B | 38.8% |
| 35 | GPT J 6B · 6B | 38.2% |
| 36 | Phi 1 5 · 1.4B | 37.2% |
| 37 | Cerebras GPT 13B · 13B | 35.8% |
| 38 | Vicuna 13B v1.1 · proprietary | 33.0% |
| 39 | Stablelm Tuned Alpha 7B · 7B | 32.4% |
| 40 | Opt 1.3b · proprietary | 24.0% |
| 41 | GPT Neo 2.7B · proprietary | 23.2% |
| 42 | Gpt2 Xl · 1.6B | 22.4% |
Score vs model size
Which models give the most quality for their size — the ones worth running locally.
- Phi 1 5, 1B, score 37.2% — on the efficiency frontier (best score at its size or smaller).
- Phi 2, 3B, score 73.6% — on the efficiency frontier (best score at its size or smaller).
- Phi 3 Mini 4k Instruct, 4B, score 88.0% — on the efficiency frontier (best score at its size or smaller).
OpenBookQA: frequently asked questions
- What is the best open LLM on OpenBookQA?
- Phi 3 Mini 4k Instruct is the top open model on OpenBookQA, scoring 88.0%. Among all models tested — including proprietary ones — it ranks #1. That puts it ahead of every proprietary model we track, including Phi 3 Medium 128K Instruct (Microsoft) at 87.4%.
- What's the best OpenBookQA model you can run on a 24 GB GPU?
- Phi 3 Mini 4k Instruct is the highest-scoring open model that fits in 24 GB at 4-bit quantization (about 2 GB), scoring 88.0% on OpenBookQA.
- What's the best OpenBookQA model you can run on a 12 GB GPU?
- Phi 3 Mini 4k Instruct is the highest-scoring open model that fits in 12 GB at 4-bit quantization (about 2 GB), scoring 88.0% on OpenBookQA.
- Can open models match proprietary models on OpenBookQA?
- Yes — the best open model (Phi 3 Mini 4k Instruct, 88.0%) matches or beats every proprietary model we track on OpenBookQA.
Scores aggregated from epoch. llmrun does not run this benchmark — see the source for methodology, or the about benchmarks for what it measures.