llmrun
Reasoning

ForecastBench Leaderboard

ForecastBench, from the Forecasting Research Institute, asks models to predict real-world future events, with questions that only resolve after the model's training cutoff so the answers can't have been memorised. It is scored as a difficulty-adjusted Brier index, where higher is better.

Source: epoch24 open models ranked+71 proprietaryData through Sep 2026

All models ranked on ForecastBench

Proprietary / closed models are shown dimmed — you can't run them locally, but they show where the open field stands.

#ModelScoreVRAM @ Q4
1GPT 5.5 (high) · proprietary
66.7
API only
2Claude Opus 5 Max · proprietary
66.6
API only
3GPT 6 Astra (high) · proprietary
66.6
API only
4Gemini 3.7 Flash (high) · proprietary
66.2
API only
5Claude Opus 4.8 Max · proprietary
66.1
API only
6GPT 5.6 Sol (medium) · proprietary
66.0
API only
7Claude Fable 5 Max · proprietary
65.9
API only
8GPT 5.6 Sol Promax · proprietary
65.8
API only
9Grok 4.5 (high) · proprietary
65.8
API only
10Claude Fable 5 (high) · proprietary
65.7
API only
11Claude Fable 5.1 (medium) · proprietary
65.7
API only
12Gemini 3.6 Flash (medium) · proprietary
65.7
API only
13GPT 5.6 Sol Max · proprietary
65.7
API only
14Muse Spark 1.3 Max · proprietary
65.5
API only
15Grok 4.6 (xhigh) · proprietary
65.4
API only
16Gemini 3.1 Pro Preview (high) · proprietary
65.2
API only
17Gemini 3 Pro Preview · proprietary
65.0
API only
18GPT 5 (Aug 07, 2025, unspecified) · proprietary
65.0
API only
19Grok 4 (Jul 09) · proprietary
64.7
API only
20Claude Opus 4.5 (Nov 01, 2025, unspecified) · proprietary
64.6
API only
21Grok 4 Fast · proprietary
64.6
API only
22Grok 4.1 Fast Reasoning · proprietary
64.6
API only
23GPT 4.5 Preview (Feb 27, 2025) · proprietary
64.5
API only
24Claude Opus 4.1 (Aug 05, 2025, unspecified) · proprietary
64.4
API only
25Gemini 3 Flash Preview · proprietary
64.4
API only
26O3 (Apr 16, 2025, unspecified) · proprietary
64.3
API only
27Gemini 2.5 Pro Preview (Mar 25) · proprietary
64.2
API only
28GPT 5.5 (unspecified) · proprietary
64.2
API only
29Kimi K3 · 2779.9B
64.2
~1674 GB
30Gemini 3.1 Pro Preview · proprietary
64.1
API only
31Grok 4.20 0309 Reasoning · proprietary
64.1
API only
32Claude Sonnet 4.6 (unspecified) · proprietary
64.0
API only
33Gemini 2.5 Pro · proprietary
64.0
API only
34Gemini 3.5 Flash (unspecified) · proprietary
64.0
API only
35MiniMax M3 · 427.0B
64.0
~257 GB
36Claude Sonnet 4 (May 14, 2025, unspecified) · proprietary
63.9
API only
37Claude Sonnet 4.5 (Sep 29, 2025, unspecified) · proprietary
63.9
API only
38Claude Sonnet 4.6 (16K) · proprietary
63.9
API only
39O4 Mini (Apr 16, 2025, unspecified) · proprietary
63.9
API only
40Qwen3 235B A22B · 235.1B
63.9
~142 GB
41GPT 5.2 (Dec 11, 2025, unspecified) · proprietary
63.8
API only
42Claude 3.7 Sonnet (Feb 19, 2025, unspecified) · proprietary
63.7
API only
43O3 Mini (Jan 31, 2025, unspecified) · proprietary
63.5
API only
44GLM 5 · 753.9B
63.4
~457 GB
45Grok 4.3 (unspecified) · proprietary
63.2
API only
46Claude Haiku 4.5 (Oct 01, 2025, unspecified) · proprietary
63.1
API only
47Claude Opus 4.6 (unspecified) · proprietary
63.1
API only
48Claude Sonnet 5 (16K) · proprietary
63.1
API only
49Grok 4.20 · proprietary
63.1
API only
50GPT 5 Mini (Aug 07, 2025, unspecified) · proprietary
63.0
API only
51GPT 5.4 (Mar 05, 2026, unspecified) · proprietary
63.0
API only
52Claude 3.5 Sonnet (Jun 20, 2024) · proprietary
62.9
API only
53Claude 3.5 Sonnet (Oct 22, 2024) · proprietary
62.9
API only
54GPT 4 Turbo (Apr 09, 2024) · proprietary
62.9
API only
55Claude Opus 4.7 (unspecified) · proprietary
62.8
API only
56Gemini 2.5 Flash · proprietary
62.7
API only
57Gemini 3.1 Flash Lite · proprietary
62.6
API only
58Claude Opus 4.8 (24K) · proprietary
62.4
API only
59DeepSeek v3 · 684.5B
62.4
~415 GB
60Gemini 2.5 Flash Preview (Apr 17) · proprietary
62.4
API only
61Kimi K2 Instruct 0905 · 1026.4B
62.4
~620 GB
62Llama 3.1 405B Instruct · 405.9B
62.3
~268 GB
63Claude 3 Opus (Feb 29, 2024) · proprietary
62.1
API only
64GPT 4.1 (Apr 14, 2025) · proprietary
62.0
API only
65GPT 5.1 (Nov 13, 2025, unspecified) · proprietary
62.0
API only
66Kimi K2 Instruct · 1026.4B
62.0
~620 GB
67Qwen3 235B A22B Thinking 2507 · 235.1B
62.0
~142 GB
68GLM 4.5 Air · 110.5B
61.7
~67 GB
69Claude Opus 4.8 (unspecified) · proprietary
61.6
API only
70Mistral Large Instruct 2407 · 122.6B
61.5
~81 GB
71Qwen2.5 72B Instruct · 72.7B
61.4
~45 GB
72Mistral Large Instruct 2411 · 122.6B
61.3
~75 GB
73GPT 4 (Jun 13) · proprietary
61.2
API only
74Gemma 4 31B IT · 31.3B
61.1
~20 GB
75GPT 4o (May 13, 2024) · proprietary
61.0
API only
76Llama 3.3 70B Instruct · 70.6B
61.0
~47 GB
77GPT 5.4 Mini (Mar 17, 2026, unspecified) · proprietary
60.8
API only
78DeepSeek V4 Pro · 1598.8B
60.7
~960 GB
79DeepSeek V3.1 · 684.5B
60.5
~415 GB
80GPT 4o (Nov 20, 2024) · proprietary
60.4
API only
81Meta Llama 3 70B · 70.6B
59.9
~47 GB
82Qwen1.5 110B Chat · 111.2B
59.3
~68 GB
83Meta Llama 3 8B Instruct · 8.0B
59.1
~5.3 GB
84Llama 4 Scout 17B 16E Instruct · 108.6B
58.9
~72 GB
85GPT 5 Nano (Aug 07, 2025, unspecified) · proprietary
58.7
API only
86Gemini 1.5 Pro 001 · proprietary
58.6
API only
87GPT 5.4 Nano (Mar 17, 2026, unspecified) · proprietary
58.6
API only
88Mixtral 8x22B Instruct v0.1 · 140.6B
58.6
~85 GB
89Claude 2.1 · proprietary
58.5
API only
90Gemini 1.5 Flash 001 · proprietary
58.4
API only
91Mixtral 8x7B Instruct v0.1 · 46.7B
58.0
~29 GB
92Gemini 2.0 Flash Lite 001 · proprietary
56.8
API only
93Claude 3 Haiku (Mar 07, 2024) · proprietary
54.6
API only
94Llama 2 70B Chat HF · 69.0B
50.2
~152 GB
95GPT 3.5 Turbo (Jan 25) · proprietary
46.2
API only

Score vs model size

Which models give the most quality for their size — the ones worth running locally.

10B100B1Tmodel size (log scale) →64.250.2GLM 5 · 754B · 63.4DeepSeek v3 · 684B · 62.4Kimi K2 Instruct 0905 · 1T · 62.4Llama 3.1 405B Instruct · 406B · 62.3Kimi K2 Instruct · 1T · 62.0Qwen3 235B A22B Thinking 2507 · 235B · 62.0Mistral Large Instruct 2407 · 123B · 61.5Mistral Large Instruct 2411 · 123B · 61.3Llama 3.3 70B Instruct · 71B · 61.0DeepSeek V4 Pro · 1.6T · 60.7DeepSeek V3.1 · 685B · 60.5Meta Llama 3 70B · 71B · 59.9Qwen1.5 110B Chat · 111B · 59.3Llama 4 Scout 17B 16E Instruct · 109B · 58.9Mixtral 8x22B Instruct v0.1 · 141B · 58.6Mixtral 8x7B Instruct v0.1 · 47B · 58.0Llama 2 70B Chat HF · 69B · 50.2Meta Llama 3 8B Instruct · 8B · 59.1Meta Llama 3 8B Instr…Gemma 4 31B IT · 31B · 61.1Gemma 4 31B ITQwen2.5 72B Instruct · 73B · 61.4Qwen2.5 72B InstructGLM 4.5 Air · 110B · 61.7GLM 4.5 AirQwen3 235B A22B · 235B · 63.9Qwen3 235B A22BMiniMax M3 · 427B · 64.0MiniMax M3Kimi K3 · 2.8T · 64.2Kimi K3
Each dot is a model. Up = higher score, left = smaller (easier to run locally). The dashed line marks the efficiency frontier — the best score you can get at each size or smaller.
  • Meta Llama 3 8B Instruct, 8B, score 59.1 — on the efficiency frontier (best score at its size or smaller).
  • Gemma 4 31B IT, 31B, score 61.1 — on the efficiency frontier (best score at its size or smaller).
  • Qwen2.5 72B Instruct, 73B, score 61.4 — on the efficiency frontier (best score at its size or smaller).
  • GLM 4.5 Air, 110B, score 61.7 — on the efficiency frontier (best score at its size or smaller).
  • Qwen3 235B A22B, 235B, score 63.9 — on the efficiency frontier (best score at its size or smaller).
  • MiniMax M3, 427B, score 64.0 — on the efficiency frontier (best score at its size or smaller).
  • Kimi K3, 2.8T, score 64.2 — on the efficiency frontier (best score at its size or smaller).

ForecastBench: frequently asked questions

What is the best open LLM on ForecastBench?
Kimi K3 is the top open model on ForecastBench, scoring 64.2. Among all models tested — including proprietary ones — it ranks #27. The top model overall is GPT 5.5 (high) (OpenAI) at 66.7.
What's the best ForecastBench model you can run on a 24 GB GPU?
Gemma 4 31B IT is the highest-scoring open model that fits in 24 GB at 4-bit quantization (~20 GB), scoring 61.1 on ForecastBench.
What's the best ForecastBench model you can run on a 12 GB GPU?
Meta Llama 3 8B Instruct is the highest-scoring open model that fits in 12 GB at 4-bit quantization (~5.3 GB), scoring 59.1 on ForecastBench.
Can open models match proprietary models on ForecastBench?
Not quite on ForecastBench: the strongest proprietary model (GPT 5.5 (high)) scores 66.7, ahead of the best open model (Kimi K3) at 64.2 — but you can run the open one yourself.

Scores aggregated from epoch. llmrun does not run this benchmark — see the source for methodology, or the about benchmarks for what it measures.