llmrun
Reasoning

ForecastBench Leaderboard

ForecastBench, from the Forecasting Research Institute, asks models to predict real-world future events, with questions that only resolve after the model's training cutoff so the answers can't have been memorised. It is scored as a difficulty-adjusted Brier index, where higher is better.

Source: epoch24 open models ranked+71 proprietaryData through Sep 2026

Open models ranked on ForecastBench

# shows rank among open models / rank overall (including proprietary).

#ModelScoreVRAM @ Q4
1 / 29Kimi K3 · 2779.9B
64.2
~1674 GB
2 / 35MiniMax M3 · 427.0B
64.0
~257 GB
3 / 40Qwen3 235B A22B · 235.1B
63.9
~142 GB
4 / 44GLM 5 · 753.9B
63.4
~457 GB
5 / 59DeepSeek v3 · 684.5B
62.4
~415 GB
6 / 61Kimi K2 Instruct 0905 · 1026.4B
62.4
~620 GB
7 / 62Llama 3.1 405B Instruct · 405.9B
62.3
~268 GB
8 / 66Kimi K2 Instruct · 1026.4B
62.0
~620 GB
9 / 67Qwen3 235B A22B Thinking 2507 · 235.1B
62.0
~142 GB
10 / 68GLM 4.5 Air · 110.5B
61.7
~67 GB
11 / 70Mistral Large Instruct 2407 · 122.6B
61.5
~81 GB
12 / 71Qwen2.5 72B Instruct · 72.7B
61.4
~45 GB
13 / 72Mistral Large Instruct 2411 · 122.6B
61.3
~75 GB
14 / 74Gemma 4 31B IT · 31.3B
61.1
~20 GB
15 / 76Llama 3.3 70B Instruct · 70.6B
61.0
~47 GB
16 / 78DeepSeek V4 Pro · 1598.8B
60.7
~960 GB
17 / 79DeepSeek V3.1 · 684.5B
60.5
~415 GB
18 / 81Meta Llama 3 70B · 70.6B
59.9
~47 GB
19 / 82Qwen1.5 110B Chat · 111.2B
59.3
~68 GB
20 / 83Meta Llama 3 8B Instruct · 8.0B
59.1
~5.3 GB
21 / 84Llama 4 Scout 17B 16E Instruct · 108.6B
58.9
~72 GB
22 / 88Mixtral 8x22B Instruct v0.1 · 140.6B
58.6
~85 GB
23 / 91Mixtral 8x7B Instruct v0.1 · 46.7B
58.0
~29 GB
24 / 94Llama 2 70B Chat HF · 69.0B
50.2
~152 GB

Score vs model size

Which models give the most quality for their size — the ones worth running locally.

10B100B1Tmodel size (log scale) →64.250.2GLM 5 · 754B · 63.4DeepSeek v3 · 684B · 62.4Kimi K2 Instruct 0905 · 1T · 62.4Llama 3.1 405B Instruct · 406B · 62.3Kimi K2 Instruct · 1T · 62.0Qwen3 235B A22B Thinking 2507 · 235B · 62.0Mistral Large Instruct 2407 · 123B · 61.5Mistral Large Instruct 2411 · 123B · 61.3Llama 3.3 70B Instruct · 71B · 61.0DeepSeek V4 Pro · 1.6T · 60.7DeepSeek V3.1 · 685B · 60.5Meta Llama 3 70B · 71B · 59.9Qwen1.5 110B Chat · 111B · 59.3Llama 4 Scout 17B 16E Instruct · 109B · 58.9Mixtral 8x22B Instruct v0.1 · 141B · 58.6Mixtral 8x7B Instruct v0.1 · 47B · 58.0Llama 2 70B Chat HF · 69B · 50.2Meta Llama 3 8B Instruct · 8B · 59.1Meta Llama 3 8B Instr…Gemma 4 31B IT · 31B · 61.1Gemma 4 31B ITQwen2.5 72B Instruct · 73B · 61.4Qwen2.5 72B InstructGLM 4.5 Air · 110B · 61.7GLM 4.5 AirQwen3 235B A22B · 235B · 63.9Qwen3 235B A22BMiniMax M3 · 427B · 64.0MiniMax M3Kimi K3 · 2.8T · 64.2Kimi K3
Each dot is a model. Up = higher score, left = smaller (easier to run locally). The dashed line marks the efficiency frontier — the best score you can get at each size or smaller.
  • Meta Llama 3 8B Instruct, 8B, score 59.1 — on the efficiency frontier (best score at its size or smaller).
  • Gemma 4 31B IT, 31B, score 61.1 — on the efficiency frontier (best score at its size or smaller).
  • Qwen2.5 72B Instruct, 73B, score 61.4 — on the efficiency frontier (best score at its size or smaller).
  • GLM 4.5 Air, 110B, score 61.7 — on the efficiency frontier (best score at its size or smaller).
  • Qwen3 235B A22B, 235B, score 63.9 — on the efficiency frontier (best score at its size or smaller).
  • MiniMax M3, 427B, score 64.0 — on the efficiency frontier (best score at its size or smaller).
  • Kimi K3, 2.8T, score 64.2 — on the efficiency frontier (best score at its size or smaller).

ForecastBench: frequently asked questions

What is the best open LLM on ForecastBench?
Kimi K3 is the top open model on ForecastBench, scoring 64.2. Among all models tested — including proprietary ones — it ranks #27. The top model overall is GPT 5.5 (high) (OpenAI) at 66.7.
What's the best ForecastBench model you can run on a 24 GB GPU?
Gemma 4 31B IT is the highest-scoring open model that fits in 24 GB at 4-bit quantization (~20 GB), scoring 61.1 on ForecastBench.
What's the best ForecastBench model you can run on a 12 GB GPU?
Meta Llama 3 8B Instruct is the highest-scoring open model that fits in 12 GB at 4-bit quantization (~5.3 GB), scoring 59.1 on ForecastBench.
Can open models match proprietary models on ForecastBench?
Not quite on ForecastBench: the strongest proprietary model (GPT 5.5 (high)) scores 66.7, ahead of the best open model (Kimi K3) at 64.2 — but you can run the open one yourself.

Scores aggregated from epoch. llmrun does not run this benchmark — see the source for methodology, or the about benchmarks for what it measures.