llmrun
Coding

METR Time Horizons Leaderboard

METR Time Horizons, from the AI-evaluation nonprofit METR, runs models on a suite of software-engineering and research tasks whose length is measured by how long they take skilled humans, from seconds to many hours. METR summarises each model as a time horizon: the human task length at which the model succeeds half the time. llmrun shows METR's average task-success score (0–1) as published by Epoch AI; higher means the model completes more of the suite, including longer tasks.

Source: epoch9 open models ranked+37 proprietaryData through Apr 2026

Open modelsAll models

Open models ranked on METR Time Horizons

# shows rank among open models / rank overall (including proprietary).

#ModelScoreVRAM @ Q4
1 / 24Kimi K2 Thinking · 1026.4B
59.2%
~620 GB
2 / 25GPT OSS 120B · 116.8B
56.6%
~71 GB
3 / 28DeepSeek R1 0528 · 684.5B
53.8%
~415 GB
4 / 29DeepSeek R1 · 684.5B
51.9%
~415 GB
5 / 31DeepSeek v3 0324 · 684.5B
49.6%
~415 GB
6 / 32DeepSeek v3 · 684.5B
47.4%
~415 GB
7 / 36Qwen2.5 72B Instruct · 72.7B
35.8%
~45 GB
8 / 39Qwen2 72B Instruct · 72.7B
29.9%
~45 GB
9 / 46Gpt2 Xl · 1.6B
10.1%
~1.1 GB

Score vs model size

Which models give the most quality for their size — the ones worth running locally.

10B100B1Tmodel size (log scale) →59.2%10.1%DeepSeek R1 0528 · 685B · 53.8%DeepSeek R1 · 684B · 51.9%DeepSeek v3 0324 · 685B · 49.6%DeepSeek v3 · 685B · 47.4%Qwen2 72B Instruct · 73B · 29.9%Gpt2 Xl · 2B · 10.1%Gpt2 XlQwen2.5 72B Instruct · 73B · 35.8%Qwen2.5 72B InstructGPT OSS 120B · 117B · 56.6%GPT OSS 120BKimi K2 Thinking · 1T · 59.2%Kimi K2 Thinking
Each dot is a model. Up = higher score, left = smaller (easier to run locally). The dashed line marks the efficiency frontier — the best score you can get at each size or smaller.
  • Gpt2 Xl, 2B, score 10.1% — on the efficiency frontier (best score at its size or smaller).
  • Qwen2.5 72B Instruct, 73B, score 35.8% — on the efficiency frontier (best score at its size or smaller).
  • GPT OSS 120B, 117B, score 56.6% — on the efficiency frontier (best score at its size or smaller).
  • Kimi K2 Thinking, 1T, score 59.2% — on the efficiency frontier (best score at its size or smaller).

METR Time Horizons: frequently asked questions

What is the best open LLM on METR Time Horizons?
Kimi K2 Thinking is the top open model on METR Time Horizons, scoring 59.2%. Among all models tested — including proprietary ones — it ranks #24. The top model overall is Claude Mythos Preview Early (Anthropic) at 85.2%.
What's the best METR Time Horizons model you can run on a 24 GB GPU?
Gpt2 Xl is the highest-scoring open model that fits in 24 GB at 4-bit quantization (~1.1 GB), scoring 10.1% on METR Time Horizons.
What's the best METR Time Horizons model you can run on a 12 GB GPU?
Gpt2 Xl is the highest-scoring open model that fits in 12 GB at 4-bit quantization (~1.1 GB), scoring 10.1% on METR Time Horizons.
Can open models match proprietary models on METR Time Horizons?
Not quite on METR Time Horizons: the strongest proprietary model (Claude Mythos Preview Early) scores 85.2%, ahead of the best open model (Kimi K2 Thinking) at 59.2% — but you can run the open one yourself.

Scores aggregated from epoch. llmrun does not run this benchmark — see the source for methodology, or the about benchmarks for what it measures.