Coding
METR Time Horizons Leaderboard
METR Time Horizons, from the AI-evaluation nonprofit METR, runs models on a suite of software-engineering and research tasks whose length is measured by how long they take skilled humans, from seconds to many hours. METR summarises each model as a time horizon: the human task length at which the model succeeds half the time. llmrun shows METR's average task-success score (0–1) as published by Epoch AI; higher means the model completes more of the suite, including longer tasks.
Source: epoch9 open models ranked+37 proprietaryData through Apr 2026
All models ranked on METR Time Horizons
Proprietary / closed models are shown dimmed — you can't run them locally, but they show where the open field stands.
| # | Model | Score | VRAM @ Q4 |
|---|---|---|---|
| 1 | Claude Mythos Preview Early · proprietary | 85.2% | API only |
| 2 | Claude Opus 4.6 (unspecified) · proprietary | 78.9% | API only |
| 3 | Gemini 3.1 Pro Preview · proprietary | 77.0% | API only |
| 4 | GPT 5.2 (Dec 11, 2025, high) · proprietary | 75.3% | API only |
| 5 | Claude Opus 4.5 (Nov 01, 2025, 16K) · proprietary | 75.0% | API only |
| 6 | GPT 5.3 Codex · proprietary | 74.5% | API only |
| 7 | GPT 5.4 (Mar 05, 2026, xhigh) · proprietary | 74.3% | API only |
| 8 | Claude Opus 4.5 (Nov 01, 2025) · proprietary | 73.0% | API only |
| 9 | Gemini 3 Pro Preview · proprietary | 71.0% | API only |
| 10 | GPT 5.1 Codex Max · proprietary | 70.8% | API only |
| 11 | GPT 5 (Aug 07, 2025, medium) · proprietary | 69.6% | API only |
| 12 | GPT 5 (Aug 07, 2025, high) · proprietary | 69.4% | API only |
| 13 | Claude Sonnet 4.5 (Sep 29, 2025, 16K) · proprietary | 67.4% | API only |
| 14 | Claude Opus 4.1 (Aug 05, 2025, 16K) · proprietary | 66.8% | API only |
| 15 | Grok 4 (Jul 09) · proprietary | 66.6% | API only |
| 16 | O3 (Apr 16, 2025, medium) · proprietary | 65.4% | API only |
| 17 | Claude Opus 4 (May 14, 2025, 16K) · proprietary | 63.9% | API only |
| 18 | O4 Mini (Apr 16, 2025, medium) · proprietary | 63.9% | API only |
| 19 | O3 (Apr 16, 2025, unspecified) · proprietary | 63.6% | API only |
| 20 | Claude Sonnet 4 (May 14, 2025, 16K) · proprietary | 62.0% | API only |
| 21 | Claude Opus 4.1 (Aug 05, 2025) · proprietary | 61.6% | API only |
| 22 | Claude Opus 4 (May 14, 2025, unspecified) · proprietary | 61.6% | API only |
| 23 | Claude 3.7 Sonnet (Feb 19, 2025, 16K) · proprietary | 60.0% | API only |
| 24 | Kimi K2 Thinking · 1026.4B | 59.2% | ~620 GB |
| 25 | GPT OSS 120B · 116.8B | 56.6% | ~71 GB |
| 26 | Claude 3.7 Sonnet (Feb 19, 2025) · proprietary | 55.8% | API only |
| 27 | Gemini 2.5 Pro Preview (Jun 05) · proprietary | 55.4% | API only |
| 28 | DeepSeek R1 0528 · 684.5B | 53.8% | ~415 GB |
| 29 | DeepSeek R1 · 684.5B | 51.9% | ~415 GB |
| 30 | O1 (Dec 17, 2024, medium) · proprietary | 51.0% | API only |
| 31 | DeepSeek v3 0324 · 684.5B | 49.6% | ~415 GB |
| 32 | DeepSeek v3 · 684.5B | 47.4% | ~415 GB |
| 33 | Claude 3.5 Sonnet (Oct 22, 2024) · proprietary | 45.2% | API only |
| 34 | O1 Preview (Sep 12, 2024) · proprietary | 45.1% | API only |
| 35 | Claude 3.5 Sonnet (Jun 20, 2024) · proprietary | 40.2% | API only |
| 36 | Qwen2.5 72B Instruct · 72.7B | 35.8% | ~45 GB |
| 37 | GPT 4 0125 Preview · proprietary | 35.2% | API only |
| 38 | GPT 4o (May 13, 2024) · proprietary | 33.8% | API only |
| 39 | Qwen2 72B Instruct · 72.7B | 29.9% | ~45 GB |
| 40 | Claude 3 Opus (Feb 29, 2024) · proprietary | 29.5% | API only |
| 41 | GPT 4 (Mar 14) · proprietary | 29.3% | API only |
| 42 | GPT 4 1106 Preview · proprietary | 28.9% | API only |
| 43 | GPT 4 Turbo (Apr 09, 2024) · proprietary | 27.2% | API only |
| 44 | GPT 3.5 Turbo Instruct · proprietary | 21.5% | API only |
| 45 | Davinci 002 · proprietary | 16.2% | API only |
| 46 | Gpt2 Xl · 1.6B | 10.1% | ~1.1 GB |
Score vs model size
Which models give the most quality for their size — the ones worth running locally.
- Gpt2 Xl, 2B, score 10.1% — on the efficiency frontier (best score at its size or smaller).
- Qwen2.5 72B Instruct, 73B, score 35.8% — on the efficiency frontier (best score at its size or smaller).
- GPT OSS 120B, 117B, score 56.6% — on the efficiency frontier (best score at its size or smaller).
- Kimi K2 Thinking, 1T, score 59.2% — on the efficiency frontier (best score at its size or smaller).
METR Time Horizons: frequently asked questions
- What is the best open LLM on METR Time Horizons?
- Kimi K2 Thinking is the top open model on METR Time Horizons, scoring 59.2%. Among all models tested — including proprietary ones — it ranks #24. The top model overall is Claude Mythos Preview Early (Anthropic) at 85.2%.
- What's the best METR Time Horizons model you can run on a 24 GB GPU?
- Gpt2 Xl is the highest-scoring open model that fits in 24 GB at 4-bit quantization (~1.1 GB), scoring 10.1% on METR Time Horizons.
- What's the best METR Time Horizons model you can run on a 12 GB GPU?
- Gpt2 Xl is the highest-scoring open model that fits in 12 GB at 4-bit quantization (~1.1 GB), scoring 10.1% on METR Time Horizons.
- Can open models match proprietary models on METR Time Horizons?
- Not quite on METR Time Horizons: the strongest proprietary model (Claude Mythos Preview Early) scores 85.2%, ahead of the best open model (Kimi K2 Thinking) at 59.2% — but you can run the open one yourself.
Scores aggregated from epoch. llmrun does not run this benchmark — see the source for methodology, or the about benchmarks for what it measures.