llmrun
Reasoning

APEX-Agents Leaderboard

APEX-Agents (AI Productivity Index) is Mercor's evaluation of AI agents on long-horizon, multi-application professional-services work — investment-banking, management-consulting, and corporate-law style tasks — graded against expert-produced solutions. The score is Pass@1, the fraction solved correctly on the first attempt.

Source: epoch20 open models ranked+40 proprietaryData through Oct 2026

Open modelsAll models

All models ranked on APEX-Agents

Proprietary / closed models are shown dimmed — you can't run them locally, but they show where the open field stands.

#ModelScoreVRAM @ Q4
1Gemini 4 Argon (unspecified) · proprietary
82.4%
API only
2Claude Sonnet 5.5 Max · proprietary
75.5%
API only
3Claude Opus 5.5 Max · proprietary
73.5%
API only
4DeepSeek V4.1 Flash 1017 · 763.2B
69.8%
~458 GB
5Claude Fable 5.1 (unspecified) · proprietary
68.6%
API only
6GPT 6.1 Terra Max · proprietary
68.3%
API only
7Gemini 3.7 Flash (unspecified) · proprietary
67.8%
API only
8Muse Wonder 1.0 Max · proprietary
66.1%
API only
9Claude Opus 5 Max · proprietary
65.8%
API only
10Grok 4.6 (unspecified) · proprietary
65.3%
API only
11GPT 6 Astra (unspecified) · proprietary
64.7%
API only
12Gemini 3.8 Flash (unspecified) · proprietary
64.3%
API only
13Claude Fable 5 · proprietary
63.6%
API only
14Qwen3.8 Max (unspecified) · proprietary
63.3%
API only
15GPT 6.1 Sol Max · proprietary
60.0%
API only
16Claude Fable 5.1 (high) · proprietary
59.7%
API only
17Muse Spark 1.3 (unspecified) · proprietary
58.6%
API only
18GPT 5.6 Terra Max · proprietary
58.2%
API only
19GLM 5.3 · 753.3B
56.6%
~456 GB
20Grok 4.5 (unspecified) · proprietary
56.2%
API only
21Claude Haiku 5.5 Max · proprietary
56.1%
API only
22GPT 5.5 (unspecified) · proprietary
55.1%
API only
23Grok 4.7 (unspecified) · proprietary
54.6%
API only
24Claude Sonnet 5 (unspecified) · proprietary
54.5%
API only
25MiMo V2.6 Pro RL · 1020B
54.4%
~613 GB
26GPT 6 Sol Max · proprietary
54.3%
API only
27GLM 5.3 Flash · 321.3B
52.8%
~195 GB
28Claude Opus 5.5 (medium) · proprietary
52.5%
API only
29GPT 5.4 (Mar 05, 2026, unspecified) · proprietary
52.4%
API only
30Mistral Large 4 1T A52B · 1000B
51.6%
~604 GB
31GPT 5.6 Sol Promax · proprietary
51.4%
API only
32Claude Opus 4.7 Max · proprietary
49.2%
API only
33Claude Opus 4.8 Max · proprietary
48.9%
API only
34Beam 1.0 501B A23B · 501.3B
48.8%
~301 GB
35Kimi K3 · 2779.9B
47.8%
~1674 GB
36Muse Spark 1.3 Max · proprietary
47.6%
API only
37Qwen3.8 27B · 27.8B
47.5%
~17 GB
38DeepSeek V4 Pro 0813 · 1650.5B
47.3%
~991 GB
39Gemini 3.6 Flash (unspecified) · proprietary
46.9%
API only
40Claude Opus 4.6 Max · proprietary
46.3%
API only
41Muse Spark 1.2 · proprietary
45.4%
API only
42GLM 5.2 · 753.3B
45.2%
~456 GB
43Claude Sonnet 5.5 (medium) · proprietary
44.6%
API only
44GPT 6 Luna Max · proprietary
44.3%
API only
45Claude Sonnet 4.6 (high) · proprietary
43.0%
API only
46GPT 5.6 Luna (unspecified) · proprietary
43.0%
API only
47GLM 5.1 · 753.9B
40.9%
~457 GB
48DeepSeek V4.1 Flash · 763.2B
39.5%
~458 GB
49MiniMax M3 · 427.0B
37.7%
~257 GB
50Kimi K2.7 Code · 1026.9B
37.6%
~620 GB
51Gemini 3.1 Pro Preview · proprietary
35.1%
API only
52Inkling · 952.4B
33.8%
~572 GB
53Muse Spark Open 816B A42B · 822.0B
33.4%
~543 GB
54Muse Spark 1.1 · proprietary
31.8%
API only
55Gemini 3.5 Flash Lite · proprietary
29.3%
API only
56Gemini 3.5 Flash (unspecified) · proprietary
27.5%
API only
57Qwen3.5 397B A17B · 403.4B
24.9%
~242 GB
58NVIDIA Nemotron 3 Ultra 550B A55B BF16 · 560.5B
22.7%
~370 GB
59DeepSeek V3.2 · 685.4B
21.3%
~415 GB
60GPT OSS 120B · 116.8B
4.4%
~71 GB

Score vs model size

Which models give the most quality for their size — the ones worth running locally.

100B1Tmodel size (log scale) →69.8%4.4%MiMo V2.6 Pro RL · 1T · 54.4%Mistral Large 4 1T A52B · 1T · 51.6%Beam 1.0 501B A23B · 501B · 48.8%Kimi K3 · 2.8T · 47.8%DeepSeek V4 Pro 0813 · 1.7T · 47.3%GLM 5.2 · 753B · 45.2%GLM 5.1 · 754B · 40.9%DeepSeek V4.1 Flash · 763B · 39.5%MiniMax M3 · 427B · 37.7%Kimi K2.7 Code · 1T · 37.6%Inkling · 952B · 33.8%Muse Spark Open 816B A42B · 822B · 33.4%Qwen3.5 397B A17B · 403B · 24.9%NVIDIA Nemotron 3 Ultra 550B A55B BF16 · 561B · 22.7%DeepSeek V3.2 · 685B · 21.3%GPT OSS 120B · 117B · 4.4%Qwen3.8 27B · 28B · 47.5%Qwen3.8 27BGLM 5.3 Flash · 321B · 52.8%GLM 5.3 FlashGLM 5.3 · 753B · 56.6%GLM 5.3DeepSeek V4.1 Flash 1017 · 763B · 69.8%DeepSeek V4.1 Flash 1…
Each dot is a model. Up = higher score, left = smaller (easier to run locally). The dashed line marks the efficiency frontier — the best score you can get at each size or smaller.
  • Qwen3.8 27B, 28B, score 47.5% — on the efficiency frontier (best score at its size or smaller).
  • GLM 5.3 Flash, 321B, score 52.8% — on the efficiency frontier (best score at its size or smaller).
  • GLM 5.3, 753B, score 56.6% — on the efficiency frontier (best score at its size or smaller).
  • DeepSeek V4.1 Flash 1017, 763B, score 69.8% — on the efficiency frontier (best score at its size or smaller).

APEX-Agents: frequently asked questions

What is the best open LLM on APEX-Agents?
DeepSeek V4.1 Flash 1017 is the top open model on APEX-Agents, scoring 69.8%. Among all models tested — including proprietary ones — it ranks #4. The top model overall is Gemini 4 Argon (unspecified) (Google DeepMind) at 82.4%.
What's the best APEX-Agents model you can run on a 24 GB GPU?
Qwen3.8 27B is the highest-scoring open model that fits in 24 GB at 4-bit quantization (~17 GB), scoring 47.5% on APEX-Agents.
Can open models match proprietary models on APEX-Agents?
Not quite on APEX-Agents: the strongest proprietary model (Gemini 4 Argon (unspecified)) scores 82.4%, ahead of the best open model (DeepSeek V4.1 Flash 1017) at 69.8% — but you can run the open one yourself.

Scores aggregated from epoch. llmrun does not run this benchmark — see the source for methodology, or the about benchmarks for what it measures.