llmrun
Reasoning

Vending-Bench 2 Leaderboard

Vending-Bench 2, from Andon Labs, is a long-horizon agent benchmark in which the model runs a simulated vending-machine business for a year, handling suppliers, pricing and cash flow. The score is the final money balance in US dollars.

Source: epoch17 open models ranked+55 proprietaryData through Nov 2026

All models ranked on Vending-Bench 2

Proprietary / closed models are shown dimmed — you can't run them locally, but they show where the open field stands.

#ModelScoreVRAM @ Q4
1Gemini 4 Carbon (unspecified) · proprietary
16484.2
API only
2GPT 6 Astra (unspecified) · proprietary
15514.7
API only
3GPT 6 Sol (unspecified) · proprietary
14427.8
API only
4Gemini 4 Argon (unspecified) · proprietary
13718.2
API only
5GPT 6.1 Sol (unspecified) · proprietary
12860.6
API only
6Claude Opus 5 (unspecified) · proprietary
11181.9
API only
7Claude Sonnet 5.5 (unspecified) · proprietary
11165.3
API only
8Claude Opus 4.7 (unspecified) · proprietary
10936.8
API only
9Grok 4.7 (unspecified) · proprietary
10536.8
API only
10GPT 5.6 Sol (unspecified) · proprietary
9619.4
API only
11Muse Wonder 1.0 (unspecified) · proprietary
9432.2
API only
12Claude Opus 5.5 (unspecified) · proprietary
9235.2
API only
13GPT 6.1 Terra (unspecified) · proprietary
9107.5
API only
14Grok 4.6 (unspecified) · proprietary
9047.0
API only
15GLM 5.2 · 753.3B
8313.8
~456 GB
16GLM 5.3 · 753.3B
8163.6
~456 GB
17Claude Opus 4.6 (unspecified) · proprietary
8017.6
API only
18GPT 5.5 (unspecified) · proprietary
7523.8
API only
19GPT 5.6 Terra (unspecified) · proprietary
7343.2
API only
20Claude Sonnet 4.6 (unspecified) · proprietary
7204.1
API only
21Muse Spark 1.1 · proprietary
6520.5
API only
22Claude Sonnet 5 (unspecified) · proprietary
6377.7
API only
23Kimi K2.6 · 1026.9B
6204.6
~620 GB
24GPT 5.4 (Mar 05, 2026, unspecified) · proprietary
6144.2
API only
25GPT 5.3 Codex · proprietary
5940.1
API only
26Claude Opus 4.8 (unspecified) · proprietary
5787.4
API only
27Claude Fable 5 (high) · proprietary
5680.3
API only
28GLM 5.1 · 753.9B
5634.4
~457 GB
29Gemini 3 Pro Preview · proprietary
5478.2
API only
30Claude Fable 5.1 (unspecified) · proprietary
5421.6
API only
31Gemini 3.5 Flash (unspecified) · proprietary
5396.4
API only
32Kimi K3 · 2779.9B
5165.0
~1674 GB
33Qwen3.6 Plus · proprietary
5114.9
API only
34Gemini 3.8 Flash (unspecified) · proprietary
5093.8
API only
35Kimi K2.7 Code · 1026.9B
5082.9
~620 GB
36Claude Fable 5 (low) · proprietary
5018.5
API only
37Claude Opus 4.5 (Nov 01, 2025, unspecified) · proprietary
4967.1
API only
38Claude Fable 5 Max · proprietary
4966.6
API only
39Grok 4.20 · proprietary
4662.8
API only
40Claude Fable 5 · proprietary
4529.9
API only
41GLM 5 · 753.9B
4432.1
~457 GB
42Claude Fable 5 (medium) · proprietary
4339.8
API only
43Qwen3.6 Max Preview · proprietary
4254.2
API only
44GPT 5.6 Luna (unspecified) · proprietary
4094.7
API only
45Grok 4.5 (unspecified) · proprietary
3887.4
API only
46Claude Sonnet 4.5 (Sep 29, 2025, unspecified) · proprietary
3838.7
API only
47Gemini 3.1 Pro Preview Customtools · proprietary
3774.3
API only
48Gemini 3 Flash Preview · proprietary
3634.7
API only
49Claude Haiku 5.5 (unspecified) · proprietary
3602.5
API only
50GPT 5.2 (Dec 11, 2025, unspecified) · proprietary
3591.3
API only
51DeepSeek V4 Pro · 1598.8B
3284.5
~960 GB
52Claude Opus 4.8 Max · proprietary
2992.3
API only
53GLM 4.7 · 358.3B
2376.8
~216 GB
54MiniMax M3 · 427.0B
2157.8
~257 GB
55GPT 5.1 (Nov 13, 2025, unspecified) · proprietary
1473.4
API only
56Kimi K2.5 · 1026.9B
1198.5
~620 GB
57Grok 4.1 Fast Reasoning · proprietary
1106.6
API only
58DeepSeek V3.2 Exp · 685.4B
1034.0
~415 GB
59Gemini 3.1 Pro Preview · proprietary
911.2
API only
60Gemini 2.5 Pro · proprietary
573.6
API only
61Gemini 2.5 Flash · proprietary
548.8
API only
62Qwen3.5 Flash · proprietary
462.7
API only
63Claude Haiku 4.5 (Oct 01, 2025, unspecified) · proprietary
458.9
API only
64Qwen3.5 27B · 27.8B
202.0
~17 GB
65MiniMax M2 · 228.7B
160.6
~138 GB
66Qwen3 Max (Sep 23, 2025) · proprietary
71.6
API only
67Grok 4.3 (unspecified) · proprietary
35.3
API only
68Qwen3.5 Plus · proprietary
0.5
API only
69Qwen3 235B A22B Thinking 2507 · 235.1B
-11.3
~142 GB
70GPT OSS 120B · 116.8B
-21.5
~71 GB
71MiniMax M2.5 · 228.7B
-23.2
~138 GB
72GPT 5 Mini (Aug 07, 2025, unspecified) · proprietary
-31.2
API only

Score vs model size

Which models give the most quality for their size — the ones worth running locally.

100B1Tmodel size (log scale) →8313.8-23.2GLM 5.3 · 753B · 8163.6Kimi K2.6 · 1T · 6204.6GLM 5.1 · 754B · 5634.4Kimi K3 · 2.8T · 5165.0Kimi K2.7 Code · 1T · 5082.9GLM 5 · 754B · 4432.1DeepSeek V4 Pro · 1.6T · 3284.5MiniMax M3 · 427B · 2157.8Kimi K2.5 · 1T · 1198.5DeepSeek V3.2 Exp · 685B · 1034.0MiniMax M2 · 229B · 160.6Qwen3 235B A22B Thinking 2507 · 235B · -11.3GPT OSS 120B · 117B · -21.5MiniMax M2.5 · 229B · -23.2Qwen3.5 27B · 28B · 202.0Qwen3.5 27BGLM 4.7 · 358B · 2376.8GLM 4.7GLM 5.2 · 753B · 8313.8GLM 5.2
Each dot is a model. Up = higher score, left = smaller (easier to run locally). The dashed line marks the efficiency frontier — the best score you can get at each size or smaller.
  • Qwen3.5 27B, 28B, score 202.0 — on the efficiency frontier (best score at its size or smaller).
  • GLM 4.7, 358B, score 2376.8 — on the efficiency frontier (best score at its size or smaller).
  • GLM 5.2, 753B, score 8313.8 — on the efficiency frontier (best score at its size or smaller).

Vending-Bench 2: frequently asked questions

What is the best open LLM on Vending-Bench 2?
GLM 5.2 is the top open model on Vending-Bench 2, scoring 8313.8. Among all models tested — including proprietary ones — it ranks #15. The top model overall is Gemini 4 Carbon (unspecified) (Google DeepMind) at 16484.2.
What's the best Vending-Bench 2 model you can run on a 24 GB GPU?
Qwen3.5 27B is the highest-scoring open model that fits in 24 GB at 4-bit quantization (~17 GB), scoring 202.0 on Vending-Bench 2.
Can open models match proprietary models on Vending-Bench 2?
Not quite on Vending-Bench 2: the strongest proprietary model (Gemini 4 Carbon (unspecified)) scores 16484.2, ahead of the best open model (GLM 5.2) at 8313.8 — but you can run the open one yourself.

Scores aggregated from epoch. llmrun does not run this benchmark — see the source for methodology, or the about benchmarks for what it measures.