llmrun
Reasoning

Vending-Bench 2 Leaderboard

Vending-Bench 2, from Andon Labs, is a long-horizon agent benchmark in which the model runs a simulated vending-machine business for a year, handling suppliers, pricing and cash flow. The score is the final money balance in US dollars.

Source: epoch17 open models ranked+55 proprietaryData through Nov 2026

Open models ranked on Vending-Bench 2

# shows rank among open models / rank overall (including proprietary).

#ModelScoreVRAM @ Q4
1 / 15GLM 5.2 · 753.3B
8313.8
~456 GB
2 / 16GLM 5.3 · 753.3B
8163.6
~456 GB
3 / 23Kimi K2.6 · 1026.9B
6204.6
~620 GB
4 / 28GLM 5.1 · 753.9B
5634.4
~457 GB
5 / 32Kimi K3 · 2779.9B
5165.0
~1674 GB
6 / 35Kimi K2.7 Code · 1026.9B
5082.9
~620 GB
7 / 41GLM 5 · 753.9B
4432.1
~457 GB
8 / 51DeepSeek V4 Pro · 1598.8B
3284.5
~960 GB
9 / 53GLM 4.7 · 358.3B
2376.8
~216 GB
10 / 54MiniMax M3 · 427.0B
2157.8
~257 GB
11 / 56Kimi K2.5 · 1026.9B
1198.5
~620 GB
12 / 58DeepSeek V3.2 Exp · 685.4B
1034.0
~415 GB
13 / 64Qwen3.5 27B · 27.8B
202.0
~17 GB
14 / 65MiniMax M2 · 228.7B
160.6
~138 GB
15 / 69Qwen3 235B A22B Thinking 2507 · 235.1B
-11.3
~142 GB
16 / 70GPT OSS 120B · 116.8B
-21.5
~71 GB
17 / 71MiniMax M2.5 · 228.7B
-23.2
~138 GB

Score vs model size

Which models give the most quality for their size — the ones worth running locally.

100B1Tmodel size (log scale) →8313.8-23.2GLM 5.3 · 753B · 8163.6Kimi K2.6 · 1T · 6204.6GLM 5.1 · 754B · 5634.4Kimi K3 · 2.8T · 5165.0Kimi K2.7 Code · 1T · 5082.9GLM 5 · 754B · 4432.1DeepSeek V4 Pro · 1.6T · 3284.5MiniMax M3 · 427B · 2157.8Kimi K2.5 · 1T · 1198.5DeepSeek V3.2 Exp · 685B · 1034.0MiniMax M2 · 229B · 160.6Qwen3 235B A22B Thinking 2507 · 235B · -11.3GPT OSS 120B · 117B · -21.5MiniMax M2.5 · 229B · -23.2Qwen3.5 27B · 28B · 202.0Qwen3.5 27BGLM 4.7 · 358B · 2376.8GLM 4.7GLM 5.2 · 753B · 8313.8GLM 5.2
Each dot is a model. Up = higher score, left = smaller (easier to run locally). The dashed line marks the efficiency frontier — the best score you can get at each size or smaller.
  • Qwen3.5 27B, 28B, score 202.0 — on the efficiency frontier (best score at its size or smaller).
  • GLM 4.7, 358B, score 2376.8 — on the efficiency frontier (best score at its size or smaller).
  • GLM 5.2, 753B, score 8313.8 — on the efficiency frontier (best score at its size or smaller).

Vending-Bench 2: frequently asked questions

What is the best open LLM on Vending-Bench 2?
GLM 5.2 is the top open model on Vending-Bench 2, scoring 8313.8. Among all models tested — including proprietary ones — it ranks #15. The top model overall is Gemini 4 Carbon (unspecified) (Google DeepMind) at 16484.2.
What's the best Vending-Bench 2 model you can run on a 24 GB GPU?
Qwen3.5 27B is the highest-scoring open model that fits in 24 GB at 4-bit quantization (~17 GB), scoring 202.0 on Vending-Bench 2.
Can open models match proprietary models on Vending-Bench 2?
Not quite on Vending-Bench 2: the strongest proprietary model (Gemini 4 Carbon (unspecified)) scores 16484.2, ahead of the best open model (GLM 5.2) at 8313.8 — but you can run the open one yourself.

Scores aggregated from epoch. llmrun does not run this benchmark — see the source for methodology, or the about benchmarks for what it measures.