llmrun
Coding

WeirdML Leaderboard

WeirdML asks models to write machine-learning code for small, deliberately unusual datasets, then runs the code and scores the accuracy it actually achieves. Built by Håvard Tveit Ihle, it rewards practical problem-solving over recognizing a familiar task.

Source: epoch46 open models ranked+115 proprietaryData through Sep 2026

Open modelsAll models

Open models ranked on WeirdML

# shows rank among open models / rank overall (including proprietary).

#ModelScore
1 / 17Kimi K3 · 2779.9B
82.6%
2 / 29GLM 5.3 · 753.3B
75.4%
3 / 33GLM 5.2 · 753.3B
70.1%
4 / 38DeepSeek V4 Pro 0813 · 1650.5B
66.2%
5 / 43DeepSeek V4 Flash 0731 · 304.2B
63.0%
6 / 55GLM 5.1 · 753.9B
57.1%
7 / 57Kimi K2.6 · 1026.9B
55.9%
8 / 60Kimi K2.7 Code · 1026.9B
54.1%
9 / 65Gemma 4 31B IT · 31.3B
52.3%
10 / 72DeepSeek V4 Pro · 1598.8B
48.9%
11 / 73GLM 5 · 753.9B
48.2%
12 / 74GPT OSS 120B · 116.8B
48.2%
13 / 77DeepSeek V3.2 Speciale · 685.4B
46.7%
14 / 84DeepSeek V4 Flash · 290.9B
45.6%
15 / 85Kimi K2.5 · 1026.9B
45.6%
16 / 92NVIDIA Nemotron 3 Ultra 550B A55B BF16 · 560.5B
43.5%
17 / 95Kimi K2 Thinking · 1026.4B
42.8%
18 / 99DeepSeek R1 0528 · 684.5B
41.6%
19 / 100Qwen3 Coder 480B A35B Instruct · 480.2B
41.2%
20 / 101Qwen3 235B A22B Thinking 2507 · 235.1B
41.0%
21 / 104GPT OSS 20B · 20.9B
40.9%
22 / 105GLM 4.5 · 358.3B
40.6%
23 / 108Qwen3.5 27B · 27.8B
39.5%
24 / 109DeepSeek V3.2 Exp · 685.4B
39.5%
25 / 111Kimi K2 Instruct · 1026.4B
39.4%
26 / 114Qwen3 235B A22B Instruct 2507 · 235.1B
38.7%
27 / 115DeepSeek V3.1 · 684.5B
38.4%
28 / 117NVIDIA Nemotron 3 Super 120B A12B BF16 · 123.6B
38.0%
29 / 121Qwen3 235B A22B · 235.1B
37.3%
30 / 123MiniMax M2.7 · 228.7B
37.0%
31 / 124Kimi K2 Instruct 0905 · 1026.5B
36.7%
32 / 125DeepSeek R1 · 684.5B
36.5%
33 / 127DeepSeek v3 0324 · 684.5B
36.1%
34 / 130Gemma 4 26B A4B IT · 25.8B
35.2%
35 / 132Qwen3.6 35B A3B · 36.0B
34.5%
36 / 133Qwen3 Coder Next · 79.7B
34.4%
37 / 135Inkling · 952.4B
32.3%
38 / 138Qwen3 30B A3B · 30.5B
29.8%
39 / 143Llama 4 Maverick 17B 128E Instruct · 401.6B
24.5%
40 / 146Llama 3.1 405B Instruct · 405.9B
21.4%
41 / 150Qwen2.5 72B Instruct · 72.7B
16.0%
42 / 151Llama 3.3 70B Instruct · 70.6B
14.4%
43 / 154Qwen2 72B Instruct · 72.7B
11.3%
44 / 157Llama 3.1 70B Instruct · 70.6B
9.0%
45 / 160Mixtral 8x22B Instruct v0.1 · 140.6B
3.2%
46 / 161Llama 3.1 8B Instruct · 8.0B
1.7%

Score vs model size

Which models give the most quality for their size — the ones worth running locally.

10B100B1Tmodel size (log scale) →82.6%1.7%GLM 5.2 · 753B · 70.1%DeepSeek V4 Pro 0813 · 1.7T · 66.2%GLM 5.1 · 754B · 57.1%Kimi K2.6 · 1T · 55.9%Kimi K2.7 Code · 1T · 54.1%DeepSeek V4 Pro · 1.6T · 48.9%GPT OSS 120B · 117B · 48.2%GLM 5 · 754B · 48.2%DeepSeek V3.2 Speciale · 685B · 46.7%DeepSeek V4 Flash · 291B · 45.6%Kimi K2.5 · 1T · 45.6%NVIDIA Nemotron 3 Ultra 550B A55B BF16 · 561B · 43.5%Kimi K2 Thinking · 1T · 42.8%DeepSeek R1 0528 · 685B · 41.6%Qwen3 Coder 480B A35B Instruct · 480B · 41.2%Qwen3 235B A22B Thinking 2507 · 235B · 41.0%GLM 4.5 · 358B · 40.6%Qwen3.5 27B · 28B · 39.5%DeepSeek V3.2 Exp · 685B · 39.5%Kimi K2 Instruct · 1T · 39.4%Qwen3 235B A22B Instruct 2507 · 235B · 38.7%DeepSeek V3.1 · 685B · 38.4%NVIDIA Nemotron 3 Super 120B A12B BF16 · 124B · 38.0%Qwen3 235B A22B · 235B · 37.3%MiniMax M2.7 · 229B · 37.0%Kimi K2 Instruct 0905 · 1T · 36.7%DeepSeek R1 · 684B · 36.5%DeepSeek v3 0324 · 685B · 36.1%Gemma 4 26B A4B IT · 26B · 35.2%Qwen3.6 35B A3B · 36B · 34.5%Qwen3 Coder Next · 80B · 34.4%Inkling · 952B · 32.3%Qwen3 30B A3B · 31B · 29.8%Llama 4 Maverick 17B 128E Instruct · 402B · 24.5%Llama 3.1 405B Instruct · 406B · 21.4%Qwen2.5 72B Instruct · 73B · 16.0%Llama 3.3 70B Instruct · 71B · 14.4%Qwen2 72B Instruct · 73B · 11.3%Llama 3.1 70B Instruct · 71B · 9.0%Mixtral 8x22B Instruct v0.1 · 141B · 3.2%Llama 3.1 8B Instruct · 8B · 1.7%Llama 3.1 8B InstructGPT OSS 20B · 21B · 40.9%GPT OSS 20BGemma 4 31B IT · 31B · 52.3%Gemma 4 31B ITDeepSeek V4 Flash 0731 · 304B · 63.0%DeepSeek V4 Flash 0731GLM 5.3 · 753B · 75.4%GLM 5.3Kimi K3 · 2.8T · 82.6%Kimi K3
Each dot is a model. Up = higher score, left = smaller (easier to run locally). The dashed line marks the efficiency frontier — the best score you can get at each size or smaller.
  • Llama 3.1 8B Instruct, 8B, score 1.7% — on the efficiency frontier (best score at its size or smaller).
  • GPT OSS 20B, 21B, score 40.9% — on the efficiency frontier (best score at its size or smaller).
  • Gemma 4 31B IT, 31B, score 52.3% — on the efficiency frontier (best score at its size or smaller).
  • DeepSeek V4 Flash 0731, 304B, score 63.0% — on the efficiency frontier (best score at its size or smaller).
  • GLM 5.3, 753B, score 75.4% — on the efficiency frontier (best score at its size or smaller).
  • Kimi K3, 2.8T, score 82.6% — on the efficiency frontier (best score at its size or smaller).

WeirdML: frequently asked questions

What is the best open LLM on WeirdML?
Kimi K3 is the top open model on WeirdML, scoring 82.6%. Among all models tested — including proprietary ones — it ranks #17. The top model overall is GPT 6 Astra Promax (OpenAI) at 93.6%.
What's the best WeirdML model you can run on a 24 GB GPU?
Gemma 4 31B IT is the highest-scoring open model that fits in 24 GB at 4-bit quantization (about 17 GB), scoring 52.3% on WeirdML.
What's the best WeirdML model you can run on a 12 GB GPU?
GPT OSS 20B is the highest-scoring open model that fits in 12 GB at 4-bit quantization (about 12 GB), scoring 40.9% on WeirdML.
Can open models match proprietary models on WeirdML?
Not quite on WeirdML: the strongest proprietary model (GPT 6 Astra Promax) scores 93.6%, ahead of the best open model (Kimi K3) at 82.6% — but you can run the open one yourself.

Scores aggregated from epoch. llmrun does not run this benchmark — see the source for methodology, or the about benchmarks for what it measures.