Coding
WeirdML Leaderboard
WeirdML asks models to write machine-learning code for small, deliberately unusual datasets, then runs the code and scores the accuracy it actually achieves. Built by Håvard Tveit Ihle, it rewards practical problem-solving over recognizing a familiar task.
Source: epoch47 open models ranked+115 proprietaryData through Sep 2026
Open models ranked on WeirdML
# shows rank among open models / rank overall (including proprietary).
| # | Model | Score | VRAM @ Q4 |
|---|---|---|---|
| 1 / 18 | Kimi K3 · 2779.9B | 82.6% | ~1674 GB |
| 2 / 30 | GLM 5.3 · 753.3B | 75.4% | ~456 GB |
| 3 / 34 | GLM 5.2 · 753.3B | 70.1% | ~456 GB |
| 4 / 39 | DeepSeek V4 Pro 0813 · 1650.5B | 66.2% | ~991 GB |
| 5 / 44 | DeepSeek V4 Flash 0731 · 304.2B | 63.0% | ~183 GB |
| 6 / 56 | GLM 5.1 · 753.9B | 57.1% | ~457 GB |
| 7 / 58 | Kimi K2.6 · 1026.9B | 55.9% | ~620 GB |
| 8 / 61 | Kimi K2.7 Code · 1026.9B | 54.1% | ~620 GB |
| 9 / 66 | Gemma 4 31B IT · 31.3B | 52.3% | ~20 GB |
| 10 / 73 | DeepSeek V4 Pro · 1598.8B | 48.9% | ~960 GB |
| 11 / 74 | GLM 5 · 753.9B | 48.2% | ~457 GB |
| 12 / 75 | GPT OSS 120B · 116.8B | 48.2% | ~71 GB |
| 13 / 78 | DeepSeek V3.2 Speciale · 685.4B | 46.7% | ~415 GB |
| 14 / 85 | DeepSeek V4 Flash · 290.9B | 45.6% | ~175 GB |
| 15 / 86 | Kimi K2.5 · 1026.9B | 45.6% | ~620 GB |
| 16 / 91 | Mistral Medium 3.5 128B · 127.7B | 43.7% | ~78 GB |
| 17 / 93 | NVIDIA Nemotron 3 Ultra 550B A55B BF16 · 560.5B | 43.5% | ~370 GB |
| 18 / 96 | Kimi K2 Thinking · 1026.4B | 42.8% | ~620 GB |
| 19 / 100 | DeepSeek R1 0528 · 684.5B | 41.6% | ~415 GB |
| 20 / 101 | Qwen3 Coder 480B A35B Instruct · 480.2B | 41.2% | ~289 GB |
| 21 / 102 | Qwen3 235B A22B Thinking 2507 · 235.1B | 41.0% | ~142 GB |
| 22 / 105 | GPT OSS 20B · 20.9B | 40.9% | ~13 GB |
| 23 / 106 | GLM 4.5 · 358.3B | 40.6% | ~216 GB |
| 24 / 109 | Qwen3.5 27B · 27.8B | 39.5% | ~17 GB |
| 25 / 110 | DeepSeek V3.2 Exp · 685.4B | 39.5% | ~415 GB |
| 26 / 112 | Kimi K2 Instruct · 1026.4B | 39.4% | ~620 GB |
| 27 / 115 | Qwen3 235B A22B Instruct 2507 · 235.1B | 38.7% | ~142 GB |
| 28 / 116 | DeepSeek V3.1 · 684.5B | 38.4% | ~415 GB |
| 29 / 118 | NVIDIA Nemotron 3 Super 120B A12B BF16 · 123.6B | 38.0% | ~75 GB |
| 30 / 122 | Qwen3 235B A22B · 235.1B | 37.3% | ~142 GB |
| 31 / 124 | MiniMax M2.7 · 228.7B | 37.0% | ~138 GB |
| 32 / 125 | Kimi K2 Instruct 0905 · 1026.4B | 36.7% | ~620 GB |
| 33 / 126 | DeepSeek R1 · 684.5B | 36.5% | ~415 GB |
| 34 / 128 | DeepSeek v3 0324 · 684.5B | 36.1% | ~415 GB |
| 35 / 131 | Gemma 4 26B A4B IT · 25.8B | 35.2% | ~16 GB |
| 36 / 133 | Qwen3.6 35B A3B · 36.0B | 34.5% | ~22 GB |
| 37 / 134 | Qwen3 Coder Next · 79.7B | 34.4% | ~48 GB |
| 38 / 136 | Inkling · 952.4B | 32.3% | ~572 GB |
| 39 / 139 | Qwen3 30B A3B · 30.5B | 29.8% | ~19 GB |
| 40 / 144 | Llama 4 Maverick 17B 128E Instruct · 401.6B | 24.5% | ~265 GB |
| 41 / 147 | Llama 3.1 405B Instruct · 405.9B | 21.4% | ~268 GB |
| 42 / 151 | Qwen2.5 72B Instruct · 72.7B | 16.0% | ~45 GB |
| 43 / 152 | Llama 3.3 70B Instruct · 70.6B | 14.4% | ~47 GB |
| 44 / 155 | Qwen2 72B Instruct · 72.7B | 11.3% | ~45 GB |
| 45 / 158 | Llama 3.1 70B Instruct · 70.6B | 9.0% | ~47 GB |
| 46 / 161 | Mixtral 8x22B Instruct v0.1 · 140.6B | 3.2% | ~85 GB |
| 47 / 162 | Llama 3.1 8B Instruct · 8.0B | 1.7% | ~5.3 GB |
Score vs model size
Which models give the most quality for their size — the ones worth running locally.
- Llama 3.1 8B Instruct, 8B, score 1.7% — on the efficiency frontier (best score at its size or smaller).
- GPT OSS 20B, 21B, score 40.9% — on the efficiency frontier (best score at its size or smaller).
- Gemma 4 31B IT, 31B, score 52.3% — on the efficiency frontier (best score at its size or smaller).
- DeepSeek V4 Flash 0731, 304B, score 63.0% — on the efficiency frontier (best score at its size or smaller).
- GLM 5.3, 753B, score 75.4% — on the efficiency frontier (best score at its size or smaller).
- Kimi K3, 2.8T, score 82.6% — on the efficiency frontier (best score at its size or smaller).
WeirdML: frequently asked questions
- What is the best open LLM on WeirdML?
- Kimi K3 is the top open model on WeirdML, scoring 82.6%. Among all models tested — including proprietary ones — it ranks #18. The top model overall is GPT 6 Astra Promax (OpenAI) at 93.6%.
- What's the best WeirdML model you can run on a 24 GB GPU?
- Gemma 4 31B IT is the highest-scoring open model that fits in 24 GB at 4-bit quantization (~20 GB), scoring 52.3% on WeirdML.
- What's the best WeirdML model you can run on a 12 GB GPU?
- Llama 3.1 8B Instruct is the highest-scoring open model that fits in 12 GB at 4-bit quantization (~5.3 GB), scoring 1.7% on WeirdML.
- Can open models match proprietary models on WeirdML?
- Not quite on WeirdML: the strongest proprietary model (GPT 6 Astra Promax) scores 93.6%, ahead of the best open model (Kimi K3) at 82.6% — but you can run the open one yourself.
Scores aggregated from epoch. llmrun does not run this benchmark — see the source for methodology, or the about benchmarks for what it measures.