Coding
WeirdML v3 Leaderboard
WeirdML v3 is the newer version of WeirdML, built by Håvard Tveit Ihle: models write machine-learning code for small, deliberately unusual datasets, and the code is run and scored on the accuracy it achieves. llmrun shows the mean score across runs, not the best run.
Source: epoch3 open models ranked+20 proprietaryData through Oct 2026
All models ranked on WeirdML v3
Proprietary / closed models are shown dimmed — you can't run them locally, but they show where the open field stands.
| # | Model | Score | VRAM @ Q4 |
|---|---|---|---|
| 1 | GPT 6 Astra (xhigh) · proprietary | 42.2% | API only |
| 2 | GPT 6.1 Sol (xhigh) · proprietary | 38.3% | API only |
| 3 | Claude Opus 5.5 (xhigh) · proprietary | 31.2% | API only |
| 4 | Claude Fable 5.1 (xhigh) · proprietary | 26.0% | API only |
| 5 | Gemini 4 Argon (high) · proprietary | 20.2% | API only |
| 6 | Claude Sonnet 5.5 (xhigh) · proprietary | 19.9% | API only |
| 7 | GPT 6 Sol (xhigh) · proprietary | 19.7% | API only |
| 8 | Claude Opus 5 (xhigh) · proprietary | 19.0% | API only |
| 9 | Claude Fable 5 (xhigh) · proprietary | 15.6% | API only |
| 10 | Muse Wonder 1.0 (xhigh) · proprietary | 15.2% | API only |
| 11 | GPT 5.6 Sol (xhigh) · proprietary | 15.0% | API only |
| 12 | Claude Haiku 5.5 (xhigh) · proprietary | 11.7% | API only |
| 13 | Grok 4.7 (xhigh) · proprietary | 8.5% | API only |
| 14 | GPT 6 Luna (xhigh) · proprietary | 7.9% | API only |
| 15 | Kimi K3 · 2779.9B | 7.5% | ~1674 GB |
| 16 | Gemini 3.8 Flash (high) · proprietary | 7.4% | API only |
| 17 | Muse Spark 1.3 (xhigh) · proprietary | 6.7% | API only |
| 18 | Claude Opus 4.5 (Nov 01, 2025, high) · proprietary | 6.5% | API only |
| 19 | DeepSeek V4.1 Flash · 763.2B | 5.9% | ~458 GB |
| 20 | GPT 5.6 Luna (xhigh) · proprietary | 5.2% | API only |
| 21 | GPT 5.2 (Dec 11, 2025, xhigh) · proprietary | 5.0% | API only |
| 22 | Mistral Large 4.0 (high) · proprietary | 4.5% | API only |
| 23 | Inkling · 952.4B | 3.9% | ~572 GB |
WeirdML v3: frequently asked questions
- What is the best open LLM on WeirdML v3?
- Kimi K3 is the top open model on WeirdML v3, scoring 7.5%. Among all models tested — including proprietary ones — it ranks #15. The top model overall is GPT 6 Astra (xhigh) (OpenAI) at 42.2%.
- Can open models match proprietary models on WeirdML v3?
- Not quite on WeirdML v3: the strongest proprietary model (GPT 6 Astra (xhigh)) scores 42.2%, ahead of the best open model (Kimi K3) at 7.5% — but you can run the open one yourself.
Scores aggregated from epoch. llmrun does not run this benchmark — see the source for methodology, or the about benchmarks for what it measures.