What is the best open LLM on WeirdML v3?

Kimi K3 is the top open model on WeirdML v3, scoring 7.5%. Among all models tested — including proprietary ones — it ranks #15. The top model overall is GPT 6 Astra (xhigh) (OpenAI) at 42.2%.

Can open models match proprietary models on WeirdML v3?

Not quite on WeirdML v3: the strongest proprietary model (GPT 6 Astra (xhigh)) scores 42.2%, ahead of the best open model (Kimi K3) at 7.5% — but you can run the open one yourself.

Coding

WeirdML v3 Leaderboard

Name: WeirdML v3 — open LLM scores
Creator: epoch

WeirdML v3 is the newer version of WeirdML, built by Håvard Tveit Ihle: models write machine-learning code for small, deliberately unusual datasets, and the code is run and scored on the accuracy it achieves. llmrun shows the mean score across runs, not the best run.

Source: epoch3 open models ranked+20 proprietaryData through Oct 2026

Open models All models

Open models ranked on WeirdML v3

# shows rank among open models / rank overall (including proprietary).

#	Model	Score	VRAM @ Q4
1 / 15	Kimi K3 · 2779.9B	7.5%	~1674 GB
2 / 19	DeepSeek V4.1 Flash · 763.2B	5.9%	~458 GB
3 / 23	Inkling · 952.4B	3.9%	~572 GB

WeirdML v3: frequently asked questions

What is the best open LLM on WeirdML v3?: Kimi K3 is the top open model on WeirdML v3, scoring 7.5%. Among all models tested — including proprietary ones — it ranks #15. The top model overall is GPT 6 Astra (xhigh) (OpenAI) at 42.2%.
Can open models match proprietary models on WeirdML v3?: Not quite on WeirdML v3: the strongest proprietary model (GPT 6 Astra (xhigh)) scores 42.2%, ahead of the best open model (Kimi K3) at 7.5% — but you can run the open one yourself.

Scores aggregated from epoch. llmrun does not run this benchmark — see the source for methodology, or the about benchmarks for what it measures.