Kimi K2.5 — Benchmarks

GPQA Diamond is a set of extremely hard, graduate-level science questions (physics, chemistry, biology) written by domain experts and filtered so that skilled non-experts with web access still fail. It measures genuine reasoning rather than memorization.

ARC-AGI-2 is the second-generation ARC Prize puzzle set, rebuilt so that tasks solved by brute-force search or memorization were removed and every problem still requires inferring a novel rule from a handful of examples. It is markedly harder than the original ARC-AGI, and most models — open and closed alike — score near zero.

ARC-AGI tests fluid, abstract reasoning on small visual grid puzzles where each task follows a novel rule the model must infer from a few examples. It deliberately resists memorization and is one of the most-watched measures of general reasoning progress.

SimpleBench is a set of everyday, common-sense and trick questions that humans answer easily but language models often get wrong. It probes basic reasoning and robustness rather than specialist knowledge.

CL-bench, built by Tencent Hunyuan and Fudan University's NLP group, tests whether a model can learn new domain knowledge, rule systems and procedures purely from a long provided context — not from pretraining — and apply them correctly, across 1,899 tasks drawn from 500 scenarios. An LLM judge grades each task against dozens of fine-grained rubrics, and a task counts as solved only when every rubric passes.

CL-bench Life, a follow-up from the CL-bench authors, tests the same context-learning skill on messier real-life material — fragmented notes, casual conversations, and behavioral logs — across 405 human-curated context-task pairs. It uses the same rubric-based LLM-judge scoring as CL-bench, requiring every rubric on a task to pass for it to count as solved.

CritPt (Critical Points) tests graduate-level physics problem-solving on research-style questions spanning mechanics, electromagnetism, quantum mechanics, and more. It was created by physicists to probe whether a model can reason through multi-step physics rather than pattern-match textbook answers.

Fiction.LiveBench is a long-context comprehension test: models answer questions that require tracking characters, events and relationships across a long story. llmrun shows the 16k-token score, the column Epoch AI reports as its headline.

Vending-Bench 2, from Andon Labs, is a long-horizon agent benchmark in which the model runs a simulated vending-machine business for a year, handling suppliers, pricing and cash flow. The score is the final money balance in US dollars.

SWE-bench Verified tests whether a model can resolve real GitHub issues from popular open-source Python projects, scored on the official swebench.com leaderboard as the percentage of human-validated issues actually fixed. It is the headline measure of practical, agentic software-engineering ability — where open-weight models like Qwen3-Coder, GLM-4.6, Kimi K2 and DeepSWE are now competitive with the frontier.

Terminal-Bench measures whether a model can complete real, end-to-end tasks in a command-line environment — running commands, editing files, and chaining steps — making it an agentic test of practical software skill.

SWE-bench Multilingual extends SWE-bench beyond Python to real GitHub issues across many programming languages, measuring whether a model can fix bugs in codebases written in Java, Go, Rust, TypeScript and more.

SciCode asks a model to write real scientific-computing code — implementing multi-step numerical methods drawn from physics, chemistry, biology, and math research — and checks the output against reference solutions. It was built by a team of scientists and ML researchers to test coding ability on genuine research problems, not typical software tasks.

WeirdML asks models to write machine-learning code for small, deliberately unusual datasets, then runs the code and scores the accuracy it actually achieves. Built by Håvard Tveit Ihle, it rewards practical problem-solving over recognizing a familiar task.

ALE-Bench, from Sakana AI and AtCoder, tests long-horizon algorithmic optimisation using problems drawn from AtCoder Heuristic Contests, where there is no single correct answer and solutions are scored on quality. Results are reported as an AtCoder-style performance rating, where higher is better.

AIME (American Invitational Mathematics Examination) is a prestigious high-school competition of hard, integer-answer problems. It's a widely-cited yardstick for multi-step mathematical reasoning, here on the 2024–2025 papers.

Humanity's Last Exam (HLE) is a set of extremely difficult, expert-written questions across many fields, designed so that even frontier models score low. It is built to stay hard as models improve, measuring the true knowledge frontier.

MMLU-Pro is a harder, cleaned-up successor to MMLU with ten answer choices and more reasoning-heavy questions across 14 subjects, measuring broad knowledge and reasoning together.

SimpleQA measures factual accuracy on short, fact-seeking questions with a single correct answer — directly probing how often a model is right versus confidently wrong (hallucination) on simple facts.

Arena (arena.ai, formerly LMArena / Chatbot Arena) ranks models by millions of blind, head-to-head human votes on real prompts. The Arena Score shown here is a Bradley-Terry rating computed with Arena's default style control, so higher means people preferred that model's answers more often — it's a relative rating, not a percentage, so compare it only to other models on this board.