Kimi K2.6 — Benchmarks

GPQA Diamond is a set of extremely hard, graduate-level science questions (physics, chemistry, biology) written by domain experts and filtered so that skilled non-experts with web access still fail. It measures genuine reasoning rather than memorization.

LiveBench Reasoning measures logical, multi-step reasoning using contamination-free questions that are refreshed regularly, so models cannot have trained on the test set.

Chess Puzzles is an Epoch AI evaluation that asks models to find the winning move in tactical chess positions drawn from real games. Because each puzzle has one verifiable correct answer and no natural-language shortcut, it isolates concrete multi-step planning.

CritPt (Critical Points) tests graduate-level physics problem-solving on research-style questions spanning mechanics, electromagnetism, quantum mechanics, and more. It was created by physicists to probe whether a model can reason through multi-step physics rather than pattern-match textbook answers.

DTBench is a set of handcrafted, expert-validated multiple-choice questions on decision theory — Newcomb-like problems and related puzzles — built by the Conceptual Reasoning Index project with Anthropic. It tests whether a model can follow an abstract argument rather than recall a fact.

EBR-bench, from Epoch AI, tests whether models learn from experience by having them play the board game Earthborne Rangers over repeated playthroughs, with the rulebook, card database and a persistent note-taking tool. Scores stay roughly flat across plays and well below an expert human baseline.

Furniture Assembly, built by Epoch AI, tests spatial reasoning with 60 photos of partly built IKEA furniture and the assembly manual: the model must say whether the build is correct so far and, if not, name the mistaken step and describe the error. Scores are accuracy across the photos, and a human baseline averaged 63%.

LMCA (Language Model Conceptual Argumentation) scores models on expert-rated arguments across philosophy, decision theory, and AI risk — domains where there is no empirical feedback loop to check an answer against. It is part of the Conceptual Reasoning Index, built with Anthropic by researchers Emery Cooper and Caspar Oesterheld.

Mystery Game Puzzles is an Epoch AI evaluation of multi-step deductive reasoning: the model works through a mystery or logic-game scenario to arrive at the correct solution. It probes sustained, structured inference rather than single-step question answering.

Vending-Bench 2, from Andon Labs, is a long-horizon agent benchmark in which the model runs a simulated vending-machine business for a year, handling suppliers, pricing and cash flow. The score is the final money balance in US dollars.

LiveBench Coding evaluates code generation and completion on fresh, contamination-free programming tasks that are updated regularly.

SciCode asks a model to write real scientific-computing code — implementing multi-step numerical methods drawn from physics, chemistry, biology, and math research — and checks the output against reference solutions. It was built by a team of scientists and ML researchers to test coding ability on genuine research problems, not typical software tasks.

WeirdML asks models to write machine-learning code for small, deliberately unusual datasets, then runs the code and scores the accuracy it actually achieves. Built by Håvard Tveit Ihle, it rewards practical problem-solving over recognizing a familiar task.

ALE-Bench, from Sakana AI and AtCoder, tests long-horizon algorithmic optimisation using problems drawn from AtCoder Heuristic Contests, where there is no single correct answer and solutions are scored on quality. Results are reported as an AtCoder-style performance rating, where higher is better.

ExploitBench, from Carnegie Mellon University researchers Seunghyun Lee and David Brumley, scores AI agents on writing working exploits for 41 real, already-patched bugs in Chrome's V8 JavaScript engine, with modern mitigations switched on. Instead of pass/fail it grades 16 stages from crash up to arbitrary code execution, and Epoch AI reports the mean capability reached.

AIME (American Invitational Mathematics Examination) is a prestigious high-school competition of hard, integer-answer problems. It's a widely-cited yardstick for multi-step mathematical reasoning, here on the 2024–2025 papers.

FrontierMath is a benchmark of exceptionally hard, original research-level mathematics problems created with professional mathematicians. Even the strongest models solve only a small fraction, making it a frontier measure of genuine mathematical ability.

LiveBench Math measures mathematical problem-solving on contamination-free, regularly-refreshed questions, including competition-style problems.

FrontierMath Tier 4 is the hardest tier of Epoch AI's research-level mathematics benchmark, holding problems that working mathematicians consider genuinely difficult research questions. Scores here stay low even for frontier models, making it the sharpest available read on mathematical ability.

ProofBench evaluates a model's ability to write and check rigorous mathematical proofs, rather than just produce a final numeric answer. It probes step-by-step logical correctness, which is a harder and more distinct skill than solving for a number.

SimpleQA measures factual accuracy on short, fact-seeking questions with a single correct answer — directly probing how often a model is right versus confidently wrong (hallucination) on simple facts.

Arena (arena.ai, formerly LMArena / Chatbot Arena) ranks models by millions of blind, head-to-head human votes on real prompts. The Arena Score shown here is a Bradley-Terry rating computed with Arena's default style control, so higher means people preferred that model's answers more often — it's a relative rating, not a percentage, so compare it only to other models on this board.