Qwen3.8 27B — Benchmarks

LiveBench Reasoning measures logical, multi-step reasoning using contamination-free questions that are refreshed regularly, so models cannot have trained on the test set.

ARC-AGI-2 is the second-generation ARC Prize puzzle set, rebuilt so that tasks solved by brute-force search or memorization were removed and every problem still requires inferring a novel rule from a handful of examples. It is markedly harder than the original ARC-AGI, and most models — open and closed alike — score near zero.

ARC-AGI tests fluid, abstract reasoning on small visual grid puzzles where each task follows a novel rule the model must infer from a few examples. It deliberately resists memorization and is one of the most-watched measures of general reasoning progress.

APEX-Agents (AI Productivity Index) is Mercor's evaluation of AI agents on long-horizon, multi-application professional-services work — investment-banking, management-consulting, and corporate-law style tasks — graded against expert-produced solutions. The score is Pass@1, the fraction solved correctly on the first attempt.

CritPt (Critical Points) tests graduate-level physics problem-solving on research-style questions spanning mechanics, electromagnetism, quantum mechanics, and more. It was created by physicists to probe whether a model can reason through multi-step physics rather than pattern-match textbook answers.

DTBench is a set of handcrafted, expert-validated multiple-choice questions on decision theory — Newcomb-like problems and related puzzles — built by the Conceptual Reasoning Index project with Anthropic. It tests whether a model can follow an abstract argument rather than recall a fact.

LiveBench 2026 Reasoning measures logical, multi-step problem solving. It belongs to the 2026 LiveBench release (first snapshot September 2, 2026), a redesigned LiveBench in which each of seven categories is scored by a single task and questions are refreshed to limit training-set contamination. It is a different test from the original LiveBench boards, so the scores are not comparable.

LMCA (Language Model Conceptual Argumentation) scores models on expert-rated arguments across philosophy, decision theory, and AI risk — domains where there is no empirical feedback loop to check an answer against. It is part of the Conceptual Reasoning Index, built with Anthropic by researchers Emery Cooper and Caspar Oesterheld.

LiveBench Coding evaluates code generation and completion on fresh, contamination-free programming tasks that are updated regularly.

SciCode asks a model to write real scientific-computing code — implementing multi-step numerical methods drawn from physics, chemistry, biology, and math research — and checks the output against reference solutions. It was built by a team of scientists and ML researchers to test coding ability on genuine research problems, not typical software tasks.

LiveBench 2026 Coding measures whether a model writes correct code for programming problems. It belongs to the 2026 LiveBench release (first snapshot September 2, 2026), a redesigned LiveBench in which each of seven categories is scored by a single task and questions are refreshed to limit training-set contamination. It is a different test from the original LiveBench boards, so the scores are not comparable.

LiveBench 2026 Long-Horizon measures multi-step tasks that a model has to carry through many steps to finish. It belongs to the 2026 LiveBench release (first snapshot September 2, 2026), a redesigned LiveBench in which each of seven categories is scored by a single task and questions are refreshed to limit training-set contamination. It is a different test from the original LiveBench boards, so the scores are not comparable.

Surface Evolver Bench tests whether a model can write a correct simulation file for Surface Evolver, a physics tool that models surfaces shaped by tension, gravity and volume constraints, defining a 3D structure in its domain-specific format from only a few examples. Scores compare the resulting geometry — volume, area and energy — against reference solutions; the benchmark was built by an independent developer (GitHub: yhenon).

LiveBench Math measures mathematical problem-solving on contamination-free, regularly-refreshed questions, including competition-style problems.

ProofBench evaluates a model's ability to write and check rigorous mathematical proofs, rather than just produce a final numeric answer. It probes step-by-step logical correctness, which is a harder and more distinct skill than solving for a number.

LiveBench 2026 Research measures finding and combining information to answer research questions. It belongs to the 2026 LiveBench release (first snapshot September 2, 2026), a redesigned LiveBench in which each of seven categories is scored by a single task and questions are refreshed to limit training-set contamination. It is a different test from the original LiveBench boards, so the scores are not comparable.

Arena (arena.ai, formerly LMArena / Chatbot Arena) ranks models by millions of blind, head-to-head human votes on real prompts. The Arena Score shown here is a Bradley-Terry rating computed with Arena's default style control, so higher means people preferred that model's answers more often — it's a relative rating, not a percentage, so compare it only to other models on this board.