Mixtral 8x22B Instruct v0.1 — Benchmarks

GPQA Diamond is a set of extremely hard, graduate-level science questions (physics, chemistry, biology) written by domain experts and filtered so that skilled non-experts with web access still fail. It measures genuine reasoning rather than memorization.

DTBench is a set of handcrafted, expert-validated multiple-choice questions on decision theory — Newcomb-like problems and related puzzles — built by the Conceptual Reasoning Index project with Anthropic. It tests whether a model can follow an abstract argument rather than recall a fact.

ForecastBench, from the Forecasting Research Institute, asks models to predict real-world future events, with questions that only resolve after the model's training cutoff so the answers can't have been memorised. It is scored as a difficulty-adjusted Brier index, where higher is better.

WeirdML asks models to write machine-learning code for small, deliberately unusual datasets, then runs the code and scores the accuracy it actually achieves. Built by Håvard Tveit Ihle, it rewards practical problem-solving over recognizing a familiar task.

MATH Level 5 covers the hardest tier of competition-mathematics problems, testing multi-step symbolic and quantitative reasoning.

MMLU-Pro is a harder, cleaned-up successor to MMLU with ten answer choices and more reasoning-heavy questions across 14 subjects, measuring broad knowledge and reasoning together.

Arena (arena.ai, formerly LMArena / Chatbot Arena) ranks models by millions of blind, head-to-head human votes on real prompts. The Arena Score shown here is a Bradley-Terry rating computed with Arena's default style control, so higher means people preferred that model's answers more often — it's a relative rating, not a percentage, so compare it only to other models on this board.