Mistral 7B v0.1 — Benchmarks

WinoGrande is a large-scale commonsense-reasoning test built around pronoun resolution: the model must pick which of two nouns a pronoun refers to in a sentence, using real-world knowledge to disambiguate. It was designed by AI2 to resist the shortcuts that let earlier models game its predecessor, Winograd.

PIQA (Physical Interaction QA) asks a model to choose the more sensible of two ways to accomplish an everyday physical task, testing practical, physical-world commonsense that isn't well captured by purely textual reasoning tests.

MMLU (Massive Multitask Language Understanding) spans 57 subjects from history to law to medicine as multiple-choice questions. It is the long-standing default for broad knowledge and remains the most widely-reported general benchmark.

ARC Challenge is a set of grade-school science questions, curated by the Allen Institute for AI (AI2) to exclude ones that simple word-matching or retrieval can answer. It measures science knowledge paired with the reasoning needed to apply it.

HellaSwag is a commonsense benchmark that asks a model to pick the most plausible continuation of an everyday situation. The wrong options are adversarially chosen to fool models, testing grounded commonsense understanding.

BoolQ is a reading-comprehension benchmark of naturally occurring yes/no questions, each paired with a short passage that contains the answer. It tests whether a model can extract and reason over information from real text rather than just recall facts.

OpenBookQA is a science-question benchmark modeled on open-book exams: the model is given a small set of core science facts and must combine them with broader common knowledge to answer multi-step questions. It targets reasoning over recall.

TriviaQA is a large set of trivia questions paired with evidence documents, testing a model's factual recall and reading comprehension across a wide range of everyday topics.