How the llmrun Score is built

1. 1

   Public leaderboards

   Results published by 4 independent sources.

2. 2

   Normalize

   Each result is rescaled to 0–1, from the benchmark's random-guess baseline to its ceiling.

3. 3

   IRT fit

   One ability θ per model, plus a difficulty b and a discrimination a per benchmark.

4. 4

   Frozen scale

   Reference 2026-Q4 fixes the parameters of 40 benchmarks, so scores don't drift.

5. 5

   llmrun Score

   A 0–100 score with a 90% range, plus 5 skill scores.

• Still in use

  A benchmark counts while models released in the last 365 days are still submitted to it. Saturated older benchmarks such as GSM8K and HellaSwag drop out this way.

• A fixed 0–100 scale

  Relative ratings such as LMArena's Elo are excluded, because they have no baseline or ceiling to normalize against.

• Plausible bounds

  Benchmarks where more than 25% of results sit at exactly 0 or 100 are excluded. In that case the scale bounds are probably wrong.

Overall llmrun Score

At least 4 eligible benchmarks across at least 2 categories, one of which is reasoning or coding.

Skill score

At least 2 benchmarks in that skill. Otherwise the skill is marked as not enough data.

• Coding

  9 benchmarks

  Writing and fixing code that must run and pass tests.

  LiveBench Coding · Aider Polyglot · SciCode · WeirdML · SWE-bench Multilingual · SWE-bench Lite · FrontierCode · ALE-Bench · Surface Evolver Bench

• Agents

  8 benchmarks

  Long multi-step tasks in real environments: repositories, terminals, simulated businesses.

  SWE-bench Verified · SWE-bench Bash Only · Terminal-Bench · DeepSWE · GSO-Bench · APEX-Agents · Vending-Bench 2 · BALROG

• Math

  6 benchmarks

  From competition problems to research-level proofs.

  AIME 2024/2025 · MATH Level 5 · LiveBench Math · FrontierMath · FrontierMath Tier 4 · ProofBench

• Science

  4 benchmarks

  Graduate-level physics, chemistry and biology, plus scientific code.

  GPQA Diamond · CritPt · SciCode · Humanity's Last Exam

• Reasoning

  9 benchmarks

  Novel puzzles, deduction, planning and forecasting.

  ARC-AGI · ARC-AGI-2 · SimpleBench · LiveBench Reasoning · Chess Puzzles · DTBench · LMCA · Mystery Game Puzzles · ForecastBench

• Planned

  Knowledge, Long context, Instruction following, Multilingual, Vision. These will be added when enough open models have public results to rank them fairly.

Contamination

A model trained on a benchmark's questions scores above its real ability. Benchmarks with fresh, rotating or private question sets (LiveBench, FrontierMath, ARC-AGI-2) reduce this effect but cannot remove it.

Self-reported vs independent

Some leaderboards accept runs submitted by developers, with their own scaffolds and settings. Independent evaluations are harder to game. We use each source's published figure as is.

Effort and thinking variants

Proprietary models are often listed at several reasoning-effort settings. We use the best variant of each, so its score reflects its strongest published configuration.

Sparse coverage for small models

Small open models are rarely submitted to frontier benchmarks, so many fall short of the minimum or get wide ranges. A missing score means there is not enough evidence. It does not mean the model is weak.

English-centric

Almost every reference benchmark is in English. Multilingual ability is not measured yet.

• Epoch AI Benchmarking Hub

  Most of the panel: independent runs of frontier reasoning, math, science, coding and agent benchmarks (CC-BY).

  32 in reference

• LiveBench

  Coding, math and reasoning splits designed to resist contamination. New questions are added on a rolling basis.

  3 in reference

• SWE-bench leaderboards

  Official SWE-bench Verified, Bash Only, Lite and Multilingual results, scored on whether the model fixes real GitHub issues.

  4 in reference

• TIGER-Lab MMLU-Pro

  MMLU-Pro, a harder ten-choice successor to MMLU with more reasoning questions and wide open-model coverage.

  1 in reference

• LMArena

  Elo ratings from blind head-to-head human votes. Shown on separate leaderboards and not used in the score, because Elo is a relative rating, not a 0–100 result.

  Not in score

Background on each benchmark is in the benchmark guide. The full ranking is on the llmrun Score leaderboard.

What is the llmrun Score?

A 0–100 capability score for language models, computed from public benchmark results. It is the model's expected result averaged over the 40 benchmarks of the frozen 2026-Q4 reference set, where 0 is random guessing and 100 is a perfect score.

Why not just average a model's benchmark scores?

Each model is tested on its own mix of benchmarks, so a plain average mostly reflects how hard that mix was. The llmrun Score fits all models and all benchmarks together with Item Response Theory, which estimates each benchmark's difficulty and takes it into account.

Does llmrun run the benchmarks itself?

No. The score uses only results published by Epoch AI Benchmarking Hub, LiveBench, SWE-bench leaderboards, TIGER-Lab MMLU-Pro. Each source is credited and linked on this page.

Why does every score come with a range?

It is the 90% interval of the estimate. A model with results on many informative benchmarks gets a narrow range, and one that barely meets the minimum gets a wide one. If two ranges overlap heavily, treat the models as roughly tied.

Will a model's score change when new models or harder benchmarks are added?

No. Benchmark parameters are frozen in reference 2026-Q4, and new benchmarks are calibrated onto that scale. The overall score changes mainly when new results for that model come in, though its range and skill scores can shift slightly as the fit is refined.

Why isn't LMArena part of the llmrun Score?

LMArena publishes an Elo-style rating built from human preference votes. The rating only says how often voters preferred one model over another, so it has no fixed baseline or ceiling to normalize against. Arena ratings are shown on their own leaderboards.

Why doesn't a model have an llmrun Score yet?

An overall score needs results on at least 4 eligible benchmarks in at least 2 categories, including reasoning or coding, and a skill score needs at least 2 benchmarks in that skill. Many small and new open models have not been submitted to enough public leaderboards yet.

How is the llmrun Score different from the S–F grade?

The llmrun Score rates the model itself and does not depend on hardware. The S–F Fit grade rates how well the model runs on a given GPU or device, based on VRAM headroom and speed. A strong model can get an F on a small GPU, and a weaker one an S.