How the llmrun Composite Score is built

1. 1

   Public leaderboards

   Results published by 4 independent sources.

2. 2

   Normalize

   Every result gets rescaled to a range of 0 through 1, moving from the benchmark's random-guess floor up to its maximum limit.

3. 3

   IRT fit

   Each model gets one ability θ, alongside a difficulty b and a discrimination a for every benchmark.

4. 4

   Frozen scale

   Reference 2026-Q4 fixes the parameters of 40 benchmarks, so scores don't drift.

5. 5

   Composite Score

   An open-ended score (100 = average model) with a 90% range, plus 5 skill scores.

• Still in use

  A benchmark counts if it has a result dated within the last 365 days (using either the source's own date or the model's release date when no source date exists). Saturated older benchmarks like GSM8K and HellaSwag drop out this way.

• A fixed baseline and ceiling

  Relative ratings such as LMArena's Elo are excluded, because they have no baseline or ceiling to normalize against.

• Plausible bounds

  Benchmarks where more than 25% of results sit at exactly 0 or 100 are excluded. In that case the scale bounds are probably wrong.

Overall Composite Score

At least 4 eligible benchmarks across at least 2 categories, one of which is reasoning or coding — or 3 benchmarks whose results already narrow the score to a 90% range of at most 15.6 points. Either way the results must come from at least 2 different benchmarks: the categories of one benchmark, such as LiveBench, count once.

Skill score

At least 2 benchmarks in that skill. Otherwise the skill is marked as not enough data.

Models without a published Composite Score are ranked on Traction in hardware lists.

• Coding

  9 benchmarks

  Writing and fixing code that must run and pass tests.

  LiveBench Coding · Aider Polyglot · SciCode · WeirdML · SWE-bench Multilingual · SWE-bench Lite · FrontierCode · ALE-Bench · Surface Evolver Bench

• Agents

  8 benchmarks

  Long multi-step tasks in real environments: repositories, terminals, simulated businesses.

  SWE-bench Verified · SWE-bench Bash Only · Terminal-Bench · DeepSWE · GSO-Bench · APEX-Agents · Vending-Bench 2 · BALROG

• Math

  6 benchmarks

  From competition problems to research-level proofs.

  AIME 2024/2025 · MATH Level 5 · LiveBench Math · FrontierMath · FrontierMath Tier 4 · ProofBench

• Science

  4 benchmarks

  Graduate-level physics, chemistry and biology, plus scientific code.

  GPQA Diamond · CritPt · SciCode · Humanity's Last Exam

• Reasoning

  9 benchmarks

  Novel puzzles, deduction, planning and forecasting.

  ARC-AGI · ARC-AGI-2 · SimpleBench · LiveBench Reasoning · Chess Puzzles · DTBench · LMCA · Mystery Game Puzzles · ForecastBench

• Planned

  Knowledge, Long context, Instruction following, Multilingual, Vision. We'll add these once enough open models share public results for a fair ranking.

Contamination

A model trained on a benchmark's questions scores above its real ability. Benchmarks with fresh, rotating or private question sets (LiveBench, FrontierMath, ARC-AGI-2) reduce this effect but cannot remove it.

Self-reported vs independent

Some leaderboards accept runs submitted by developers, with their own scaffolds and settings. Independent evaluations are harder to game. We use each source's published figure as is.

Effort and thinking variants

Sources list proprietary models across multiple reasoning-effort settings, but we consolidate them into a single row per model. For every benchmark, we take the result from the highest setting available; if nothing higher exists, we fall back to the default or an even lower one.

Sparse coverage for small models

Small open models are rarely submitted to frontier benchmarks, so many fall short of the minimum or get wide ranges. A missing score means there is not enough evidence. It does not mean the model is weak.

English-centric

Almost every reference benchmark is in English. Multilingual ability is not measured yet.

• Epoch AI Benchmarking Hub

  Most of the panel: independent runs of frontier reasoning, math, science, coding and agent benchmarks (CC-BY).

  32 in reference

• LiveBench

  Coding, math and reasoning splits designed to resist contamination. New questions are added on a rolling basis.

  3 in reference

• SWE-bench leaderboards

  Official SWE-bench Verified, Bash Only, Lite and Multilingual results, scored on whether the model fixes real GitHub issues.

  4 in reference

• TIGER-Lab MMLU-Pro

  MMLU-Pro, a harder ten-choice successor to MMLU with more reasoning questions and wide open-model coverage.

  1 in reference

• LMArena

  Elo ratings from blind head-to-head human votes. Shown on separate leaderboards and not used in the score, because Elo is a relative rating, not a 0–100 result.

  Not in score

Details for every benchmark are in the benchmark guide, while the complete rankings appear on the llmrun Composite Score leaderboard.

1. 2026-10-17v3-irt-4Method

   Second publication route, benchmark families, LiveBench 2026

   A model can now also get an overall score from 3 benchmarks when they already narrow it to a 90% range of at most 15.6 points. Every published score must now draw on at least 2 different benchmarks: the categories of one benchmark, such as LiveBench, count once. Three categories of the 2026 LiveBench release (Reasoning, Coding, Long-Horizon) joined the score as new benchmarks, and Chess Puzzles follows Epoch AI's re-run of that benchmark.

   Effect: Open models with a published score went from 94 to 100: eight gained one and two lost theirs (Llama 3.2 1B and Gemma 3 1B, whose Chess Puzzles results the source no longer publishes). The largest change to an existing score is 3.4 points (Muse Spark Open). The top 20 contains the same models, with two pairs of neighbours swapping places.

2. 2026-10-10v3-irt-3Method

   Active panel dated by each result's own date

   A benchmark stays in the active panel while it has a result dated within the last 365 days. Each result is now dated by its own source date; a model's release date only dates results whose source gives none. Before, a recent model release could keep an old board active.

   Effect: SWE-bench Lite (results last dated 2025) left the panel. No model lost its score; the largest change is about 1 point (DeepSeek V3.2).

3. 2026-10-04namingDisplay

   Renamed to llmrun Composite Score

   Same method, scale and numbers; only the name changed to make clear the score combines public benchmark results.

   Effect: None on values or order.

4. 2026-10-04ranking

   Hardware lists rank on the llmrun Composite Score

   GPU, device and homepage lists use a model's llmrun Composite Score as its quality signal where one is published. Models without one are ranked on Traction. Fit now reserves memory for a 16K-token context.

   Effect: Hardware list order changed. Fit grades now include room for a 16K-token context, so some tight fits grade lower.

   How Traction works

5. 2026-10-04v3-irt-3Display

   Open scale: 100 + 25·θ

   The Score is now a straight-line mapping of ability: 100 is the average model of the 2026-Q4 reference field and every 25 points is one standard deviation of ability. There is no upper limit. Same reference and same fit as v3-irt-2.

   Effect: Every model's number changed, but the order of models did not. Old 0–100 values are not comparable with new ones, and there is no exact linear conversion, because the old display curved off towards 100. Compare models within one version.

6. 2026-10-03v3-irt-2Launch

   llmrun Score launched

   An Item Response Theory fit over 40 reference benchmarks, with the scale frozen as reference 2026-Q4, a 90% range on every score and five skill scores. Displayed 0–100 as the expected result on a typical benchmark.

   Effect: Replaced the composite benchmark rating on the benchmark pages. A model's score moves only when new results for that model arrive.

7. 2026-09-20v2Method

   Active panel

   The composite reads only the benchmarks current models are still submitted to, and a model needs at least 4 of them for an overall rating.

   Effect: Models rated mostly on saturated older benchmarks dropped. Models with fewer than 4 active benchmarks lost their overall rating.

8. 2026-06v1Launch

   Benchmark rating

   A composite of normalized benchmark scores: the average within each category, then an equal-weight average across categories.

   Effect: First overall rating on llmrun.

What is the llmrun Composite Score?

A capability score for language models, derived from public benchmark results using Item Response Theory. 100 represents the average frozen 2026-Q4 reference field model, with every 25 points equaling one standard deviation in ability. The scale lacks an upper limit, meaning stronger future models will simply achieve higher scores.

Why not just average a model's benchmark scores?

Each model is tested on its own mix of benchmarks, so a plain average mostly reflects how hard that mix was. The llmrun Composite Score fits all models and all benchmarks together with Item Response Theory, which estimates each benchmark's difficulty and takes it into account.

Does llmrun run the benchmarks itself?

No. The score relies solely on results from Epoch AI Benchmarking Hub, LiveBench, SWE-bench leaderboards, TIGER-Lab MMLU-Pro. Every source gets credit here and includes links.

Why does every score come with a range?

It is the 90% interval of the estimate. A model with results on many informative benchmarks gets a narrow range, and one that barely meets the minimum gets a wide one. If two ranges overlap heavily, treat the models as roughly tied.

Will a model's score change when new models or harder benchmarks are added?

No. Benchmark parameters are frozen in reference 2026-Q4, and new benchmarks are calibrated onto that scale. The overall score changes mainly when new results for that model come in, though its range and skill scores can shift slightly as the fit is refined.

Why is the score not capped at 100?

A capped score clumps the strongest models at the top, leaving no space for better ones. The llmrun Composite Score converts ability directly into a straight line: 100 + 25 × θ. This means differences carry equal weight across the entire scale, so a much stronger model can reach scores like 160 or 200 without needing to rebase the scale.

Why isn't LMArena part of the llmrun Composite Score?

LMArena publishes an Elo-style rating built from human preference votes. The rating only says how often voters preferred one model over another, so it has no fixed baseline or ceiling to normalize against. Arena ratings are shown on their own leaderboards.

Why doesn't a model have an llmrun Composite Score yet?

An overall score requires results from at least 4 eligible benchmarks across at least 2 categories, including reasoning or coding, or from 3 benchmarks that already narrow the score to a 90% range of at most 15.6 points. Either way the results must come from at least 2 different benchmarks, so the categories of one benchmark (such as LiveBench) never publish a score on their own; meanwhile, a skill score needs at least 2 benchmarks within that specific area. Many small or new open models haven't been submitted to enough public leaderboards yet.

How is the llmrun Composite Score different from Fit (the S–F rating)?

The llmrun Composite Score rates the model itself and does not depend on hardware. Fit, the S–F rating, rates how well the model runs on a given GPU or device, based on VRAM headroom and speed. A strong model can get an F on a small GPU, and a weaker one an S.