About LLM Benchmarks

How llmrun aggregates scores

llmrun does not run any benchmark itself. We collect published results from public, permissively-licensed sources, attribute each score to its origin, and line them up next to the VRAM and GPU requirements for every open model. Each board keeps the field's own numbers; on top of them we publish one derived index, the llmrun Composite Score, fitted across all boards so models tested on different benchmarks stay comparable — mapped to what you can run.

The categories

We group benchmarks by the skill they probe. Pick a leaderboard to see the ranked open models and which ones fit your hardware.

Reading the scores

Most scores are an accuracy or pass-rate (a percentage of items answered correctly); suites like LiveBench use a normalized 0–100 score. Higher is always better here. Each benchmark has its own scale and difficulty, so compare models within a single leaderboard rather than across different ones. On every leaderboard, the score-vs-size chart highlights models that punch above their weight — high quality at a small size is what makes a model worth running locally.

Open vs proprietary models

Open models have downloadable weights — those are llmrun's focus, and each links to its VRAM and GPU requirements. Proprietary models (GPT, Gemini, Claude and friends) are shown dimmed behind the All models toggle on each leaderboard, so you can see where the open field stands against the closed frontier without pretending you can run them at home.

FAQ

Does llmrun run these benchmarks itself?

No. llmrun is an aggregator — it collects published scores from public sources (Epoch AI, LiveBench, Aider, LMArena and others), attributes each to its origin, and presents them in one place alongside the hardware you'd need to run each open model. We never re-run a benchmark or invent a composite score.

What does a benchmark score actually mean?

Most scores are an accuracy or pass-rate: the fraction of test items the model got right (shown as a percentage), or a normalized 0–100 score for suites like LiveBench. Higher is better. Because each benchmark uses its own scale and methodology, compare models within a single benchmark rather than across different ones.

How is the llmrun Composite Score on a model's page calculated?

It is the llmrun Composite Score: an open-ended capability index (100 is the average model of the reference field, with no upper limit) fitted across the benchmarks current models are still submitted to with Item Response Theory, so a hard benchmark counts as hard instead of being averaged away. Every score comes with a 90% range, and a model needs results on at least 4 qualifying benchmarks across 2 categories (including reasoning or coding), or on 3 that pin its score down to a 90% range of at most 15.6 points, before we publish it — and the results must come from at least two different benchmarks, not just categories of one. Saturated classics like MMLU and HellaSwag keep their own leaderboards but do not feed the score, and Arena (LMArena) ratings are shown as their own leaderboards only. See the methodology page for the full method.

Why do some leaderboards show fewer models than others?

We only list a model on a benchmark when a trustworthy public score exists for it. Newer or niche open models may not have been evaluated on every benchmark yet, so coverage varies by leaderboard.

What's the difference between open and proprietary models here?

Open models have downloadable weights you can run on your own hardware — those are llmrun's focus, and each links to a page with VRAM and GPU requirements. Proprietary (closed) models are shown dimmed behind the 'All models' toggle for context: you can see where the open field stands relative to the frontier, even though you can't run them locally.

How fresh are the scores?

Each leaderboard shows when its data was last refreshed. Several sources (LiveBench, Aider) update on a rolling basis with contamination-free questions, and we re-import periodically.