How It Works
llmrun checks whether your GPU or device is powerful enough to run a given LLM locally — and predicts how fast it will be. This page explains every number and Fit rating you see on the site.
Parameters
A model's parameter count (measured in billions) is the number of learned weights it contains. Larger models tend to be more capable but require more memory and generate text more slowly.
Size → capability trade-off
↑ Smarter & more capable · ↓ Faster & less VRAM
Quantization
Quantization reduces the numerical precision of a model's weights, making the file smaller and inference faster at the cost of some quality. The format name tells you roughly how many bits each weight uses.
For example, a full-precision (FP16) 7 B model weighs ~13 GB. At Q4_K_M it shrinks to ~4 GB — small enough for an 8 GB GPU.
Quality retention vs file size (7 B model)
★ Best balance of quality and file size — the most popular choice for local inference.
GGUF Format
GGUF is the standard file format for quantized models, used by llama.cpp, Ollama, LM Studio, and other local inference tools. It bundles the model weights, tokenizer vocabulary, and metadata into a single file that runs on CPU, GPU, or both.
Single file
.gguf file contains everything needed to run the model — no separate config or tokenizer downloads.Ready to run
CPU + GPU splits
VRAM & Estimation
VRAM (Video RAM) is the dedicated memory on your GPU. To run a model entirely on GPU, the quantized weights need to fit in VRAM — if they don't, inference falls back to the CPU, which is dramatically slower.
llmrun estimates the VRAM footprint by breaking it into three components:
VRAM = Model Weights + KV Cache + ~0.3 GB
Model Weights = parameters × bitsPerWeight ÷ 8
KV Cache = 2 × kv_heads × head_dim × layers × context_tokens × 2 bytes
The ~0.3 GB accounts for framework overhead (llama.cpp, Ollama, etc.).
When we have the model's architecture details (number of KV heads, hidden size, layers), we compute the KV cache precisely. Models using Grouped-Query Attention (GQA) — like Llama 3, Qwen 2.5, and Mistral — have fewer KV heads than attention heads, significantly reducing their KV cache VRAM. When architecture data is unavailable, we fall back to a simple 10% overhead estimate.
Some newer models keep a full-length KV cache on only part of their layers, and we count only those layers. Qwen3.5 and later and Qwen3-Next use linear attention on 3 of every 4 layers. Gemma 3 and 4 and gpt-oss limit most layers to a short sliding window. DeepSeek V3 and similar models store a compressed latent cache.
Unified memory. Apple Silicon Macs use one shared memory pool between CPU and GPU rather than separate VRAM. llmrun counts RAM minus 3.5 GB (reserved for macOS) as available for models, so it checks a 32 GB Mac against 28.5 GB. Other unified-memory devices with a fixed GPU cap are checked against that limit instead.
Measured values win
Headroom matters
Tokens per Second
During text generation (the "decode" phase), the model reads its entire weight tensor once per output token. This makes inference memory-bandwidth bound — speed is dictated by how fast data can stream from VRAM, not by raw compute power.
tok/s = (bandwidth GB/s ÷ modelSize GB) × efficiency
The efficiency factor accounts for memory contention, software stack overhead, and batch-size-1 conditions. It varies by platform because each vendor's inference stack has different levels of optimisation.
Efficiency factor by platform
These factors are calibrated against community benchmarks from llama.cpp and Ollama. They may change as software stacks improve.
How fast does it feel?
Prompt Processing vs Generation
Inference has two distinct phases. Prefill (prompt processing) reads the entire prompt at once and is compute-bound — limited by how many floating-point operations per second the device can sustain. Decode (generation) produces one token at a time and is memory-bandwidth-bound — every token re-reads the entire model from VRAM.
The single "tok/s" number shown everywhere on this site describes decode only, at batch size 1. That is the right number for one person chatting locally — it says nothing about how fast a long prompt gets ingested before the model starts replying.
Time to first token
Time to first token (TTFT) is what a long prompt actually costs you: how long you wait after hitting send before the model starts streaming a reply. It's governed by the prefill rate, not the decode rate — the whole prompt has to be processed before the first output token can be produced.
Reference prompt sizes
512
tokens — Short chat
4,096
tokens — Long chat
32,768
tokens — Document / codebase
TTFT = prompt tokens ÷ prefill tok/s. Longer prompts cost proportionally more time before the first token appears — independent of how fast the model decodes afterward.
The prefill formula
prefill tok/s ≈ (dense FP16 TFLOPS × MFU) ÷ (2 × active parameters)
The factor of 2 is one multiply and one accumulate per parameter per token — the standard FLOPs-per-parameter convention for a transformer forward pass. MFU (model-FLOPs-utilization) is the fraction of a device's theoretical peak FP16 throughput that a real prefill pass actually sustains.
Prefill MFU by platform
Calibrated against llama.cpp pp512 prompt-processing benchmarks on Q4_K_M-quantized weights. Real-world results vary by roughly ±30% with prompt length, batch size, SKU, and driver/runtime version.
Why a cheap card can out-prefill an expensive one
A pro card's real edge is capacity
Mixture-of-Experts (MoE) models only spend their active parameters per token — see Dense vs MoE. When a model's name doesn't label its active-parameter count, we fall back to the total parameter count in the formula above, which understates its real prefill speed.
Memory Bandwidth
Memory bandwidth (GB/s) is the throughput between a GPU's processor and its VRAM. Because LLM decode reads the full model on every token, bandwidth is the primary bottleneck for generation speed — more bandwidth means more tokens per second for the same model.
Bandwidth comparison (GB/s)
Higher bandwidth = faster tok/s at the same model size. This is why Apple Silicon Macs with unified high-bandwidth memory can outperform discrete GPUs that have more VRAM but lower bandwidth.
Explore bandwidth comparisons on any hardware listing page or VRAM tier page.
Dense vs Mixture of Experts (MoE)
Most LLMs are dense — every parameter is used on every token. A Mixture of Experts (MoE) model splits its parameters into groups called experts and only activates a subset per token. The result: higher quality with better speed, but the entire model still has to fit in memory.
Dense Architecture
All parameters active on every token. VRAM = total params. Speed scales with total params.
Example: Llama 3 70B — 70 B total, 70 B active per token.
MoE Architecture
Only 2 of 8 experts active per token. VRAM = total params. Speed ≈ active params only.
Example: Mixtral 8×7B — 46.7 B total, ~12.9 B active per token.
On llmrun, MoE models show both the total parameter count (for VRAM sizing) and the active count (for speed intuition).
Context Length
Context length defines the maximum number of tokens (input + output) the model can handle in a single conversation. A "128K context" model can process roughly 100,000 words at once — enough for entire codebases or long documents.
The catch: longer contexts consume additional VRAM through the KV cache, which grows with every token in the conversation.
Fit check (16K)
Longer contexts
Max context
Fit ratings and the VRAM shown in model lists assume a 16K-token context. Longer prompts and conversations grow the KV cache and need more memory. The Context column in model lists shows each model's maximum; model pages show the VRAM needed at that full context.
Efficiency & Value
Raw tok/s isn't the only number worth optimizing for. llmrun also surfaces two efficiency metrics that matter more once a card is already fast enough to feel usable: how much throughput you get per watt, and how much throughput you get per dollar.
Tokens per watt
tok/W = decode tok/s ÷ TDP watts
Higher is better. Matters most for small-form-factor, laptop, and always-on builds, where power and thermal budget are as constrained as raw speed.
Dollars per tok/s
$/tok-s = hardware MSRP ÷ decode tok/s
Lower is better — it means fewer hardware dollars spent per token per second of decode throughput.
Both metrics matter more than raw tok/s once a card clears the point where it "feels" fast: past that point the question shifts from "can I use this interactively" to "did I get good value for the power and money spent."
$/tok-s is MSRP-based. It uses list price, not street price, so it understates the value of cards that regularly sell below MSRP (previous-generation or secondhand cards) and overstates the value of cards that sell above MSRP (scalped or supply-constrained launches).
Cost per Token & Cost of Ownership
The hardware pages and the cost chart answer one question: what does it cost to generate a million tokens on this card, and what does it cost to own it? Here is exactly how those numbers are built.
What we compute
- Power per 1M tokens
= W / 1000 × $/kWh ÷ (tok/s × 3600) × 1e6 - Hardware per 1M tokens
= price ÷ (tok/s × 3600 × hours/day × 365 × years) × 1e6 - Total per 1M tokens is the sum of the two.
- N-year cost of ownership = price + electricity over those N years.
Defaults are $0.15/kWh (a typical US rate), 4 hours a day of generation, and 3 years. You can change all three on the chart, and every figure updates.
One yardstick model
Prices
Power
Limits
Fit (S–F)
Fit tells you how well a model runs on a specific GPU or device. Each hardware and model pair gets a letter from S to F and a number from 0 to 100. The letter is the number cut at the thresholds shown below.
Fit and the llmrun Composite Score answer different questions. Fit depends on the hardware: the same model can be S on one GPU and F on another. The llmrun Composite Score rates the model itself from benchmark results and is the same on every GPU. A strong model can get a low Fit on a small card, and a weaker model can get a high one.
Fit blends two things: how much VRAM headroom the model leaves after loading, and its estimated decode speed in tok/s. Speed is weighted more heavily, because the cap for tight fits (below) already covers the case where headroom is the problem.
Headroom has a plateau. Once a model leaves enough spare VRAM for the KV cache and OS overhead, extra headroom does not raise the score, and spare VRAM is never penalised. A model that only barely fits is capped, even if its decode speed looks good on paper, because there is no room left for a longer conversation.
Fit does not decide the order of recommendation lists. Lists are ordered mainly by quality (the llmrun Composite Score, or, for a model without a Composite Score, an estimate from its size adjusted by its Traction), then by recency and the Fit number, then by popularity. A separate VRAM-use value, which peaks at around 75% of a card's VRAM, is only a light tie-breaker at the end. It exists because a 1B model "fitting" on a 48GB card is a poor recommendation even though it technically runs. An older model that a newer, same-size model from the same publisher replaces, base (non-instruct) checkpoints, and specialised models (translation, reward models, and similar) are moved down the list.
Earlier versions of llmrun sorted lists mainly by VRAM usage, grading them on that same metric. A large-VRAM professional card might have shown a mediocre letter for a small model running fast with plenty of headroom simply because the model didn't fill the card. Fit and the VRAM-use value are now calculated separately, and lists lead with quality.
Traction
Many new or niche models lack enough benchmarks for an llmrun Composite Score. Hardware lists must still place them, so they are ranked by Traction: an index of indirect signals showing a model is good and in real use. Once a model gathers sufficient benchmark results, its Composite Score replaces Traction.
Traction combines three inputs. The LMArena rating, if present for the model, carries the most weight because it's the sole quality signal available here. Downloads in the last 30 days follow, adjusted for release age so a fresh week-old model isn't penalized for having less download time. Likes add a small, slower check on those downloads. Without an LMArena rating, a model can only reach part of the scale; heavy downloads alone cannot push it above one with a solid rating.
In lists, Traction is shown as five bars, filled in proportion to its value. Opening it shows the value on a scale from 0 to 500. It is a ranking signal, not a quality score: it is not on the same scale as the llmrun Composite Score, and the two should not be compared.
When These Numbers Were Calibrated
A benchmark without a date and a software stack attached is not reproducible. Every tok/s, time-to-first-token, tok/W, and $/tok-s figure on llmrun is a model calibrated against a specific inference runtime and vendor driver stack — not a live measurement of your exact card. Here is exactly what that calibration currently is.
- Last reviewed
- 2026-09-21
- Inference runtime
- llama.cpp / Ollama (GGUF)
Vendor compute / driver stack
- NVIDIA
- CUDA 12.x
- AMD
- ROCm 7.x
- Intel
- oneAPI / SYCL
- Apple
- Metal
GPU driver releases and llama.cpp/Ollama runtime updates shift real achievable throughput over time — sometimes significantly. Treat every figure derived from this basis as accurate as of the date above, not as a permanent ground truth.
Accuracy & Limitations
Every figure on llmrun is a calibrated estimate, not a measurement taken on your exact hardware. Here is what the model does not account for.
Single user, batch size 1
Multi-GPU setups
Runtime optimizations
MoE routing
Error bars: expect roughly ±20% on decode tok/s and roughly ±30% on prefill throughput — and wider than that on any hardware without a verified compute figure.
These are estimates from a calibrated model, not measurements taken on this hardware. GPU driver and inference-runtime releases move real throughput over time — that's why we publish exactly what the calibration is and when it was last reviewed in When These Numbers Were Calibrated.
Frequently Asked Questions
- How do you calculate VRAM requirements?
- We compute VRAM as Model Weights + KV Cache + Framework Overhead. Model weights = parameters × bits-per-weight ÷ 8. KV cache depends on the model's attention architecture (number of KV heads, head dimension, and layers). When architecture data is available, we compute the precise KV cache size; otherwise we fall back to a 10% overhead estimate. Measured values from Ollama or llama.cpp always take priority.
- How accurate are the tok/s estimates?
- They're based on the theoretical bandwidth formula: tok/s ≈ (bandwidth GB/s ÷ model size GB) × efficiency. The efficiency factor varies by platform: 65% for NVIDIA (CUDA), 60% for AMD (ROCm 7.x), 50% for Intel (oneAPI), and 70% for Apple Silicon (Metal). In practice, results typically land within ±20% of these figures. This covers decode (generation) speed only — see "What is time to first token?" for prompt-processing speed.
- What does the S–F Fit rating mean?
- Fit shows how well a model runs on a specific GPU or device, as a letter from S to F and a number from 0 to 100. It combines VRAM headroom and estimated decode speed. S means the model runs fast with room to spare. F means the hardware can't load the model. Fit is separate from the llmrun Composite Score, which rates the model's quality from benchmarks and does not depend on hardware. See the Fit section above for details.
- Why does VRAM headroom matter?
- A model that barely fits in VRAM leaves nothing for the KV cache, which grows with conversation length. Models with Grouped-Query Attention (GQA) are more memory-efficient for long contexts. Fit ratings reserve room for a 16K-token context; longer prompts and conversations need more memory.
- What is the GGUF format?
- GGUF is the standard file format for quantized models used by llama.cpp, Ollama, and LM Studio. It packages weights, tokenizer, and metadata into a single ready-to-run file.
- What is time to first token, and what affects it?
- Time to first token (TTFT) is how long you wait after sending a prompt before the model starts streaming a reply. Unlike tok/s (which measures decode, one token at a time), TTFT is governed by prefill — a compute-bound pass that processes the whole prompt at once. It scales with prompt length and with the device's dense FP16 compute throughput and model-FLOPs-utilization (MFU), not with memory bandwidth. Longer prompts and lower-MFU hardware both increase TTFT.
- Why can an expensive professional GPU show a lower tok/s than a cheap gaming GPU?
- The tok/s figure shown across the site measures decode speed, which is memory-bandwidth bound — a consumer card with fast, narrow VRAM can decode a small model very quickly per token. A professional card's real advantage is capacity: how large a model it can load at all, not raw decode or prefill speed. For a model that fits both cards, the cheaper card can legitimately be faster.
- What does llmrun's performance model not account for?
- Batch size above 1 / concurrent requests, multi-GPU tensor and pipeline parallelism and interconnect cost, speculative decoding, flash-attention variants, quantized KV cache, CPU/RAM offload, and Mixture-of-Experts routing beyond the active-parameter-plus-routing-overhead approximation. See Accuracy & Limitations below for the full list and our stated error bars.
- How often are the performance estimates recalibrated?
- Every tok/s, TTFT, tok/W, and $/tok-s figure on llmrun is calibrated against a specific inference runtime and vendor driver stack, not measured live on your exact card. That basis was last reviewed on 2026-09-21 against llama.cpp / Ollama (GGUF). GPU driver releases and runtime updates shift real achievable throughput over time, so we periodically re-review these constants — see When These Numbers Were Calibrated below for the current basis.
Data Sources
Hardware specs come from manufacturer datasheets and are cross-referenced with community benchmarks. Model VRAM figures are collected from:
- Ollama — model library and runtime measurements
- llama.cpp — community benchmarks and perplexity data
- Hugging Face — model cards and architecture metadata
Spot an error? Email contact@llmrun.dev — data corrections are always welcome.