How It Works
llmrun checks whether your GPU or device is powerful enough to run a given LLM locally — and predicts how fast it will be. This page explains every number and grade you see on the site.
Parameters
A model's parameter count (measured in billions) is the number of learned weights it contains. Larger models tend to be more capable but require more memory and generate text more slowly.
Size → capability trade-off
↑ Smarter & more capable · ↓ Faster & less VRAM
Quantization
Quantization reduces the numerical precision of a model's weights, making the file smaller and inference faster at the cost of some quality. The format name tells you roughly how many bits each weight uses.
For example, a full-precision (FP16) 7 B model weighs ~13 GB. At Q4_K_M it shrinks to ~4 GB — small enough for an 8 GB GPU.
Quality retention vs file size (7 B model)
★ Best balance of quality and file size — the most popular choice for local inference.
GGUF Format
GGUF is the standard file format for quantized models, used by llama.cpp, Ollama, LM Studio, and other local inference tools. It bundles the model weights, tokenizer vocabulary, and metadata into a single file that runs on CPU, GPU, or both.
Single file
.gguf file contains everything needed to run the model — no separate config or tokenizer downloads.Ready to run
CPU + GPU splits
VRAM & Estimation
VRAM (Video RAM) is the dedicated memory on your GPU. To run a model entirely on GPU, the quantized weights need to fit in VRAM — if they don't, inference falls back to the CPU, which is dramatically slower.
llmrun estimates the VRAM footprint by breaking it into three components:
VRAM = Model Weights + KV Cache + ~0.3 GB
Model Weights = parameters × bitsPerWeight ÷ 8
KV Cache = 2 × kv_heads × head_dim × layers × context_tokens × 2 bytes
The ~0.3 GB accounts for framework overhead (llama.cpp, Ollama, etc.).
When we have the model's architecture details (number of KV heads, hidden size, layers), we compute the KV cache precisely. Models using Grouped-Query Attention (GQA) — like Llama 3, Qwen 2.5, and Mistral — have fewer KV heads than attention heads, significantly reducing their KV cache VRAM. When architecture data is unavailable, we fall back to a simple 10% overhead estimate.
Measured values win
Headroom matters
Tokens per Second
During text generation (the "decode" phase), the model reads its entire weight tensor once per output token. This makes inference memory-bandwidth bound — speed is dictated by how fast data can stream from VRAM, not by raw compute power.
tok/s = (bandwidth GB/s ÷ modelSize GB) × efficiency
The efficiency factor accounts for memory contention, software stack overhead, and batch-size-1 conditions. It varies by platform because each vendor's inference stack has different levels of optimisation.
Efficiency factor by platform
These factors are calibrated against community benchmarks from llama.cpp and Ollama. They may change as software stacks improve.
How fast does it feel?
Prompt Processing vs Generation
Inference has two distinct phases. Prefill (prompt processing) reads the entire prompt at once and is compute-bound — limited by how many floating-point operations per second the device can sustain. Decode (generation) produces one token at a time and is memory-bandwidth-bound — every token re-reads the entire model from VRAM.
The single "tok/s" number shown everywhere on this site describes decode only, at batch size 1. That is the right number for one person chatting locally — it says nothing about how fast a long prompt gets ingested before the model starts replying.
Time to first token
Time to first token (TTFT) is what a long prompt actually costs you: how long you wait after hitting send before the model starts streaming a reply. It's governed by the prefill rate, not the decode rate — the whole prompt has to be processed before the first output token can be produced.
Reference prompt sizes
512
tokens — Short chat
4,096
tokens — Long chat
32,768
tokens — Document / codebase
TTFT = prompt tokens ÷ prefill tok/s. Longer prompts cost proportionally more time before the first token appears — independent of how fast the model decodes afterward.
The prefill formula
prefill tok/s ≈ (dense FP16 TFLOPS × MFU) ÷ (2 × active parameters)
The factor of 2 is one multiply and one accumulate per parameter per token — the standard FLOPs-per-parameter convention for a transformer forward pass. MFU (model-FLOPs-utilization) is the fraction of a device's theoretical peak FP16 throughput that a real prefill pass actually sustains.
Prefill MFU by platform
Calibrated against llama.cpp pp512 prompt-processing benchmarks on Q4_K_M-quantized weights. Real-world results vary by roughly ±30% with prompt length, batch size, SKU, and driver/runtime version.
Why a cheap card can out-prefill an expensive one
A pro card's real edge is capacity
Mixture-of-Experts (MoE) models only spend their active parameters per token — see Dense vs MoE. When a model's name doesn't label its active-parameter count, we fall back to the total parameter count in the formula above, which understates its real prefill speed.
Memory Bandwidth
Memory bandwidth (GB/s) is the throughput between a GPU's processor and its VRAM. Because LLM decode reads the full model on every token, bandwidth is the primary bottleneck for generation speed — more bandwidth means more tokens per second for the same model.
Bandwidth comparison (GB/s)
Higher bandwidth = faster tok/s at the same model size. This is why Apple Silicon Macs with unified high-bandwidth memory can outperform discrete GPUs that have more VRAM but lower bandwidth.
Explore bandwidth comparisons on any hardware listing page or VRAM tier page.
Dense vs Mixture of Experts (MoE)
Most LLMs are dense — every parameter is used on every token. A Mixture of Experts (MoE) model splits its parameters into groups called experts and only activates a subset per token. The result: higher quality with better speed, but the entire model still has to fit in memory.
Dense Architecture
All parameters active on every token. VRAM = total params. Speed scales with total params.
Example: Llama 3 70B — 70 B total, 70 B active per token.
MoE Architecture
Only 2 of 8 experts active per token. VRAM = total params. Speed ≈ active params only.
Example: Mixtral 8×7B — 46.7 B total, ~12.9 B active per token.
On llmrun, MoE models show both the total parameter count (for VRAM sizing) and the active count (for speed intuition).
Context Length
Context length defines the maximum number of tokens (input + output) the model can handle in a single conversation. A "128K context" model can process roughly 100,000 words at once — enough for entire codebases or long documents.
The catch: longer contexts consume additional VRAM through the KV cache, which grows with every token in the conversation.
Default (2K–4K)
Extended (16K+)
Max context
Our compatibility grades assume a 2K-token context. If you need long-context inference, look for extra VRAM headroom or check the "+Context" VRAM on model pages.
Efficiency & Value
Raw tok/s isn't the only number worth optimizing for. llmrun also surfaces two efficiency metrics that matter more once a card is already fast enough to feel usable: how much throughput you get per watt, and how much throughput you get per dollar.
Tokens per watt
tok/W = decode tok/s ÷ TDP watts
Higher is better. Matters most for small-form-factor, laptop, and always-on builds, where power and thermal budget are as constrained as raw speed.
Dollars per tok/s
$/tok-s = hardware MSRP ÷ decode tok/s
Lower is better — it means fewer hardware dollars spent per token per second of decode throughput.
Both metrics matter more than raw tok/s once a card clears the point where it "feels" fast: past that point the question shifts from "can I use this interactively" to "did I get good value for the power and money spent."
$/tok-s is MSRP-based. It uses list price, not street price, so it understates the value of cards that regularly sell below MSRP (previous-generation or secondhand cards) and overstates the value of cards that sell above MSRP (scalped or supply-constrained launches).
Compatibility Grades
Every hardware–model combination on llmrun receives a letter grade from S to F that answers one question: how well will this model run for me?
Two different scores feed into what you see on llmrun, and they do different jobs. The displayed grade blends how much VRAM headroom a model leaves with its estimated decode speed (tok/s), weighting speed more heavily since the tight-fit cap below already handles the danger case.
Headroom is scored with a deliberate plateau: once a model leaves "enough" spare VRAM for the KV cache and OS overhead, extra headroom stops adding to the score — spare VRAM is never penalised. A model that only barely fits is hard-capped, even if its raw decode speed looks good on paper, because there's no room left for a longer conversation.
A separate hardware-fit score measures how fully a model uses a card's VRAM — it peaks around ~75% utilization and drops off at both extremes, since a 1B model "fitting" on a 48GB card is a poor recommendation even though it technically runs. This fit score is used only to order recommendation lists, never to grade them.
Earlier versions of llmrun used the fit score for both jobs — ordering and grading. That conflation is why a large-VRAM professional card could show a mediocre letter for a small model it ran excellently: the model didn't use enough of the card's capacity, even though it ran fast with huge headroom to spare. The grade and the fit score are now computed separately for exactly this reason.
When These Numbers Were Calibrated
A benchmark without a date and a software stack attached is not reproducible. Every tok/s, time-to-first-token, tok/W, and $/tok-s figure on llmrun is a model calibrated against a specific inference runtime and vendor driver stack — not a live measurement of your exact card. Here is exactly what that calibration currently is.
- Last reviewed
- 2026-07-30
- Inference runtime
- llama.cpp / Ollama (GGUF)
Vendor compute / driver stack
- NVIDIA
- CUDA 12.x
- AMD
- ROCm 7.x
- Intel
- oneAPI / SYCL
- Apple
- Metal
GPU driver releases and llama.cpp/Ollama runtime updates shift real achievable throughput over time — sometimes significantly. Treat every figure derived from this basis as accurate as of the date above, not as a permanent ground truth.
Accuracy & Limitations
Every figure on llmrun is a calibrated estimate, not a measurement taken on your exact hardware. Here is what the model does not account for.
Single user, batch size 1
Multi-GPU setups
Runtime optimizations
MoE routing
Error bars: expect roughly ±20% on decode tok/s and roughly ±30% on prefill throughput — and wider than that on any hardware without a verified compute figure.
These are estimates from a calibrated model, not measurements taken on this hardware. GPU driver and inference-runtime releases move real throughput over time — that's why we publish exactly what the calibration is and when it was last reviewed in When These Numbers Were Calibrated.
Frequently Asked Questions
- How do you calculate VRAM requirements?
- We compute VRAM as Model Weights + KV Cache + Framework Overhead. Model weights = parameters × bits-per-weight ÷ 8. KV cache depends on the model's attention architecture (number of KV heads, head dimension, and layers). When architecture data is available, we compute the precise KV cache size; otherwise we fall back to a 10% overhead estimate. Measured values from Ollama or llama.cpp always take priority.
- How accurate are the tok/s estimates?
- They're based on the theoretical bandwidth formula: tok/s ≈ (bandwidth GB/s ÷ model size GB) × efficiency. The efficiency factor varies by platform: 65% for NVIDIA (CUDA), 60% for AMD (ROCm 7.x), 50% for Intel (oneAPI), and 70% for Apple Silicon (Metal). In practice, results typically land within ±20% of these figures. This covers decode (generation) speed only — see "What is time to first token?" for prompt-processing speed.
- What does the S–F grade mean?
- It summarises speed and VRAM headroom into one letter. S means the model runs fast with plenty of room to spare. F means your hardware can't load the model at all. See Compatibility Grades below for how the grade and the separate hardware-fit ranking score differ.
- Why does VRAM headroom matter?
- A model that barely fits in VRAM leaves nothing for the KV cache, which grows with conversation length. Models with Grouped-Query Attention (GQA) are more memory-efficient for long contexts. We recommend at least 1–2 GB above the model weight size for comfortable use.
- What is the GGUF format?
- GGUF is the standard file format for quantized models used by llama.cpp, Ollama, and LM Studio. It packages weights, tokenizer, and metadata into a single ready-to-run file.
- What is time to first token, and what affects it?
- Time to first token (TTFT) is how long you wait after sending a prompt before the model starts streaming a reply. Unlike tok/s (which measures decode, one token at a time), TTFT is governed by prefill — a compute-bound pass that processes the whole prompt at once. It scales with prompt length and with the device's dense FP16 compute throughput and model-FLOPs-utilization (MFU), not with memory bandwidth. Longer prompts and lower-MFU hardware both increase TTFT.
- Why can an expensive professional GPU show a lower tok/s than a cheap gaming GPU?
- The tok/s figure shown across the site measures decode speed, which is memory-bandwidth bound — a consumer card with fast, narrow VRAM can decode a small model very quickly per token. A professional card's real advantage is capacity: how large a model it can load at all, not raw decode or prefill speed. For a model that fits both cards, the cheaper card can legitimately be faster.
- What does llmrun's performance model not account for?
- Batch size above 1 / concurrent requests, multi-GPU tensor and pipeline parallelism and interconnect cost, speculative decoding, flash-attention variants, quantized KV cache, CPU/RAM offload, and Mixture-of-Experts routing beyond a simple active-parameter approximation. See Accuracy & Limitations below for the full list and our stated error bars.
- How often are the performance estimates recalibrated?
- Every tok/s, TTFT, tok/W, and $/tok-s figure on llmrun is calibrated against a specific inference runtime and vendor driver stack, not measured live on your exact card. That basis was last reviewed on 2026-07-30 against llama.cpp / Ollama (GGUF). GPU driver releases and runtime updates shift real achievable throughput over time, so we periodically re-review these constants — see When These Numbers Were Calibrated below for the current basis.
Data Sources
Hardware specs come from manufacturer datasheets and are cross-referenced with community benchmarks. Model VRAM figures are collected from:
- Ollama — model library and runtime measurements
- llama.cpp — community benchmarks and perplexity data
- Hugging Face — model cards and architecture metadata
Spot an error? Email data@llmrun.dev — data corrections are always welcome.