llmrun estimates that Qwen3.6 35B A3B writes about 151 tokens a second on an RTX 4090. The estimate for the smaller, dense Qwen3.8 27B is 37.6 on the same card. The bigger model owes its fourfold estimated speed to a mixture of experts (MoE), with all 36 billion parameters held in memory and only a small slice read for each token. MoE has been around for 35 years and is why a 24 GB gaming card can now run a 35B model at an estimated speed of well over 100 tokens a second. All experts still have to be kept in memory, with the speed advantage getting smaller as the conversation grows longer.
Key takeaways
- We estimate MoE to give local inference its biggest speed-up, about 4× over a dense model of similar size on a 24 GB card, as long as the whole model fits on the GPU.
- Even though far less is read per token, a 35B MoE with all experts loaded requires more memory than a 27B dense model.
- Our estimates charge more fixed work per token on Macs, so their gain is the smallest. Discrete cards, AMD's included, have the greatest gain.
- Long contexts and offloading to system RAM remove much of the MoE advantage. In one public benchmark, the MoE advantage was 4.6× with short prompts and 2× with a 64K context.
A 1991 idea that waited for big models
Chatbots were outside the scope of the first paper. In 1991, Robert Jacobs, Michael Jordan, Steven Nowlan and Geoffrey Hinton published "Adaptive Mixtures of Local Experts", describing small networks, "each of which learns to handle a subset of the complete set of training cases", and "a gating network that decides which of the experts should be used for each training case". The demonstration dealt with a "vowel discrimination task".
The idea found a use once models became too large to run whole. In 2017, Noam Shazeer and co-authors including Hinton and Jeff Dean published "Outrageously Large Neural Networks", a paper that lived up to its title. It combined "up to 137 billion parameters" in thousands of experts, claiming more than 1000× the model capacity with "only minor losses in computational efficiency". Google's Switch Transformer took that to a trillion in 2021. These were all data-centre models.
MoE reached local models on Friday 8 December 2023, when Mistral AI, as Simon Willison put it, "tweeted a mysterious magnet (BitTorrent) link". Mixtral 8x7B was the 87 GB download. The announcement three days later gave a total of 46.7 billion parameters, of which only 12.9 billion were used per token, with a claim of "6x faster inference" than Llama 2 70B.
The ratio of total to active parameters has continued to grow since then. DeepSeek-V3 has 37 of its 671 billion parameters "activated for each token". The "A3B" in Qwen's April 2025 Qwen3-30B-A3B refers to about "3 billion activated parameters", and Qwen claimed it beat its own dense QwQ-32B with a tenth of the active parameters. OpenAI's gpt-oss pair was released in August, and Google's Gemma 4 followed in April 2026 with its 26B MoE focused "on latency" and its 31B dense sibling "maximizing raw quality".
A library where each token reads a few shelves
The model is rather like a library of weights. A dense model is reading the entire library for each word that it writes. A small router picks a handful of experts, the shelves of this library, for each token from the many that make up most of each MoE layer. Each token reads only a few shelves, but since the next may need any shelf the whole library stays in the building.
The speed of reading weights from memory is the bottleneck to text generation on a GPU. The fewer bytes you read per token, the more tokens you produce per second. Qwen3.6 35B A3B has 36.0B parameters, of which our catalogue counts 4.7B as active, a bit more than the name suggests.
Yet the MoE is less than six times as fast, even though it reads one-sixth of the data. The routing time for each token does not depend on memory speed. The MoE is about four times faster than the dense model on an RTX 4090 in our estimates. Our speeds are estimates based on llmrun's model calibrated against llama.cpp measurements and described in How llmrun determines model requirements.
Four times faster on the same card
Most 24 GB owners choose between Qwen3.8 27B (dense) and Qwen3.6 35B A3B (MoE). Both models fit fully in GPU memory at Q4_K_M.
- Dense 27B
- MoE 35B A3B
RX 7900 XTX
Dense 27B33.1MoE 35B A3B138.3RTX 3090
Dense 27B34.9MoE 35B A3B143.6RTX 4090
Dense 27B37.6MoE 35B A3B151RTX 5090
Dense 27B66.9MoE 35B A3B213.3Mac Studio M5 Max 64 GB
Dense 27B24.7MoE 35B A3B67.8Mac Studio M5 Ultra 96 GB
Dense 27B49.3MoE 35B A3B87.9
| Item | Dense 27B | MoE 35B A3B |
|---|---|---|
| RX 7900 XTX | 33.1 | 138.3 |
| RTX 3090 | 34.9 | 143.6 |
| RTX 4090 | 37.6 | 151 |
| RTX 5090 | 66.9 | 213.3 |
| Mac Studio M5 Max 64 GB | 24.7 | 67.8 |
| Mac Studio M5 Ultra 96 GB | 49.3 | 87.9 |
The estimated RTX 4090 time for the dense model to write a whole 600-token reply is about 16 seconds, compared to about 4 seconds for the MoE. The writing speed of either model exceeds anyone's reading speed. Adults average about 238 words a minute for non-fiction. The difference is in coding agents and tool loops, where many replies go unread before the next step starts.
Hardware Corner measured short-prompt RTX 3090 speeds of 153.6 tok/s for the older Qwen3 30B A3B and 33.5 for the dense Qwen3.5 27B, a 4.6× gap. With different models used, this comparison is only a sanity check.
The dense Qwen is a newer-generation model, which makes the quality comparison uneven. Its llmrun Score is 121.9 and the MoE's is 106.8. The Gemma 4 comparison is cleaner in that both models are from the same family and release.
- Dense 31B
- MoE 26B A4B
RX 7900 XTX
Dense 31B28.2MoE 26B A4B155.8RTX 3090
Dense 31B29.8MoE 26B A4B162.2RTX 4090
Dense 31B32.1MoE 26B A4B171.1RTX 5090
Dense 31B57.1MoE 26B A4B249.5Mac Studio M5 Max 64 GB
Dense 31B21.1MoE 26B A4B80.7Mac Studio M5 Ultra 96 GB
Dense 31B42.2MoE 26B A4B108.7
| Item | Dense 31B | MoE 26B A4B |
|---|---|---|
| RX 7900 XTX | 28.2 | 155.8 |
| RTX 3090 | 29.8 | 162.2 |
| RTX 4090 | 32.1 | 171.1 |
| RTX 5090 | 57.1 | 249.5 |
| Mac Studio M5 Max 64 GB | 21.1 | 80.7 |
| Mac Studio M5 Ultra 96 GB | 42.2 | 108.7 |
The llmrun Score is 105.8 for Gemma 4 26B A4B and 106.3 for the dense Gemma 4 31B. The estimated speed gain comes at the cost of half a point. So much for MoE trading quality for speed.
Biggest on AMD cards, smallest on Macs
In either chart, the ratios vary from machine to machine. This is due to the fixed cost of routing each token. With slow memory, most of each token's time is spent reading weights, so there is a lot to gain from reading less. The share of time spent routing each token increases as memory reads get faster.
- RX 7900 XTX4.2
- RTX 30904.1
- RTX 40904
- RTX 50903.2
- Mac Studio M5 Max 64 GB2.7
- Mac Studio M5 Ultra 96 GB1.8
| Item | Qwen3.6 35B A3B speed ÷ Qwen3.8 27B speed, estimated, Q4_K_M |
|---|---|
| RX 7900 XTX | 4.2 |
| RTX 3090 | 4.1 |
| RTX 4090 | 4 |
| RTX 5090 | 3.2 |
| Mac Studio M5 Max 64 GB | 2.7 |
| Mac Studio M5 Ultra 96 GB | 1.8 |
Of these cards, the RTX 5090 has the smallest estimated gain, so its lead over the 4090 narrows on MoE models. For AMD, the position has changed. Until 15 November, a single old ROCm result was the basis of our AMD MoE estimates, and the cards were well behind in those estimates. For gpt-oss-20b on an RX 9070 XT, the newer llama.cpp results published on GitHub were 148.5 tok/s for ROCm and 180.8 for Vulkan. We re-fitted our estimates after these runs showed no extra MoE overhead. AMD is now assigned the same estimated routing cost as NVIDIA; with its slower memory, the RX 7900 XTX gets the largest estimated speed-up in the chart. We have only one secondary report for the Intel estimates, so Arc is left out here, and our AMD and Intel guide covers the software.
Our Mac routing cost is higher and is based on llama.cpp runs on Apple chips. That is why an M5 Ultra gives twice the dense speed of an M5 Max in our estimates, but only 30% more MoE speed. We consider these Apple ratios the least settled figures in the article, and our Mac guide has more detail.
The bill arrives in memory
Google's Gemma documentation describes the 26B model as one that "only activates 4 billion parameters per token during generation", while "all 26 billion parameters must be loaded into memory". All of the experts stay loaded in memory in case the next token requires any of them. Memory follows the total parameter count.
| Model | Total / active | Memory |
|---|---|---|
| gpt-oss-20b | 20.9B / 4.2B | 13.1 GB |
| Gemma 4 26B A4B | 25.8B / 4.4B | 16.5 GB |
| Qwen3.8 27B (dense) | 27.8B / 27.8B | 17.9 GB |
| Gemma 4 31B (dense) | 31.3B / 31.3B | 21.7 GB |
| Qwen3.6 35B A3B | 36.0B / 4.7B | 22.0 GB |
| gpt-oss-120b | 116.8B / 5.8B | 70.8 GB |
| DeepSeek V4.1 Flash | 763.2B / 16.0B | 458.4 GB |
Q4_K_M with a 16K context, llmrun estimates, 2026-11-16. The active counts in our catalogue are higher than the vendors' figures (OpenAI gives 5.1B and 3.6B for gpt-oss). There is disagreement on DeepSeek V4.1 Flash, so the figures quoted are our own.
When the MoE is larger than the dense model it replaces, its speed comes at the cost of memory. We estimate the memory requirement of Qwen3.6 35B A3B to be 4.2 GB higher than that of Qwen3.8 27B. The MoE uses 92% of a 24 GB card, with little room for a long context. The Gemma MoE saves memory as well as time, as it is the smaller model.
| Model | RTX 5070 Ti | RTX 4090 | RTX 5090 | Mac Studio M5 Max (64 GB) | Mac Studio M5 Ultra (96 GB) | Mac Studio M5 Ultra (512 GB) |
|---|---|---|---|---|---|---|
| GPT OSS 20B | S, Runs greatQ4_K_M · ~173 tok/s | S, Runs greatQ4_K_M · ~190 tok/s | S, Runs greatQ4_K_M · ~282 tok/s | S, Runs greatQ4_K_M · ~92 tok/s | S, Runs greatQ4_K_M · ~128 tok/s | S, Runs greatQ4_K_M · ~128 tok/s |
| Gemma 4 26B A4B IT | B, DecentIQ4_NL · ~164 tok/s | S, Runs greatQ4_K_M · ~171 tok/s | S, Runs greatQ4_K_M · ~250 tok/s | S, Runs greatQ4_K_M · ~81 tok/s | S, Runs greatQ4_K_M · ~109 tok/s | S, Runs greatQ4_K_M · ~109 tok/s |
| Qwen3.6 35B A3B | C, UsableIQ3_S · ~174 tok/s | A, Runs wellQ4_K_M · ~151 tok/s | S, Runs greatQ4_K_M · ~213 tok/s | S, Runs greatQ4_K_M · ~68 tok/s | S, Runs greatQ4_K_M · ~88 tok/s | S, Runs greatQ4_K_M · ~88 tok/s |
| GPT OSS 120B | Doesn't fitneeds 40 GB+ | Doesn't fitneeds 40 GB+ | Doesn't fitneeds 40 GB+ | C, UsableQ4_0 · ~72 tok/s | S, Runs greatQ4_K_M · ~89 tok/s | S, Runs greatQ4_K_M · ~89 tok/s |
| DeepSeek V4.1 Flash | Doesn't fitneeds 210 GB+ | Doesn't fitneeds 210 GB+ | Doesn't fitneeds 210 GB+ | Doesn't fitneeds 210 GB+ | Doesn't fitneeds 210 GB+ | A, Runs wellQ4_K_M · ~52 tok/s |
- S, Runs greatGPT OSS 20BQ4_K_M · ~173 tok/s
- B, DecentGemma 4 26B A4B ITIQ4_NL · ~164 tok/s
- C, UsableQwen3.6 35B A3BIQ3_S · ~174 tok/s
- —GPT OSS 120BDoesn't fit · needs 40 GB+
- —DeepSeek V4.1 FlashDoesn't fit · needs 210 GB+
- S, Runs greatGPT OSS 20BQ4_K_M · ~190 tok/s
- S, Runs greatGemma 4 26B A4B ITQ4_K_M · ~171 tok/s
- A, Runs wellQwen3.6 35B A3BQ4_K_M · ~151 tok/s
- —GPT OSS 120BDoesn't fit · needs 40 GB+
- —DeepSeek V4.1 FlashDoesn't fit · needs 210 GB+
- S, Runs greatGPT OSS 20BQ4_K_M · ~282 tok/s
- S, Runs greatGemma 4 26B A4B ITQ4_K_M · ~250 tok/s
- S, Runs greatQwen3.6 35B A3BQ4_K_M · ~213 tok/s
- —GPT OSS 120BDoesn't fit · needs 40 GB+
- —DeepSeek V4.1 FlashDoesn't fit · needs 210 GB+
- S, Runs greatGPT OSS 20BQ4_K_M · ~92 tok/s
- S, Runs greatGemma 4 26B A4B ITQ4_K_M · ~81 tok/s
- S, Runs greatQwen3.6 35B A3BQ4_K_M · ~68 tok/s
- C, UsableGPT OSS 120BQ4_0 · ~72 tok/s
- —DeepSeek V4.1 FlashDoesn't fit · needs 210 GB+
- S, Runs greatGPT OSS 20BQ4_K_M · ~128 tok/s
- S, Runs greatGemma 4 26B A4B ITQ4_K_M · ~109 tok/s
- S, Runs greatGPT OSS 120BQ4_K_M · ~89 tok/s
- S, Runs greatQwen3.6 35B A3BQ4_K_M · ~88 tok/s
- —DeepSeek V4.1 FlashDoesn't fit · needs 210 GB+
- S, Runs greatGPT OSS 20BQ4_K_M · ~128 tok/s
- S, Runs greatGemma 4 26B A4B ITQ4_K_M · ~109 tok/s
- S, Runs greatGPT OSS 120BQ4_K_M · ~89 tok/s
- S, Runs greatQwen3.6 35B A3BQ4_K_M · ~88 tok/s
- A, Runs wellDeepSeek V4.1 FlashQ4_K_M · ~52 tok/s
The gpt-oss-120b row is of particular interest. It reads only a little more per token than the 20b model, so it is fast for its size on a machine that can hold its 71 GB, at an estimated 89 tok/s on a Mac Studio M5 Ultra (96 GB). At Q4_0, a 64 GB Mac can just fit it with 98% of memory in use. The 128 GB boxes were built around this model class.
Long prompts and offload eat the lead
The estimates above are for short prompts with the whole model in fast memory. Both assumptions break down in real use.
The likely reason is the same attention cost over a long context for both kinds of model, with a much larger share of the MoE's small token budget taken up by it. In those runs, the MoE was still about twice as fast at 64K tokens, a real gain though smaller than the headline one. The estimated answer-start time on the RTX 4090 for a 4,000-token prompt is about half a second for the MoE, compared to about three seconds for the dense 27B. Since our prompt-reading estimates look about twice as cautious as community measurements, both are likely quicker in practice.
For weights that do not fit, MoE has an option that dense models lack. Since August 2025, llama.cpp has had a --n-cpu-moe flag (pull request #15077) that keeps the expert weights of some layers in system RAM and the rest on the GPU. As it reads only a few experts per token, MoE accesses slow RAM less than a dense model would. It is slow all the same. The 8K-context measurements for Qwen3.6 35B A3B on a 24 GB NVIDIA L4 from localllm.in were 65.6 tok/s fully on the GPU and 22.3 tok/s with 12 layers of experts on the CPU. The same setting let the card run at 22.7 tok/s with a 64K context, where the all-GPU run ran out of memory. The speeds estimated by llmrun exclude offloading.
Which MoE model fits your memory
How much memory you have comes first, followed by the choice between speed and Score.
| Memory | Model | llmrun Score | What to expect |
|---|---|---|---|
| 16 GB | gpt-oss-20b | 96.4 | fits at Q4_K_M with room for context |
| 24 GB | Gemma 4 26B A4B | 105.8 | the fastest pick at 24 GB, near its dense sibling's Score |
| 24 GB | Qwen3.6 35B A3B | 106.8 | fills 92% of the card, little left for context |
| 64 to 128 GB | gpt-oss-120b | 97.5 | Q4_0 at 64 GB, Q4_K_M from 96 GB |
| 512 GB | DeepSeek V4.1 Flash | 129.4 | the highest Score here, only on a 512 GB Mac |
The 26B and 35B models load only at 3 to 4 bits and nearly fill a 16 GB card such as the RTX 5070 Ti while gpt-oss-20b can fit comfortably, as our 16 GB guide shows. For 24 GB, it depends on the work. Our pick for chat and one-off questions where the answer matters most is the dense Qwen3.8 27B. It scores 121.9 in llmrun Score at an estimated 37.6 tok/s on an RTX 4090. In repeated agent loops, the 24 GB MoE models give roughly four times the estimated speed for about 15 points less Score. For the same split, see the November list of best local LLMs; the cards that run both are covered in our used RTX 3090 review.