llmrun

Tech13 min read

MoE models: why local LLMs suddenly got fast

llmrun estimates that a MoE model is 3 to 5.5 times as fast as a dense model on one GPU.

llmrun estimates that Qwen3.6 35B A3B writes about 151 tokens a second on an RTX 4090. The estimate for the smaller, dense Qwen3.8 27B is 37.6 on the same card. The bigger model owes its fourfold estimated speed to a mixture of experts (MoE), with all 36 billion parameters held in memory and only a small slice read for each token. MoE has been around for 35 years and is why a 24 GB gaming card can now run a 35B model at an estimated speed of well over 100 tokens a second. All experts still have to be kept in memory, with the speed advantage getting smaller as the conversation grows longer.

Key takeaways

  1. We estimate MoE to give local inference its biggest speed-up, about 4× over a dense model of similar size on a 24 GB card, as long as the whole model fits on the GPU.
  2. Even though far less is read per token, a 35B MoE with all experts loaded requires more memory than a 27B dense model.
  3. Our estimates charge more fixed work per token on Macs, so their gain is the smallest. Discrete cards, AMD's included, have the greatest gain.
  4. Long contexts and offloading to system RAM remove much of the MoE advantage. In one public benchmark, the MoE advantage was 4.6× with short prompts and 2× with a 64K context.

A 1991 idea that waited for big models

Chatbots were outside the scope of the first paper. In 1991, Robert Jacobs, Michael Jordan, Steven Nowlan and Geoffrey Hinton published "Adaptive Mixtures of Local Experts", describing small networks, "each of which learns to handle a subset of the complete set of training cases", and "a gating network that decides which of the experts should be used for each training case". The demonstration dealt with a "vowel discrimination task".

The idea found a use once models became too large to run whole. In 2017, Noam Shazeer and co-authors including Hinton and Jeff Dean published "Outrageously Large Neural Networks", a paper that lived up to its title. It combined "up to 137 billion parameters" in thousands of experts, claiming more than 1000× the model capacity with "only minor losses in computational efficiency". Google's Switch Transformer took that to a trillion in 2021. These were all data-centre models.

MoE reached local models on Friday 8 December 2023, when Mistral AI, as Simon Willison put it, "tweeted a mysterious magnet (BitTorrent) link". Mixtral 8x7B was the 87 GB download. The announcement three days later gave a total of 46.7 billion parameters, of which only 12.9 billion were used per token, with a claim of "6x faster inference" than Llama 2 70B.

The ratio of total to active parameters has continued to grow since then. DeepSeek-V3 has 37 of its 671 billion parameters "activated for each token". The "A3B" in Qwen's April 2025 Qwen3-30B-A3B refers to about "3 billion activated parameters", and Qwen claimed it beat its own dense QwQ-32B with a tenth of the active parameters. OpenAI's gpt-oss pair was released in August, and Google's Gemma 4 followed in April 2026 with its 26B MoE focused "on latency" and its 31B dense sibling "maximizing raw quality".

A library where each token reads a few shelves

The model is rather like a library of weights. A dense model is reading the entire library for each word that it writes. A small router picks a handful of experts, the shelves of this library, for each token from the many that make up most of each MoE layer. Each token reads only a few shelves, but since the next may need any shelf the whole library stays in the building.

The speed of reading weights from memory is the bottleneck to text generation on a GPU. The fewer bytes you read per token, the more tokens you produce per second. Qwen3.6 35B A3B has 36.0B parameters, of which our catalogue counts 4.7B as active, a bit more than the name suggests.

17.4GB
read per token: dense Qwen3.8 27B, Q4_K_M
2.9GB
read per token: Qwen3.6 35B A3B, Q4_K_M
13%
share of the MoE model's weights each token reads

Yet the MoE is less than six times as fast, even though it reads one-sixth of the data. The routing time for each token does not depend on memory speed. The MoE is about four times faster than the dense model on an RTX 4090 in our estimates. Our speeds are estimates based on llmrun's model calibrated against llama.cpp measurements and described in How llmrun determines model requirements.

Four times faster on the same card

Most 24 GB owners choose between Qwen3.8 27B (dense) and Qwen3.6 35B A3B (MoE). Both models fit fully in GPU memory at Q4_K_M.

SpeedComparison chart: The Qwen MoE runs 3.2× to 4.2× faster on NVIDIA and AMD cards, 1.8× to 2.7× on Macsestimated tok/s
  • Dense 27B
  • MoE 35B A3B
  • RX 7900 XTX

    Dense 27B33.1
    MoE 35B A3B138.3
  • RTX 3090

    Dense 27B34.9
    MoE 35B A3B143.6
  • RTX 4090

    Dense 27B37.6
    MoE 35B A3B151
  • RTX 5090

    Dense 27B66.9
    MoE 35B A3B213.3
  • Mac Studio M5 Max 64 GB

    Dense 27B24.7
    MoE 35B A3B67.8
  • Mac Studio M5 Ultra 96 GB

    Dense 27B49.3
    MoE 35B A3B87.9
Source: llmrun speed model, Q4_K_M, active parameters from llmrun catalogue · Data as of Nov 16, 2026

The estimated RTX 4090 time for the dense model to write a whole 600-token reply is about 16 seconds, compared to about 4 seconds for the MoE. The writing speed of either model exceeds anyone's reading speed. Adults average about 238 words a minute for non-fiction. The difference is in coding agents and tool loops, where many replies go unread before the next step starts.

Hardware Corner measured short-prompt RTX 3090 speeds of 153.6 tok/s for the older Qwen3 30B A3B and 33.5 for the dense Qwen3.5 27B, a 4.6× gap. With different models used, this comparison is only a sanity check.

The dense Qwen is a newer-generation model, which makes the quality comparison uneven. Its llmrun Score is 121.9 and the MoE's is 106.8. The Gemma 4 comparison is cleaner in that both models are from the same family and release.

SpeedComparison chart: Gemma 4: the 26B A4B is 4.4× to 5.5× faster on the cards than the dense 31B, at a near-equal Scoreestimated tok/s
  • Dense 31B
  • MoE 26B A4B
  • RX 7900 XTX

    Dense 31B28.2
    MoE 26B A4B155.8
  • RTX 3090

    Dense 31B29.8
    MoE 26B A4B162.2
  • RTX 4090

    Dense 31B32.1
    MoE 26B A4B171.1
  • RTX 5090

    Dense 31B57.1
    MoE 26B A4B249.5
  • Mac Studio M5 Max 64 GB

    Dense 31B21.1
    MoE 26B A4B80.7
  • Mac Studio M5 Ultra 96 GB

    Dense 31B42.2
    MoE 26B A4B108.7
Source: llmrun speed model, Q4_K_M, active parameters from llmrun catalogue · Data as of Nov 16, 2026

The llmrun Score is 105.8 for Gemma 4 26B A4B and 106.3 for the dense Gemma 4 31B. The estimated speed gain comes at the cost of half a point. So much for MoE trading quality for speed.

Biggest on AMD cards, smallest on Macs

In either chart, the ratios vary from machine to machine. This is due to the fixed cost of routing each token. With slow memory, most of each token's time is spent reading weights, so there is a lot to gain from reading less. The share of time spent routing each token increases as memory reads get faster.

SpeedBar chart: The MoE speed-up is largest on the RX 7900 XTX and smallest on the M5 UltraQwen3.6 35B A3B speed ÷ Qwen3.8 27B speed, estimated, Q4_K_M
  • RX 7900 XTX4.2
  • RTX 30904.1
  • RTX 40904
  • RTX 50903.2
  • Mac Studio M5 Max 64 GB2.7
  • Mac Studio M5 Ultra 96 GB1.8
Source: llmrun speed model, ratios of the estimates above · Data as of Nov 16, 2026

Of these cards, the RTX 5090 has the smallest estimated gain, so its lead over the 4090 narrows on MoE models. For AMD, the position has changed. Until 15 November, a single old ROCm result was the basis of our AMD MoE estimates, and the cards were well behind in those estimates. For gpt-oss-20b on an RX 9070 XT, the newer llama.cpp results published on GitHub were 148.5 tok/s for ROCm and 180.8 for Vulkan. We re-fitted our estimates after these runs showed no extra MoE overhead. AMD is now assigned the same estimated routing cost as NVIDIA; with its slower memory, the RX 7900 XTX gets the largest estimated speed-up in the chart. We have only one secondary report for the Intel estimates, so Arc is left out here, and our AMD and Intel guide covers the software.

Our Mac routing cost is higher and is based on llama.cpp runs on Apple chips. That is why an M5 Ultra gives twice the dense speed of an M5 Max in our estimates, but only 30% more MoE speed. We consider these Apple ratios the least settled figures in the article, and our Mac guide has more detail.

The bill arrives in memory

Google's Gemma documentation describes the 26B model as one that "only activates 4 billion parameters per token during generation", while "all 26 billion parameters must be loaded into memory". All of the experts stay loaded in memory in case the next token requires any of them. Memory follows the total parameter count.

ModelTotal / activeMemory
gpt-oss-20b20.9B / 4.2B13.1 GB
Gemma 4 26B A4B25.8B / 4.4B16.5 GB
Qwen3.8 27B (dense)27.8B / 27.8B17.9 GB
Gemma 4 31B (dense)31.3B / 31.3B21.7 GB
Qwen3.6 35B A3B36.0B / 4.7B22.0 GB
gpt-oss-120b116.8B / 5.8B70.8 GB
DeepSeek V4.1 Flash763.2B / 16.0B458.4 GB

Q4_K_M with a 16K context, llmrun estimates, 2026-11-16. The active counts in our catalogue are higher than the vendors' figures (OpenAI gives 5.1B and 3.6B for gpt-oss). There is disagreement on DeepSeek V4.1 Flash, so the figures quoted are our own.

When the MoE is larger than the dense model it replaces, its speed comes at the cost of memory. We estimate the memory requirement of Qwen3.6 35B A3B to be 4.2 GB higher than that of Qwen3.8 27B. The MoE uses 92% of a 24 GB card, with little room for a long context. The Gemma MoE saves memory as well as time, as it is the smaller model.

FitFit matrix: At 16 GB the 26B and 35B MoE models drop below 4-bit; of these machines, gpt-oss-120b fits from 64 GBFit grade S (best) to F · best quant that fits with 16K context · speeds estimated
Source: llmrun catalogue · Data as of Nov 21, 2026

The gpt-oss-120b row is of particular interest. It reads only a little more per token than the 20b model, so it is fast for its size on a machine that can hold its 71 GB, at an estimated 89 tok/s on a Mac Studio M5 Ultra (96 GB). At Q4_0, a 64 GB Mac can just fit it with 98% of memory in use. The 128 GB boxes were built around this model class.

Long prompts and offload eat the lead

The estimates above are for short prompts with the whole model in fast memory. Both assumptions break down in real use.

The likely reason is the same attention cost over a long context for both kinds of model, with a much larger share of the MoE's small token budget taken up by it. In those runs, the MoE was still about twice as fast at 64K tokens, a real gain though smaller than the headline one. The estimated answer-start time on the RTX 4090 for a 4,000-token prompt is about half a second for the MoE, compared to about three seconds for the dense 27B. Since our prompt-reading estimates look about twice as cautious as community measurements, both are likely quicker in practice.

For weights that do not fit, MoE has an option that dense models lack. Since August 2025, llama.cpp has had a --n-cpu-moe flag (pull request #15077) that keeps the expert weights of some layers in system RAM and the rest on the GPU. As it reads only a few experts per token, MoE accesses slow RAM less than a dense model would. It is slow all the same. The 8K-context measurements for Qwen3.6 35B A3B on a 24 GB NVIDIA L4 from localllm.in were 65.6 tok/s fully on the GPU and 22.3 tok/s with 12 layers of experts on the CPU. The same setting let the card run at 22.7 tok/s with a 64K context, where the all-GPU run ran out of memory. The speeds estimated by llmrun exclude offloading.

Which MoE model fits your memory

How much memory you have comes first, followed by the choice between speed and Score.

MemoryModelllmrun ScoreWhat to expect
16 GBgpt-oss-20b96.4fits at Q4_K_M with room for context
24 GBGemma 4 26B A4B105.8the fastest pick at 24 GB, near its dense sibling's Score
24 GBQwen3.6 35B A3B106.8fills 92% of the card, little left for context
64 to 128 GBgpt-oss-120b97.5Q4_0 at 64 GB, Q4_K_M from 96 GB
512 GBDeepSeek V4.1 Flash129.4the highest Score here, only on a 512 GB Mac

The 26B and 35B models load only at 3 to 4 bits and nearly fill a 16 GB card such as the RTX 5070 Ti while gpt-oss-20b can fit comfortably, as our 16 GB guide shows. For 24 GB, it depends on the work. Our pick for chat and one-off questions where the answer matters most is the dense Qwen3.8 27B. It scores 121.9 in llmrun Score at an estimated 37.6 tok/s on an RTX 4090. In repeated agent loops, the 24 GB MoE models give roughly four times the estimated speed for about 15 points less Score. For the same split, see the November list of best local LLMs; the cards that run both are covered in our used RTX 3090 review.

Questions

Why is a MoE model faster if it has more parameters?
Text generation on a GPU is limited by the speed of the available memory. The router selects only a few experts to be read for each token.
Does a MoE model need less VRAM because only a few parameters are active?
All of the experts are kept in memory because the router may select any of them for the next token. Memory follows the total parameter count.
Are MoE models fast on AMD GPUs?
Yes, in our estimates. An RX 7900 XTX runs Qwen3.6 35B A3B at an estimated 138.3 tok/s, close to the 143.6 of an RTX 3090.
Is MoE worth it on a Mac?
The estimated gains for Qwen3.6 35B A3B over the dense 27B are 2.7× on an M5 Max and 1.8× on an M5 Ultra, smaller than on a discrete GPU. The bigger advantage is the Mac's memory size, which allows models such as gpt-oss-120b to run at all.
Can I run a MoE model bigger than my VRAM?
Some experts can be kept in system RAM through llama.cpp, at a large speed cost. The speeds estimated by llmrun exclude offloading.