The default 4-bit file for Qwen3.8 27B on its model page is Q4_K_M, which requires 17.9 GB with a 16K-token context by our estimate. This makes a 24 GB card its natural home, with enough room to support a 64K context. There is room for a 3-bit file on 16 GB, just enough on 12 GB, and no room for the model on 8 GB. What surprises most people is how little the context costs. With only one layer out of four retaining a growing memory of the conversation, the move from 16K to 64K tokens costs under 3 GB.
Key takeaways
- No compromise is needed at 24 GB. Our estimate for Q4_K_M with a 64K context is 20.6 GB in total.
- For a 16 GB card, we recommend a 3-bit quant and an 8-bit cache. Four-bit will only fit with a short context and nothing else on the card.
- This model is using about a third as much cache as last year's Qwen3 32B for the same context length.
- Q4_K_M worked without any loss of quality against full precision in outside tests. There is currently little evidence below 4-bit, with a collapse in quality at 1-bit.
- Time is a factor as well as memory. At the default effort, the model can spend thousands of tokens on reasoning, which takes minutes on an RTX 4090 by our estimate.
Where the 18 GB goes
The card has three demands on its memory. The size of the weights file at Q4_K_M is 16.7 GB, which changes with the quant you download. The key-value cache of the conversation so far grows as more tokens of context are allowed. The runtime takes about 0.3 GB of memory for itself. Loading the vision projector for reading images takes extra memory as well, so one 16 GB owner on Hacker News simply switched it off.
With about 3 GB saved at each step down, the quant decides whether the model fits.
| Quant | Bits | Weights | KV 16K | KV 32K | KV 64K | Total 16K | Total 32K | Total 64K | Fits in |
|---|---|---|---|---|---|---|---|---|---|
| IQ3_XXS | 3.10 | 10.8 | 0.9 | 1.8 | 3.6 | 12.0 | 12.8 | 14.6 | 16 GB+ |
| Q3_K_M | 3.90 | 13.5 | 0.9 | 1.8 | 3.6 | 14.7 | 15.6 | 17.4 | 24 GB+ |
| IQ4_XS | 4.30 | 14.9 | 0.9 | 1.8 | 3.6 | 16.1 | 17.0 | 18.8 | 24 GB+ |
| Q4_K_M | 4.80 | 16.7 | 0.9 | 1.8 | 3.6 | 17.9 | 18.8 | 20.6 | 24 GB+ |
| Q5_K_M | 5.70 | 19.8 | 0.9 | 1.8 | 3.6 | 21.0 | 21.9 | 23.7 | 24 GB+ |
| Q6_K | 6.60 | 22.9 | 0.9 | 1.8 | 3.6 | 24.1 | 25.0 | 26.8 | 32 GB+ |
| Q8_0 | 8.00 | 27.8 | 0.9 | 1.8 | 3.6 | 29.0 | 29.9 | 31.7 | 32 GB+ |
- IQ3_XXSweights 10.8 · fits 16 GB12.0 / 12.8 / 14.6GB at 16K / 32K / 64K
- Q3_K_Mweights 13.5 · fits 24 GB14.7 / 15.6 / 17.4GB at 16K / 32K / 64K
- IQ4_XSweights 14.9 · fits 24 GB16.1 / 17.0 / 18.8GB at 16K / 32K / 64K
- Q4_K_Mweights 16.7 · fits 24 GB17.9 / 18.8 / 20.6GB at 16K / 32K / 64K
- Q5_K_Mweights 19.8 · fits 24 GB21.0 / 21.9 / 23.7GB at 16K / 32K / 64K
- Q6_Kweights 22.9 · fits 32 GB24.1 / 25.0 / 26.8GB at 16K / 32K / 64K
- Q8_0weights 27.8 · fits 32 GB29.0 / 29.9 / 31.7GB at 16K / 32K / 64K
The difference between the 16K and 64K columns is less than 3 GB in any row. The difference between quant steps is about 3 GB in any column. How much fits depends on the quant, and how much room remains depends on the context. The Qwen3.8 27B page shows all 25 quants we track, ranging from the 7.6 GB IQ2_XXS to the 55.6 GB BF16 original. The figures in Unsloth's own table are 16 to 19 GB of RAM plus VRAM for its 4-bit files, in agreement with ours.
Alibaba · Qwen 3.8
Qwen3.8 27B
From 8 GB to 48 GB: what each card size gets
Below 24 GB, a 3-bit file is the best fit and memory bandwidth is the main influence on speed. The largest quant and our estimated speed for each machine are shown in the grid, with a 16K context throughout.
| Model | RTX 3060 12GB | RTX 5060 Ti 16GB | RTX 4090 | RTX 5090 | Mac Mini M5 Pro (24 GB) |
|---|---|---|---|---|---|
| Qwen3.8 27B | D, Barely runsIQ3_XXS · ~20 tok/s | C, UsableQ3_K_L · ~19 tok/s | A, Runs wellQ4_K_M · ~38 tok/s | S, Runs greatQ4_K_M · ~67 tok/s | C, UsableQ4_K_M · ~12 tok/s |
- D, Barely runsQwen3.8 27BIQ3_XXS · ~20 tok/s
- C, UsableQwen3.8 27BQ3_K_L · ~19 tok/s
- A, Runs wellQwen3.8 27BQ4_K_M · ~38 tok/s
- S, Runs greatQwen3.8 27BQ4_K_M · ~67 tok/s
- C, UsableQwen3.8 27BQ4_K_M · ~12 tok/s
8 GB: pick a different model
The smallest file in our list is IQ2_XXS, which requires 8.8 GB with a 16K context. None of these files fits wholly in memory. Our first choice by Score for 8 GB is Qwen3.5 9B, which takes 6.6 GB at Q4_K_M (8 GB tier).
12 GB: it loads, but there's no slack
With a 16K context, IQ3_XXS requires 12.0 GB of memory on an RTX 3060 12GB, leaving a few hundredths of a gigabyte free. That will not work if the card also drives your monitor. The pairing has a fit grade of D and an estimated speed of 20.3 tok/s. It's fine as an experiment. Our ranking still puts the 9B first for daily use on 12 GB (12 GB tier).
16 GB: 3-bit, and a choice about context
This is the main point of debate. In response to Quesma's quantization tests, one Hacker News reader said "there's a real hole here at Q3", referring to the 3-bit files that owners of 16 GB cards actually run. Another reader's view was "You can't run any of the 4-bit quants on a 16GB gpu with enough context to be useful for all but the most basic tasks."
Our estimate leaves less than 1 GB free in all three cases, with a cache figure that is a little low (discussed below). The approach taken by 16 GB owners is to reduce the cache size. One owner's report was Q3 "on 5060ti with 64k context. It runs great", and another got a 12.3 GiB 3-bit file onto an RX 9070 XT with a 4-bit cache, no vision projector and a 64K window. We would suggest a 3-bit quant first, and an 8-bit cache alongside it, using the setup in the last section.
24 GB: the home it was built for
Q4_K_M requires 20.6 GB with a 64K context, and Q5_K_M requires 21.9 GB with a 32K context. Quesma's Piotr Migdał reached the same conclusion: Q4_K_M "fits on a 24 GB card such as RTX 4090, still leaving room for about 64k tokens of context". To get a 100K context for coding, one owner used an 8-bit cache, and an RTX 3090 owner running agents on the 4-bit file called it "fast enough to be usable". It is first by llmrun Score among the models that fit 24 GB (24 GB tier).
32 and 48 GB: more bits or the whole window
With 32 GB, it is a question of precision. With a 32K context, Q8_0 requires 29.9 GB of memory. Our estimates squeeze Q8_0 at 64K and Q4_K_M with the full 262K window just under 32 GB, but since the real cache is a little larger, both need an 8-bit cache. On 48 GB, Q8_0 fits the full window, at an estimated 42.4 GB. The other uses for 32 GB are covered in our RTX 5090 review.
Macs
Our allowance for a Mac is its RAM minus 3.5 GB, giving a 24 GB Mac 20.5 GB for Q4_K_M at up to a 32K context, with no room for 64K. According to the long-running llama.cpp discussion, you can raise macOS's default GPU limit of about two-thirds of RAM on smaller machines with sudo sysctl iogpu.wired_limit_mb. That setting is lost at reboot. The estimated speed of a Mac mini M5 Pro at Q4_K_M is about 12 tok/s, with an M5 Ultra at about four times that. A comparison of the chips is given in Which Mac for local LLMs.
Why long context stays cheap on this model
The keys and values for every token are stored in every layer of a standard transformer. The layout of Qwen3.8 27B is three Gated DeltaNet layers for every ordinary attention layer, according to its model card. Gated DeltaNet, the linear-attention design published at ICLR 2025 by Songlin Yang, Jan Kautz and Ali Hatamizadeh, has a fixed-size state for conversations of any length. The growing cache is required by only 16 of the 64 layers. The first use of this layout by Qwen was in Qwen3-Next, for "ultra-long context length". Our estimates for a 32K context are 5.4 GB of cache for last year's dense Qwen3 32B and 1.8 GB for this model.
Dashed line: 24 GB card (24 GB, weights + 16-bit cache + runtime)
- 16K17.9
- 32K18.8
- 64K20.6
- 128K24.1
- 262K (full)31.3
| Item | GB, weights + 16-bit cache + runtime |
|---|---|
| 16K | 17.9 |
| 32K | 18.8 |
| 64K | 20.6 |
| 128K | 24.1 |
| 262K (full) | 31.3 |
Treat the long end of the chart as a floor. Our estimate is about a fifth lower than Migdał's 16-bit cache measurement in llama.cpp of "around 2.3 GB per 32k tokens". A 24 GB card has room for the difference of about 1 GB at 64K. At the full window the gap is about 4 GB, which is why you cannot count on 32 GB to hold all 262K tokens at Q4_K_M.
A smaller cache is the cheaper solution. You can halve the cache in llama.cpp by storing it at 8 bits. A September study by Sergii Kozyrev and Davyd Maiboroda found an 8-bit (FP8) cache on Qwen3.8 27B "essentially free on tasks", with a small perplexity cost at 32K. Their test used FP8, not llama.cpp's own 8-bit format, so it is only a strong hint for your setup.
What each quant costs in answers
The quality results here are from Quesma's test, with no measurements of our own. About $3,000 of cloud GPU time went into Migdał's tests. To test Unsloth's files, he used three benchmarks, GPQA Diamond for graduate-level science, IFBench for instruction following and Terminal-Bench 2.1 for agentic coding. He published the findings on 26 August 2026.
| File | Size | What happened |
|---|---|---|
| Q4_K_M | 17 GB | matched the BF16 original on Terminal-Bench, little change elsewhere |
| UD-Q2_K_XL | 10.7 GB | same on IFBench, a bit lower on GPQA, a clear drop on Terminal-Bench, about 25% more output tokens for the same solved tasks |
| UD-IQ1_S | 6.2 GB | GPQA around random guessing |
Migdał wrote of 4-bit that "you won't notice a difference on these benchmarks." For the smallest files, his conclusion was "Quality drops off a cliff at 1-bit."
There are two things worth noting about these tests. He did not test any 3-bit files used by owners of 16 GB cards. One HN reader noted the different kinds of "4-bit" and "2-bit" files, the plain K-quant Q4_K_M and Unsloth's smaller dynamic ones. According to one commenter, only Unsloth's Q6_K_XL solved their private coding problem, and "anything Q5 doesn't". This is only one person's example, but an average across benchmarks can miss the hard problem you need to solve. Unsloth says that its Dynamic V3.1 files, released on 11 November, are 5 to 10% more accurate at the same size, but we have not seen an independent test of them yet.
Our rule: take Q4_K_M so long as it fits with the context you need. All that Q6_K or Q8_0 on 32 GB will cost is speed. On 16 GB, 3-bit is the honest choice, even though nobody has tested its quality carefully.
How fast it feels, and the thinking tax
A dense model reads the whole weights file for every generated token, and thus runs faster with a smaller quant.
- Q5_K_M (21.0 GB)31.9
- Q4_K_M (17.9 GB)37.6
- IQ4_XS (16.1 GB)41.8
- Q3_K_M (14.7 GB)45.9
- IQ3_XXS (12.0 GB)56.9
| Item | estimated tok/s, 16K context |
|---|---|
| Q5_K_M (21.0 GB) | 31.9 |
| Q4_K_M (17.9 GB) | 37.6 |
| IQ4_XS (16.1 GB) | 41.8 |
| Q3_K_M (14.7 GB) | 45.9 |
| IQ3_XXS (12.0 GB) | 56.9 |
IQ3_XXS is about half again as fast as Q4_K_M according to our estimate. The speeds given for plain decoding are estimated by llmrun's calibrated model, not measured on our own cards (how we estimate). Trained with multi-token prediction, Qwen3.8 27B allows llama.cpp to draft several tokens at once. On a 5060 Ti, one owner reported 30 to 50 tok/s of generation using it on a 3-bit file, against llmrun's Q3_K_L estimate of 19.4 tok/s.
At an estimated 37.6 tok/s, the model writes faster than anyone reads. The wait comes from thinking. By default the model uses the highest reasoning effort, and the best science answers in Quesma's runs used around 8,000 reasoning tokens. On a 4090 that means about three and a half minutes before the answer starts, by our estimate. "Even at 30 t/s, 3.8 thinks so long, even on medium, it still takes 3x or more longer than any cloud model," one HN commenter wrote. For chat, you can choose a reasoning effort of medium or low, as allowed by the model card. Prompt reading on a 4090 takes about 3 seconds for 4,000 tokens by our estimate, which seems cautious compared with community measurements.
Setting it up so it fits
Even if it should fit, the model can run out of memory at the default settings. If you specify no context, llama.cpp's server reserves cache for 262,144 tokens, the model's full window (server README).
- Pick the quant from the table above: Q4_K_M for 24 GB and up, a 3-bit file for 16 GB. You should have about 1 GB free for the desktop and a real cache that is a little larger than our estimate.
- You can set the context yourself. You can use
-c 32768or-c 65536to set a smaller context for llama-server. - If you need more room, quantize the cache to reduce its memory requirement. The cache is halved with
-ctk q8_0 -ctv q8_0, and some 16 GB owners use q4_0. - If you do not need images, you can leave out the vision projector (the mmproj file) to free that memory for context.