llmrun
llm-jpQwen3MoeForCausalLM

Llm Jp 4.1 32B A3b Thinking — Hardware Requirements & GPU Compatibility

ChatFunctions

llm-jp-4.1-32b-a3b-thinking is a Mixture-of-Experts reasoning model from Japan's National Institute of Informatics Research and Development Center for Large Language Models, featuring 32 billion total parameters with 3.8 billion active ones. It employs 128 routed experts where only 8 stay active, was pre-trained and mid-trained on 11.7 trillion tokens, then post-trained using supervised fine-tuning and direct preference optimization without reinforcement learning. The model targets Japanese and English while the card notes code data across several programming languages; with only 3.8 billion parameters active per token, decoding runs fast even though full weights still demand a large GPU or a 32 GB-class unified-memory machine, even when quantized. The architecture supports a 65,536 token context window. Released under the Apache 2.0 license for unrestricted commercial and research use, it came out in September 2026. As part of the LLM-jp-4.1 thinking series, this MoE model joins 8B and 33B dense variants and serves as an update to the earlier llm-jp-4 version.

32.1B params · MoE3.8B active66K contextApache 2.02.1K downloads26 likes
S

Runs great on a 24 GB GPU

Runs from16 GB Q2_K · 32KBest on24 GB Q4_K_M · ~188 tok/s · 64KMacs from18 GB Q2_K · ~45 tok/s

Apache 2.0

Reference GPU · RTX 4090 24 GB

Estimates, ±20% · how we estimate

Run Llm Jp 4.1 32B A3b Thinking

One recommended way. Every other method, OS and quant is in the step-by-step guide.Each way opens a step-by-step guide for your OS and quant.

Recommended

vLLM
one command · needs vLLM installed
$vllm serve llm-jp/llm-jp-4.1-32b-a3b-thinking --max-model-len 16384

vLLM repo with its own quantisation

Download: Weights on Hugging Face

install, OS, other quants

Ways to runOther ways

each opens step-by-step

Run in cloud

Runs on RTX A6000 (48 GB) (27 GB headroom) · Q4_K_M

Generation speed
~155 tok/s
generation speed
Cost per 1M output tokens
$0.59
per 1M output tokens

How Much VRAM Does Llm Jp 4.1 32B A3b Thinking Need?

Select a quantization level; the listed hardware adheres to it.

QuantizationVRAM+32KOn RTX 4090 (reference)
14.0 GB15 GB2-bit · K-quant — 2-bit quantization with K-quant improvementsFits: 64K context · ~232 tok/s
16.0 GB17 GB3-bit medium — 3-bit medium quantizationFits: 64K context · ~214 tok/s
19.6 GB20.7 GB4-bit medium — 4-bit medium quantization — most popular sweet spotFits: 64K context · ~188 tok/s
23.3 GB24.3 GB5-bit medium — 5-bit medium quantization — good quality/size tradeoffFits: 16K context · ~167 tok/s
26.9 GB27.9 GB6-bit, very good — 6-bit quantization, very good qualityDoes not fit: needs 27.4 GB
32.5 GB33.5 GB8-bit, near-lossless — 8-bit quantization, near-losslessDoes not fit: needs 33 GB
64.7 GB65.7 GB16-bit · bfloat — Brain floating point 16 — preferred for trainingDoes not fit: needs 65.1 GB
est.calculated, no published file yet

Selected Q4_K_M

4-bit medium quantization

20.1GBat 16K context

Weights
19.3 GB
KV cache
0.5 GB
Overhead
0.3 GB

Fits RTX 4090 (reference) · 3.9 GB free · ~188 tok/s

≥50 tok/s needs ≥197 GB/s · RTX 5090 ~265, RTX 3090 Ti ~188

Which Hardware Can Run Llm Jp 4.1 32B A3b Thinking?

Q4_K_M · 19.6 GB

Llm Jp 4.1 32B A3b Thinking (Q4_K_M) requires 19.6 GB of VRAM to load the model weights. For comfortable inference with headroom for KV cache and system overhead, 26+ GB is recommended. Using the full 66K context window can add up to 2.1 GB, bringing total usage to 21.7 GB. 8 GPUs can run it, including NVIDIA GeForce RTX 5090, NVIDIA GeForce RTX 3090 Ti. 137 devices with unified memory can run Llm Jp 4.1 32B A3b Thinking, including Mac Studio M5 Ultra (256 GB), Mac Studio M5 Ultra (512 GB), NVIDIA RTX Spark N1X Laptop (32 GB).

By VRAM tier · typical GPUs and Macs at Q4_K_M
8–12 GB · RTX 4060, RTX 3060 12GBDoesn't fit — smallest quant needs 14.5 GB8 GB tier →12 GB tier →
16 GB · RTX 5060 Ti 16GBQ2_K · 14.5 GB · ~133 tok/s · 32K16 GB tier →
24 GB · RTX 4090referenceQ4_K_M · 20.1 GB · ~188 tok/s · 64K24 GB tier →
Mac 32 GB · Mac Mini M4Q4_K_M · 20.1 GB · ~29 tok/s · 64KMacs →
Mac 64 GB · Mac Studio M4 MaxQ4_K_M · 20.1 GB · ~80 tok/s · 64KMacs →

Alternatives to Llm Jp 4.1 32B A3b Thinking

Similar models

6
ModelScoreVRAMtok/sWhy it is similar
Hy Memory 30B A3BTencent—19.1 GB~184same size
Granite Switch 4.1 30BIBM—22.7 GB~35same size
Muse Glimmer 30BMeta—19.6 GB~36same size
NVIDIA Nemotron 3.5 Lightning 30B A3B BF16NVIDIA—19.3 GB~197same size
Hy MT2 30B A3BTencent—19.1 GB~170same size
EXAONE 4.5 33BLGAI-EXAONE—23.9 GB~33same size

Specifications

Publisher
llm-jp
Parameters
32.1B
Active parameters
3.8B per token
Architecture
Qwen3MoeForCausalLM
Context Length
65,536 tokens
Vocabulary Size
196,608
Release Date
2026-09-16
License
Apache 2.0

Frequently Asked Questions

How much VRAM does Llm Jp 4.1 32B A3b Thinking need?

Llm Jp 4.1 32B A3b Thinking requires 19.6 GB of VRAM at Q4_K_M, or 64.7 GB at BF16. Full 66K context adds up to 2.1 GB (21.7 GB total).

VRAM = Weights + KV Cache + Overhead

Weights = 32.1B × 4.8 bits ÷ 8 = 19.3 GB

KV Cache + Overhead ≈ 0.3 GB (at 2K context + ~0.3 GB framework)

Fit ratings and hardware model lists check this model with room for a 16K-token context, which needs a little more memory.

KV Cache + Overhead ≈ 2.4 GB (at full 66K context)

VRAM usage by quantization

19.6 GB
21.7 GB

Learn more about VRAM estimation →

Can NVIDIA GeForce RTX 4090 run Llm Jp 4.1 32B A3b Thinking?

Yes, at Q5_K_M (23.3 GB) or lower. Higher quantizations like Q6_K (26.9 GB) exceed the NVIDIA GeForce RTX 4090's 24 GB.

What's the best quantization for Llm Jp 4.1 32B A3b Thinking?

For Llm Jp 4.1 32B A3b Thinking, Q4_K_M (19.6 GB) offers the best balance of quality and VRAM usage. Q5_K_M (23.3 GB) provides better quality if you have the VRAM. The smallest option is Q2_K at 14.0 GB.

VRAM requirement by quantization

Q2_K
14.0 GB
Q4_K_M ★
19.6 GB
Q5_K_M
23.3 GB
Q6_K
26.9 GB
Q8_0
32.5 GB
BF16
64.7 GB

★ Recommended — best balance of quality and VRAM usage.

Learn more about quantization →

Can I run Llm Jp 4.1 32B A3b Thinking on a Mac?

Yes — MacBook Pro 14" M3 Pro (18 GB) and 117 other Macs can run Llm Jp 4.1 32B A3b Thinking, the smallest at Q2_K. Apple Silicon uses unified memory, so the model shares RAM with the system. At Q4_K_M you need at least 19.6 GB of usable unified memory (RAM minus macOS overhead), from Mac Mini M5 Pro (24 GB).

Can I run Llm Jp 4.1 32B A3b Thinking locally?

Yes — Llm Jp 4.1 32B A3b Thinking can run locally on consumer hardware. Popular tools include Ollama, LM Studio, and llama.cpp.

How fast is Llm Jp 4.1 32B A3b Thinking?

At Q4_K_M, Llm Jp 4.1 32B A3b Thinking can reach ~145 tok/s on AMD Instinct MI350X. On NVIDIA GeForce RTX 4090: ~188 tok/s. Speed depends mainly on GPU memory bandwidth. Real-world results typically within ±20%.

tok/s = 1000 ÷ (active GB ÷ (bandwidth GB/s × efficiency) × 1000 + layers × routing ms)

Mixture-of-Experts: just the active experts get read for each token, along with a fixed routing cost per layer.

Example: NVIDIA B200 → 2.3 GB active ÷ (8000 × 0.65) = 0.45003 ms, plus 32 layers × 0.055 ms = 1.76000 ms, so 1000 ÷ 2.21003 ms = ~452 tok/s

Estimated speed at Q4_K_M (19.6 GB)

~453 tok/s
~188 tok/s
~453 tok/s
~401 tok/s

Real-world results typically within ±20%. Speed depends on batch size, quantization kernel, and software stack.

Learn more about tok/s estimation →

What's the download size of Llm Jp 4.1 32B A3b Thinking?

At Q4_K_M, the download is about 19.28 GB. The full-precision BF16 version is 64.28 GB. The smallest option (Q2_K) is 13.66 GB.

Which GPUs can run Llm Jp 4.1 32B A3b Thinking?

8 consumer GPUs can run Llm Jp 4.1 32B A3b Thinking at Q4_K_M. Top options include NVIDIA GeForce RTX 5090, AMD Radeon RX 7900 XT, AMD Radeon RX 7900 XTX. 1 GPU has plenty of headroom for comfortable inference.

Which devices can run Llm Jp 4.1 32B A3b Thinking?

137 devices with unified memory can run Llm Jp 4.1 32B A3b Thinking at Q4_K_M (19.6 GB), including AMD Ryzen 7 8845HS (Hawk Point) Laptop, AMD Ryzen AI 7 350 (Krackan Point) Laptop, AMD Ryzen AI 9 HX 370 (Strix Point) Laptop, ASUS Ascent GX10. Apple Silicon Macs use unified memory shared between CPU and GPU, making them well-suited for local LLM inference.