Inkling 2 Small — Hardware Requirements & GPU Compatibility
VisionChat
Inkling 2 Small is an open-weight multimodal model from Thinking Machines Lab, released under Apache 2.0. It accepts text and images and produces text, with a context window of up to one million tokens. Its sparse mixture-of-experts architecture has 594 billion total parameters and activates about 17 billion per token, routing each token to eight of 384 experts. The model targets coding, conversation and applications that use tools. Despite the Small name, this is a model for a GPU cluster. The developer distributes a native NVFP4 checkpoint and an alternative MXFP8 version, and recommends at least 500 GB of aggregate VRAM for the NVFP4 deployment. Supported serving options include SGLang, vLLM, TokenSpeed and Unsloth. Community GGUF conversions provide other download formats; memory estimates depend on the selected format and context length.
Inkling 2 Small runs on a 512 GB+ Mac or a multi-GPU server
Best local pick → Mac Studio (M3 Ultra, 512GB) · Q4_K_M · 359.1 GB · ~37 tok/s
Macs from 192 GB (IQ2_XXS · ~56 tok/s)
Score 134.7#2 of 107 open#1 of 103 that fit ≤512 GBApache 2.0
Not set up for it?
Reference GPU · RTX 4090 24 GB
Estimates, ±20% · how we estimate
llmrun Composite Score →
134.7
- 90% range 130.1–139.3
- #2 of 107 open models
- #1 of 103 that fit ≤512 GB
Run Inkling 2 Small
One recommended way. Every other method, OS and quant is in the step-by-step guide.Each way opens a step-by-step guide for your OS and quant.
Recommended
vllm serve thinkingmachines/Inkling-2-Small --max-model-len 16384 --tensor-parallel-size 8Size the nodes from the table above
Download: Inkling-2-Small-Q4_K_M.gguf356.40 GBunsloth
Ways to runOther ways
each opens step-by-stepRun in cloud
Rent 5× H100 SXM · Q4_K_M
- Estimated hourly price
- ~$13.45/hr
- est., whole server
- GPUs
- 5 GPUs
- 400 GB of VRAM
$5+ bonus for new users
How Much VRAM Does Inkling 2 Small Need?
Select a quantization level; the listed hardware adheres to it.
| Quantization | VRAM | +32K | Runs on | |
|---|---|---|---|---|
| 163.9 GB | 168.5 GB | 2-bit · i-quant extreme compression — Importance-weighted 2-bit, extreme compression — significant quality loss | Runs on B200 192 GB | |
| 230.8 GB | 235.3 GB | 3-bit · i-quant — Importance-weighted 3-bit | Runs on Instinct MI325X 256 GB | |
| 253.1 GB | 257.6 GB | 2-bit · K-quant — 2-bit quantization with K-quant improvements | Runs on Instinct MI325X 256 GB | |
| 290.2 GB | 294.7 GB | 3-bit medium — 3-bit medium quantization | Runs on 5× H100 SXM 80 GB | |
| 357 GB | 361.5 GB | 4-bit medium — 4-bit medium quantization — most popular sweet spot | Runs on 5× H100 SXM 80 GB | |
| 423.8 GB | 428.4 GB | 5-bit medium — 5-bit medium quantization — good quality/size tradeoff | Runs on 6× H100 SXM 80 GB | |
| 490.6 GB | 495.2 GB | 6-bit, very good — 6-bit quantization, very good quality | Runs on 7× H100 SXM 80 GB | |
| 594.6 GB | 599.1 GB | 8-bit, near-lossless — 8-bit quantization, near-lossless | Runs on 5× H200 SXM 141 GB | |
| 1188.6 GB | 1193.1 GB | 16-bit · bfloat — Brain floating point 16 — preferred for training | Runs on 7× B200 192 GB |
Selected Q4_K_M
4-bit medium quantization359.1GBat 16K context
- Weights
- 356.4 GB
- KV cache
- 2.4 GB
- Overhead
- 0.3 GB
Server-class: ~359.1 GB at 16K context · about 5× H100 SXM 80 GB
Which Hardware Can Run Inkling 2 Small?
Q4_K_M · 357 GBInkling 2 Small (Q4_K_M) requires 357 GB of VRAM to load the model weights. For comfortable inference with headroom for KV cache and system overhead, 465+ GB is recommended. Using the full 1049K context window can add up to 154.3 GB, bringing total usage to 511.3 GB. No consumer GPU has enough memory. 1 device with unified memory can run Inkling 2 Small, including Mac Studio (M3 Ultra, 512GB).
192+ GB
Q4_K_M · best fit IQ2_XXS (166.1 GB)
6 GPUs · 1 Mac class · 2 device families
B200~282Simulate output on B200B300~282Simulate output on B300Instinct MI350X~95Simulate output on Instinct MI350XM3 Ultra256–512 GB~56Simulate output on Mac Studio (M3 Ultra, 256GB)8x H100 SXM5~344Simulate output on NVIDIA DGX H1008x A100 SXM4~324Simulate output on NVIDIA DGX A100 640GB
By VRAM tier · typical GPUs and Macs at Q4_K_M
Consumer GPUs (up to 32 GB) and Macs up to 128 GB: too small for this model (needs ≥ 166.1 GB)8 GB tier →12 GB tier →16 GB tier →24 GB tier →Macs →
How good is Inkling 2 Small?
The llmrun Score merges 4 public benchmark boards into a single figure, where 100 represents the average for open models.
Score vs VRAM in the ≤512 GB tier
- Inkling 2 Small
- Other models
- Best Score for its VRAM
#1 of 103 that fit ≤512 GB
Each dot is a model at the best quant that fits ≤512 GB. The line steps up to the best Score available at each amount of VRAM; a dot below it is beaten by a model that needs no more memory.
Full ≤512 GB ranking →Benchmarks
Here are all 6 boards that have results for this model, sorted by their weight in the llmrun Score; badges show who ran the test.
- LMArena Text
LMArena · Nov 2026Arena
Arena1411.8
#46/191 open
- LMArena WebDev
LMArena · Nov 2026Arena
Arena1647.1
#4/58 open
- ARC-AGI-2
Epoch AI · Nov 2026Independent
Independent81.7%
#1/23 open
- SciCode
Epoch AI · Nov 2026Lab-reported
Lab-reported52.3%
#10/74 open
- ARC-AGI
Epoch AI · Nov 2026Independent
Independent96.5%
#1/23 open
- APEX-Agents
Epoch AI · Nov 2026Lab-reported
Lab-reported46.8%
#11/22 open
In the ≤512 GB class
103 models fit ≤512 GBSame Score, only models that fit ≤512 GB at their best quant: this model's peers by memory need.
- #2
DeepSeek V4.1 Flash 1017DeepSeek
134.5
458.4 GB
- #3
MiMo V2.6 Pro RLXiaomi
133.5
499.3 GB
- #4
MiMo V2.6 Flash RLXiaomi
131.9
186.5 GB
- #5
GLM 5.3Z.ai
130.2
453.8 GB
- #6
DeepSeek V4.1 FlashDeepSeek
129.4
458.4 GB
Official models with an llmrun Score, each answering one question; sizes are each model's default quant (usually Q4_K_M) with a 16K-token context.
Runner-up at this size
Step 5 Preview
StepFun
- 6.7 lower Score
- 48.4 GB more VRAM
Score 128.0 · 407.5 GB
Fits in less VRAM
GLM 5.3 Flash
Z.ai
- 10.7 lower Score
- 153.9 GB less VRAM
Score 124.0 · 205.2 GB
Other vendor
DeepSeek V4.1 Flash 1017
DeepSeek
- 0.2 lower Score
- 99.3 GB more VRAM
Score 134.5 · 458.4 GB
Similar models
10| Model | Score | VRAM | tok/s | Why it is similar |
|---|---|---|---|---|
| MiMo V2.6 Flash RLXiaomi | 131.9 | 186.5 GB | —needs > 24 GB | similar score |
| GLM 5.3Z.ai | 130.2 | 453.8 GB | —needs > 24 GB | similar score |
| DeepSeek V4.1 FlashDeepSeek | 129.4 | 458.4 GB | —needs > 24 GB | similar score |
| DeepSeek V4 Flash 0731DeepSeek | 127.6 | 183 GB | —needs > 24 GB | similar score |
| GLM 5.2Z.ai | 122.9 | 453.8 GB | —needs > 24 GB | similar score |
| Qwen3.8 27BAlibaba | 121.9 | 17.9 GB | ~38 | similar score |
| Beam 1.0 501B A23Breflection | 119.5 | 304 GB | —needs > 24 GB | same size |
| DeepSeek V3.2DeepSeek | 110.4 | 412.7 GB | —needs > 24 GB | same size |
| NVIDIA Nemotron 3 Ultra 550B A55B BF16NVIDIA | 109.1 | 378.4 GB | —needs > 24 GB | same size |
| DeepSeek V3.2 ExpDeepSeek | 109.0 | 412.7 GB | —needs > 24 GB | same size |
Specifications
- Publisher
- Thinking Machines
- Family
- Inkling 2
- Parameters
- 594B
- Active parameters
- 17B per token
- Architecture
- Inkling2ForConditionalGeneration
- Context Length
- 1,048,576 tokens
- Vocabulary Size
- 201,024
- Release Date
- 2026-11-04
- License
- Apache 2.0
Frequently Asked Questions
How much VRAM does Inkling 2 Small need?
Inkling 2 Small requires 357 GB of VRAM at Q4_K_M, or 1188.6 GB at BF16. Full 1049K context adds up to 154.3 GB (511.3 GB total).
VRAM = Weights + KV Cache + Overhead
Weights = 594B × 4.8 bits ÷ 8 = 356.4 GB
KV Cache + Overhead ≈ 0.6 GB (at 2K context + ~0.3 GB framework)
Fit ratings and hardware model lists check this model with room for a 16K-token context, which needs a little more memory.
KV Cache + Overhead ≈ 154.9 GB (at full 1049K context)
VRAM usage by quantization
Can NVIDIA GeForce RTX 5090 run Inkling 2 Small?
No — Inkling 2 Small requires at least 163.9 GB at IQ2_XXS, which exceeds the NVIDIA GeForce RTX 5090's 32 GB of VRAM.
What's the best quantization for Inkling 2 Small?
For Inkling 2 Small, Q4_K_M (357 GB) offers the best balance of quality and VRAM usage. Q5_K_M (423.8 GB) provides better quality if you have the VRAM. The smallest option is IQ2_XXS at 163.9 GB.
VRAM requirement by quantization
★ Recommended — best balance of quality and VRAM usage.
Can I run Inkling 2 Small on a Mac?
Yes — Mac Pro M2 Ultra (192 GB) and 3 other Macs can run Inkling 2 Small, the smallest at IQ2_XXS. Apple Silicon uses unified memory, so the model shares RAM with the system. At Q4_K_M you need at least 357 GB of usable unified memory (RAM minus macOS overhead), from Mac Studio (M3 Ultra, 512GB).
Can I run Inkling 2 Small locally?
Yes — Inkling 2 Small can run locally on consumer hardware. Popular tools include Ollama, LM Studio, and llama.cpp.
What's the download size of Inkling 2 Small?
At Q4_K_M, the download is about 356.40 GB. The full-precision BF16 version is 1188.00 GB. The smallest option (IQ2_XXS) is 163.35 GB.
Which GPUs can run Inkling 2 Small?
No single consumer GPU has enough VRAM to run Inkling 2 Small at Q4_K_M (357 GB). Multi-GPU or professional hardware is required.
Which devices can run Inkling 2 Small?
1 device with unified memory can run Inkling 2 Small at Q4_K_M (357 GB), including Mac Studio (M3 Ultra, 512GB). Apple Silicon Macs use unified memory shared between CPU and GPU, making them well-suited for local LLM inference.