llmrun
Thinking MachinesInkling 2Inkling2ForConditionalGeneration

Inkling 2 Small — Hardware Requirements & GPU Compatibility

VisionChat

Inkling 2 Small is an open-weight multimodal model from Thinking Machines Lab, released under Apache 2.0. It accepts text and images and produces text, with a context window of up to one million tokens. Its sparse mixture-of-experts architecture has 594 billion total parameters and activates about 17 billion per token, routing each token to eight of 384 experts. The model targets coding, conversation and applications that use tools. Despite the Small name, this is a model for a GPU cluster. The developer distributes a native NVFP4 checkpoint and an alternative MXFP8 version, and recommends at least 500 GB of aggregate VRAM for the NVFP4 deployment. Supported serving options include SGLang, vLLM, TokenSpeed and Unsloth. Community GGUF conversions provide other download formats; memory estimates depend on the selected format and context length.

594B params · MoE17B active1049K contextApache 2.01.1K downloads265 likes92 quant downloads

Inkling 2 Small runs on a 512 GB+ Mac or a multi-GPU server

Best local pick → Mac Studio M5 Ultra (512 GB) · Q4_K_M · 359.1 GB · ~47 tok/s

Macs from192 GB IQ2_XXS · ~56 tok/s

Score 134.7#2 of 107 open#1 of 103 that fit ≤512 GBApache 2.0

Not set up for it?

Reference GPU · RTX 4090 24 GB

Estimates, ±20% · how we estimate

SOn your NVIDIA DGX H200: Q4_K_M fits · 768.9 GB free · ~289 tok/s · up to 1M contextNVIDIA DGX H200 page →

llmrun Composite Score →

134.7

Benchmarks · tier chart ↓
Among 107 open modelsHow the Composite Score is built

Run Inkling 2 Small

Best fit for your NVIDIA DGX H200. Every other method, OS and quant is in the step-by-step guide.Each way opens a step-by-step guide for your OS and quant.

Recommended for your NVIDIA DGX H200

Ollama
one command · needs Ollama installed
$ollama run hf.co/unsloth/Inkling-2-Small-GGUF

Q4_K_M fits fully · 768.9 GB free · 16K context

Download: Inkling-2-Small-Q4_K_M.gguf356.40 GBunsloth

install, OS, other quants

Ways to runOther ways

each opens step-by-step

Run in cloud

Rent 5× H100 SXM · Q4_K_M

Estimated hourly price
~$13.45/hr
est., whole server
GPUs
5 GPUs
400 GB of VRAM
Deployon RunPodRent H100 SXM$2.69/hr per GPU (opens in new tab)

$5+ bonus for new users

Compare GPUs →

How Much VRAM Does Inkling 2 Small Need?

Select a quantization level; the listed hardware adheres to it.

QuantizationVRAM+32KOn your device
163.9 GB168.5 GB2-bit · i-quant extreme compression — Importance-weighted 2-bit, extreme compression — significant quality lossFits: 1M context · ~295 tok/s
230.8 GB235.3 GB3-bit · i-quant — Importance-weighted 3-bitFits: 1M context · ~270 tok/s
253.1 GB257.6 GB2-bit · K-quant — 2-bit quantization with K-quant improvementsFits: 1M context · ~263 tok/s
290.2 GB294.7 GB3-bit medium — 3-bit medium quantizationFits: 1M context · ~303 tok/s
357 GB361.5 GB4-bit medium — 4-bit medium quantization — most popular sweet spotFits: 1M context · ~289 tok/s
423.8 GB428.4 GB5-bit medium — 5-bit medium quantization — good quality/size tradeoffFits: 1M context · ~277 tok/s
490.6 GB495.2 GB6-bit, very good — 6-bit quantization, very good qualityFits: 1M context · ~266 tok/s
594.6 GB599.1 GB8-bit, near-lossless — 8-bit quantization, near-losslessFits: 1M context · ~301 tok/s
1188.6 GB1193.1 GB16-bit · bfloat — Brain floating point 16 — preferred for trainingDoes not fit: needs 1190.7 GB
est.calculated, no published file yet

Selected Q4_K_M

4-bit medium quantization

359.1GBat 16K context

Weights
356.4 GB
KV cache
2.4 GB
Overhead
0.3 GB

Fits NVIDIA DGX H200 · 768.9 GB free · ~289 tok/s

≥30 tok/s needs ≥511 GB/s · M5 Ultra ~47, M3 Ultra ~37

Which Hardware Can Run Inkling 2 Small?

Q4_K_M · 357 GB

Inkling 2 Small (Q4_K_M) requires 357 GB of VRAM to load the model weights. For comfortable inference with headroom for KV cache and system overhead, 465+ GB is recommended. Using the full 1049K context window can add up to 154.3 GB, bringing total usage to 511.3 GB. No consumer GPU has enough memory. 2 devices with unified memory can run Inkling 2 Small, including Mac Studio M5 Ultra (512 GB).

By VRAM tier · typical GPUs and Macs at Q4_K_M

Consumer GPUs (up to 32 GB) and Macs up to 128 GB: too small for this model (needs ≥ 166.1 GB)8 GB tier →12 GB tier →16 GB tier →24 GB tier →Macs →

How good is Inkling 2 Small?

The llmrun Score merges 4 public benchmark boards into a single figure, where 100 represents the average for open models.

Score vs VRAM in the ≤512 GB tier

  • Inkling 2 Small
  • Other models
  • Best Score for its VRAM

#1 of 103 that fit ≤512 GB

Each dot is a model at the best quant that fits ≤512 GB. The line steps up to the best Score available at each amount of VRAM; a dot below it is beaten by a model that needs no more memory.

Full ≤512 GB ranking →

Benchmarks

Here are all 6 boards that have results for this model, sorted by their weight in the llmrun Score; badges show who ran the test.

  • LMArena Text

    LMArena · Nov 2026Arena

    1411.8

    #46/191 open

    Arena
  • LMArena WebDev

    LMArena · Nov 2026Arena

    1647.1

    #4/58 open

    Arena
  • ARC-AGI-2

    Epoch AI · Nov 2026Independent

    81.7%

    #1/23 open

    Independent
  • SciCode

    Epoch AI · Nov 2026Lab-reported

    52.3%

    #10/74 open

    Lab-reported
  • ARC-AGI

    Epoch AI · Nov 2026Independent

    96.5%

    #1/23 open

    Independent
  • APEX-Agents

    Epoch AI · Nov 2026Lab-reported

    46.8%

    #11/22 open

    Lab-reported
All 6 Inkling 2 Small benchmarks

In the ≤512 GB class

103 models fit ≤512 GB

Same Score, only models that fit ≤512 GB at their best quant: this model's peers by memory need.

  1. #1

    Inkling 2 SmallThinking Machines

    134.7

    359.1 GB

  2. #2

    DeepSeek V4.1 Flash 1017DeepSeek

    134.5

    458.4 GB

  3. #3

    MiMo V2.6 Pro RLXiaomi

    133.5

    499.3 GB

  4. #4

    MiMo V2.6 Flash RLXiaomi

    131.9

    186.5 GB

  5. #5

    GLM 5.3Z.ai

    130.2

    453.8 GB

  6. #6

    DeepSeek V4.1 FlashDeepSeek

    129.4

    458.4 GB

Full ≤512 GB ranking →

Alternatives to Inkling 2 Small

Official models with an llmrun Score, each answering one question; sizes are each model's default quant (usually Q4_K_M) with a 16K-token context.

Similar models

10
ModelScoreVRAMWhy it is similar
MiMo V2.6 Flash RLXiaomi131.9186.5 GBsimilar score
GLM 5.3Z.ai130.2453.8 GBsimilar score
DeepSeek V4.1 FlashDeepSeek129.4458.4 GBsimilar score
DeepSeek V4 Flash 0731DeepSeek127.6183 GBsimilar score
GLM 5.2Z.ai122.9453.8 GBsimilar score
Qwen3.8 27BAlibaba121.917.9 GBsimilar score
Beam 1.0 501B A23Breflection119.5304 GBsame size
DeepSeek V3.2DeepSeek110.4412.7 GBsame size
NVIDIA Nemotron 3 Ultra 550B A55B BF16NVIDIA109.1378.4 GBsame size
DeepSeek V3.2 ExpDeepSeek109.0412.7 GBsame size

Specifications

Publisher
Thinking Machines
Family
Inkling 2
Parameters
594B
Active parameters
17B per token
Architecture
Inkling2ForConditionalGeneration
Context Length
1,048,576 tokens
Vocabulary Size
201,024
Release Date
2026-11-04
License
Apache 2.0

Resources

Frequently Asked Questions

How much VRAM does Inkling 2 Small need?

Inkling 2 Small requires 357 GB of VRAM at Q4_K_M, or 1188.6 GB at BF16. Full 1049K context adds up to 154.3 GB (511.3 GB total).

VRAM = Weights + KV Cache + Overhead

Weights = 594B × 4.8 bits ÷ 8 = 356.4 GB

KV Cache + Overhead ≈ 0.6 GB (at 2K context + ~0.3 GB framework)

Fit ratings and hardware model lists check this model with room for a 16K-token context, which needs a little more memory.

KV Cache + Overhead ≈ 154.9 GB (at full 1049K context)

VRAM usage by quantization

357.0 GB
511.3 GB

Learn more about VRAM estimation →

Can NVIDIA GeForce RTX 5090 run Inkling 2 Small?

No — Inkling 2 Small requires at least 163.9 GB at IQ2_XXS, which exceeds the NVIDIA GeForce RTX 5090's 32 GB of VRAM.

What's the best quantization for Inkling 2 Small?

For Inkling 2 Small, Q4_K_M (357 GB) offers the best balance of quality and VRAM usage. Q5_K_M (423.8 GB) provides better quality if you have the VRAM. The smallest option is IQ2_XXS at 163.9 GB.

VRAM requirement by quantization

IQ2_XXS
163.9 GB
Q2_K
253.1 GB
Q4_K_M ★
357.0 GB
Q5_K_M
423.8 GB
Q6_K
490.6 GB
BF16
1188.6 GB

★ Recommended — best balance of quality and VRAM usage.

Learn more about quantization →

Can I run Inkling 2 Small on a Mac?

Yes — Mac Pro M2 Ultra (192 GB) and 5 other Macs can run Inkling 2 Small, the smallest at IQ2_XXS. Apple Silicon uses unified memory, so the model shares RAM with the system. At Q4_K_M you need at least 357 GB of usable unified memory (RAM minus macOS overhead), from Mac Studio M5 Ultra (512 GB).

Can I run Inkling 2 Small locally?

Yes — Inkling 2 Small can run locally on consumer hardware. Popular tools include Ollama, LM Studio, and llama.cpp.

What's the download size of Inkling 2 Small?

At Q4_K_M, the download is about 356.40 GB. The full-precision BF16 version is 1188.00 GB. The smallest option (IQ2_XXS) is 163.35 GB.

Which GPUs can run Inkling 2 Small?

No single consumer GPU has enough VRAM to run Inkling 2 Small at Q4_K_M (357 GB). Multi-GPU or professional hardware is required.

Which devices can run Inkling 2 Small?

2 devices with unified memory can run Inkling 2 Small at Q4_K_M (357 GB), including Mac Studio (M3 Ultra, 512GB), Mac Studio M5 Ultra (512 GB). Apple Silicon Macs use unified memory shared between CPU and GPU, making them well-suited for local LLM inference.