Can NVIDIA GeForce RTX 4090 run Mistral Small 24B Instruct 2501?

Yes, at Q6_K (20.5 GB) or lower. Higher quantizations like Q8_0 (24.7 GB) exceed the NVIDIA GeForce RTX 4090's 24 GB.

What's the best quantization for Mistral Small 24B Instruct 2501?

For Mistral Small 24B Instruct 2501, Q4_K_M (15.1 GB) offers the best balance of quality and VRAM usage. Q4_K_L (15.4 GB) provides better quality if you have the VRAM. The smallest option is IQ2_XS at 7.9 GB.

Can I run Mistral Small 24B Instruct 2501 on a Mac?

Mistral Small 24B Instruct 2501 requires at least 7.9 GB at IQ2_XS, which exceeds the unified memory of most consumer Macs. You would need a Mac Studio or Mac Pro with a high-memory configuration.

Can I run Mistral Small 24B Instruct 2501 locally?

Yes — Mistral Small 24B Instruct 2501 can run locally on consumer hardware. At Q4_K_M quantization it needs 15.1 GB of VRAM. Popular tools include Ollama, LM Studio, and llama.cpp.

How fast is Mistral Small 24B Instruct 2501?

At Q4_K_M, Mistral Small 24B Instruct 2501 can reach ~193 tok/s on AMD Instinct MI300X. On NVIDIA GeForce RTX 4090: ~43 tok/s. Speed depends mainly on GPU memory bandwidth. Real-world results typically within ±20%.

What's the download size of Mistral Small 24B Instruct 2501?

At Q4_K_M, the download is about 14.40 GB. The full-precision Q8_0 version is 24.00 GB. The smallest option (IQ2_XS) is 7.20 GB.

Mistral AI·Mistral·MistralForCausalLM

Mistral Small 24B Instruct 2501 — Hardware Requirements & GPU Compatibility

Chat

Mistral Small 24B Instruct is Mistral AI's January 2025 release targeting the mid-range parameter sweet spot. At 24 billion parameters it sits between lightweight 7B models and heavier 70B-class offerings, delivering strong instruction-following, reasoning, and coding performance without demanding top-tier hardware. This model fits comfortably on a single GPU with 16–24 GB of VRAM at common quantization levels, making it an attractive option for users with cards like the RTX 4090 or RTX 3090 who want a noticeable step up from 7B models. It strikes an appealing balance between quality and resource requirements for serious local use.

178.1K downloads 955 likes33K context

Specifications

Publisher: Mistral AI
Family: Mistral
Parameters: 24B
Architecture: MistralForCausalLM
Context Length: 32,768 tokens
Vocabulary Size: 131,072
License: Apache 2.0

Get Started

HuggingFace

mistralai/Mistral-Small-24B-Instruct-2501

How Much VRAM Does Mistral Small 24B Instruct 2501 Need?

Select a quantization to see compatible GPUs below.

Quantization	Bits	VRAM	+ Context	File Size	Quality
IQ2_XS	2.40	7.9 GB	14.2 GB	7.20 GB	Importance-weighted 2-bit, extra small
IQ2_S	2.50	8.2 GB	14.5 GB	7.50 GB	Importance-weighted 2-bit, small
IQ2_M	2.70	8.8 GB	15.1 GB	8.10 GB	Importance-weighted 2-bit, medium
IQ3_XS	3.30	10.6 GB	16.9 GB	9.90 GB	Importance-weighted 3-bit, extra small
Q2_K	3.40	10.9 GB	17.2 GB	10.20 GB	2-bit quantization with K-quant improvements
Q3_K_S	3.50	11.2 GB	17.5 GB	10.50 GB	3-bit small quantization
IQ3_M	3.60	11.5 GB	17.8 GB	10.80 GB	Importance-weighted 3-bit, medium
Q3_K_M	3.90	12.4 GB	18.7 GB	11.70 GB	3-bit medium quantization
Q4_0	4.00	12.7 GB	19.0 GB	12.00 GB	4-bit legacy quantization
Q3_K_L	4.10	13.0 GB	19.3 GB	12.30 GB	3-bit large quantization
IQ4_XS	4.30	13.6 GB	19.9 GB	12.90 GB	Importance-weighted 4-bit, compact
Q4_1	4.50	14.2 GB	20.5 GB	13.50 GB	4-bit legacy quantization with offset
Q4_K_S	4.50	14.2 GB	20.5 GB	13.50 GB	4-bit small quantization
IQ4_NL	4.50	14.2 GB	20.5 GB	13.50 GB	Importance-weighted 4-bit, non-linear
Q4_K_M	4.80	15.1 GB	21.4 GB	14.40 GB	4-bit medium quantization — most popular sweet spot
Q4_K_L	4.90	15.4 GB	21.7 GB	14.70 GB	4-bit large quantization
Q5_K_S	5.50	17.2 GB	23.5 GB	16.50 GB	5-bit small quantization
Q5_K_M	5.70	17.8 GB	24.1 GB	17.10 GB	5-bit medium quantization — good quality/size tradeoff
Q5_K_L	5.80	18.1 GB	24.4 GB	17.40 GB	5-bit large quantization
Q6_K	6.60	20.5 GB	26.8 GB	19.80 GB	6-bit quantization, very good quality
Q8_0	8.00	24.7 GB	31.0 GB	24.00 GB	8-bit quantization, near-lossless

Which GPUs Can Run Mistral Small 24B Instruct 2501?

Q4_K_M · 15.1 GB

Show professional

Mistral Small 24B Instruct 2501 (Q4_K_M) requires 15.1 GB of VRAM to load the model weights. For comfortable inference with headroom for KV cache and system overhead, 20+ GB is recommended. Using the full 33K context window can add up to 6.3 GB, bringing total usage to 21.4 GB. 17 GPUs can run it, including NVIDIA GeForce RTX 5090, NVIDIA GeForce RTX 3090 Ti, NVIDIA GeForce RTX 5080.

Runs great

— Plenty of headroom

NVIDIA GeForce RTX 5090~77 tok/s NVIDIA GeForce RTX 3090 Ti~43 tok/s NVIDIA GeForce RTX 4090~43 tok/s NVIDIA GeForce RTX 3090~40 tok/s AMD Radeon RX 7900 XTX~35 tok/s

Decent

— Enough VRAM, may be tight

NVIDIA GeForce RTX 5080~41 tok/s NVIDIA GeForce RTX 5070 Ti~39 tok/s NVIDIA GeForce RTX 4080 SUPER~32 tok/s NVIDIA GeForce RTX 4080~31 tok/s AMD Radeon RX 7900 XT~29 tok/s NVIDIA GeForce RTX 4070 Ti SUPER~29 tok/s AMD Radeon RX 7800 XT~23 tok/s AMD Radeon RX 6800~19 tok/s AMD Radeon RX 6800 XT~19 tok/s AMD Radeon RX 6900 XT~19 tok/s Intel Arc A770 16GB~19 tok/s NVIDIA GeForce RTX 4060 Ti 16GB~12 tok/s

Which Devices Can Run Mistral Small 24B Instruct 2501?

Q4_K_M · 15.1 GB

27 devices with unified memory can run Mistral Small 24B Instruct 2501, including NVIDIA DGX H100, NVIDIA DGX A100 640GB, Mac Mini M4 (16 GB).

Runs great

— Plenty of headroom

Decent

— Enough memory, may be tight

Mac Mini M4 (16 GB)~5 tok/s MacBook Air 13" M4 (16 GB)~5 tok/s MacBook Air 15" M4 (16 GB)~5 tok/s MacBook Pro 14" M4 (16 GB)~5 tok/s iPad Pro M4 13" (16 GB)~5 tok/s MacBook Air 13" M3 (16 GB)~4 tok/s

Related Models

Derivatives (9)

Dolphin Mistral 24B Venice Edition Mistral Small 24B Instruct 2501 GGUF Mistral Small 24B Instruct 2501 GGUF Mistral Small 24B Instruct 2501 GGUF Mistral Small 24B Instruct 2501 GGUF Mistral Small 24B Instruct 2501 GGUF Mistral Small 24B Instruct 2501 Q2 K GGUF Mistral Small 24B Instruct 2501 Q3 K S GGUF Mistral Small 24B Instruct 2501 Q8 0 GGUF

Frequently Asked Questions

How much VRAM does Mistral Small 24B Instruct 2501 need?: Mistral Small 24B Instruct 2501 requires 15.1 GB of VRAM at Q4_K_M, or 24.7 GB at Q8_0. Full 33K context adds up to 6.3 GB (21.4 GB total).
VRAM = Weights + KV Cache + Overhead
Weights = 24B × 4.8 bits ÷ 8 = 14.4 GB
KV Cache + Overhead ≈ 0.7 GB (at 2K context + ~0.3 GB framework)
KV Cache + Overhead ≈ 7 GB (at full 33K context)
VRAM usage by quantization
Q4_K_M
15.1 GB
Q4_K_M + full context
21.4 GB
Learn more about VRAM estimation →
Can NVIDIA GeForce RTX 4090 run Mistral Small 24B Instruct 2501?: Yes, at Q6_K (20.5 GB) or lower. Higher quantizations like Q8_0 (24.7 GB) exceed the NVIDIA GeForce RTX 4090's 24 GB.
What's the best quantization for Mistral Small 24B Instruct 2501?: For Mistral Small 24B Instruct 2501, Q4_K_M (15.1 GB) offers the best balance of quality and VRAM usage. Q4_K_L (15.4 GB) provides better quality if you have the VRAM. The smallest option is IQ2_XS at 7.9 GB.
VRAM requirement by quantization
IQ2_XS
7.9 GB~57%
Q3_K_S
11.2 GB~77%
IQ4_XS
13.6 GB~87%
Q4_K_M ★
15.1 GB~89%
Q4_K_L
15.4 GB~90%
Q8_0
24.7 GB~99%
★ Recommended — best balance of quality and VRAM usage.
Learn more about quantization →
Can I run Mistral Small 24B Instruct 2501 on a Mac?: Mistral Small 24B Instruct 2501 requires at least 7.9 GB at IQ2_XS, which exceeds the unified memory of most consumer Macs. You would need a Mac Studio or Mac Pro with a high-memory configuration.
Can I run Mistral Small 24B Instruct 2501 locally?: Yes — Mistral Small 24B Instruct 2501 can run locally on consumer hardware. At Q4_K_M quantization it needs 15.1 GB of VRAM. Popular tools include Ollama, LM Studio, and llama.cpp.
How fast is Mistral Small 24B Instruct 2501?: At Q4_K_M, Mistral Small 24B Instruct 2501 can reach ~193 tok/s on AMD Instinct MI300X. On NVIDIA GeForce RTX 4090: ~43 tok/s. Speed depends mainly on GPU memory bandwidth. Real-world results typically within ±20%.
tok/s = (bandwidth GB/s ÷ model GB) × efficiency
Example: AMD Instinct MI300X → 5300 ÷ 15.1 × 0.55 = ~193 tok/s
Estimated speed at Q4_K_M (15.1 GB)
AMD Instinct MI300X
~193 tok/s
NVIDIA GeForce RTX 4090
~43 tok/s
NVIDIA H100 SXM
~144 tok/s
AMD Instinct MI250X
~119 tok/s
Real-world results typically within ±20%. Speed depends on batch size, quantization kernel, and software stack.
Learn more about tok/s estimation →
What's the download size of Mistral Small 24B Instruct 2501?: At Q4_K_M, the download is about 14.40 GB. The full-precision Q8_0 version is 24.00 GB. The smallest option (IQ2_XS) is 7.20 GB.