llmrun
fastinoNemotronNemotronHForCausalLM

Fastino Nemotron 3.5 Lightning Finance — Hardware Requirements & GPU Compatibility

Chat

Fastino Nemotron 3.5 Lightning Finance is a 31.6B-parameter open language model from fastino in the Nemotron family. It supports a context window of up to 262,144 tokens. At Q4_K_M it needs about 19.32 GB of VRAM — see which GPUs and Macs can run it below.

31.6B params · MoE2.4B active262K contextApache 2.06.7K downloads34 likes
S

Runs great on a 24 GB GPU

Runs from16 GB Q3_K_M · 32KBest on24 GB Q4_K_M · ~197 tok/s · 256KMacs from18 GB Q2_K · ~49 tok/s

Apache 2.0

Reference GPU · RTX 4090 24 GB

Estimates, ±20% · how we estimate

Run Fastino Nemotron 3.5 Lightning Finance

One recommended way. Every other method, OS and quant is in the step-by-step guide.

Recommended

vLLM
one command · needs vLLM installed
$vllm serve fastino/Fastino-Nemotron-3.5-Lightning-Finance --max-model-len 16384

vLLM repo with its own quantisation

Download: Weights on Hugging Face

install, OS, other quants

Other ways

each opens step-by-step

Run in cloud

Fits on RTX 3090 (24 GB) (4 GB headroom) · Q4_K_M

Generation speed
~191 tok/s
generation speed
Cost per 1M output tokens
$0.31
per 1M output tokens
Deployon Vast.aiBest optionfrom $0.22/hr (opens in new tab)
Compare GPUs →
or

How Much VRAM Does Fastino Nemotron 3.5 Lightning Finance Need?

Select a quantization level; the listed hardware adheres to it.

QuantizationVRAM+32KOn RTX 4090 (reference)
13.8 GB13.9 GB2-bit · K-quant — 2-bit quantization with K-quant improvementsFits: 256K context · ~226 tok/s
15.8 GB15.9 GB3-bit medium — 3-bit medium quantizationFits: 256K context · ~215 tok/s
19.3 GB19.4 GB4-bit medium — 4-bit medium quantization — most popular sweet spotFits: 256K context · ~197 tok/s
22.9 GB23 GB5-bit medium — 5-bit medium quantization — good quality/size tradeoffFits: 128K context · ~183 tok/s
26.4 GB26.5 GB6-bit, very good — 6-bit quantization, very good qualityDoes not fit: needs 26.4 GB
31.9 GB32 GB8-bit, near-lossless — 8-bit quantization, near-losslessDoes not fit: needs 32 GB
63.5 GB63.6 GB16-bit · bfloat — Brain floating point 16 — preferred for trainingDoes not fit: needs 63.5 GB
est.calculated, no published file yet

Selected Q4_K_M

4-bit medium quantization

19.3GBat 16K context

Weights
19.0 GB
KV cache
0.1 GB
Overhead
0.3 GB

Fits RTX 4090 (reference) · 4.7 GB free · ~197 tok/s

≥50 tok/s needs ≥130 GB/s · RTX 5090 ~244, RTX 3090 Ti ~197

Which Hardware Can Run Fastino Nemotron 3.5 Lightning Finance?

Q4_K_M · 19.3 GB

Fastino Nemotron 3.5 Lightning Finance (Q4_K_M) requires 19.3 GB of VRAM to load the model weights. For comfortable inference with headroom for KV cache and system overhead, 26+ GB is recommended. Using the full 262K context window can add up to 1.2 GB, bringing total usage to 20.5 GB. 8 GPUs can run it, including NVIDIA GeForce RTX 5090, NVIDIA GeForce RTX 3090 Ti. 122 devices with unified memory can run Fastino Nemotron 3.5 Lightning Finance, including NVIDIA DGX Spark, NVIDIA Jetson AGX Thor Developer Kit, Mac Studio M1 Max (32 GB).

By VRAM tier · typical GPUs and Macs at Q4_K_M
8–12 GB · RTX 4060, RTX 3060 12GBDoesn't fit — smallest quant needs 13.8 GB8 GB tier →12 GB tier →
16 GB · RTX 5060 Ti 16GBQ3_K_M · 15.8 GB · ~145 tok/s · 32K16 GB tier →
24 GB · RTX 4090referenceQ4_K_M · 19.3 GB · ~197 tok/s · 256K24 GB tier →
Mac 32 GB · Mac Mini M4Q4_K_M · 19.3 GB · ~36 tok/s · 256KMacs →
Mac 64 GB · Mac Studio M4 MaxQ4_K_M · 19.3 GB · ~71 tok/s · 256KMacs →

Alternatives to Fastino Nemotron 3.5 Lightning Finance

Nemotron family · 42 models →

Similar models

6
ModelScoreVRAMtok/sWhy it is similar
Hy Memory 30B A3BTencent—19.1 GB~184same size
Llm Jp 4.1 32B A3b Thinkingllm-jp—20.1 GB~188same size
Llm Jp 4.1 33B Thinkingllm-jp—22.5 GB~35same size
Agnes 3.0 FlashAgnes-AI—22.1 GB~35same size
Muse Glimmer 30BMeta—19.6 GB~36same size
Hy MT2 30B A3BTencent—19.1 GB~170same size

Specifications

Publisher
fastino
Family
Nemotron
Parameters
31.6B
Active parameters
2.4B per token
Architecture
NemotronHForCausalLM
Context Length
262,144 tokens
Vocabulary Size
131,072
Release Date
2026-08-06
License
Apache 2.0

Resources

Frequently Asked Questions

How much VRAM does Fastino Nemotron 3.5 Lightning Finance need?

Fastino Nemotron 3.5 Lightning Finance requires 19.3 GB of VRAM at Q4_K_M, or 63.5 GB at BF16. Full 262K context adds up to 1.2 GB (20.5 GB total).

VRAM = Weights + KV Cache + Overhead

Weights = 31.6B × 4.8 bits ÷ 8 = 18.9 GB

KV Cache + Overhead ≈ 0.4 GB (at 2K context + ~0.3 GB framework)

Fit ratings and hardware model lists check this model with room for a 16K-token context, which needs a little more memory.

KV Cache + Overhead ≈ 1.6 GB (at full 262K context)

VRAM usage by quantization

19.3 GB
20.5 GB

Learn more about VRAM estimation →

Can NVIDIA GeForce RTX 4090 run Fastino Nemotron 3.5 Lightning Finance?

Yes, at Q5_K_M (22.9 GB) or lower. Higher quantizations like Q6_K (26.4 GB) exceed the NVIDIA GeForce RTX 4090's 24 GB.

What's the best quantization for Fastino Nemotron 3.5 Lightning Finance?

For Fastino Nemotron 3.5 Lightning Finance, Q4_K_M (19.3 GB) offers the best balance of quality and VRAM usage. Q5_K_M (22.9 GB) provides better quality if you have the VRAM. The smallest option is Q2_K at 13.8 GB.

VRAM requirement by quantization

Q2_K
13.8 GB
Q4_K_M ★
19.3 GB
Q5_K_M
22.9 GB
Q6_K
26.4 GB
Q8_0
31.9 GB
BF16
63.5 GB

★ Recommended — best balance of quality and VRAM usage.

Learn more about quantization →

Can I run Fastino Nemotron 3.5 Lightning Finance on a Mac?

Yes — MacBook Pro 14" M3 Pro (18 GB) and 108 other Macs can run Fastino Nemotron 3.5 Lightning Finance, the smallest at Q2_K. Apple Silicon uses unified memory, so the model shares RAM with the system. At Q4_K_M you need at least 19.3 GB of usable unified memory (RAM minus macOS overhead), from Mac Mini M5 Pro (24 GB).

Can I run Fastino Nemotron 3.5 Lightning Finance locally?

Yes — Fastino Nemotron 3.5 Lightning Finance can run locally on consumer hardware. Popular tools include Ollama, LM Studio, and llama.cpp.

How fast is Fastino Nemotron 3.5 Lightning Finance?

At Q4_K_M, Fastino Nemotron 3.5 Lightning Finance can reach ~93 tok/s on AMD Instinct MI350X. On NVIDIA GeForce RTX 4090: ~197 tok/s. Speed depends mainly on GPU memory bandwidth. Real-world results typically within ±20%.

tok/s = 1000 ÷ (active GB ÷ (bandwidth GB/s × efficiency) × 1000 + layers × routing ms)

Mixture-of-Experts: just the active experts get read for each token, along with a fixed routing cost per layer.

Example: NVIDIA B200 → 1.4 GB active ÷ (8000 × 0.65) = 0.278 ms, plus 52 layers × 0.055 ms = 2.860 ms, so 1000 ÷ 3.138 ms = ~319 tok/s

Estimated speed at Q4_K_M (19.3 GB)

~319 tok/s
~197 tok/s
~319 tok/s
~302 tok/s

Real-world results typically within ±20%. Speed depends on batch size, quantization kernel, and software stack.

Learn more about tok/s estimation →

What's the download size of Fastino Nemotron 3.5 Lightning Finance?

At Q4_K_M, the download is about 18.95 GB. The full-precision BF16 version is 63.16 GB. The smallest option (Q2_K) is 13.42 GB.

Which GPUs can run Fastino Nemotron 3.5 Lightning Finance?

8 consumer GPUs can run Fastino Nemotron 3.5 Lightning Finance at Q4_K_M. Top options include NVIDIA GeForce RTX 5090, AMD Radeon RX 7900 XT, AMD Radeon RX 7900 XTX. 1 GPU has plenty of headroom for comfortable inference.

Which devices can run Fastino Nemotron 3.5 Lightning Finance?

122 devices with unified memory can run Fastino Nemotron 3.5 Lightning Finance at Q4_K_M (19.3 GB), including AMD Ryzen AI 9 HX 370 (Strix Point) Laptop, ASUS Ascent GX10, Beelink GTR9 Pro (Ryzen AI Max+ 395, 128 GB), Framework Desktop (Ryzen AI Max+ 395, 128 GB). Apple Silicon Macs use unified memory shared between CPU and GPU, making them well-suited for local LLM inference.