Devstral Small 2 24B Instruct 2512 — Benchmarks

Benchmark scores for Devstral Small 2 24B Instruct 2512 aggregated from public leaderboards (3 boards), with how it ranks among open models and how it compares with its nearest neighbours. See hardware requirements for what you need to run it.

No llmrun Composite Score for Devstral Small 2 24B Instruct 2512: the score is only published for a model measured on at least 4 of the boards in the score panel, spanning at least 2 benchmark categories and including a reasoning or coding board — or on 3 boards that narrow its score to a 90% range of at most 15.6 points. Either way the results must come from at least two different benchmarks. Its individual scores below stand on their own — see how the llmrun Composite Score gets calculated.

Devstral Small 2 24B Instruct 2512 reasoning benchmarks

Multi-step logic and problem-solving — can the model think through a hard problem rather than recall an answer.

CritPt
CritPt (Critical Points) tests graduate-level physics problem-solving on research-style questions spanning mechanics, electromagnetism, quantum mechanics, and more. It was created by physicists to probe whether a model can reason through multi-step physics rather than pattern-match textbook answers.
0.0%
0.0%
median 0.9%open leader 26.6%
Overall best: Gemini 4 Carbon (high) 36.0%
19th percentile · #71 of 84 open · #226 of 244 all
#71 of 84
#226 of 244
Lab-reported
Epoch AI · Dec 2025

Devstral Small 2 24B Instruct 2512 coding benchmarks

Writing, editing, and fixing real code — measured by whether the resulting program actually runs and passes tests.

SWE-bench Verified
SWE-bench Verified tests whether a model can resolve real GitHub issues from popular open-source Python projects, scored on the official swebench.com leaderboard as the percentage of human-validated issues actually fixed. It is the headline measure of practical, agentic software-engineering ability — where open-weight models like Qwen3-Coder, GLM-4.6, Kimi K2 and DeepSWE are now competitive with the frontier.
56.4%
56.4%
median 62.2%open leader 75.8%
Overall best: live-SWE-agent + Claude 4.5 Opus medium (20251101) 79.2%
32nd percentile · #14 of 20 open · #84 of 162 all
#14 of 20
#84 of 162
Independent
SWE-bench · Dec 2025
SciCode
SciCode asks a model to write real scientific-computing code — implementing multi-step numerical methods drawn from physics, chemistry, biology, and math research — and checks the output against reference solutions. It was built by a team of scientists and ML researchers to test coding ability on genuine research problems, not typical software tasks.
28.8%
28.8%
median 40.1%open leader 60.9%
Overall best: Claude Opus 5.5 Max 66.9%
15th percentile · #61 of 72 open · #202 of 217 all
#61 of 72
#202 of 217
Lab-reported
Epoch AI · Dec 2025

Scores aggregated from public benchmark sources (Epoch AI: 2 boards, SWE-bench: 1 board) — each linked from the benchmark pages. llmrun does not run these benchmarks. Percentiles and ranks are among open models in the llmrun catalogue; the median and leader marks refer to the same field.