llmrun

Benchmarks10 min read

SWE-bench Verified leaderboard: open models for 24 GB

The best open result for 24 GB is Qwen3-Coder 30B A3B at 60.4%. Read the date as well as the score: the last open results are from early 2026.

When SWE-bench was released in October 2023, the best model in its authors' comparison, Claude 2, fixed 1.96% of the assigned GitHub issues. By February 2026, the open model MiniMax M2.5 had fixed 75.8% of the human-checked Verified set, and the best closed model was a point ahead. In the same month, Verified co-author OpenAI also announced that it would stop reporting the score. This is the most quoted coding benchmark, and something its own co-author no longer trusts at the top. For a 24 GB card, Qwen3-Coder 30B A3B at 60.4% is the best open result. We would start a local coding agent with the newer Qwen3.8 27B, despite its absence from the board.

Key takeaways

  1. A result represents both the model and the agent. Scores on Claude 4 Sonnet range from 64.9% to 76.8%, depending on the agent.
  2. A one-point gap, five issues out of 500 tasks, is well within the sampling noise; it is of little use. For reading the board, the gaps should be ten points or more.
  3. Qwen3-Coder 30B A3B has no higher-scoring open rival between 24 GB and 140 GB, with a score of 60.4%. Adding more memory has no benefit on this board without enough room for a 229B model.
  4. The board has no open results after February 2026 and no entries for open models released since, including our choice for 24 GB.

Claude 2 solved 2% of it

The strategy behind SWE-bench was quite simple and hard to fake. Carlos Jimenez and colleagues collected 2,294 software engineering problems from real GitHub issues in 12 popular Python projects, including Django, sympy and pytest, and paired them with the repositories before the fix. "Given a codebase along with a description of an issue … a language model is tasked with editing the codebase to address the issue", the authors wrote. The success of a patch depends on passing the human maintainers' tests for that fix. Nobody grades the patch's style. The tests decide.

The early models scored poorly, and the original set was unfair in places. Some tests required details absent from the issue, while some issues had more than one possible reading. OpenAI worked with expert software engineers in August 2024 to review 1,699 problems, with three reviewers per problem, and kept 500. The resulting SWE-bench Verified subset is available on swebench.com as a "human-validated subset of 500 SWE-bench instances".

1.96%
Claude 2 on the original SWE-bench, 2023
75.8%
MiniMax M2.5, best open model on Verified, Feb 2026
79.2%
best entry on our board, an agent system on Claude 4.5 Opus

The climb in scores was very fast. As for the entries on our copy of the board, it holds 20 open models and 142 entries from closed models and agent systems, with agent companies joining what OpenAI says "became a standard metric reported in frontier model releases".

Why OpenAI walked away

"Why SWE-bench Verified no longer measures frontier coding capabilities" was published by OpenAI on 23 February 2026. The gains at the top were down to six points in six months, and OpenAI looked at the tasks themselves for an explanation. The researchers analyzed 138 problems that o3 had not consistently solved over 64 runs to draw their conclusions. Of those problems, 59.4% had material flaws in the tests or the description. A test can fail on an import error if the function name it requires is not specified in the issue, even for a correct fix. Other tests check the behaviour from two other bugs fixed in the same pull request.

For anyone reading leaderboards, the second finding was worse. The ability to reproduce the human fix or recite the original issue on some tasks was present in every frontier model OpenAI probed. In one case, its own GPT-5.2 remembered from Django's release notes that edit_only had arrived in version 4.1, the detail required by the hidden tests. With reporting stopped and other labs asked to do the same, OpenAI recommended Scale AI's SWE-bench Pro until better tests exist. Pro uses copyleft and private code that is less likely to have been in training data. GPT-5 was the best model at the September 2025 launch, with 23.3% of the public tasks solved.

Some warning signs had already appeared before this. In the June 2025 paper The SWE-Bench Illusion, the authors wrote that models "achieve up to 76% accuracy in identifying buggy file paths using only issue descriptions" on Verified, compared with at most 53% on repositories outside SWE-bench. In September 2025, a GitHub issue on the SWE-bench repo reported agents using git log --all to find future commits containing the fix. It gave examples of Claude 4 Sonnet on a pytest task and Qwen3-Coder 480B on a Django task. Eight days later, the maintainers rebuilt the task images to "remove all future git history".

Despite these problems, the board is still useful for local models. OpenAI's audit examined the hardest tasks that decide the top of the table. For the 24 GB owner, the gap to consider is 15 points rather than one.

The same model, six different scores

A SWE-bench entry combines a model with an agent, a program responsible for choosing files to open, commands to run and the number of attempts. Labs, agent startups and benchmark maintainers all submit entries, and an agent can count for as much as a model generation. On the Bash Only board, Claude 4 Sonnet scores 12 points above Claude 3.7 Sonnet. For Claude 4 Sonnet, changing the agent makes the same difference.

QualityBar chart: One model, six setups: Claude 4 Sonnet scores from 64.9% to 76.8%% of tasks resolved
  • EPAM AI/Run Developer Agent76.8
  • Tools + Claude 4 Sonnet72.4
  • Moatless Tools70.8
  • OpenHands70.4
  • SWE-agent66.6
  • mini-SWE-agent, Bash Only board64.9
Source: llmrun.dev/benchmark/swe-bench-verified (five entries); llmrun.dev/benchmark/swe-bench-bash (mini-SWE-agent); submissions dated 2025-05-22 to 2025-08-04 · Data as of Nov 16, 2026

With identical weights, the best wrapper scores about 12 points above the simplest. The authors describe the simplest, mini-SWE-agent, as "Just some 100 lines of python", using only a shell. The SWE-bench team uses this software for every model on the Bash Only board, the closest thing to a fair race.

Older open models can show an enormous difference between the two boards. Each Verified entry's author chose the agent, and those choices are unknown.

QualityComparison chart: Same open weights, two boards: from equal to 38 points apart (Verified setups unknown)% of tasks resolved
  • Bash Only
  • Verified board
  • Qwen2.5 Coder 32B

    Bash Only9
    Verified board47
  • Kimi K2 Instruct

    Bash Only43.8
    Verified board65.4
  • Qwen3-Coder 480B A35B

    Bash Only55.4
    Verified board69.6
  • GLM 4.6

    Bash Only55.4
    Verified board68.2
  • DeepSeek V3.2

    Bash Only60
    Verified board70
  • GPT OSS 120B

    Bash Only26
    Verified board26
Source: llmrun.dev/benchmark/swe-bench-bash and /benchmark/swe-bench-verified · Data as of Nov 16, 2026

Qwen2.5 Coder 32B is the outlier, with 47% for a custom agent compared with 9% for the bare shell. The February 2026 rows are the exception that helps. MiniMax M2.5 at 75.8% and Claude 4.5 Opus (high) at 76.8% have the same results on both boards, from the same measurement setup.

How much a point is worth

Each solved issue moves the score by 0.2 points in this 500-task test. The usual binomial standard error of a 60% score is about 2.2 points, even for a perfectly fair run. However, with contamination, flawed tests and unknown agents, a one-point lead is of little use.

That changes how the February headline reads. MiniMax M2.5 is one point, five tasks, behind Claude 4.5 Opus (high) in the same harness. We would call that a remarkable tie for weights anyone can download.

Here is how we would read the board:

  • Use the same test setup for all models. The Bash Only board and February 2026 Verified rows meet that requirement, whereas a custom agent alongside a plain one does not.
  • Treat a gap under about five points as a draw, and one over ten as real.
  • Look at the date. A model missing from the list has simply not been run; its absence says nothing about its quality.
  • Spend some time trying the shortlisted models on your own repository. Local models have no cost per attempt.

The open leaders need 140 GB or more

Every open model with a score above 65% is a mixture-of-experts giant. The leader needs about 140 GB at Q4_K_M, while the next four need 413 GB or more.

QualityBenchmark leaderboard: The top open models on SWE-bench Verified all need 140 GB or moreSWE-bench Verified score · higher is better · memory at Q4_K_M with 16K context
  1. MiniMax M2.5#1 · 228.7B · ~140 GB75.8%
  2. GLM 5#2 · 753.9B · ~454 GB72.8%
  3. Kimi K2 Instruct 0905#3 · 1026.4B · ~617 GB71.2%
  4. Kimi K2.5#4 · 1026.9B · ~618 GB70.8%
  5. DeepSeek V3.2#5 · 685.4B · ~413 GB70.0%
  6. Qwen3 Coder 480B A35B Instruct#6 · 480.2B · ~290 GB69.6%
  7. GLM 4.6#7 · 356.8B · ~217 GB68.2%
  8. Kimi K2 Instruct#8 · 1026.4B · ~617 GB65.4%
Source: SWE-bench · Data as of Feb 26, 2026SWE-bench Verified: all 20 open models →

Of these, MiniMax M2.5 alone fits on a single desktop machine. The Mac Studio M5 Ultra with 256 GB has room for it at Q4_K_M and a long context. All the other models on this list require a multi-GPU server.

What fits a 24 GB card

With the same board filtered for 24 GB at Q4_K_M and a 16K context, the top score is about 15 points lower.

QualityBenchmark leaderboard: Qwen3-Coder 30B A3B leads the open results that fit 24 GBSWE-bench Verified score · higher is better · memory at Q4_K_M with 16K context
  1. Qwen3 Coder 30B A3B Instruct#1 · 30.5B · ~19 GB60.4%
  2. DeepSWE Preview#2 · 32.8B · ~23 GB58.8%
  3. Devstral Small 2 24B Instruct 2512#3 · 24.0B · ~18 GB56.4%
  4. Devstral Small 2505#4 · 23.6B · ~18 GB46.8%
  5. Devstral Small 2507#5 · 23.6B · ~18 GB38.0%
Source: SWE-bench · Data as of Feb 26, 2026SWE-bench Verified: all 20 open models →

The Qwen3-Coder 30B A3B mixture-of-experts model needs about 19 GB and reads only around 3 billion parameters per token. Our estimated generation rate on an RTX 4090 is about 173 tokens a second, more than five times that of the dense DeepSWE Preview just below it. For an agent with dozens of steps, the difference can add up to minutes per task.

Extra memory buys surprisingly little here. No open model between 24 GB and 140 GB scores higher on this board. Mistral's Devstral 2 123B needs 81 GB and scores 53.8%, compared with 56.4% for its own 24B sibling, Devstral Small 2. Both results are dated the same day in December 2025. A 48 GB build or a 128 GB Mac allows larger models, but the board offers no evidence of better patches until the MiniMax class.

What we'd run for a coding agent on 24 GB

The problem with the board's winner is its date. The Qwen3-Coder 30B A3B result dates from September 2025, for a model released in mid-2025. No open-model submissions have arrived since February, leaving the August 2026 release Qwen3.8 27B without a Verified entry. It has an llmrun Score of 122.3 for agents and 118.4 for coding from other boards, and is our 24 GB choice in the November roundup of local models. It leads on the evidence available, but we cannot say it beats Qwen3-Coder on SWE-bench.

We would start with Qwen3.8 27B, at about 38 tok/s estimated on an RTX 4090 and 17.9 GB at Q4_K_M (memory by quant and context). Keep Qwen3-Coder 30B A3B for long agent loops when speed matters more than the last few points of quality.

SWE-bench Verified is in the Score's agents domain, with a result from the last 365 days required for a board to remain (how the panel works). The latest Verified result is from 26 February 2026. It leaves the panel next February unless the maintainers publish new runs, and SWE-bench Lite has already left.

Questions

Which open model is best on SWE-bench Verified?
MiniMax M2.5 leads the open results, using the same harness as Claude 4.5 Opus (high). It fits on a 256 GB Mac Studio M5 Ultra or a multi-GPU server at Q4_K_M.
What is the best SWE-bench Verified model for a 24 GB GPU?
First place goes to Qwen3-Coder 30B A3B. We would still start with the newer Qwen3.8 27B, although it has no Verified entry.
Why do two leaderboards give different scores for the same model?
The boards can use different agents around the same model. Bash Only runs every model through one minimal agent, mini-SWE-agent; other Verified entries can use custom agents, more attempts or a lab's own tooling.
Is SWE-bench Verified still worth looking at?
Yes, for rough ranking: a 15-point gap is real. At the top, flawed tests and memorized answers limit its value.