When SWE-bench was released in October 2023, the best model in its authors' comparison, Claude 2, fixed 1.96% of the assigned GitHub issues. By February 2026, the open model MiniMax M2.5 had fixed 75.8% of the human-checked Verified set, and the best closed model was a point ahead. In the same month, Verified co-author OpenAI also announced that it would stop reporting the score. This is the most quoted coding benchmark, and something its own co-author no longer trusts at the top. For a 24 GB card, Qwen3-Coder 30B A3B at 60.4% is the best open result. We would start a local coding agent with the newer Qwen3.8 27B, despite its absence from the board.
Key takeaways
- A result represents both the model and the agent. Scores on Claude 4 Sonnet range from 64.9% to 76.8%, depending on the agent.
- A one-point gap, five issues out of 500 tasks, is well within the sampling noise; it is of little use. For reading the board, the gaps should be ten points or more.
- Qwen3-Coder 30B A3B has no higher-scoring open rival between 24 GB and 140 GB, with a score of 60.4%. Adding more memory has no benefit on this board without enough room for a 229B model.
- The board has no open results after February 2026 and no entries for open models released since, including our choice for 24 GB.
Claude 2 solved 2% of it
The strategy behind SWE-bench was quite simple and hard to fake. Carlos Jimenez and colleagues collected 2,294 software engineering problems from real GitHub issues in 12 popular Python projects, including Django, sympy and pytest, and paired them with the repositories before the fix. "Given a codebase along with a description of an issue … a language model is tasked with editing the codebase to address the issue", the authors wrote. The success of a patch depends on passing the human maintainers' tests for that fix. Nobody grades the patch's style. The tests decide.
The early models scored poorly, and the original set was unfair in places. Some tests required details absent from the issue, while some issues had more than one possible reading. OpenAI worked with expert software engineers in August 2024 to review 1,699 problems, with three reviewers per problem, and kept 500. The resulting SWE-bench Verified subset is available on swebench.com as a "human-validated subset of 500 SWE-bench instances".
The climb in scores was very fast. As for the entries on our copy of the board, it holds 20 open models and 142 entries from closed models and agent systems, with agent companies joining what OpenAI says "became a standard metric reported in frontier model releases".
Why OpenAI walked away
"Why SWE-bench Verified no longer measures frontier coding capabilities" was published by OpenAI on 23 February 2026. The gains at the top were down to six points in six months, and OpenAI looked at the tasks themselves for an explanation. The researchers analyzed 138 problems that o3 had not consistently solved over 64 runs to draw their conclusions. Of those problems, 59.4% had material flaws in the tests or the description. A test can fail on an import error if the function name it requires is not specified in the issue, even for a correct fix. Other tests check the behaviour from two other bugs fixed in the same pull request.
For anyone reading leaderboards, the second finding was worse. The ability to reproduce the human fix or recite the original issue on some tasks was present in every frontier model OpenAI probed. In one case, its own GPT-5.2 remembered from Django's release notes that edit_only had arrived in version 4.1, the detail required by the hidden tests. With reporting stopped and other labs asked to do the same, OpenAI recommended Scale AI's SWE-bench Pro until better tests exist. Pro uses copyleft and private code that is less likely to have been in training data. GPT-5 was the best model at the September 2025 launch, with 23.3% of the public tasks solved.
Some warning signs had already appeared before this. In the June 2025 paper The SWE-Bench Illusion, the authors wrote that models "achieve up to 76% accuracy in identifying buggy file paths using only issue descriptions" on Verified, compared with at most 53% on repositories outside SWE-bench. In September 2025, a GitHub issue on the SWE-bench repo reported agents using git log --all to find future commits containing the fix. It gave examples of Claude 4 Sonnet on a pytest task and Qwen3-Coder 480B on a Django task. Eight days later, the maintainers rebuilt the task images to "remove all future git history".
Despite these problems, the board is still useful for local models. OpenAI's audit examined the hardest tasks that decide the top of the table. For the 24 GB owner, the gap to consider is 15 points rather than one.
The same model, six different scores
A SWE-bench entry combines a model with an agent, a program responsible for choosing files to open, commands to run and the number of attempts. Labs, agent startups and benchmark maintainers all submit entries, and an agent can count for as much as a model generation. On the Bash Only board, Claude 4 Sonnet scores 12 points above Claude 3.7 Sonnet. For Claude 4 Sonnet, changing the agent makes the same difference.
- EPAM AI/Run Developer Agent76.8
- Tools + Claude 4 Sonnet72.4
- Moatless Tools70.8
- OpenHands70.4
- SWE-agent66.6
- mini-SWE-agent, Bash Only board64.9
| Item | % of tasks resolved |
|---|---|
| EPAM AI/Run Developer Agent | 76.8 |
| Tools + Claude 4 Sonnet | 72.4 |
| Moatless Tools | 70.8 |
| OpenHands | 70.4 |
| SWE-agent | 66.6 |
| mini-SWE-agent, Bash Only board | 64.9 |
With identical weights, the best wrapper scores about 12 points above the simplest. The authors describe the simplest, mini-SWE-agent, as "Just some 100 lines of python", using only a shell. The SWE-bench team uses this software for every model on the Bash Only board, the closest thing to a fair race.
Older open models can show an enormous difference between the two boards. Each Verified entry's author chose the agent, and those choices are unknown.
- Bash Only
- Verified board
Qwen2.5 Coder 32B
Bash Only9Verified board47Kimi K2 Instruct
Bash Only43.8Verified board65.4Qwen3-Coder 480B A35B
Bash Only55.4Verified board69.6GLM 4.6
Bash Only55.4Verified board68.2DeepSeek V3.2
Bash Only60Verified board70GPT OSS 120B
Bash Only26Verified board26
| Item | Bash Only | Verified board |
|---|---|---|
| Qwen2.5 Coder 32B | 9 | 47 |
| Kimi K2 Instruct | 43.8 | 65.4 |
| Qwen3-Coder 480B A35B | 55.4 | 69.6 |
| GLM 4.6 | 55.4 | 68.2 |
| DeepSeek V3.2 | 60 | 70 |
| GPT OSS 120B | 26 | 26 |
Qwen2.5 Coder 32B is the outlier, with 47% for a custom agent compared with 9% for the bare shell. The February 2026 rows are the exception that helps. MiniMax M2.5 at 75.8% and Claude 4.5 Opus (high) at 76.8% have the same results on both boards, from the same measurement setup.
How much a point is worth
Each solved issue moves the score by 0.2 points in this 500-task test. The usual binomial standard error of a 60% score is about 2.2 points, even for a perfectly fair run. However, with contamination, flawed tests and unknown agents, a one-point lead is of little use.
That changes how the February headline reads. MiniMax M2.5 is one point, five tasks, behind Claude 4.5 Opus (high) in the same harness. We would call that a remarkable tie for weights anyone can download.
Here is how we would read the board:
- Use the same test setup for all models. The Bash Only board and February 2026 Verified rows meet that requirement, whereas a custom agent alongside a plain one does not.
- Treat a gap under about five points as a draw, and one over ten as real.
- Look at the date. A model missing from the list has simply not been run; its absence says nothing about its quality.
- Spend some time trying the shortlisted models on your own repository. Local models have no cost per attempt.
The open leaders need 140 GB or more
Every open model with a score above 65% is a mixture-of-experts giant. The leader needs about 140 GB at Q4_K_M, while the next four need 413 GB or more.
| # | Model | Score | Memory |
|---|---|---|---|
| 1 | MiniMax M2.5228.7B | 75.8% | ~140 GB |
| 2 | GLM 5753.9B | 72.8% | ~454 GB |
| 3 | Kimi K2 Instruct 09051026.4B | 71.2% | ~617 GB |
| 4 | Kimi K2.51026.9B | 70.8% | ~618 GB |
| 5 | DeepSeek V3.2685.4B | 70.0% | ~413 GB |
| 6 | Qwen3 Coder 480B A35B Instruct480.2B | 69.6% | ~290 GB |
| 7 | GLM 4.6356.8B | 68.2% | ~217 GB |
| 8 | Kimi K2 Instruct1026.4B | 65.4% | ~617 GB |
- MiniMax M2.5#1 · 228.7B · ~140 GB75.8%
- GLM 5#2 · 753.9B · ~454 GB72.8%
- Kimi K2 Instruct 0905#3 · 1026.4B · ~617 GB71.2%
- Kimi K2.5#4 · 1026.9B · ~618 GB70.8%
- DeepSeek V3.2#5 · 685.4B · ~413 GB70.0%
- Qwen3 Coder 480B A35B Instruct#6 · 480.2B · ~290 GB69.6%
- GLM 4.6#7 · 356.8B · ~217 GB68.2%
- Kimi K2 Instruct#8 · 1026.4B · ~617 GB65.4%
Of these, MiniMax M2.5 alone fits on a single desktop machine. The Mac Studio M5 Ultra with 256 GB has room for it at Q4_K_M and a long context. All the other models on this list require a multi-GPU server.
What fits a 24 GB card
With the same board filtered for 24 GB at Q4_K_M and a 16K context, the top score is about 15 points lower.
| # | Model | Score | Memory |
|---|---|---|---|
| 1 | Qwen3 Coder 30B A3B Instruct30.5B | 60.4% | ~19 GB |
| 2 | DeepSWE Preview32.8B | 58.8% | ~23 GB |
| 3 | Devstral Small 2 24B Instruct 251224.0B | 56.4% | ~18 GB |
| 4 | Devstral Small 250523.6B | 46.8% | ~18 GB |
| 5 | Devstral Small 250723.6B | 38.0% | ~18 GB |
- Qwen3 Coder 30B A3B Instruct#1 · 30.5B · ~19 GB60.4%
- DeepSWE Preview#2 · 32.8B · ~23 GB58.8%
- Devstral Small 2 24B Instruct 2512#3 · 24.0B · ~18 GB56.4%
- Devstral Small 2505#4 · 23.6B · ~18 GB46.8%
- Devstral Small 2507#5 · 23.6B · ~18 GB38.0%
The Qwen3-Coder 30B A3B mixture-of-experts model needs about 19 GB and reads only around 3 billion parameters per token. Our estimated generation rate on an RTX 4090 is about 173 tokens a second, more than five times that of the dense DeepSWE Preview just below it. For an agent with dozens of steps, the difference can add up to minutes per task.
Extra memory buys surprisingly little here. No open model between 24 GB and 140 GB scores higher on this board. Mistral's Devstral 2 123B needs 81 GB and scores 53.8%, compared with 56.4% for its own 24B sibling, Devstral Small 2. Both results are dated the same day in December 2025. A 48 GB build or a 128 GB Mac allows larger models, but the board offers no evidence of better patches until the MiniMax class.
What we'd run for a coding agent on 24 GB
The problem with the board's winner is its date. The Qwen3-Coder 30B A3B result dates from September 2025, for a model released in mid-2025. No open-model submissions have arrived since February, leaving the August 2026 release Qwen3.8 27B without a Verified entry. It has an llmrun Score of 122.3 for agents and 118.4 for coding from other boards, and is our 24 GB choice in the November roundup of local models. It leads on the evidence available, but we cannot say it beats Qwen3-Coder on SWE-bench.
We would start with Qwen3.8 27B, at about 38 tok/s estimated on an RTX 4090 and 17.9 GB at Q4_K_M (memory by quant and context). Keep Qwen3-Coder 30B A3B for long agent loops when speed matters more than the last few points of quality.
SWE-bench Verified is in the Score's agents domain, with a result from the last 365 days required for a board to remain (how the panel works). The latest Verified result is from 26 February 2026. It leaves the panel next February unless the maintainers publish new runs, and SWE-bench Lite has already left.