Coding
SWE-bench Verified Leaderboard
SWE-bench Verified tests whether a model can resolve real GitHub issues from popular open-source Python projects, scored on the official swebench.com leaderboard as the percentage of human-validated issues actually fixed. It is the headline measure of practical, agentic software-engineering ability — where open-weight models like Qwen3-Coder, GLM-4.6, Kimi K2 and DeepSWE are now competitive with the frontier.
Source: swebench20 open models ranked+142 proprietaryData through Feb 2026
All models ranked on SWE-bench Verified
Proprietary / closed models are shown dimmed — you can't run them locally, but they show where the open field stands.
| # | Model | Score | VRAM @ Q4 |
|---|---|---|---|
| 1 | live-SWE-agent + Claude 4.5 Opus medium (20251101) · proprietary | 79.2% | API only |
| 2 | Sonar Foundation Agent + Claude 4.5 Opus · proprietary | 79.2% | API only |
| 3 | TRAE + Doubao-Seed-Code · proprietary | 78.8% | API only |
| 4 | live-SWE-agent + Gemini 3 Pro Preview (2025-11-18) · proprietary | 77.4% | API only |
| 5 | Atlassian Rovo Dev (2025-09-02) · proprietary | 76.8% | API only |
| 6 | Claude 4.5 Opus (high) · proprietary | 76.8% | API only |
| 7 | EPAM AI/Run Developer Agent v20250719 + Claude 4 Sonnet · proprietary | 76.8% | API only |
| 8 | ACoder · proprietary | 76.4% | API only |
| 9 | Gemini 3 Flash (high) · proprietary | 75.8% | API only |
| 10 | MiniMax M2.5 · 228.7B | 75.8% | ~138 GB |
| 11 | Claude 4.6 Opus · proprietary | 75.6% | API only |
| 12 | Warp · proprietary | 75.6% | API only |
| 13 | TRAE + Claude Sonnet 4 + Opus 4 + Sonnet 3.7 + Gemini 2.5 Pro · proprietary | 75.2% | API only |
| 14 | Harness AI · proprietary | 74.8% | API only |
| 15 | Sonar Foundation Agent + Claude 4.5 Sonnet · proprietary | 74.8% | API only |
| 16 | JoyCode + Claude 4 Sonnet + GPT-4.1 · proprietary | 74.6% | API only |
| 17 | Lingxi-v1.5_claude-4-sonnet-20250514 · proprietary | 74.6% | API only |
| 18 | Claude 4.5 Opus (20251101) (medium) · proprietary | 74.4% | API only |
| 19 | Prometheus-v1.2.1 + GPT-5 · proprietary | 74.4% | API only |
| 20 | Refact.ai Agent + Claude 4 Sonnet + o4-mini · proprietary | 74.4% | API only |
| 21 | Gemini 3 Pro Preview (2025-11-18) · proprietary | 74.2% | API only |
| 22 | Salesforce AI Research SAGE (OpenHands) · proprietary | 73.8% | API only |
| 23 | Tools + Claude 4 Opus (2025-05-22) · proprietary | 73.2% | API only |
| 24 | Salesforce AI Research SAGE (bash-only) · proprietary | 73.0% | API only |
| 25 | GLM 5 · 753.9B | 72.8% | ~457 GB |
| 26 | GPT 5.2 (high) · proprietary | 72.8% | API only |
| 27 | GPT 5.2 Codex · proprietary | 72.8% | API only |
| 28 | Tools + Claude 4 Sonnet (2025-05-22) · proprietary | 72.4% | API only |
| 29 | GPT 5.2 (2025-12-11) (high) · proprietary | 71.8% | API only |
| 30 | OpenHands + GPT-5 · proprietary | 71.8% | API only |
| 31 | Claude 4.5 Sonnet (high) · proprietary | 71.4% | API only |
| 32 | Bloop · proprietary | 71.2% | API only |
| 33 | Kimi K2 Instruct 0905 · 1026.5B | 71.2% | ~620 GB |
| 34 | Prometheus-v1.2 + GPT-5 · proprietary | 71.2% | API only |
| 35 | Qodo Command · proprietary | 71.2% | API only |
| 36 | Kimi K2.5 · 1026.9B | 70.8% | ~620 GB |
| 37 | Moatless Tools + Claude 4 Sonnet · proprietary | 70.8% | API only |
| 38 | Claude 4.5 Sonnet (20250929) · proprietary | 70.6% | API only |
| 39 | TRAE · proprietary | 70.6% | API only |
| 40 | Augment Agent v1 · proprietary | 70.4% | API only |
| 41 | OpenHands + Claude 4 Sonnet · proprietary | 70.4% | API only |
| 42 | Refact.ai Agent · proprietary | 70.4% | API only |
| 43 | devlo · proprietary | 70.2% | API only |
| 44 | DeepSeek V3.2 · 685.4B | 70.0% | ~415 GB |
| 45 | Zencoder (2025-04-30) · proprietary | 70.0% | API only |
| 46 | Gemini 3 Pro · proprietary | 69.6% | API only |
| 47 | Qwen3 Coder 480B A35B Instruct · 480.2B | 69.6% | ~289 GB |
| 48 | GPT 5.2 (2025-12-11) · proprietary | 69.0% | API only |
| 49 | GLM 4.6 · 356.8B | 68.2% | ~215 GB |
| 50 | Nemotron-CORTEXA · proprietary | 68.2% | API only |
| 51 | Claude 4 Opus (20250514) · proprietary | 67.6% | API only |
| 52 | Claude 4.5 Haiku (high) · proprietary | 66.6% | API only |
| 53 | SWE-agent + Claude 4 Sonnet · proprietary | 66.6% | API only |
| 54 | Aime-coder v1 + Anthopic Claude 3.7 Sonnet · proprietary | 66.4% | API only |
| 55 | GPT 5.1 (2025-11-13) (medium) · proprietary | 66.0% | API only |
| 56 | GPT 5.1 Codex (medium) · proprietary | 66.0% | API only |
| 57 | OpenHands · proprietary | 65.8% | API only |
| 58 | Amazon Q Developer Agent (v20250405-dev) · proprietary | 65.4% | API only |
| 59 | Augment Agent v0 · proprietary | 65.4% | API only |
| 60 | Kimi K2 Instruct · 1026.4B | 65.4% | ~620 GB |
| 61 | GPT 5 (2025-08-07) (medium) · proprietary | 65.0% | API only |
| 62 | Claude 4 Sonnet (20250514) · proprietary | 64.9% | API only |
| 63 | PatchPilot-v1.1 · proprietary | 64.6% | API only |
| 64 | W&B Programmer O1 crosscheck5 · proprietary | 64.6% | API only |
| 65 | GLM 4.5 · 358.3B | 64.2% | ~216 GB |
| 66 | AgentScope · proprietary | 63.4% | API only |
| 67 | Kimi K2 Thinking · 1026.4B | 63.4% | ~620 GB |
| 68 | Tools + Claude 3.7 Sonnet (2025-02-24) · proprietary | 63.2% | API only |
| 69 | Blackbox AI Agent · proprietary | 62.8% | API only |
| 70 | EPAM AI/Run Developer Agent v20250219 + Anthopic Claude 3.5 Sonnet · proprietary | 62.8% | API only |
| 71 | SWE-agent + Claude 3.7 Sonnet w/ Review Heavy · proprietary | 62.4% | API only |
| 72 | CodeStory Midwit Agent + swe-search · proprietary | 62.2% | API only |
| 73 | MiniMax M2 · 228.7B | 61.0% | ~138 GB |
| 74 | OpenHands + 4x Scaled (2024-02-03) · proprietary | 60.8% | API only |
| 75 | Qwen3 Coder 30B A3B Instruct · 30.5B | 60.4% | ~19 GB |
| 76 | Learn-by-interact · proprietary | 60.2% | API only |
| 77 | GPT 5 mini (2025-08-07) (medium) · proprietary | 59.8% | API only |
| 78 | DeepSWE Preview · 32.8B | 58.8% | ~20 GB |
| 79 | o3 (2025-04-16) · proprietary | 58.4% | API only |
| 80 | Emergent E1 (v2024-12-23) · proprietary | 57.2% | API only |
| 81 | Artemis Agent v2 (2025-09-24) · proprietary | 57.0% | API only |
| 82 | Gru(2024-12-08) · proprietary | 57.0% | API only |
| 83 | SWE-Rizzo · proprietary | 56.6% | API only |
| 84 | Devstral Small 2 24B Instruct 2512 · 24.0B | 56.4% | ~15 GB |
| 85 | GPT 5 mini · proprietary | 56.2% | API only |
| 86 | EPAM AI/Run Developer Agent v20241212 + Anthopic Claude 3.5 Sonnet · proprietary | 55.4% | API only |
| 87 | Amazon Q Developer Agent (v20241202-dev) · proprietary | 55.0% | API only |
| 88 | Devstral 2 123B Instruct 2512 · 125.0B | 53.8% | ~76 GB |
| 89 | FrogBoss-32B-2510 · proprietary | 53.6% | API only |
| 90 | Gemini 2.5 Pro (2025-05-06) · proprietary | 53.6% | API only |
| 91 | Bracket.sh · proprietary | 53.2% | API only |
| 92 | OpenHands + CodeAct v2.1 (claude-3-5-sonnet-20241022) · proprietary | 53.0% | API only |
| 93 | Claude 3.7 Sonnet (20250219) · proprietary | 52.8% | API only |
| 94 | Google Jules + Gemini 2.0 Flash (v20241212-experimental) · proprietary | 52.2% | API only |
| 95 | Engine Labs (2024-11-25) · proprietary | 51.8% | API only |
| 96 | AutoCodeRover-v2.1 (Claude-3.5-Sonnet-20241022) · proprietary | 51.6% | API only |
| 97 | Agentless-1.5 + Claude-3.5 Sonnet (20241022) · proprietary | 50.8% | API only |
| 98 | Bytedance MarsCode Agent · proprietary | 50.0% | API only |
| 99 | Solver (2024-10-28) · proprietary | 50.0% | API only |
| 100 | nFactorial (2024-11-05) · proprietary | 49.2% | API only |
| 101 | Tools + Claude 3.5 Sonnet (2024-10-22) · proprietary | 49.0% | API only |
| 102 | Composio SWE-Kit (2024-10-25) · proprietary | 48.6% | API only |
| 103 | AppMap Navie v2 · proprietary | 47.2% | API only |
| 104 | Qwen2.5 Coder 32B Instruct · 32.8B | 47.0% | ~21 GB |
| 105 | Devstral Small 2505 · 23.6B | 46.8% | ~15 GB |
| 106 | Emergent E1 (v2024-10-12) · proprietary | 46.6% | API only |
| 107 | AutoCodeRover-v2.0 (Claude-3.5-Sonnet-20241022) · proprietary | 46.2% | API only |
| 108 | PatchPilot + Co-PatcheR · proprietary | 46.0% | API only |
| 109 | Solver (2024-09-12) · proprietary | 45.4% | API only |
| 110 | Gru(2024-08-24) · proprietary | 45.2% | API only |
| 111 | FrogMini-14B-2510 · proprietary | 45.0% | API only |
| 112 | o4-mini (2025-04-16) · proprietary | 45.0% | API only |
| 113 | CodeShellAgent + Gemini 2.0 Flash (Experimental) · proprietary | 44.2% | API only |
| 114 | Agentless Lite + O3 Mini (20250214) · proprietary | 42.4% | API only |
| 115 | Amazon Nova Premier 1.0 (2025-04-30) · proprietary | 42.4% | API only |
| 116 | DeepSeek v3 0324 · 684.5B | 42.0% | ~415 GB |
| 117 | nFactorial (2024-10-30) · proprietary | 41.6% | API only |
| 118 | ugaiforge · proprietary | 41.6% | API only |
| 119 | SWE-RL (Llama3-SWE-RL-70B + Agentless Mini) (20250226) · proprietary | 41.2% | API only |
| 120 | Composio SWEkit + Claude 3.5 Sonnet (2024-10-16) · proprietary | 40.6% | API only |
| 121 | Honeycomb · proprietary | 40.6% | API only |
| 122 | Nebius AI Qwen 2.5 72B Generator + LLama 3.1 70B Critic · proprietary | 40.6% | API only |
| 123 | Tools + Claude 3.5 Haiku · proprietary | 40.6% | API only |
| 124 | Qwen 2.5 · proprietary | 40.2% | API only |
| 125 | EPAM AI/Run Developer Agent v20241029 + Anthopic Claude 3.5 Sonnet · proprietary | 39.6% | API only |
| 126 | GPT 4.1 (2025-04-14) · proprietary | 39.6% | API only |
| 127 | Agentless-1.5 + GPT 4o (2024-05-13) · proprietary | 38.8% | API only |
| 128 | Amazon Q Developer Agent (v20240719-dev) · proprietary | 38.8% | API only |
| 129 | AutoCodeRover (v20240620) + GPT 4o (2024-05-13) · proprietary | 38.4% | API only |
| 130 | Devstral Small 2507 · 23.6B | 38.0% | ~15 GB |
| 131 | Factory Code Droid · proprietary | 37.0% | API only |
| 132 | GPT 5 nano (2025-08-07) (medium) · proprietary | 34.8% | API only |
| 133 | SWE-agent + Claude 3.5 Sonnet · proprietary | 33.6% | API only |
| 134 | MASAI + GPT 4o (2024-06-12) · proprietary | 32.6% | API only |
| 135 | Artemis Agent v1 (2024-11-20) · proprietary | 32.0% | API only |
| 136 | nFactorial (2024-10-07) · proprietary | 31.6% | API only |
| 137 | SWE-Fixer (Qwen2.5-7b retriever + Qwen2.5-72b editor) 20241128 · proprietary | 30.2% | API only |
| 138 | Lingma Agent + Lingma SWE-GPT 72b (v0925) · proprietary | 28.8% | API only |
| 139 | Gemini 2.5 Flash (2025-04-17) · proprietary | 28.7% | API only |
| 140 | EPAM AI/Run Developer Agent + GPT4o · proprietary | 27.0% | API only |
| 141 | AppMap Navie + GPT 4o (2024-05-13) · proprietary | 26.2% | API only |
| 142 | GPT OSS 120B · 116.8B | 26.0% | ~71 GB |
| 143 | nFactorial (2024-10-01) · proprietary | 25.8% | API only |
| 144 | Amazon Q Developer Agent (v20240430-dev) · proprietary | 25.6% | API only |
| 145 | Lingma Agent + Lingma SWE-GPT 72b (v0918) · proprietary | 25.0% | API only |
| 146 | GPT 4.1 mini (2025-04-14) · proprietary | 23.9% | API only |
| 147 | MCTS-Refine-7B · proprietary | 23.2% | API only |
| 148 | SWE-agent + GPT 4o (2024-05-13) · proprietary | 23.2% | API only |
| 149 | SWE-agent + GPT 4 (1106) · proprietary | 22.4% | API only |
| 150 | GPT 4o (2024-11-20) · proprietary | 21.6% | API only |
| 151 | Llama 4 Maverick Instruct · proprietary | 21.0% | API only |
| 152 | Lingma Agent + Lingma SWE-GPT 7b (v0925) · proprietary | 18.2% | API only |
| 153 | SWE-agent + Claude 3 Opus · proprietary | 15.8% | API only |
| 154 | Gemini 2.0 Flash · proprietary | 13.5% | API only |
| 155 | Lingma Agent + Lingma SWE-GPT 7b (v0918) · proprietary | 10.2% | API only |
| 156 | Llama 4 Scout Instruct · proprietary | 9.1% | API only |
| 157 | RAG + Claude 3 Opus · proprietary | 7.0% | API only |
| 158 | RAG + Claude 2 · proprietary | 4.4% | API only |
| 159 | RAG + GPT 4 (1106) · proprietary | 2.8% | API only |
| 160 | RAG + SWE-Llama 7B · proprietary | 1.4% | API only |
| 161 | RAG + SWE-Llama 13B · proprietary | 1.2% | API only |
| 162 | RAG + ChatGPT 3.5 · proprietary | 0.4% | API only |
Score vs model size
Which models give the most quality for their size — the ones worth running locally.
- Devstral Small 2505, 24B, score 46.8% — on the efficiency frontier (best score at its size or smaller).
- Devstral Small 2 24B Instruct 2512, 24B, score 56.4% — on the efficiency frontier (best score at its size or smaller).
- Qwen3 Coder 30B A3B Instruct, 31B, score 60.4% — on the efficiency frontier (best score at its size or smaller).
- MiniMax M2, 229B, score 61.0% — on the efficiency frontier (best score at its size or smaller).
- MiniMax M2.5, 229B, score 75.8% — on the efficiency frontier (best score at its size or smaller).
SWE-bench Verified: frequently asked questions
- What is the best open LLM on SWE-bench Verified?
- MiniMax M2.5 is the top open model on SWE-bench Verified, scoring 75.8%. Among all models tested — including proprietary ones — it ranks #9. The top model overall is Sonar Foundation Agent + Claude 4.5 Opus at 79.2%.
- What's the best SWE-bench Verified model you can run on a 24 GB GPU?
- Qwen3 Coder 30B A3B Instruct is the highest-scoring open model that fits in 24 GB at 4-bit quantization (~19 GB), scoring 60.4% on SWE-bench Verified.
- Can open models match proprietary models on SWE-bench Verified?
- Not quite on SWE-bench Verified: the strongest proprietary model (Sonar Foundation Agent + Claude 4.5 Opus) scores 79.2%, ahead of the best open model (MiniMax M2.5) at 75.8% — but you can run the open one yourself.
Scores aggregated from swebench. llmrun does not run this benchmark — see the source for methodology, or the about benchmarks for what it measures.