DeepSWE Leaderboard
DeepSWE, built by Datacurve, evaluates coding agents on 113 original software-engineering tasks across 91 repositories and five languages — written from scratch rather than adapted from real commits, so a model can't have seen the solution during pretraining, and graded Pass@1 with hand-written behavioral verifiers. It's a benchmark name only, unrelated to the open-weight DeepSWE-Preview coding model.
Source: epoch5 open models ranked+63 proprietaryData through Sep 2026
Open models ranked on DeepSWE
# shows rank among open models / rank overall (including proprietary).
| # | Model | Score |
|---|---|---|
| 1 / 16 | GLM 5.3 · 753.3B | 69.0% |
| 2 / 19 | Kimi K3 · 2779.9B | 68.5% |
| 3 / 30 | GLM 5.3 Flash · 321.3B | 63.4% |
| 4 / 54 | GLM 5.2 · 753.3B | 43.8% |
| 5 / 61 | Kimi K2.7 Code · 1026.9B | 30.5% |
Score vs model size
Which models give the most quality for their size — the ones worth running locally.
- GLM 5.3 Flash, 321B, score 63.4% — on the efficiency frontier (best score at its size or smaller).
- GLM 5.3, 753B, score 69.0% — on the efficiency frontier (best score at its size or smaller).
DeepSWE: frequently asked questions
- What is the best open LLM on DeepSWE?
- GLM 5.3 is the top open model on DeepSWE, scoring 69.0%. Among all models tested — including proprietary ones — it ranks #16. The top model overall is GPT 6 Astra (xhigh) (OpenAI) at 74.1%.
- Can open models match proprietary models on DeepSWE?
- Not quite on DeepSWE: the strongest proprietary model (GPT 6 Astra (xhigh)) scores 74.1%, ahead of the best open model (GLM 5.3) at 69.0% — but you can run the open one yourself.
Scores aggregated from epoch. llmrun does not run this benchmark — see the source for methodology, or the about benchmarks for what it measures.