AI rose from 31% to 83.5% on fixing real code bugs, then OpenAI dropped the test as flawed
Built from real bug reports, SWE-bench Verified was a standard coding test. OpenAI dropped it in 2026 over broken tests and memorised fixes.
Fixing real software bugs (SWE-bench Verified)
Share of 500 real GitHub issues resolved. Independent runs (Epoch AI) vs labs' own reports.
What this measures: The AI gets a real bug report from a popular open-source project and must change the code so the project's own tests pass.
- Independent (Epoch AI, fixed setup)
- Self-reported by labs
Independent (Epoch AI, fixed setup), Known flaws · last data Apr 2026: Epoch AI stopped adding runs after June 2026, and OpenAI stopped reporting this test in February 2026 after finding flawed tests and signs that models had memorised the fixes. Top scores were also near the ceiling. Now watch: SWE-bench Pro (Scale AI) and MirrorCode.
Self-reported by labs, Replaced by a newer test · last data Apr 2026: Labs largely stopped reporting SWE-bench Verified in 2026 after OpenAI's audit found flawed tests and memorised fixes; Anthropic's last reported score was 93.9% (April 2026). Now watch: SWE-bench Pro (Scale AI).
Epoch AI stopped adding SWE-bench runs after June 2026, and OpenAI stopped reporting the benchmark in February 2026 after finding flawed tests and signs that models had memorised fixes. Lab numbers use their own agent setups and run higher; the successor benchmark is SWE-bench Pro.
Show data table and sources
| Series | Date | Label | Value |
|---|---|---|---|
| Independent (Epoch AI, fixed setup) | 20 Nov 2024 | GPT-4o (Nov 2024) | 31% |
| Independent (Epoch AI, fixed setup) | 24 Feb 2025 | Claude 3.7 Sonnet | 61% |
| Independent (Epoch AI, fixed setup) | 16 Apr 2025 | o3 (medium) | 62% |
| Independent (Epoch AI, fixed setup) | 22 May 2025 | Claude Opus 4 | 71% |
| Independent (Epoch AI, fixed setup) | 5 Aug 2025 | Claude Opus 4.1 | 73% |
| Independent (Epoch AI, fixed setup) | 7 Aug 2025 | GPT-5 (high) | 74% |
| Independent (Epoch AI, fixed setup) | 24 Nov 2025 | Claude Opus 4.5 | 77% |
| Independent (Epoch AI, fixed setup) | 5 Feb 2026 | Claude Opus 4.6 | 79% |
| Independent (Epoch AI, fixed setup) | 16 Apr 2026 | Claude Opus 4.7 (max) | 84% |
| Self-reported by labs | 13 Aug 2024 | GPT-4o + Agentless | 33% |
| Self-reported by labs | 22 Oct 2024 | Claude 3.5 Sonnet (new) | 49% |
| Self-reported by labs | 29 Sept 2025 | Claude Sonnet 4.5 | 77% |
| Self-reported by labs | 11 Dec 2025 | GPT-5.2 Thinking | 80% |
| Self-reported by labs | 7 Apr 2026 | Claude Mythos Preview | 94% |
- SWE-bench Verified (Epoch AI) (Epoch AI)
- Epoch AI benchmark runs (CSV, CC BY 4.0) (Epoch AI)
- Introducing SWE-bench Verified (OpenAI)
- Introducing computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku (Anthropic)
- Introducing Claude Sonnet 4.5 (Anthropic)
- Introducing GPT-5.2 (OpenAI)
- Project Glasswing (Claude Mythos Preview results) (Anthropic)
SWE-bench Verified asks an AI model to fix real problems reported on popular Python projects on GitHub, a site where programmers share code. Each fix is checked with the project’s own tests. OpenAI released it in August 2024: 500 issues screened by 93 developers, on which GPT-4o solved 33.2%.
In Epoch AI’s independent runs with a fixed setup, the best score rose from 31% (GPT-4o, November 2024) to 74% (GPT-5, August 2025) and 83.5% (Claude Opus 4.7, April 2026). Epoch stopped adding SWE-bench runs after June 2026. Labs’ own reported scores run higher: 80.0% for GPT-5.2 (December 2025) and 93.9% for Claude Mythos Preview (April 2026, a model with restricted access).
In February 2026, OpenAI said it no longer evaluates on the test. It audited 138 problems its o3 model didn’t solve consistently and found at least 59.4% had flaws that mattered. In 35.5%, the tests were too narrow to accept valid fixes; in 18.8%, they checked for features the task never asked for. Every frontier model it tested could also reproduce the original human-written fix for some tasks, a sign of memorisation from training. Anthropic says its own memorisation checks flag a subset of problems. OpenAI now recommends Scale AI’s harder SWE-bench Pro, where the best public score is about 61%.
What it means
Top scores on SWE-bench Verified no longer tell you which model codes best; harder tests have taken over. Independent runs show real progress, but part of the top end reflects broken tests and memorised answers. Fixing one isolated bug is also a long way from building whole software.
Caveats
- Scores depend heavily on the agent setup, retries and compute; vendor and independent numbers aren’t directly comparable.
- The contamination findings come from OpenAI, which created the benchmark and competes with the labs whose models it tested. It also wasn’t leading the leaderboard when it retired the benchmark, which some read as a motive.
- Epoch rates SWE-bench Verified as “flawed” and estimates 5–10% of tasks are ambiguous.
- These benchmarks measure isolated bug fixes, not whole software projects.