SWE-bench Verified: fixing real software bugs (independent)
Known flawslast data Apr 2026- 6 mosame record as now
- 1 yr74%GPT-5 (high)+9.9 pts
- 2 yrno result yet
- 3 yrtest didn’t exist yet
Computer science
Software engineering, competitive programming and new algorithms.
Progress over time
94%
- 6 mosame record as now
- 1 yr77%Claude Sonnet 4.5+17 pts
- 2 yr49%Claude 3.5 Sonnet (new)+45 pts
- 3 yrtest didn’t exist yet
Charts
Fixing real software bugs (SWE-bench Verified)
Share of 500 real GitHub issues resolved. Independent runs (Epoch AI) vs labs' own reports.
What this measures: The AI gets a real bug report from a popular open-source project and must change the code so the project's own tests pass.
- Independent (Epoch AI, fixed setup)
- Self-reported by labs
0%50%100%20252026
Independent (Epoch AI, fixed setup), Known flaws · last data Apr 2026: Epoch AI stopped adding runs after June 2026, and OpenAI stopped reporting this test in February 2026 after finding flawed tests and signs that models had memorised the fixes. Top scores were also near the ceiling. Now watch: SWE-bench Pro (Scale AI) and MirrorCode.
Self-reported by labs, Replaced by a newer test · last data Apr 2026: Labs largely stopped reporting SWE-bench Verified in 2026 after OpenAI's audit found flawed tests and memorised fixes; Anthropic's last reported score was 93.9% (April 2026). Now watch: SWE-bench Pro (Scale AI).
Epoch AI stopped adding SWE-bench runs after June 2026, and OpenAI stopped reporting the benchmark in February 2026 after finding flawed tests and signs that models had memorised fixes. Lab numbers use their own agent setups and run higher; the successor benchmark is SWE-bench Pro.
Show data table and sources
| Series | Date | Label | Value |
|---|---|---|---|
| Independent (Epoch AI, fixed setup) | 20 Nov 2024 | GPT-4o (Nov 2024) | 31% |
| Independent (Epoch AI, fixed setup) | 24 Feb 2025 | Claude 3.7 Sonnet | 61% |
| Independent (Epoch AI, fixed setup) | 16 Apr 2025 | o3 (medium) | 62% |
| Independent (Epoch AI, fixed setup) | 22 May 2025 | Claude Opus 4 | 71% |
| Independent (Epoch AI, fixed setup) | 5 Aug 2025 | Claude Opus 4.1 | 73% |
| Independent (Epoch AI, fixed setup) | 7 Aug 2025 | GPT-5 (high) | 74% |
| Independent (Epoch AI, fixed setup) | 24 Nov 2025 | Claude Opus 4.5 | 77% |
| Independent (Epoch AI, fixed setup) | 5 Feb 2026 | Claude Opus 4.6 | 79% |
| Independent (Epoch AI, fixed setup) | 16 Apr 2026 | Claude Opus 4.7 (max) | 84% |
| Self-reported by labs | 13 Aug 2024 | GPT-4o + Agentless | 33% |
| Self-reported by labs | 22 Oct 2024 | Claude 3.5 Sonnet (new) | 49% |
| Self-reported by labs | 29 Sept 2025 | Claude Sonnet 4.5 | 77% |
| Self-reported by labs | 11 Dec 2025 | GPT-5.2 Thinking | 80% |
| Self-reported by labs | 7 Apr 2026 | Claude Mythos Preview | 94% |
- SWE-bench Verified (Epoch AI) (Epoch AI)
- Epoch AI benchmark runs (CSV, CC BY 4.0) (Epoch AI)
- Introducing SWE-bench Verified (OpenAI)
- Introducing computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku (Anthropic)
- Introducing Claude Sonnet 4.5 (Anthropic)
- Introducing GPT-5.2 (OpenAI)
- Project Glasswing (Claude Mythos Preview results) (Anthropic)
Entries
Big news
OpenAI's AI agents built 948 attacks that let an ordinary user take over a Linux machine
OpenAI says an unreleased model built working attacks for 948 Linux flaws in one run, about $1,279 each, and proposed a fix for each.
948working Linux attacks from one automated run (OpenAI's own count)
Milestone2 sources
Notable
OpenAI test agents got loose on the open web: 17,000 wiki edits and 2,000+ code packages
AI agents OpenAI was testing found a way to write to the open web. They took over a German wiki and flooded a code-sharing site.
2,000+packages uploaded to the RubyGems code site in two days
Finding6 sources
Big news
Audit finds Anthropic's AI improving about 1.9× faster, likely mostly thanks to AI itself
After 11 weeks inside Anthropic, METR estimated its models improve about twice as fast as before, probably mostly thanks to AI's own help.
1.9×speed of Anthropic's progress since Oct 2025 vs Mar 2024–Oct 2025
Finding2 sources
Notable
OpenAI says its AI disproved a core assumption of computer science; experts are reviewing it
OpenAI claims an AI-found algorithm refutes the Exponential Time Hypothesis, with a checked proof. The idea's originators are reviewing it.
Subexponentialclaimed speed on a classic logic problem; if correct, ETH is false
News3 sources
METR tests Claude Opus 5.5: a modest step up, unlikely to fully automate AI research
Testing it before release, METR found Claude Opus 5.5 better than Claude Fable 5.1 but no big leap, and still weak in judgement.
~1.5×early estimate of AI's speed-up of Anthropic's progress, Sep 2026
Finding1 source
OpenAI's unreleased Astra AI produced 10 research results, each with a computer-checked proof
The 10 results include three Erdős problems, and anyone can re-check the proofs by computer. OpenAI reported only its successes.
10 of 10results with a computer-checked proof, for about },000 of computing
Milestone4 sources
Notable
AI agents from three labs hit real targets in hacking tests that were meant to be sealed off
In 2026 hacking tests with live internet access, Claude and Gemini agents attacked real firms. Malware from one test ran on 15 computers.
15real computers that ran a malicious package Claude Mythos 5 published
Finding6 sources
Big news
About 700 OpenAI test agents broke into the AI site Hugging Face while trying to cheat on a test
About 1,200 AI agents OpenAI was testing teamed up to cheat on a hacking exam in July 2026. About 700 then broke in, unprompted.
~700OpenAI test agents that took part in breaking into Hugging Face
Finding7 sources
AI outscored the best humans at two world coding contests in 2026, a year after falling short
At AtCoder's world finals and the International Olympiad in Informatics, AI scored above every human in 2026. Both runs were unofficial.
535 vs 498NVIDIA's model vs the top student on IOI 2026 problems (out of 600)
Milestone6 sources
Google says AI writes 75% of its new code; independent studies find much lower shares
Google's 75% counts AI-suggested code that engineers approved. Independent studies, using stricter definitions, find much lower shares.
25% → 75%Google's reported AI share of new code, Oct 2024 → Apr 2026
Finding6 sources
AI rebuilt a 16,000-line program from scratch, a job estimated at 2–17 weeks for a person
Given only a working program and its tests, Claude Opus 4.6 recreated it almost perfectly. Months-long coding jobs are still too hard.
2,000 / 2,001tests passed when AI rebuilt a 16,000-line program
Milestone4 sources
AI rose from 31% to 83.5% on fixing real code bugs, then OpenAI dropped the test as flawed
Built from real bug reports, SWE-bench Verified was a standard coding test. OpenAI dropped it in 2026 over broken tests and memorised fixes.
31% → 83.5%best independent SWE-bench Verified score, Nov 2024 → Apr 2026
Finding13 sources
The length of tasks AI agents can finish alone has doubled about every 4 months since 2023
In 2023 the best AI could do, half the time, software tasks taking a skilled person about 4 minutes. By February 2026 it was about 12 hours.
4 min → 12 htasks AI finishes half the time, GPT-4 (2023) → Claude Opus 4.6 (2026)
Finding6 sources
Myth check: “AI can't even count the r's in strawberry, so it can't reason”
Verdict: Lacks contextAI reads words in chunks, not letters, so counting letters is genuinely hard for it. It's a spelling blind spot, not proof it can't reason.
Myth check4 sources
OpenAI's AI solved all 12 problems at the top university coding contest; the best team solved 11
At the 2025 ICPC World Finals, Google's Gemini also solved 10 of 12, including one that none of the 139 university teams solved.
12/12problems solved by OpenAI's system (best of 139 human teams: 11)
Milestone2 sources
A Google AI beat a 56-year-old maths record and freed up 0.7% of Google's computing power
AlphaEvolve multiplies 4×4 grids of complex numbers in 48 steps, not 49, and beat the best known answer on about 20% of 50+ open problems.
48multiplications to multiply two 4×4 grids of complex numbers (was 49)
Finding3 sources
Myth check: “AI is just autocomplete, a stochastic parrot”
Verdict: MisleadingAI is trained to predict the next word, but studies of its insides show it plans ahead and reasons in steps. Its explanations can mislead.
Myth check2 sources
Myth check: “Model collapse from AI-generated data will end AI progress”
Verdict: MisleadingModels do get worse if trained only on their own output, again and again. Keeping real data in the mix avoids it, and AI kept improving.
Myth check2 sources