Skip to content

Fixing real software bugs (SWE-bench Verified)

Share of 500 real GitHub issues resolved. Independent runs (Epoch AI) vs labs' own reports.

Topics: Computer science, Benchmarks

What this measures: The AI gets a real bug report from a popular open-source project and must change the code so the project's own tests pass.

  • Independent (Epoch AI, fixed setup)
  • Self-reported by labs
0%50%100%20252026

Independent (Epoch AI, fixed setup), Known flaws · last data Apr 2026: Epoch AI stopped adding runs after June 2026, and OpenAI stopped reporting this test in February 2026 after finding flawed tests and signs that models had memorised the fixes. Top scores were also near the ceiling. Now watch: FrontierSWE V2, Terminal-Bench 4.1 and MirrorCode.

Self-reported by labs, Replaced by a newer test · last data Apr 2026: Labs largely stopped reporting SWE-bench Verified in 2026 after OpenAI's audit found flawed tests and memorised fixes; Anthropic's last reported score was 93.9% (April 2026). Now watch: SWE-bench Pro (Scale AI).

Lab numbers use their own agent setups and run higher.

Sources: Epoch AI, OpenAI, AnthropicData as of 9 Nov 2026
Show data table and sources
SeriesDateLabelValue
Independent (Epoch AI, fixed setup)20 Nov 2024GPT-4o (Nov 2024)31%
Independent (Epoch AI, fixed setup)24 Feb 2025Claude 3.7 Sonnet61%
Independent (Epoch AI, fixed setup)16 Apr 2025o3 (medium)62%
Independent (Epoch AI, fixed setup)22 May 2025Claude Opus 471%
Independent (Epoch AI, fixed setup)5 Aug 2025Claude Opus 4.173%
Independent (Epoch AI, fixed setup)7 Aug 2025GPT-5 (high)74%
Independent (Epoch AI, fixed setup)24 Nov 2025Claude Opus 4.577%
Independent (Epoch AI, fixed setup)5 Feb 2026Claude Opus 4.679%
Independent (Epoch AI, fixed setup)16 Apr 2026Claude Opus 4.7 (max)84%
Self-reported by labs13 Aug 2024GPT-4o + Agentless33%
Self-reported by labs22 Oct 2024Claude 3.5 Sonnet (new)49%
Self-reported by labs29 Sept 2025Claude Sonnet 4.577%
Self-reported by labs11 Dec 2025GPT-5.2 Thinking80%
Self-reported by labs7 Apr 2026Claude Mythos Preview94%

Entries with this chart