AI tests meant to last for years are being maxed out within one to two years

A benchmark is a standard test used to compare AI models. It is “saturated” when the best models score so close to the maximum that it can no longer tell them apart. That now happens within one to two years of a test’s release.

Stanford’s 2025 AI Index looked at tests introduced in 2023. In a single year, top scores rose by 18.8 percentage points on MMMU (college-level questions with images), 48.9 on GPQA (PhD-level science questions) and 67.3 on SWE-bench (fixing real software bugs). The 2026 edition says SWE-bench Verified went “from 60% to near 100% in a single year”.

Examples tracked on this site:

• MATH (competition maths): 6.9% at release in 2021, about 98% by 2025. Labs then stopped reporting it.
• AIME (a US maths contest): 100% without tools (GPT-5.2, December 2025).
• ARC-AGI-2 (visual puzzles): about 4% to 95.0% in 18 months.
• FrontierMath Tiers 1–3 (hard problems written by mathematicians): under 2% at launch (November 2024) to 93.7% on Epoch AI’s corrected version (September 2026).
• ARC-AGI-3 (puzzle games): 0.51% at launch (March 2026) to 62.7% six months later.
• Humanity’s Last Exam (expert questions that AI failed when they were written): about 9% (January 2025) to 54.8% in independent testing.

Test makers respond with harder or cleaned-up versions: FrontierMath v2 (June 2026, which fixed errors in 42% of problems), HLE-Rolling, HLE-Diamond and SWE-bench Pro.

What it means

When you read that AI aced a test, check how old the test is: tests from 2023 onwards have been topped within one to two years. Fast saturation reflects real progress, but also leaked test questions and mistakes in the tests. A maxed-out test shows a model can answer that kind of question, not that it can do the job the test stands for.

Caveats

• Saturation partly reflects flawed tests and leaked test data, not only ability: SWE-bench Verified’s flawed tests, wrong answers in Humanity’s Last Exam and ARC Prize’s concern that models are tuned to its puzzles.
• Labs’ own scores run higher than independent runs.
• Progress is uneven: the 2026 AI Index notes the top model read analogue clocks correctly only 50.1% of the time, and agents still fail about 1 in 3 structured tasks.
• Test scores aren’t real-world outcomes; even job-style tests like GDPval use one-shot, clearly specified tasks.