An exam built to stump AI: best scores rose from about 9% in Jan 2025 to 54.8% by Sept 2026
It is 2,500 expert questions that top AI got wrong when it was made. Independent tests without web search put the best at 54.8%.
Humanity's Last Exam
Best independently measured score without tools (Scale AI leaderboard), by model release date.
What this measures: 2,500 expert-written questions chosen because the best AI of early 2025 got them wrong. At launch, top models scored under 10%.
Labs report higher scores with tools (e.g. 64.7% for Claude Mythos Preview, self-reported). Those are not comparable with these no-tools runs. Independent reviews estimate 18–29% of the biology/chemistry answers conflict with the literature, so scores above ~70–80% may reflect label noise rather than skill.
Show data table and sources
| Date | Label | Value |
|---|---|---|
| 17 Dec 2024 | o1 | 8% |
| 16 Apr 2025 | o3 (high) | 20% |
| 5 Jun 2025 | Gemini 2.5 Pro | 22% |
| 7 Aug 2025 | GPT-5 | 25% |
| 6 Oct 2025 | GPT-5 Pro | 32% |
| 18 Nov 2025 | Gemini 3 Pro | 38% |
| 19 Feb 2026 | Gemini 3.1 Pro (high) | 46% |
| 2 Sept 2026 | GPT-6 Astra | 55% |
- Scale Labs Leaderboard: Humanity's Last Exam (Scale AI)
- Humanity's Last Exam (paper) (CAIS / Scale AI)
- About 30% of Humanity's Last Exam chemistry/biology answers are likely wrong (FutureHouse)
Humanity’s Last Exam (HLE) was built by the Center for AI Safety and Scale AI and published in January 2025 (later in Nature). It has 2,500 questions with clear-cut answers, written by nearly 1,000 subject experts. A question was only accepted if leading AI models got it wrong. At release, OpenAI’s o1 scored 9.1% and GPT-4o 3.3%.
Scale AI runs an independent leaderboard with no tools, meaning no web search or running code, and grades answers automatically. The best score there climbed from 20.3% (o3, April 2025) to 37.5% (Gemini 3 Pro, November 2025), 46.4% (Gemini 3.1 Pro, February 2026) and 54.8% (GPT-6 Astra, September 2026, ±1.9).
Labs’ own numbers run higher, especially when tools are allowed. Anthropic reported Claude Mythos Preview at 56.8% without tools and 64.7% with tools (April 2026). Models are still often confidently wrong: Scale lists calibration errors (the gap between how sure a model says it is and how often it is right) of 20–57% for the top entries. The maintainers have released successors, HLE-Rolling and HLE-Diamond, for when models reach the original set’s noise ceiling.
What it means
Questions chosen because AI got them wrong are now about half answered by the best model, without looking anything up. But some of the exam’s own answers are wrong and models are often overconfident, so a high score is no reason to trust any single answer. Scale also notes that a high score alone does not show a model can do research on its own.
Caveats
- Some answers are wrong. FutureHouse estimated 29% (±3.7%) of text-only chemistry and biology answers conflict with published research; the HLE team’s own review found about 18% of a bio/chem subset problematic. Scores above about 70–80% may mostly reflect these errors.
- With-tools and no-tools scores aren’t comparable, and lab self-reports use different settings and graders.
- Anthropic says Mythos Preview does well on HLE even at low effort, which could point to some memorisation. The questions are public; a private held-out set exists, but Scale doesn’t report scores on it.
- Scale’s page notes that a high HLE score would not on its own suggest a model can do research autonomously.