Method
AI Tracker exists because both hype and denial about AI do better without numbers. Here is how the numbers get here.
Sourcing rules
- Every number links to a source. Entries cannot be published without at least one source; the build fails otherwise.
- Original sources first. Papers, lab announcements, benchmark leaderboards and the evaluating organisations (METR, Epoch AI, ARC Prize, SecureBio…). News articles, X posts and Reddit threads are linked too, since they're often where results surface first. Each entry is marked as either traced to the original source or based on secondary reporting.
- Unverified items aren't published. A screenshot without a traceable origin stays a draft until the original is found.
- Caveats are part of the entry. Self-reported vs independently checked results, compute budgets, contamination worries, and evidence that cuts the other way (for example, studies where AI tools slowed people down) are included rather than left out.
- Like-for-like comparisons. Myth checks state the claim fairly, and compare footprints in the same units and scope (direct vs supply-chain water, per prompt vs per year).
Saturation and human baselines
What does it mean when a test is “saturated”?
A test is saturated when the best AI scores close to the maximum, or as high as mistakes in the test's own answer key allow. It can no longer tell newer models apart, so researchers build harder tests.Full glossary.
Tests designed to stay hard for years have often fallen in one or two. Skill isn't the only cause: some tests leak into training data or have flawed questions. Each chart says when and why a test stopped being useful, and what replaced it.
What is a “human baseline”?
The score people get on the same test, measured by the test's creators. PhD experts get about 70% of GPQA's science questions right; the average person solves about 60% of ARC-AGI-2's puzzles.
Baselines depend on who was tested (specialists or the general public), how long they had and whether they were paid to try hard. “AI beats humans” only means something when it says which humans, on which test. Many tests have no published human baseline.
Badges
- New: added in the last 21 days.
- Updated: meaningfully changed in the last 21 days. The entry shows what changed and when. Typo fixes don't count.
News levels
Most entries carry no level. A few are marked on the day they are added, so readers who check in rarely can see what mattered:
- Big news: a large jump in what AI can do, a serious incident or an important discovery.
- Notable: worth a second look.
The level is a judgement call, reviewed daily, and can change as more is known.
Progress over time
For each tracked measure, the comparison takes the best publicly reported result on or before each date (today, 6 months ago, 1, 2 and 3 years ago). Percent scores are compared in percentage points; durations and prices as multiples. When a test didn't exist yet at a given date, it says so instead of showing zero.
Updated daily
The source list (benchmark trackers, feeds from labs and independent testers, and the communities where results first surface) is checked every day. New results become entries, chart points or model scores, each with its source. The home page lists what changed each day, and every entry shows when it was added and when it last changed.
Follow along
Subscribe to the main RSS feed, or to a single topic: each topic page has its own feed.
Corrections
Found an error, a better source, or something missing? Open an issue. Corrections are applied with an Updated note so changes stay visible.