Glossary

A test is saturated when the best AI scores close to the maximum, or close to the point where mistakes in the test's own answer key cap what anyone can score. After that it can't tell newer models apart, so researchers build harder tests.

Saturation is a sign of progress: tests designed to stay hard for years have often fallen in one or two. But it isn't only skill. Some tests leak into training data or turn out to have flawed questions, which is why each chart says what replaced the test.

The people who run this measure have stopped publishing new results, or nothing new has appeared for more than five months. The chart shows the last data point and says why it stopped, for example because the newest models finish nearly every task.

Researchers and labs have moved to a harder or cleaner version of the test. The old results stay on the site for history; the note on the chart names the newer test to watch.

Problems were found in the test itself, such as wrong answer keys, tests that reject correct solutions, or questions that models had already seen during training. Scores still mean something, but treat small differences with care.

Newer evidence has overtaken this entry, usually a later study with newer AI tools. We keep it because it was true for its time and is still cited, but the banner at the top says what changed and links to the newer result.

Nobody has submitted a new result for a while, usually because a different approach overtook this one and there's little reason to keep trying.

The score people get on the same test, measured by the test's creators. For example, PhD experts get about 70% of GPQA's science questions right, and the average person solves about 60% of ARC-AGI-2's puzzles.

Baselines depend on which people were tested (specialists or the general public), how long they had, and whether they were paid to try hard. So "AI beats humans" only means something when it says which humans, on which test.

A fixed set of tasks with known answers, used to compare AI systems the same way. Think of it as a standardised exam for AI.

Independent results come from an organisation that doesn't sell the model (for example Epoch AI, METR, ARC Prize or Scale AI) running it in a fixed, published way. Self-reported results come from the company that made the model. They're often higher, because companies choose their best settings.

The software wrapped around a model while it's tested: what it can see, what it remembers between steps, whether it can run code, and how many attempts it gets. The same model can score very differently with different harnesses, so the site keeps them apart.

How big a job an AI agent can finish on its own. A task's "length" is how long it takes a skilled person. A 2-hour time horizon means the AI succeeds half the time on tasks that take a skilled person about 2 hours.

One of the most capable AI models available at a given moment, usually each major lab's newest top model.

The chunks of text a model reads and writes, roughly three quarters of a word each. AI is priced per million tokens, about 750,000 words.

A computer program that checks a mathematical proof step by step. If Lean accepts a proof, it is correct, as long as the statement being proved was written down correctly. That makes AI maths results much easier to trust.

A chart axis where each step is 10 times the previous one (1, 10, 100, 1,000). It lets very small and very large numbers fit on one chart. On a log scale, a straight upward line means something grows by the same multiple every year.

A single score from Epoch AI that combines a model's results on many benchmarks, so models can be compared over time even as individual tests saturate.

Extending a past trend into the future to see where it would lead if it continued. It isn't a forecast: trends can speed up, slow down or stop. The site always shows a low, middle and high version so the uncertainty is visible.