Standardised tests of AI ability, expert baselines, and how fast tests get saturated.https://www.justaparrot.com/enAI Trackerhttps://www.justaparrot.com/e/claude-fable-5-2-release/https://www.justaparrot.com/e/claude-fable-5-2-release/Anthropic released it on 9 November 2026. Independent tests put it level with rivals on hard maths and behind them on PhD-level science.Tue, 10 Nov 2026 00:00:00 GMTbenchmarksmathhttps://www.justaparrot.com/e/ai-2027-vs-reality/https://www.justaparrot.com/e/ai-2027-vs-reality/Reframed around METR's November audit: on AI speeding up AI research, reality has caught up with the scenario's March 2027. Added Daniel Kokotajlo's and Ryan Greenblatt's reactions.. METR found AI roughly doubling Anthropic's pace, a level the scenario reaches in March 2027. Its lead author says the team is ‘spooked’.Tue, 10 Nov 2026 00:00:00 GMTautonomymythsbenchmarkshttps://www.justaparrot.com/e/arc-agi-harness-matters/https://www.justaparrot.com/e/arc-agi-harness-matters/Added Claude Fable 5.2, the new best score with the standard harness (81.2%).. How a model is run in a test can matter as much as the model. GPT-6 Astra scored 62.7% or 99.9% depending on the software wrapped around it.Tue, 10 Nov 2026 00:00:00 GMTbenchmarksautonomycost-of-intelligencehttps://www.justaparrot.com/e/arc-agi-2-and-arc-agi-3-progress/https://www.justaparrot.com/e/arc-agi-2-and-arc-agi-3-progress/Claude Fable 5.2 raised the best ARC-AGI-3 standard-harness score from 62.7% to 81.2%.. Puzzles ordinary people can solve stumped AI in early 2025. By September 2026 the best model solved 95%, checked by the ARC Prize team.Tue, 10 Nov 2026 00:00:00 GMTbenchmarksautonomyhttps://www.justaparrot.com/e/felonybench-ai-incident-leaderboard/https://www.justaparrot.com/e/felonybench-ai-incident-leaderboard/Now uses felonybench.org's counts (13 and 10 probable felonies) and explains how it counts; felonybench.com's incident tally (24) is given alongside.. It counts published cases where an AI agent did what would probably be a US felony for a person. Labs that disclose more score higher.Tue, 10 Nov 2026 00:00:00 GMTfunbenchmarksautonomyhttps://www.justaparrot.com/e/blueprint-bench-2-floor-plans/https://www.justaparrot.com/e/blueprint-bench-2-floor-plans/Shown about 20 photos of a flat, Gemini 4 Carbon draws its floor plan as well as the people tested. In 2025, people beat every model.Mon, 09 Nov 2026 00:00:00 GMTfunbenchmarkshttps://www.justaparrot.com/e/metr-anthropic-rd-acceleration-oct-2026/https://www.justaparrot.com/e/metr-anthropic-rd-acceleration-oct-2026/After 11 weeks inside Anthropic, METR estimated its models improve about twice as fast as before, probably mostly thanks to AI's own help.Mon, 09 Nov 2026 00:00:00 GMTautonomycomputer-sciencebenchmarkshttps://www.justaparrot.com/e/iq-style-test-scores/https://www.justaparrot.com/e/iq-style-test-scores/The quiz is kept offline so models can't have trained on it. Scores rose about 50 points since May 2024, but an AI ‘IQ’ means little.Mon, 09 Nov 2026 00:00:00 GMTcost-of-intelligencebenchmarkshttps://www.justaparrot.com/e/bullshitbench-nonsense-detection/https://www.justaparrot.com/e/bullshitbench-nonsense-detection/BullshitBench asks 100 expert-sounding questions that make no sense. Newer top models play along more often than an older Claude did.Mon, 09 Nov 2026 00:00:00 GMTfunbenchmarkshttps://www.justaparrot.com/e/gpqa-diamond-models-pass-phd-experts/https://www.justaparrot.com/e/gpqa-diamond-models-pass-phd-experts/On 198 hard multiple-choice science questions, independent tests put the best AI at 96.4% in 2026. PhD experts get 69.7%; guessing 25%.Mon, 09 Nov 2026 00:00:00 GMTbenchmarkshttps://www.justaparrot.com/e/vending-bench-2-cartels-and-refunds/https://www.justaparrot.com/e/vending-bench-2-cartels-and-refunds/The leader earns a quarter of what a good strategy could. Against rival AI shops, Claude Opus 5 proposed or joined price cartels.Mon, 09 Nov 2026 00:00:00 GMTfunbenchmarksautonomyhttps://www.justaparrot.com/e/epoch-price-of-thought/https://www.justaparrot.com/e/epoch-price-of-thought/The price of answering at a fixed standard halves about every quarter: one science level got 725× cheaper in 18 months. Top models did not.Mon, 09 Nov 2026 00:00:00 GMTcost-of-intelligencebenchmarkshttps://www.justaparrot.com/e/robot-household-task-success/https://www.justaparrot.com/e/robot-household-task-success/Figure and Physical Intelligence report about half of chores fully done in their own tests. In an independent challenge, the best got 12%.Mon, 09 Nov 2026 00:00:00 GMTroboticsbenchmarkshttps://www.justaparrot.com/e/frontiermath-tier-4-saturated/https://www.justaparrot.com/e/frontiermath-tier-4-saturated/In July 2025 the best AI solved 6% of Epoch AI's hardest, professor-written maths problems. By September 2026 top models scored 97.6–100%.Mon, 09 Nov 2026 00:00:00 GMTmathbenchmarkshttps://www.justaparrot.com/e/humanitys-last-exam-progress/https://www.justaparrot.com/e/humanitys-last-exam-progress/It is 2,500 expert questions that top AI got wrong when it was made. Independent tests without web search put the best at 54.8%.Mon, 09 Nov 2026 00:00:00 GMTbenchmarkshttps://www.justaparrot.com/e/erdos-problems-ai-scorecard/https://www.justaparrot.com/e/erdos-problems-ai-scorecard/Counting every effort, AI has solved 17 of 68 major open Erdős problems. Under Epoch AI's strict test rules, the best model solves 3.Mon, 09 Nov 2026 00:00:00 GMTmathbenchmarkshttps://www.justaparrot.com/e/ai-cheating-on-tests-reward-hacking/https://www.justaparrot.com/e/ai-cheating-on-tests-reward-hacking/Some AI models game how a test is checked instead of doing the task, so a high score can hide work that was never done.Mon, 09 Nov 2026 00:00:00 GMTfunbenchmarksautonomyhttps://www.justaparrot.com/e/osworld-computer-use/https://www.justaparrot.com/e/osworld-computer-use/On a standard test of everyday computer tasks, AI passed the human score in December 2025. On tasks taking people 1.6 hours, it got 20.6%.Mon, 09 Nov 2026 00:00:00 GMTautonomybenchmarkshttps://www.justaparrot.com/e/arc-agi-cost-per-task/https://www.justaparrot.com/e/arc-agi-cost-per-task/Passing 75% on a set of visual puzzles cost $200 a puzzle in December 2024 and $0.012 by July 2026. The best models still cost far more.Mon, 09 Nov 2026 00:00:00 GMTcost-of-intelligencebenchmarkshttps://www.justaparrot.com/e/imo-2026-perfect-scores/https://www.justaparrot.com/e/imo-2026-perfect-scores/AI from Xiaohongshu and Huawei was reported as officially graded 42 of 42. Seven of 666 students also got full marks. In 2024 AI scored 28.Mon, 09 Nov 2026 00:00:00 GMTmathbenchmarkshttps://www.justaparrot.com/e/pelican-on-a-bicycle-benchmark/https://www.justaparrot.com/e/pelican-on-a-bicycle-benchmark/Simon Willison asks every new AI to draw a pelican riding a bicycle in code. A study of 48 similar prompts found no sign labs train for it.Mon, 09 Nov 2026 00:00:00 GMTfunbenchmarkshttps://www.justaparrot.com/e/competitive-programming-2026-ai-beats-humans/https://www.justaparrot.com/e/competitive-programming-2026-ai-beats-humans/At AtCoder's world finals and the International Olympiad in Informatics, AI scored above every human in 2026. Both runs were unofficial.Mon, 09 Nov 2026 00:00:00 GMTcomputer-sciencebenchmarkshttps://www.justaparrot.com/e/first-proof-second-batch/https://www.justaparrot.com/e/first-proof-second-batch/Four AI systems tried 10 research problems that had never been published. Referees graded the answers like journal papers.Mon, 09 Nov 2026 00:00:00 GMTmathbenchmarkshttps://www.justaparrot.com/e/metr-frontier-2026-saturation/https://www.justaparrot.com/e/metr-frontier-2026-saturation/In early 2026 the best public AI succeeded half the time on tasks a skilled person needs 12 hours for. METR's tests can't measure further.Mon, 09 Nov 2026 00:00:00 GMTautonomybenchmarkshttps://www.justaparrot.com/e/myth-ai-progress-hit-a-wall/https://www.justaparrot.com/e/myth-ai-progress-hit-a-wall/AI kept improving fast through 2026, from 4-minute tasks in 2023 to 12-hour ones. Just making models bigger did slow, and real use lags.Mon, 09 Nov 2026 00:00:00 GMTmythsbenchmarksautonomyhttps://www.justaparrot.com/e/claude-beats-pokemon-red/https://www.justaparrot.com/e/claude-beats-pokemon-red/Old Pokémon games became an informal test of long tasks for AI. Claude stalled for months; Gemini, first to win, had a more helpful setup.Mon, 09 Nov 2026 00:00:00 GMTfunbenchmarksautonomyhttps://www.justaparrot.com/e/gdpval-models-vs-industry-experts/https://www.justaparrot.com/e/gdpval-models-vs-industry-experts/Experts blind-graded AI and professional work, such as legal briefs, from 44 jobs. The 84.9% is OpenAI's own count on its own test.Mon, 09 Nov 2026 00:00:00 GMTbenchmarksadoptionhttps://www.justaparrot.com/e/benchmark-saturation/https://www.justaparrot.com/e/benchmark-saturation/On tests launched in 2025–26, the best AI scores went from near 0% to over 50% within months, so test makers keep building harder ones.Mon, 09 Nov 2026 00:00:00 GMTbenchmarkshttps://www.justaparrot.com/e/mirrorcode-weeks-long-coding-tasks/https://www.justaparrot.com/e/mirrorcode-weeks-long-coding-tasks/Given only a working program and its tests, Claude Opus 4.6 recreated it almost perfectly. Months-long coding jobs are still too hard.Mon, 09 Nov 2026 00:00:00 GMTautonomycomputer-sciencebenchmarkshttps://www.justaparrot.com/e/swe-bench-verified-saturation/https://www.justaparrot.com/e/swe-bench-verified-saturation/Built from real bug reports, SWE-bench Verified was a standard coding test. OpenAI dropped it in 2026 over broken tests and memorised fixes.Mon, 09 Nov 2026 00:00:00 GMTcomputer-sciencebenchmarkshttps://www.justaparrot.com/e/metr-time-horizon-doubling/https://www.justaparrot.com/e/metr-time-horizon-doubling/In 2023 the best AI could do, half the time, software tasks taking a skilled person about 4 minutes. By February 2026 it was about 12 hours.Mon, 09 Nov 2026 00:00:00 GMTautonomybenchmarkscomputer-sciencehttps://www.justaparrot.com/e/icpc-world-finals-2025/https://www.justaparrot.com/e/icpc-world-finals-2025/At the 2025 ICPC World Finals, Google's Gemini also solved 10 of 12, including one that none of the 139 university teams solved.Mon, 09 Nov 2026 00:00:00 GMTcomputer-sciencebenchmarkshttps://www.justaparrot.com/e/imo-2025-gold-level/https://www.justaparrot.com/e/imo-2025-gold-level/Two AI systems solved 5 of 6 problems with plain-language proofs, in the time limits. The contest graded Google's; OpenAI chose its graders.Mon, 09 Nov 2026 00:00:00 GMTmathbenchmarkshttps://www.justaparrot.com/e/virology-capabilities-test/https://www.justaparrot.com/e/virology-capabilities-test/On written lab-problem questions, o3 scored 43.8% to virologists' 22.1%. In a 2026 trial, AI did not significantly help novices in the lab.Mon, 09 Nov 2026 00:00:00 GMTbenchmarkshttps://www.justaparrot.com/e/price-of-gpt4-level-performance/https://www.justaparrot.com/e/price-of-gpt4-level-performance/The cheapest model matching GPT-4 on PhD-level science fell from $37.50 to $0.12 per million tokens by late 2024. Top models stay pricey.Mon, 09 Nov 2026 00:00:00 GMTcost-of-intelligencebenchmarks