METR's 2026 trial: developers took 4–18% less time with AI tools, but the signal is weak
With late-2025 AI tools, returning developers took 18% less time and new recruits 4% less. Neither is certain; METR expects more.
METR, an independent AI evaluation group, ran a larger follow-up to its 2025 developer trial, starting in August 2025. This time 57 developers worked on more than 800 tasks across 143 code projects (repositories). Some tasks allowed AI tools and some did not.
For 10 developers from the original study, tasks with AI allowed took 18% less time. For newly recruited developers, they took 4% less. Both come with wide confidence intervals (the range the true value probably sits in): −38% to +9% and −15% to +9%. Both ranges include zero, so neither rules out no effect at all.
METR says the study now gives an unreliable signal, for a telling reason. Some developers declined to take part because they didn’t want to work without AI. Others held back tasks they didn’t want to do without it. METR therefore sees its estimate as likely a lower bound (the true effect is probably larger), and is redesigning the study.
A separate METR survey of 349 technical workers (February–April 2026) found a median self-reported 2× boost, up from 1.3× a year earlier. METR cautions that self-reports are “not necessarily grounded in reality”, given how wrong developers’ own estimates were in 2025.
What it means
For people who write software, the best controlled evidence now leans towards AI tools saving time rather than costing it. It does not yet say by how much. Developers’ own sense of their speed-up is not a reliable guide either way.
Caveats
- Both confidence intervals include zero, so the trial alone does not prove a speed-up.
- Who took part, and which tasks they submitted, skewed the sample in ways that can push the estimate in either direction.
- The survey is a convenience sample (whoever chose to reply) with about a 2% response rate.