Skip to content
AI Tracker
← Back to Agents & METR

Outdated: developers took 19% longer with early-2025 AI; METR's 2026 data points to speed-ups

Finding

In 2025, 16 skilled developers took 19% longer with AI while feeling 20% faster. With late-2025 tools, a follow-up found 18% less time.

+19% → −18%task time with AI: early-2025 tools (2025) vs late-2025 tools (2026)

In early 2025 METR found that experienced developers took 19% longer when they were allowed to use AI tools (95% CI +2% to +39%). METR itself now calls that result out of date.

The study was a randomised controlled trial. 16 developers worked on 246 real issues in large open-source projects they knew well, and AI tools were randomly allowed or forbidden on each (mainly Cursor with Claude 3.5 and 3.7 Sonnet). Before the study the developers expected a 24% speed-up. Afterwards they still believed AI had made them 20% faster.

What newer data shows

  • METR’s follow-up (late-2025 tools, February 2026). With 57 developers and 800+ tasks, returning developers took 18% less time with AI (CI −38% to +9%) and new recruits 4% less (CI −15% to +9%). METR thinks this understates the effect, because some developers declined to take part rather than work without AI.
  • METR’s survey (February–April 2026). 349 technical workers reported a median 2× boost, up from 1.3× a year earlier. These are self-reports, the kind of estimate that was wrong in 2025.
  • Microsoft’s field study (January–April 2026). Engineers who adopted Claude Code or GitHub Copilot CLI merged about 24% more pull requests (finished code changes) than a modelled comparison group.
  • Inside a lab. METR put AI’s speed-up of Anthropic’s research at about 1.5× in September 2026 and about 1.9× in November. That measures how fast a lab’s models improve, not one developer’s speed.

We found no independent study yet of developer speed with GPT-6 Astra (September 2026).

What it means

The 2025 slowdown should not be quoted as today’s picture: with newer tools, the evidence leans towards modest speed-ups, though no single study proves one yet. The lesson that has lasted is the perception gap. How fast people feel with AI is not a reliable measure of how fast they are.

Caveats

  • Both 2026 follow-up intervals include zero, so the newer trial alone does not prove a speed-up. METR is redesigning the study.
  • The 2025 trial covered 16 highly experienced maintainers in mature codebases, not all software work.
  • Microsoft’s study is observational, and a merged pull request is not the same as useful output.
Suggest a correction