A new study conducted by Model Evaluation & Threat Research (METR) reveals that experienced open-source developers were slower when using generative AI tools, which was contrary to their initial expectations.
The study observed a 19% increase in task completion time when developers used AI systems, such as Claude 3.5/3.7 Sonnet and Cursor Pro.
The randomised controlled trial involved 16 veteran open-source contributors working on 246 real-world tasks across mature repositories they had contributed to for an average of five years.
Developers forecasted that AI assistance would make them 24% faster, and later estimated a perceived speed-up of 20%. However, actual results showed the opposite — developers completed tasks slower with AI enabled.
This suggests a large disconnect between perceived and actual AI productivity benefits, the researchers noted in the paper.
Experts from economics and machine learning also misjudged AI’s impact, forecasting a 39% and 38% speed-up, respectively. The METR team analysed 143 hours of screen recordings to understand the slowdown, identifying factors such as the low reliability of AI outputs, difficulty in adapting to complex codebases, and the AI’s failure to leverage repository context.
Developers spent more time reviewing AI suggestions and waiting for generations rather than coding directly. While 93% had prior experience with LLMs, only 44% had used Cursor Pro before the study.
Despite this, the slowdown was consistent even among Cursor veterans.
METR cautioned against overgeneralising the results, noting that AI may still be useful in other contexts, especially for novice developers or greenfield projects. But the findings challenge the assumption that state-of-the-art AI tools inherently boost productivity for experienced engineers.
The report emphasises the importance of carefully evaluating AI tools in real-world settings and cautions against relying solely on benchmark performance or anecdotal success stories.
Read more: AI Wrote the Code—Now Developers are Stuck Fixing it
