Research and benchmarks: 6 sourced coding agent stories
Studies and benchmarks that measure what agents do. Of the 6 stories, 1 was published by a tool vendor and 5 by independent researchers or press. 3 involve an existing codebase and 0 started from scratch. Every number links to where it was published.
- Independent Research / benchmark
Harvard study of 718 firms: AI agents raise code volume but not shipped work, review becomes the bottleneck
718 firms (Jellyfish data) · 2026-08-04
- 718 Firms
- ~300 million Work events
- +30% Lines of code after agent adoption
Various tools
- Independent Research / benchmark
NBER study of 500,000+ GitHub developers: coding agents multiply commits and code, but releases rise only ~20%
NBER working paper (Demirer, Musolff, Yang) · 2026-05
- 500,000+ Developers studied
- +30% autocomplete, ~+180% with sync agents, ~+240% with async agents (cumulative) Weekly commits
- +958% Sync agents: lines of code
GitHub Copilot, Claude Code, OpenAI Codex
- Independent Research / benchmark brownfield
MSR 2026 study: coding agents give open-source repos a velocity bump only as their first AI tool, but lasting technical debt either way
Shyam Agarwal, Hao He, Bogdan Vasilescu (MSR 2026) · 2026-04-13
- +36.3% Commits (agents as first AI tool)
- +76.6% Lines added (agents as first AI tool)
- +3.1% / −6.3% Commits / lines (prior AI IDE users)
Various tools
- Vendor-published Research / benchmark brownfield
Legacy-Bench: frontier agents pass only 16.9–42.5% of COBOL, Java 7 and Fortran tasks
Factory · 2026-04-01
- 16.9%–42.5% Pass rate range (12 model–agent combos)
- >70% Same models on Terminal-Bench 2 / SWE-bench Verified
- 42.5% Best overall (GPT-5.3-Codex)
Factory Droid, OpenAI Codex, Gemini CLI
- Independent Research / benchmark
METR's late-2025 follow-up: raw data hints at speed-up, but developers refusing to work without AI breaks the study
METR (57 open-source developers) · 2026-02-24
- 57 Developers
- 143 Repositories
- 800+ Tasks
Various tools
- Independent Research / benchmark brownfield
METR RCT: experienced open-source developers took 19% longer with early-2025 AI tools
METR (16 open-source developers) · 2025-07-10
- 16 Developers
- 246 Tasks
- 19% longer with AI Measured effect
Cursor