✓ ShippedByAgents
Independent Research / benchmark

Study of 26,999 PRs: agent changes that touch both app code and deployment config wait 3.24x longer when reviewed, yet 43–56% merge with no human review

Researchers looked at 26,999 pull requests from the AIDev v5 and MOSAIC-3M datasets, covering six coding agents and a matched human baseline in repositories with 100+ stars. They focused on 'dual-scope' PRs that change both application code and deployment infrastructure (CI pipelines, containers, provisioning manifests). Only about 4.5% of agent PRs are dual-scope, but those PRs split two ways. When reviewed, they add a median of 13 hours and take 3.24 times as long to merge, mostly because of CI and provisioning changes. Otherwise, agent identity cuts the odds of human review by 63.7% compared with human contributors, so many merge silently.

The numbers (as reported)

PRs studied26,999 (AIDev v5: 21,855; MOSAIC-3M: 5,144)
Dual-scope share4.48% to 4.55% of agent PRs
Review delayMedian +13.13 hours, 3.24x time-to-merge for reviewed dual-scope PRs
Unreviewed merges43.1% to 56.2% of dual-scope PRs merge without inspection, median under 31 minutes
Review oddsAgent identity lowers odds of human review by 63.7% vs. human peers

Facts

Company / subject
Mohammad Wathiq Soualhi and Basem Suleiman (UNSW Sydney)
Tools
Claude Code, OpenAI Codex, GitHub Copilot, Cursor, Devin, Jules
Codebase
brownfield
Work type
Research / benchmark
Stack
—
Published
2026-10-09
Last checked
2026-10-26

What the source claims Independent

Reported by an independent source, such as a study or the press.

The source doesn't claim specific benefits.

Caveats

Preprint, not yet peer reviewed. One author is listed as an independent researcher. Open-source repositories only. Measures merge and review outcomes, not whether the merged changes broke anything. The six agents are taken from the paper's text; check the per-agent breakdown before publishing.

More stories by tool: Claude CodeOpenAI CodexCursorGitHub CopilotDevin

Related topics: Research and benchmarksBrownfield codebasesIndependent sources

Source

arXiv 2610.12849

One email a week: what actually shipped

New sourced stories, the numbers behind them, and what didn't work. No hype.

We'll only use this to send the newsletter. See the privacy page.