✓ ShippedByAgents
Independent Research / benchmark

CodeMeepo multi-agent migrations with Copilot CLI pass 18% more tests than ReCodeAgent across nine repos, including a full SONiC xcvrd Python-to-Rust port

Updated on Set published to 2026-10-25 from the arXiv submission date on abs/2610.34633.

CodeMeepo is a two-phase multi-agent repository migration workflow (translation then benchmark-guided optimization) orchestrated by a deterministic state machine over eight Copilot CLI custom-agent roles. Agents neither author nor edit a fixed external acceptance oracle; a parity agent turns coverage gaps into new milestones. On nine subjects in three language pairs (AlphaTrans Java→Python, CRUST-Bench C→Rust, SONiC xcvrd Python→Rust), all arms used claude-sonnet-5: CodeMeepo had the best pass rate on every subject and passed 1,334 vs ReCodeAgent’s 1,129 tests (+18%). On production xcvrd (15,554 LOC) it passed all 105 oracle tests vs ReCodeAgent’s 53; five optimization rounds then improved five of ten benchmarks by >5%, averaging 25% on those.

The numbers (as reported)

Subjects9 repos, 3 language pairs
Tests passed vs ReCodeAgent1,334 vs 1,129 (+18%)
xcvrd (production daemon)105/105 oracle tests vs ReCodeAgent 53/105
Optimization5 of 10 xcvrd benchmarks improved >5%, avg 25% on those
Model (all arms)claude-sonnet-5 via Copilot CLI

Facts

Company / subject
Microsoft / UC Irvine / OpenAI researchers (while at Microsoft)
Tools
GitHub Copilot
Models
claude-sonnet-5
Codebase
brownfield
Work type
Research / benchmark
Stack
Java, Python, C, Rust, SONiC, Apache Burr, GitHub Copilot CLI
Published
2026-10-25
Last checked
2026-11-16
Updated
2026-11-16

What the source claims Independent

Reported by an independent source, such as a study or the press.

  • Best pass rate on every subject; 18% more tests passed than ReCodeAgent on the same model +18% tests (1,334 vs 1,129)
    “passes 18% more tests than a state-of-the-art multi-agent baseline using the same model”
  • Full 105/105 pass on production SONiC xcvrd Python→Rust vs 53 for ReCodeAgent 105/105 vs 53/105
  • Optimization phase improved half of xcvrd benchmarks by 25% on average 5/10 benchmarks, avg +25%
    “its optimization phase then improves half of the benchmarks, by 25% on average”

Caveats

Research paper; authors affiliated with UC Irvine, Microsoft, and OpenAI, with contributions while at Microsoft. Benchmarks are controlled migrations with fixed oracles, not live product ship stories.

More stories by tool: GitHub Copilot

Related topics: Research and benchmarksBrownfield codebasesIndependent sources

Source

arXiv 2610.34633 (Habibi, Wu, Jiang, Lu)

One email a week: what actually shipped

New sourced stories, the numbers behind them, and what didn't work. No hype.

We'll only use this to send the newsletter. See the privacy page.