✓ ShippedByAgents
Practitioner (own blog/paper) Research / benchmark

Ultrathink’s production agent fleet: after 30 days on Codex vs a frozen Claude Code baseline, Codex did not win on completion or QA

Ultrathink, an e-commerce company operated by AI agents, moved its autonomous coding fleet from Claude Code to Codex and compared two consecutive ~30-day operating periods with frozen windows and preserved denominators. The frozen Claude period completed 1,286 of 1,317 terminal tasks (97.65%) with a strict first-pass QA proxy of 134/206 (65.05%) and median wall time 422s. The Codex period completed 574 of 631 terminal tasks (90.97%) with QA proxy 70/144 (48.61%) and median 779s. The authors call the verdict mixed evidence, no Codex win, and warn workload mix and operating policy also changed, so it is not a universal ranking. Cost, token economics, human-intervention rate and availability were left blank as unavailable.

The numbers (as reported)

Claude completed / terminal1,286 / 1,317 (97.65%); started 1,321
Codex completed / terminal574 / 631 (90.97%); started 656
Strict first-pass QA proxyClaude 134/206 (65.05%); Codex 70/144 (48.61%)
Median wall timeClaude 422s; Codex 779s
p90 wall timeClaude 1,477s; Codex 3,126s
WindowsClaude ~2026-07-08 to 2026-08-07; Codex ~2026-08-07 to 2026-09-06

Facts

Company / subject
Ultrathink
Tools
Claude Code, OpenAI Codex
Codebase
Not applicable
Work type
Research / benchmark
Stack
—
Published
2026-09-07
Last checked
2026-11-05

What the source claims Practitioner (own blog/paper)

Claimed by the builder, writing about their own work. Not checked independently.

  • Frozen Claude cohort completed a higher share of terminal tasks than the Codex period 97.65% vs 90.97%
    “Codex did not win.”
  • Strict first-pass QA proxy favoured the Claude period by 16.44 points 65.05% vs 48.61%

Caveats

Self-reported by Ultrathink about its own fleet. Consecutive periods, not a controlled A/B; social vs operations workload mix shifted sharply. Missing cost and intervention metrics prevent a price-adjusted verdict. Not a claim that Claude Code is better for every team.

More stories by tool: Claude CodeOpenAI Codex

Related topics: Research and benchmarks

Source

Ultrathink Field Notes

One email a week: what actually shipped

New sourced stories, the numbers behind them, and what didn't work. No hype.

We'll only use this to send the newsletter. See the privacy page.