Ultrathink’s production agent fleet: after 30 days on Codex vs a frozen Claude Code baseline, Codex did not win on completion or QA
Ultrathink, an e-commerce company operated by AI agents, moved its autonomous coding fleet from Claude Code to Codex and compared two consecutive ~30-day operating periods with frozen windows and preserved denominators. The frozen Claude period completed 1,286 of 1,317 terminal tasks (97.65%) with a strict first-pass QA proxy of 134/206 (65.05%) and median wall time 422s. The Codex period completed 574 of 631 terminal tasks (90.97%) with QA proxy 70/144 (48.61%) and median 779s. The authors call the verdict mixed evidence, no Codex win, and warn workload mix and operating policy also changed, so it is not a universal ranking. Cost, token economics, human-intervention rate and availability were left blank as unavailable.
The numbers (as reported)
| Claude completed / terminal | 1,286 / 1,317 (97.65%); started 1,321 |
|---|---|
| Codex completed / terminal | 574 / 631 (90.97%); started 656 |
| Strict first-pass QA proxy | Claude 134/206 (65.05%); Codex 70/144 (48.61%) |
| Median wall time | Claude 422s; Codex 779s |
| p90 wall time | Claude 1,477s; Codex 3,126s |
| Windows | Claude ~2026-07-08 to 2026-08-07; Codex ~2026-08-07 to 2026-09-06 |
Facts
- Company / subject
- Ultrathink
- Tools
- Claude Code, OpenAI Codex
- Codebase
- Not applicable
- Work type
- Research / benchmark
- Stack
- —
- Published
- 2026-09-07
- Last checked
- 2026-11-05
What the source claims Practitioner (own blog/paper)
Claimed by the builder, writing about their own work. Not checked independently.
- Frozen Claude cohort completed a higher share of terminal tasks than the Codex period 97.65% vs 90.97%
“Codex did not win.”
- Strict first-pass QA proxy favoured the Claude period by 16.44 points 65.05% vs 48.61%
Caveats
Self-reported by Ultrathink about its own fleet. Consecutive periods, not a controlled A/B; social vs operations workload mix shifted sharply. Missing cost and intervention metrics prevent a price-adjusted verdict. Not a claim that Claude Code is better for every team.
More stories by tool: Claude CodeOpenAI Codex
Related topics: Research and benchmarks