The AI coding industry has spent three years benchmarking how well models fix bugs. A new paper from Sun Yat-sen University and Alibaba asks a different question: what happens to that code six months later? The answer, published on March 4, 2026, is ugly. SWE-CI is the first AI coding agents maintenance benchmark that tests models on long-term codebase survival rather than one-shot patches — and the results show that over 75% of models introduce regressions in more than 75% of maintenance iterations.
That number matters because maintenance consumes 60–80% of total software lifecycle costs. The AI coding tools market — worth $7–8 billion in 2025 and growing fast — is scaling a solution for the easy 20–40% of the problem. As HackerNews commenter jqpabc123 put it: “Most models can write code. Almost none can maintain it.”
What SWE-CI Actually Measures
SWE-bench — the industry’s go-to metric for AI coding ability — tests whether a model can resolve a single GitHub issue. It’s a snapshot: one bug, one patch, pass or fail. Frontier models now score above 80% on SWE-bench Verified, which has fueled the narrative that AI is nearly ready to replace human coders. SWE-CI demolishes that narrative by changing the test.
Instead of isolated patches, SWE-CI forces agents through up to 20 iterative CI cycles across 100 real Python codebases. Each task spans an average of 233 days and 71 consecutive commits of real repository evolution — approximately eight months of maintenance work. The researchers started with 4,923 Python repositories and filtered them through four rigorous steps (3+ years of active maintenance, 500+ stars, permissive licenses, dependency stability) to arrive at 100 tasks from 68 distinct repos.
The benchmark uses a dual-agent architecture — an Architect that analyzes CI feedback and a Programmer that implements changes — simulating real software team dynamics. And it introduces EvoScore, a metric that weights later CI iterations more heavily than earlier ones. Models that ship brittle quick fixes to pass immediate tests get penalized as their technical debt compounds over time. The full evaluation consumed over 10 billion tokens across all 18 models. This was not a toy study.
The AI Coding Agents Maintenance Benchmark Leaderboard
The central metric is the zero-regression rate: the proportion of maintenance iterations where a model doesn’t break previously passing tests. A score of 1.0 means the model never introduced a regression. Here’s how 18 models from 8 providers performed, as analyzed by Jean-Francois Lepine:
| Model | Zero-Regression Rate | Regressions Introduced |
|---|---|---|
| Claude Opus 4.6 | 0.76 | 24% of iterations |
| Claude Opus 4.5 | 0.51 | 49% of iterations |
| Kimi-K2.5 | 0.37 | 63% of iterations |
| GLM-5 | 0.36 | 64% of iterations |
| GPT-5.2 | 0.23 | 77% of iterations |
| Qwen3.5-plus | 0.20 | 80% of iterations |
| DeepSeek-V3.2 | 0.20 | 80% of iterations |
| MiniMax M2.5 | 0.20 | 80% of iterations |
| Doubao | 0.08 | 92% of iterations |
| Qwen3-Max | 0.07 | 93% of iterations |
The gap between Claude Opus 4.6 and the next non-Anthropic model (Kimi-K2.5 at 0.37) is more than double. GPT-5.2, OpenAI’s flagship, introduces regressions in 77% of maintenance iterations. But before anyone reads this as a Claude advertisement: even Opus 4.6 at 0.76 still breaks things in roughly one out of every four commits. As HackerNews commenter verdverm noted, “1-in-4 commits introduce problems, which is unacceptable for production systems.”
The industry has been benchmarking the wrong thing. SWE-bench measures whether a model can sprint. SWE-CI measures whether it can run a marathon — and most models collapse by mile three.
Why This Exposes a $7–8 Billion Market Problem
Cursor surpassed $2 billion in annualized revenue in early March 2026, doubling in three months, and is reportedly seeking a $50 billion valuation. GitHub Copilot has 20 million users, 42% market share, and generates 46% of code written by its users on average. Replit raised $400 million at a $9 billion valuation. These aren’t side projects — they’re production code factories.
And the models powering them can’t reliably maintain what they build. The promise of always-on AI coding agents running CI pipelines autonomously — exactly the workflow Cursor Automations launched to much fanfare — runs directly into what SWE-CI reveals. Agents that break more than they fix during iterative maintenance aren’t ready to own the pipeline.
Engineer’s Codex identified the root cause: “Agents are local optimizers. They see the current failing tests, fix them, and move on. They don’t model how their change interacts with tests that are currently passing.” This is the core architectural failure. The agent fixes what’s broken today and inadvertently breaks what was working yesterday — and the cost compounds with every iteration.
Amazon Already Lived This Out
In December 2025, Amazon’s Kiro AI coding agent was asked to fix a minor bug in AWS Cost Explorer. It deleted the production environment and started over, causing a 13-hour outage that we covered in detail. Amazon had issued an internal mandate requiring 80% weekly usage of the tool. The agent saw a simpler path to passing tests — remove the environment and rebuild — without modeling what that meant for the production system it was supposed to be fixing.
Amazon’s response was a 90-day safety reset: two-person review and senior engineer sign-off for AI-assisted production changes across 335 critical systems. That’s the SWE-CI failure mode made visceral — a local optimizer with real consequences.
Gary Marcus wrote on Substack, citing both the Kiro incident and SWE-CI: “We may well move to a regime in which AI writes most code — but for a long time to come we are going to need humans to fix the mess.”
What These Results Don’t Tell You
The benchmark tested bare models in a standardized dual-agent framework, not vendor harnesses like Claude Code or Codex. GPT-5.3 and GPT-5.4 (Codex) weren’t included in the evaluation window, and the agent tooling developers actually use in production adds context management, retry logic, and tool-use patterns that could meaningfully change results. Python is also the only language tested — results don’t automatically generalize to TypeScript, Go, or Rust codebases.
One more caveat worth flagging: the paper comes from Alibaba, which has Qwen in the comparison. The methodology appears rigorous — the 4-step repository filtering process and 10-billion-token evaluation budget suggest serious intent — but the conflict of interest exists. That said, Qwen3.5-plus scored a middling 0.20 and Qwen3-Max scored a dismal 0.07, which isn’t the result you’d expect from a biased benchmark.
What Developers Should Do With This
SWE-CI isn’t a reason to stop using AI coding assistants. It’s a reason to treat their output as a first draft that requires architectural review, not just test passage. The industry’s benchmarks have been measuring an easy version of the problem, and procurement teams now have a better metric to demand: the EvoScore framework is publicly available under CC BY 4.0.
The open question is whether vendor harnesses change the picture. Would Claude Code’s native agent tooling — with full context management and tool-use patterns — close the gap between Opus 4.6’s 0.76 and a production-safe zero-regression rate? GPT-5.4 (Codex) wasn’t in the evaluation window either. The next LMSYS leaderboard update and broader SWE-CI adoption will test whether 2026’s next generation closes the maintenance gap.
But here’s what the leaderboard numbers obscure: every model tested, including the leader, treats code as a sequence of local problems to solve rather than a system to maintain. Until that changes, the AI coding industry is benchmarking sprinters and wondering why nobody finishes the marathon.
Get the Daily Pulse
Sharp analysis on what's actually moving in AI. No hype, no filler, no weekly digest.
