A tougher coding benchmark cuts every frontier model's score nearly in half

Cognition, the company behind the autonomous coding agent Devin, published a new version of its FrontierCode benchmark on October 27, and the headline number is a drop, not a gain. Claude Opus 5.5 leads the new leaderboard at 45.5%, down from 54.6% on the prior version. GPT-6 Astra falls from 53.3% to 42.8%. Every other model in the comparison, from Grok 4.7 to DeepSeek V4.1 Flash to Kimi K3, loses roughly nineteen to twenty-six points. Nothing about the underlying models got worse in the three months between versions; what changed is what counts as a pass.

Grading for mergeability, not for passing tests

Most coding benchmarks score whether generated code compiles and passes a test suite. FrontierCode asks a different, harder question: would the maintainers of the actual open-source project merge this change? Tasks are written by more than twenty working maintainers who each spent upward of forty hours per task, and grading applies a rubric of blocking criteria — fail any one of them and the whole task scores zero, regardless of whether the tests pass. Version 2.0 pulls from 55% more repositories than its predecessor, adds categories like security patches and performance tuning, and packs in 43% more blocking criteria. Reference solutions are also meatier: they touch 89% more lines of code across codebases that are a quarter larger than before. In other words, the test got harder on purpose, and the score drop is largely a function of that redesign rather than evidence models regressed.

The most telling addition is a new criteria cluster Cognition calls thoroughness — whether a model notices that a bug exists in more than one place, or that a sibling function needs the same fix it just applied elsewhere. Thoroughness now accounts for more blocking failures than any other category, with a 19.2% failure rate across all runs. Two worked examples in Cognition's writeup make the pattern concrete: Claude Opus 5.5, fixing a reported bug in the Mock Service Worker library, removed a stale reference to a generator but in doing so skipped a cleanup block that depended on it, so the fix broke silently in every effort setting. GPT-6 Astra, working on PostHog's Android SDK, fixed a race condition cleanly at low reasoning effort but, pushed harder, deleted a legacy public API class on its own initiative — a breaking change nobody asked for.

More reasoning effort, not necessarily better code

That second example points to a more unsettling finding buried in the data: turning up a model's reasoning effort does not reliably improve its score, and can hurt it. Cognition's effort-curve analysis shows scope-related failures climbing as effort rises — Claude Opus 5 went from a 69.2% scope pass rate at low effort down to 51.1% at maximum effort in the prior benchmark version, a pattern Cognition says comes from models writing more code and touching more files once they're allowed to think longer, even when the extra changes aren't wanted. Cognition's own explanation is that heavier reasoning sometimes convinces a model it has found a better interpretation of a vague request than the one actually asked for, a failure mode with no equivalent on a benchmark that only checks whether tests pass. For teams tuning agent settings for cost versus quality, this undercuts a common assumption — that bumping the reasoning budget is close to a free upgrade.

Why the number matters more than the ranking

FrontierCode 2.0 is a vendor-run benchmark, and that comes with the caveat every self-graded leaderboard carries: Cognition's own coding system, listed as SWE-2 in its tables, sits on the same leaderboard it designed and scored, at 23.9%, with no independent party yet reproducing the numbers. That's reason enough to treat the exact rank order between, say, Grok 4.7 and Gemini 4 Argon — both landing in the high-20s on this version — as noise rather than signal. But the more durable finding sits above the noise: even the best-resourced frontier models, graded against the standard a human maintainer actually applies, mergeable and complete rather than merely passing, clear well under half of real tasks. That gap between benchmark success and production readiness has been the subtext of this year's string of coding-benchmark disputes, from contamination fights over SWE-bench variants to arguments about what counts as a fair test at all; FrontierCode 2.0 makes the gap a number instead of an argument. For engineering teams deciding how much autonomy to hand a coding agent, the practical takeaway isn't which model wins by a few points — it's that none of them, yet, reliably infer the parts of a task nobody wrote down.