An AI Almost No One Can Use Just Topped the Industry's Hardest New Benchmark
On October 27, Artificial Analysis shipped what it called its biggest overhaul of the Intelligence Index since v4 launched in January: new private test sets, a rebuilt coding benchmark, and a methodology explicitly designed to punish models that look smart mainly because they've memorized the test. One day later, the new No. 1 model on that harder, more contamination-resistant index was one almost nobody can actually run.
That model is Google's Gemini 4 Carbon, which scored 59 on Intelligence Index v5 and topped four of its nine component evaluations — HLE-Diamond, Terminal-Bench-Science, AA-Omniscience, and AutomationBench-AA. Artificial Analysis says plainly that Carbon "is being rolled out to selected users and is not yet publicly available." Google ran it through the benchmark; the rest of the world is still working from leaked screenshots. Reports from testers in Google's early-access program had already been circulating for roughly a week before the index update, describing a model noticeably better at 3D generation than its predecessor, Gemini 4 Argon, but also two to four times slower and still rough on frontend UI work — hands-on impressions that outpaced anything Google itself has confirmed.
The other half of the story
The flip side of v5's debut is Kimi K3.1 Pro. Moonshot AI's model had placed 4th on the old v4.3 index at launch, trailing the top-ranked model by just 3 points. Under v5's harder private tests, that gap widened to 12 points, and Gizmodo reported the model's effective ranking sliding from 4th to 9th. It remains the strongest model among those with released weights — at 46 points, ahead of DeepSeek V4.1 Flash and GLM-5.3 — but Moonshot hasn't actually published those weights yet; it says they're coming November 4.
The mechanism behind both stories is the same change: roughly 60% of Intelligence Index v5 now runs on private, held-out question sets, including a new coding benchmark from Cognition, FrontierCode 2.0, that Artificial Analysis says will stay permanently unpublished specifically to prevent labs from training on it. That's a defensible response to "benchmaxxing" — tuning a model to ace public test questions rather than to actually reason well. But it lands asymmetrically. A model whose weights are about to go public gets scored against tests designed to be contamination-proof forever; a model that will likely never ship its weights at all gets scored under a vendor-arranged early access arrangement, evaluated before the public gets a chance to independently verify anything Google said.
A win that comes with caveats
Carbon's numbers aren't uniformly dominant, either. On the Coding Agent Index, Claude Opus 5.5 still beats it, 62 to 59. On AA-Briefcase, a knowledge-work benchmark, Kimi K3.1 Pro actually outscores Carbon, 1,678 Elo to 1,609. And Carbon's hallucination rate nearly tripled from Argon's 4% to 11%, even as its accuracy sits at a middling 58%. Independent testers' complaints about Carbon's speed — two to four times slower than Argon on some tasks — aren't captured in the top-line score at all. A composite index number makes for a clean headline; it compresses away exactly the tradeoffs that matter to anyone actually choosing a model to build on.
There's a broader industry pattern underneath both halves of this story: output token usage per task jumped from a median of 35,000 in v4.3 to 59,000 in v5, and score spreads between a model's low-effort and max-effort settings roughly doubled for several frontier systems. Models are getting "smarter" on these indices partly by thinking — and billing — a lot more: Claude Opus 5.5's cost per index task now runs $10.04, while Moonshot continues to price Kimi K3.1 Pro as a budget option by comparison.
Artificial Analysis is a three-year-old private benchmarking firm, not a standards body, and its rankings move markets and headlines in ways that outstrip that status. A week where the chart-topper is a model the public cannot download, test, or buy at listed prices — while a transparent, soon-to-be-open competitor gets bumped down by design changes aimed at fairness — is worth reading as a snapshot of benchmark politics as much as of model capability.