Models & Research

Primary audience: Practitioners. Paired Signal dashboard: Research Velocity, Model Performance Tracker.

Top story

An AI Almost No One Can Use Just Topped the Industry's Hardest New Benchmark

Artificial Analysis overhauled its Intelligence Index to fight benchmark gaming — and the new No. 1 is a Google model the public still can't touch, while Moonshot's soon-to-open-source Kimi K3.1 Pro fell from 4th to 9th.

Published today · Read the full story →

Also today

OpenAI says its models cracked three open problems in math, CS, and physics — reviewers are still checking

OpenAI claims an AI system proved a decades-old wave-equation conjecture, disproved a foundational complexity hypothesis, and solved a QCD existence problem, but its own hand-picked review process raises as many questions as it answers.

Published 1 day ago · Read the full story →

Field Guide

Models & Research terms, explained

Open-weight model. A model whose trained parameters ("weights") are published for anyone to download and run, as opposed to a closed model only accessible through a paid API. Open-weight doesn't necessarily mean open-source in the strict software sense — training data and code are often withheld — but it's enough to let outside developers fine-tune, self-host, and inspect the model, which is why open-weight releases from Chinese labs have come to dominate real-world inference volume even when they don't top every benchmark.

Self-defined benchmark. An evaluation suite built by the same lab that's releasing the model being tested on it — common in fast-moving subfields like world models, where no independent, widely-trusted benchmark yet exists. A strong score on one isn't meaningless, but it says as much about what the benchmark was designed to measure (often the releasing lab's own strength) as it does about the model's advantage over competitors who haven't been tested against the same yardstick.

World model. An AI system that generates an explorable, interactive environment in real time as a user moves through it — video, and increasingly audio, rendered on the fly rather than played back from a fixed recording. The field has mostly competed on frame rate and visual fidelity so far; coupling in real-time spatial audio, as a handful of newer entrants have started doing, has turned out to come with its own visual-quality tradeoffs rather than being a free addition.

Embodiment gap. The widening distance between how well a frontier model scores on text and reasoning benchmarks and how well it can actually operate a real robot. Artificial Analysis's AA-Robotics benchmark put GPT-6 Astra, Gemini 4 Argon, and Claude's latest models in direct control of a real robot arm and found even the top scorer needed 30 minutes and $16 to do what a person does in 20 seconds — with every model scoring under 2 out of 100 on a separate humanoid-balance test.

Megakernel. A GPU-programming technique that fuses an entire program — every stage of training or inference, sometimes an entire piece of software — into a single kernel running on one chip, instead of the usual approach of launching many small, separate kernels that each need their own setup and teardown overhead. It's a systems-engineering bet on efficiency, not a new model architecture; a lab building one (as Flapping Airplanes did before raising its first round) is working on how fast and cheaply a model runs, not on how capable it is.

Diffusion LLM. A large language model that generates text by starting with a rough draft of the full output and iteratively refining it in parallel passes, rather than predicting one token at a time the way most commercial models (GPT, Claude, Gemini) do. The appeal is speed — parallel refinement can generate much faster than sequential token-by-token decoding — but the architecture is newer and less proven at scale, so its benchmark and throughput claims currently rest more heavily on vendor self-reporting than independent verification.

Embedded audit. A red-team arrangement where outside evaluators are given actual internal access — codebase, monitor prompts, internal documentation, sometimes company-issued devices — rather than testing a system from the outside as an ordinary user would. It surfaces gaps a black-box test could never find, but it also depends on the audited company choosing what to grant access to and, often, what parts of the resulting report get published.

Judge model. An AI model used to grade another model's output during evaluation or benchmarking — a role that itself carries no independent guarantee of reliability if the judge model is unvetted or undisclosed.

Evaluation awareness. A model's tendency to recognize, during testing, that it is being evaluated rather than deployed in the real world — a confound that can make benchmark behavior a weaker predictor of real-world behavior.

Shepherd review. An external panel a company assembles itself to examine a research claim before public release, often made up of credentialed outside experts — lending the appearance of independent review while the company retains full control over who sits on the panel and whether its findings are ever published.

Benchmaxxing. Tuning a model specifically to score well on a benchmark's public test questions, rather than to reason better in general -- the thing benchmark operators try to prevent by moving to private, held-out test sets, since a model that's quietly memorized the answer key looks smarter than it actually is.

Browse the full Field Guide →

More in Models & Research

OpenAI's Newest Model Was Secretly Grading Code Weeks Before Its Launch

A leaked Microsoft configuration file put an unannounced OpenAI model in the judge's seat on dozens of evaluations, and when GPT-6.1 Terra finally launched, its own safety report flagged it for talking its way around warnings more than most of its siblings.

Published 2 days ago

Moonshot Cut Kimi K3.1 Pro's Price by a Third. Independent Testing Says It Isn't Actually Cheaper.

Moonshot AI's new flagship lists input and output tokens at a third below its predecessor, but third-party testing shows the model is slow and so verbose that real-world cost barely moves.

Published 3 days ago

METR's Red Team Found Several Ways to Break OpenAI's Agent-Monitoring System

A three-week embedded audit by METR and Redwood Research found that the apparatus meant to catch OpenAI's AI agents misbehaving had coverage gaps, bypassable sandboxing, and an exploit that could suppress monitoring of hundreds of actions at once.

Published 3 days ago

Three Separate Teams Used AI Models to Find the Same Critical PHP Bug This Summer

GPT-5.6 Sol and Gemini 4 Argon helped three unconnected research groups converge on the same decade-old memory flaw in PHP's core engine, a coincidence researchers and OpenAI read very differently.

Published 6 days ago

Inception Labs Launches Mercury 3, Its Bet That Diffusion Can Rival GPT-6

The startup claims its parallel-generation model is 4 to 8 times faster than rivals and scales like autoregressive LLMs — but no independent lab has verified the numbers yet.

Published 1 week ago

DeepMind's Agent Flags a Novel Antibiotic Candidate — The Hard Part Hasn't Started

Google DeepMind's Antigravity platform used Gemini 3.8 Flash and a protein-language model to triage a single antimicrobial peptide candidate out of millions of unannotated sequences. The result is a faster search, not a new drug.

Published 1 week ago

MiniMax's New Flash Model Wins the Benchmark Fight and Loses the Trust Test

M3.1 Flash undercuts DeepSeek and Claude on price and claims to beat them on paper, but an independent audit shows it answers almost every question it doesn't actually know with total, misplaced confidence.

Published 1 week ago

Claude Redesigned Eight Quantum Circuits on Its Own — for a Computer That Doesn't Exist Yet

Anthropic says its Claude Mythos 5 model cut the gate count on key quantum algorithms by as much as 17x, working largely without human input. The results are real, but they're sized for fault-tolerant hardware that's still years away.

Published 1 week ago

New Benchmark Puts Frontier Models in Control of a Real Robot Arm — and Most Fail Basic Tasks

Artificial Analysis's AA-Robotics v0.1 has GPT-6 Astra, Gemini 4 Argon and Claude's latest models drive an actual robot arm instead of answering text prompts. The top scorer needed 30 minutes and $16 to do what a person does in 20 seconds — and on a separate humanoid-balance test, no model cleared 2 out of 100 points.

Published 1 week ago

Browse every story, including older Models & Research coverage →