New Benchmark Puts Frontier Models in Control of a Real Robot Arm — and Most Fail Basic Tasks

Artificial Analysis, the independent group behind the widely cited AI Intelligence Index, launched a new evaluation category on October 14: AA-Robotics v0.1, a research preview that drops general-purpose language models into direct control of physical hardware instead of a chat window. Each model runs inside a coding-agent harness, reads camera images, and writes Python that sends joint commands to a real six-axis arm over an HTTP API. There is no simulator standing in for the real world, and no worked examples to crib from.

Eight tabletop tasks, one real arm

The arm is a Hiwonder NexArm with five joints and a gripper, watched by three fixed tabletop cameras and one wrist-mounted camera. Models attempt eight tasks grouped into control and dexterity, spatial reasoning, and adaptability: pressing a button, placing a toy crocodile in a marked rectangle, pouring a drink into a cup, tidying blocks into a box, matching shapes, writing "hello world," stacking the tallest tower, and measuring a box's dimensions. A human judge scores the final state out of 100, with partial credit for incomplete attempts. A separate, newer set of eleven humanoid tasks tests balance and walking rather than tabletop manipulation.

GPT-6 Astra tops a field that mostly struggles

On the arm tasks, GPT-6 Astra led with 69.4 points, ahead of GPT-6.1 Sol (64.6), Gemini 4 Argon (55.3), Claude Opus 5.5 (48.8) and Claude Fable 5.1 (46.3). Models reliably aced the button-press and crocodile-placement tasks but mostly earned partial credit elsewhere; stacking a tower and measuring the box were the hardest, with grasp-success rates across all models ranging from just 28% to 45%. The humanoid suite was far more unforgiving: no model scored above 2 out of 100, and coverage of the run indicates Grok 4.7 and Gemini 4 Argon failed to move the robot at all in most episodes. Artificial Analysis itself frames the overall result plainly, writing that models remain unsuited to the job as things stand: "even if frontier language models are unsuitable for production robotics use on their own."

A 100x speed gap and an uneven cost picture

The bigger story may be economics rather than accuracy. Models averaged roughly 30 minutes per task — most of it spent interpreting the camera feed and planning a first move — against a human baseline of about 20 seconds, a gap Artificial Analysis describes as "nearly two orders of magnitude worse than human baselines," adding that "today's language models are roughly 100 times slower than a human on AA-Robotics v0.1." Cost per task ran from $1.93 for GPT-6.1 Sol to $16.36 for GPT-6 Astra, meaning Astra's four-point edge over Sol cost more than eight times as much to produce; roughly 70% of spending went to processing visual input rather than reasoning text. Artificial Analysis notes the cost comparison to humans is "narrower on cost" than on time, but still concludes flatly that today's models "are far too slow and expensive for any kind of production usage," while adding "we expect this to change quickly."

Why this cuts against the embodied-AI narrative

The result lands at an awkward moment for a sector that has spent 2026 raising enormous sums on the premise that frontier reasoning models are close to general-purpose physical labor. AA-Robotics is deliberately narrow — one arm, one tabletop setup, a handful of tasks, end-state grading that doesn't penalize unsafe or wasteful intermediate motion, and no published confidence intervals on the score gaps — so the numbers shouldn't be read as a verdict on robotics overall. What it does establish is a same-conditions comparison across vendors that didn't exist before: a model that tops text and math leaderboards is not thereby close to competent, affordable manipulation. Commercial humanoid and manipulation systems still lean on purpose-built low-level controllers, with a language model at most supplying a plan a faster system executes; the humanoid-balance results reinforce why that division of labor persists rather than giving way to an end-to-end language model in the control loop.

What to watch

Artificial Analysis has signaled AA-Robotics will expand past eight arm tasks and continue tracking the humanoid suite, and says it expects the current gaps to narrow quickly as vendors optimize for embodied use cases specifically rather than treating robot control as a side effect of general reasoning ability. The open questions for the next version are whether grasp-success rates and per-task cost improve as fast as text benchmarks have, and whether any vendor starts training or fine-tuning specifically against this kind of real-hardware loop rather than relying on a general chat model to generate robot code on the fly.