Red Hat-Backed Open-Source Inference Stack llm-d Hits 1.0, Claims Big Cost Cuts

llm-d, the open-source project for running large language models across Kubernetes clusters, shipped its first production-grade release on October 20, calling it the most important milestone in the effort's 17-month history. The timing matters: llm-d has quietly become one of the few pieces of AI infrastructure that NVIDIA, AMD, Intel and Google's TPU team are all building for at once, alongside cloud rivals Google Cloud, AWS, Oracle and Microsoft Azure.

The project began in May 2025 as a joint effort between Red Hat, IBM Research, Google Cloud, NVIDIA and CoreWeave, aimed at a problem that has grown more acute as agentic AI workloads multiply: serving frontier-scale models reliably, for many tenants at once, without the cost and fragility of treating each GPU cluster as a bespoke system. It became a Cloud Native Computing Foundation Sandbox project in March 2026, licensed under Apache 2.0, and now counts more than 500 contributors from over 25 organizations, including Hugging Face, Cisco, Lambda, Mistral AI and researchers from UC Berkeley and the University of Chicago.

What changed in 1.0

Earlier versions of llm-d proved the core architecture: splitting prompt processing (prefill) from token generation (decode) across separate hardware pools, and routing requests to whichever server already holds a conversation's cached context. Version 1.0 is explicitly aimed at what the maintainers call "Day 2" operations — the unglamorous work of running that architecture in production. It adds a stable, versioned configuration API; zero-downtime upgrades via a hot-standby routing layer and a new "DisaggregatedSet" abstraction that keeps paired prefill and decode replicas in lockstep; and multi-tenant traffic controls that classify requests by priority and shed load gracefully when a cluster saturates.

The release also leans harder into agentic workloads specifically, with routing that keeps a multi-turn conversation pinned to the replica holding its cache and native support for the chained-request pattern used by "Responses API"-style agent frameworks — a tacit acknowledgment that coding agents and tool-calling loops, not single chat turns, are now the workload driving inference infrastructure decisions.

The cost claims, and their inconsistencies

The most attention-grabbing numbers come from IBM Research, which reported serving up to 3,000 concurrent coding agents on 544 Nvidia H100 GPUs using a 744-billion-parameter open-weight model, with a prefix-cache hit rate near 98% and a blended cost of roughly $0.11 per million tokens — a figure IBM and the project describe as about 11.5 times cheaper than an equivalent proprietary API. Google Cloud separately reported cutting time-to-first-token by more than 35% in Vertex AI production traffic, while AWS said disaggregated serving on SageMaker HyperPod pushed throughput up to 70% above standard vLLM, and a Tesla-Red Hat deployment on AMD hardware reported triple the output-token rate for Llama 3.1 70B.

None of these figures have been independently audited, and they don't all describe the same thing. Reporting on IBM's own deployment has cited both an 11.5x cost reduction and, elsewhere, a narrower 5-to-10x range for the identical workload — a discrepancy neither IBM nor the project has reconciled publicly. The comparisons also rest on different baselines (standard vLLM, a generic "proprietary API," or a plain Kubernetes Service), making the across-the-board framing of llm-d as a uniform double-digit-multiple cost cutter harder to verify than the marketing suggests.

Why it matters anyway

What's harder to dispute is the breadth of buy-in. Competing GPU and chip vendors, competing cloud providers, and a foundation-model maker (Mistral AI, which is contributing code on open disaggregated-serving standards) are now building against the same open interface rather than each maintaining proprietary serving stacks. That mirrors how Kubernetes itself displaced a decade of bespoke container orchestration — and it suggests the open-source contest in AI infrastructure is no longer only about who publishes the best open-weight model, but about who controls the plumbing that makes any model, proprietary or open, cheap enough to run at agentic scale.