DeepSeek open-sources its production inference engine, minus the hardest parts

On October 21, DeepSeek quietly pushed a new repository to GitHub: DeepInfer, an Apache 2.0-licensed inference engine the company says powers the large-scale production serving of its DeepSeek-V4.1-Flash model. It arrived with a companion repo for tokenization and API compatibility, a nod to the open-source 3FS storage system behind DeepSeek's caching layer, and a set of aggressive performance and economics claims. It's the kind of release that would normally be filed under routine open-source housekeeping, except DeepSeek paired it with numbers suggesting its inference stack runs at a strikingly high margin — and left out the two components that would let anyone actually verify that at scale.

What's actually in the repository

DeepInfer's own README describes it as the framework that 'powers DeepSeek's large-scale production services,' and says it was originally built primarily on vLLM, with SGLang and TensorRT-LLM credited as additional influences. The code includes optimized kernels, attention/MoE parallelism, DeepSeek's DSpark speculative-decoding scheme, a technique called SWA Bounded Replay, and structured-output support, running on both NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. What isn't there, by DeepSeek's own account: KV-cache management, prefill/decode disaggregation, and load balancing across data-parallel replicas — in other words, the orchestration logic that turns a fast kernel into a service that can actually absorb real traffic. DeepSeek calls the project fully open source; the gap between that framing and the repository's actual contents is the crux of the story.

The numbers DeepSeek published alongside the code are all first-party and unaudited: roughly 155 trillion tokens processed per day from a single cluster, a 96% input-cache hit rate, about 400 milliseconds to first token, and roughly $38 in revenue per accelerator-hour at off-peak pricing. Outside analysis of those figures (not a DeepSeek disclosure itself) framed the implied payback period on the underlying hardware at around 81 days, with an inference margin estimated in the 80-96% range depending on which costs are counted. Those are the kind of numbers that, if they held up under scrutiny, would say something important about how far ahead Chinese labs have pushed inference economics under hardware constraints.

The open-source community's reality check

It didn't take long for an answer. On October 23, the vLLM project — the framework DeepInfer is built on top of — published its own benchmark post, 'Serving DeepSeek-V4.1-Flash at Scale,' using prefill/decode disaggregation and wide expert-parallelism on an NVIDIA GB300 NVL72 rack. vLLM reported decode throughput on matched GB200 hardware coming within about 6% of DeepInfer's published curves across batch sizes from 8 to 256, and said its own prefill curve on GB300 tracked DeepInfer's published numbers. It reported 954,000 tokens per second per GPU at a 196-tokens-per-second-per-user latency target, and a cost figure of $38.96 per GPU-hour — close enough to DeepSeek's own $38 figure to suggest the economics aren't a DeepSeek-only trick so much as a property of the hardware generation and software techniques now available across the open ecosystem.

vLLM's team was careful to caveat the comparison rather than claim an outright win: DeepInfer's published configuration used a different expert-parallel topology, a different count of redundant experts, and a longer bounded-replay window (SWA Bounded Replay, an attention/KV-cache parameter rather than a speculative-decoding one), so the two results aren't a clean apples-to-apples reproduction of DeepSeek's actual production service end to end. That caution is itself notable — it's a more rigorous read than the 'we matched the market leader' headline the comparison could have supported, and it underscores how much of DeepInfer's advantage is specifically in the parts DeepSeek didn't release.

Why the gaps matter more than the gains

The pattern here is becoming familiar in open-weight AI releases more broadly: a splashy 'fully open source' framing attached to a repository that withholds exactly the pieces that are hardest to build and most commercially sensitive. For a company running a cache-heavy service at DeepSeek's claimed scale, KV-cache management and cross-replica load balancing aren't peripheral — they're a large share of why the published latency and margin numbers look the way they do. Releasing the kernels and the parallelism code while keeping the caching and routing logic closed lets DeepSeek claim credit for openness and community goodwill (vLLM's own post thanked DeepSeek for contributions back to the upstream project) without handing competitors the operational playbook that actually makes the service cheap to run.

It's also a data point in the broader contest between Chinese AI labs and the Western-dominated open-source inference stack. DeepInfer explicitly builds on and credits vLLM and SGLang rather than competing with them from scratch, and its Huawei Ascend support is a visible signal of China's push toward an inference stack that doesn't depend entirely on NVIDIA hardware. Whether DeepSeek eventually releases the missing cache and load-balancing components — or whether those stay the durable edge it keeps to itself — is the detail worth watching next, since it will say a lot about whether 'open source' here means a genuine architecture contribution or a well-timed marketing layer over a tightly held production system.