DeepSeek open-sources production inference engine DeepInfer, with Huawei Ascend support from day one

Apache 2.0 release lets developers run V4.1 Flash with DeepSeek's own serving optimizations, pairing Nvidia Blackwell with Huawei's Ascend 950

By LineZotpaper

Published20 October 2026

Read Time3 min

DeepSeek has open-sourced DeepInfer, the inference engine that serves its models in production, in a move the Chinese AI lab said last year was not yet practical. The Apache 2.0 framework, announced Tuesday, is designed to let developers run DeepSeek's V4.1 Flash model on their own infrastructure with many of the same optimizations the company uses behind its API. It also gives day-one hardware support to Huawei's Ascend 950 NPUs alongside Nvidia's Blackwell data center GPUs, the latest sign of DeepSeek's deepening investment in Chinese chips amid US export controls.

DeepSeek has long published its model weights for anyone to download and run using third-party software such as vLLM. But the software that manages requests, distributes work across chips and generates responses has a significant impact on speed, throughput and computing costs. DeepInfer gives developers access to DeepSeek's own approach to those problems. "We're sharing our production stack to help the community lower compute costs, reduce energy use, and make AI more affordable," the company said on X.

The framework includes optimized computing kernels and communication libraries, including the previously open-sourced FlashMLA, DeepGEMM and DeepEP, alongside mechanisms for distributing inference across multiple chips. It supports speculative decoding, a technique that predicts and verifies multiple tokens at once to speed up response generation, plus asynchronous scheduling and structured outputs. Alongside DeepInfer, DeepSeek is open-sourcing deepseek-recipe, a companion Rust and Python library designed to help developers reproduce the behavior of DeepSeek's official API by handling the conversion of API requests into model inputs and outputs. The release follows an updated version of V4.1 Flash and the company's new Search and Fetch APIs for its in-house web search technology this week.

The move reverses a position DeepSeek took in April 2025, when it said its inference engine was too tightly coupled to internal infrastructure to share. "The engine is tightly coupled with our internal infrastructure, including cluster management tools, making it impractical for public deployment without significant modifications," the company wrote at the time. It has not detailed how it overcame those difficulties, although the scope of the release offers clues. DeepInfer currently supports offline inference with, in the company's words, "the core features needed for serving at scale," but lacks a KV cache management system, infrastructure for separating inference stages across machines, and load balancing across replicas. Developers must supply much of the surrounding infrastructure for a full production service.

The hardware line-up is the release's most striking detail. Nvidia support is restricted to the SM100 architecture used in Blackwell data center GPUs, with older architectures such as Hopper absent, while Huawei's Ascend 950 NPU series is supported from day one. DeepSeek committed in 2025 to working with hardware makers so its models run across different chip architectures from launch, and it has since been reported to be planning large-scale Ascend deployments, including a new data center in Inner Mongolia, greater use of Huawei chips for training, and collaboration on Ascend programming tools. Feature support differs between the two platforms: batch-invariant inference and deterministic sampling are available only on Nvidia hardware.

The Huawei inclusion drew attention in developer communities. On Reddit, one commenter singled out the two supported platforms as the highlight of the news, while another pointed to DeepSeek's origins in quantitative finance as an explanation for its optimization expertise. A third joked about reported safeguards around Anthropic's Claude models and Huawei kernel programming tasks.

Analysis

Why This Matters

DeepInfer moves DeepSeek's production software itself into the open, not just model weights, giving developers a chance to run V4.1 Flash with the same optimizations used behind the company's API.
The day-one pairing of Huawei Ascend with Nvidia highlights how US export controls are reshaping the AI hardware market and pushing Chinese labs toward domestic chips.
The release also tests whether an open-source inference stack can meaningfully cut the compute and energy costs of running large models.

Background

DeepSeek, a Chinese AI lab, has built a reputation for publishing model weights openly while running a commercial API on top of them. In April 2025 it explained that its inference software had evolved from an early version of vLLM into a highly customized system, and that its small research team and tight coupling with internal cluster management tools made a full release impractical at the time. Instead, it contributed individual components such as FlashMLA, DeepGEMM and DeepEP to existing projects, and committed to working with hardware makers ahead of future model launches. US export controls have limited Chinese access to Nvidia's most advanced chips, and DeepSeek has been reported to be investing heavily in Huawei's Ascend processors for both inference and, increasingly, training.

Key Perspectives

DeepSeek: The company says it is sharing its production stack to help the community lower compute costs, reduce energy use and make AI more affordable. It positions DeepInfer as the framework powering V4.1 Flash at scale with high throughput and low latency. Developers and operators: The Apache 2.0 release offers a production-grade inference framework, but the current version supports offline inference and omits several components needed for a complete service, including KV cache management, cross-machine stage separation and load balancing. Users on Nvidia hardware also get features, such as deterministic sampling, that are not yet available on Ascend. Industry observers: The dual Nvidia and Huawei support is read as the clearest sign yet of DeepSeek's bet on Chinese chips. Some in developer communities credit DeepSeek's quantitative finance background for its performance work; others note the constraints US export policy has created for Chinese AI infrastructure.

What to Watch

Whether future DeepInfer releases add the missing production components, including KV cache management and load balancing across replicas.
Reports of DeepSeek-Huawei cooperation on Ascend training deliveries and the planned Inner Mongolia data center.
Independent benchmarks from developers running V4.1 Flash on Ascend versus Nvidia hardware, and how DeepSeek responds to any feature gaps.

Sources

DeepSeek open-sources the inference engine behind its API. And Huawei gets equal billing with Nvidia. · The New Stack

Checked

12 claims in this story checked against 1 source: 9 supported.

Show each claim

Partly supportedDeepInfer is DeepSeek’s production inference engine, and the release makes that engine available to developers.The New Stack: “The DeepInfer documentation makes clear that the publicly available software currently supports offline inference, with “the core features needed for serving at scale,” as the company puts it. However, several components needed to operate a complete production service are still missing.”
SupportedDeepInfer supports Huawei Ascend 950 alongside Nvidia Blackwell hardware from day one.The New Stack: “Nvidia is included, as expected, though support is restricted to its SM100 architecture, used in Blackwell data center GPUs. Notably, the other supported platform is Huawei’s Ascend 950 NPU series”
SupportedDeepInfer is released under Apache 2.0 and is designed to let developers run DeepSeek’s V4.1 Flash on their own infrastructure with many of the company’s API optimizations.The New Stack: “Released under the Apache 2.0 license, DeepInfer includes optimized computing kernels and communication libraries”
SupportedDeepSeek said on X, “We're sharing our production stack to help the community lower compute costs, reduce energy use, and make AI more affordable.”The New Stack: ““We’re sharing our production stack to help the community lower compute costs, reduce energy use, and make AI more affordable,” the company writes.”
SupportedDeepSeek said in April 2025 that its engine was too tightly coupled to internal infrastructure for public deployment without significant modifications.The New Stack: ““The engine is tightly coupled with our internal infrastructure, including cluster management tools, making it impractical for public deployment without significant modifications,” the company wrote at the time.”
SupportedThe DeepInfer announcement was made Tuesday.The New Stack: “Taking to X on Tuesday, DeepSeek said it was making its internal inference technology available to outside developers”
Partly supportedDeepSeek is open-sourcing deepseek-recipe alongside DeepInfer as a companion Rust and Python library.The New Stack: “The library was quietly released in September, with an updated version published on Tuesday.”
SupportedDeepInfer supports speculative decoding, asynchronous scheduling and structured outputs.The New Stack: “It also supports speculative decoding, a technique that speeds up response generation by predicting and verifying multiple tokens at once, and asynchronous scheduling”
SupportedThe public version supports offline inference but lacks a KV cache management system, infrastructure for separating inference stages across machines, and load balancing across replicas.The New Stack: “These include a KV cache management system, infrastructure for separating different stages of inference across machines, and load balancing across replicas.”
SupportedNvidia support is limited to SM100, used in Blackwell data center GPUs; older architectures such as Hopper are absent.The New Stack: “support is restricted to its SM100 architecture, used in Blackwell data center GPUs. [...] Older Nvidia architectures, including Hopper, are conspicuously absent.”
SupportedBatch-invariant inference and deterministic sampling are available only on Nvidia hardware.The New Stack: “DeepInfer’s batch-invariant inference and deterministic sampling features, which help ensure reproducible outputs under specified conditions, are available only on Nvidia hardware.”
Partly supportedA Reddit commenter cited DeepSeek’s quantitative-finance origins as an explanation for its optimization expertise.The New Stack: “another pointed to DeepSeek’s origins in quantitative finance as a possible explanation for its technical expertise.”