DeepInfer is DeepSeek's open-source inference framework, released under the Apache-2.0 license, designed for serving the DeepSeek-V4.1-Flash model at scale. This independent field guide provides a comprehensive overview of the framework, detailing its architecture, performance benchmarks, and the critical components that were included versus those that were not. It serves as a valuable resource for understanding DeepSeek's approach to deploying frontier open-weight models.
Key Features and Architecture:
DeepInfer is a production-derived inference engine incorporating highly optimized kernels and advanced parallelism techniques. It includes:
- Optimized Kernel Stack: Integrates FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, and custom TileLang kernels, all wired into a single efficient engine.
- Wide-EP and EPLB Parallelism: Utilizes data parallelism for attention and expert parallelism (EPLB) for Mixture-of-Experts (MoE) layers. In DeepSeek's production setup, decode operations can spread experts across 64–128 GPUs, with hot experts rebalanced using redundant copies.
- DSpark Speculative Decoding: Employs a confidence-scheduled speculation mechanism where draft tokens are proposed ahead, and a confidence head predicts which ones are worth verifying. This technique reportedly boosts throughput by 4–7% at DeepSeek's production operating point.
- SWA Bounded Replay Cache: Instead of rebuilding the entire sliding-window cache across every layer, the engine replays only the recent tail. This results in an approximate cache but with a bounded recompute cost, optimizing memory and computation.
- Asynchronous Scheduling: Designed for asynchronous operation by default, allowing the CPU to prepare the next batch while the GPU processes the current one. This eliminates idle gaps between decode steps, significantly improving throughput.
- Deterministic Modes: Offers batch-invariant inference and deterministic sampling on NVIDIA hardware, crucial features for auditable benchmarks and agent runs.
- Structured Outputs: Supports structured outputs via regex, JSON Schema, and EBNF.
- deepseek-recipe: A companion library for API protocol conversion and tokenization, released separately under an MIT license.
Hardware Support and Limitations:
DeepInfer is specifically designed for NVIDIA SM100 and Huawei Ascend 950 hardware paths, primarily for Linux environments. It is an offline inference engine and not a hosted API or a general-purpose engine for other models.
The Production Gap:
A crucial aspect highlighted by this guide is what DeepInfer does not include, which is essential for a complete online inference service:
- KV Cache System: The release lacks a persistent prefix/KV cache that can survive across requests and machines.
- Prefill/Decode Disaggregation: It does not include separate prefill and decode pools with mechanisms for transfer and routing.
- Load Balancing: There is no built-in load balancing solution that is aware of both request count and cache length across data-parallel replicas.
- User-Facing API: It is not an OpenAI-compatible server or a complete serving stack with features like authentication, quotas, billing, or observability.
Users intending to deploy DeepInfer in a production online environment would need to build these missing components themselves.
Benchmarks and Caveats:
The site presents official benchmarks, including Pareto charts for prefill (non-cached input throughput vs. latency on 4 SM100 GPUs) and decode (output throughput vs. per-user speed on 64 SM100 GPUs). These benchmarks use a 131,072-token context length, not the model's 1M-token maximum. The guide emphasizes that these numbers are specific to DeepSeek's internal testing environment and assumptions (e.g., 75% draft acceptance rate for decode) and s...
