DeepSeek Introduces DualPath Architecture to Optimize AI Agent Inference Throughput

DeepSeek has released a new inference framework, DualPath, designed to enhance the performance of AI agents by addressing GPU idling caused by I/O bottlenecks. Developed in collaboration with researchers from Peking University and Tsinghua University, the framework aims to improve the efficiency of large language model (LLM) inference in multi-turn agentic scenarios.

Visualizing the 'dual-path KV-Cache loading' mechanism with a GPU and storage unit.

The core of DualPath involves a "dual-path KV-Cache loading" mechanism, which introduces a second loading path, "Storage-to-Decode." This design seeks to alleviate the I/O bottleneck that arises from reading large-scale Key-Value (KV) Cache data from external storage, thereby preventing computing resources from being underutilized due to slow data loading speeds.

DeepSeek's research indicates that in multi-turn agentic inference, GPUs often experience significant downtime while waiting for data. The company reported that DualPath can boost offline inference throughput by up to 1.87 times and increase the number of agent runs per second in online scenarios by 1.96 times.

Addressing GPU Underutilization in Agent Tasks

The development of DualPath stems from observations of severe GPU underutilization during agent inference tasks. Agent operations, which can span minutes or hours, involve complex processes such as code generation, documentation review, and testing. These tasks require rapid context access, often involving millions of tokens, leading to substantial KV Cache demands.

Abstract depiction of KV Cache as a central memory hub for AI agent tasks.

KV Cache functions as a temporary storage for the model's "thought traces," which are accessed each time a new token is generated. While storing this data in high-bandwidth memory (HBM) is crucial for context retention, the limited supply of HBM has created a bottleneck in the AI industry. This has led to strategies where less immediately needed memories are offloaded to cheaper storage like SSDs or main memory and retrieved when required.

Traditional inference architectures process data serially, meaning the compute unit must wait for data to be fully loaded into HBM before processing can begin. DeepSeek's analysis revealed that this serial process causes GPUs to idle while data is transferred via bandwidth-limited interfaces such as PCIe.

DeepSeek collected data from agent coding tasks and introduced a "cache-to-compute ratio" metric. Trajectories from representative coding tasks showed an average of 157 interaction turns, an average context length of 32.7k, and an average append length of 429. This indicates a KV cache hit rate of 98.7% and a cache-to-compute ratio of approximately 22GB/PFLOP for DeepSeek-V3.2.

The company noted that the growth of network bandwidth and HBM capacity has not kept pace with the increase in GPU FLOPS, leading to a 14.4-fold decrease in the I/O-to-compute ratio. This imbalance, coupled with limited HBM capacity, restricts the batch size of tokens that GPUs can process simultaneously and affects the utilization of compute units.

Visual metaphor for the growing imbalance between GPU FLOPS and I/O-to-compute ratio.

DualPath Architecture and Implementation

DualPath transforms the serial process of "thinking" and "recalling" into parallel operations by decoupling compute and memory access. Unlike traditional architectures that wait for complete data loading before computation, DualPath allows for simultaneous data downloading and processing.

The architecture employs two independent pipelines: an Access Path responsible for moving KV Cache data blocks from SSD/DRAM, and a Compute Path that immediately begins processing already moved data blocks. DeepSeek defines Prefill GPUs as Prefill Engines (PEs) and Decode GPUs similarly. Storage NICs (SNICs) and compute NICs (CNICs) are also utilized.

DeepSeek's approach involves staging KV data to the DRAM of the decode GPU server and then transferring it to the prefill GPU via GDRDMA, rather than loading all KV directly from local NVMe or DRAM to the GPU.

Technically, DualPath uses Chunk-based Streaming to divide the KV Cache into smaller segments. While the compute unit processes the N-th chunk, the storage unit preloads the N+1-th chunk. DRAM buffers (PE buffer and DE buffer) are used to construct full blocks from layer chunks. This allows the GPU memory to accommodate single-layer KV vectors, with subsequent layers loaded from the CPU and then from disk in a sliding window fashion.

Diagram illustrating DualPath's parallel Access and Compute pipelines with chunk-based streaming.

The architecture is designed to adapt to its infrastructure, with analyses of NIC and DRAM bandwidth guiding the configuration of Prefill:Decode ratios. Optimal results were observed across P:D ratios ranging from 1:7 to 7:2.

Performance Improvements and Ablation Studies

In agent inference benchmarks, DualPath demonstrated a 1.96-fold increase in system throughput. The company evaluated DeepSeek V3.2's 660B and 27B versions, as well as Qwen 2.5-32B, using an agent trajectory dataset. Comparisons were made against a baseline without DualPath and against SGLang with HiCache and Mooncake.

Results showed that DualPath achieved up to 1.87x acceleration for DS 660B compared to the basic configuration, largely eliminating KV-cache I/O overhead. For DS 27B, it achieved up to 1.78x improvement, though performance was still 1.09–1.85x slower than an ideal scenario due to storage bandwidth limitations in the 1P1D configuration. DualPath's advantages were more pronounced with larger batch sizes and longer Maximum Append Lengths (MALs).

The framework demonstrated 1.82–1.99x acceleration at different append ratios compared to the basic configuration. Across all configurations, DualPath achieved an average of 1.64x acceleration, with a peak of 2.46x, confirming that storage bandwidth is a primary bottleneck in agent scenarios. For online services, performance improvement was more significant on larger models.

An ablation study was conducted to determine the contribution of different techniques to improvements in Time to First Token (TTFT) and Job Completion Time (JCT). Layerwise prefill accounted for 45% of the improvement, dual-path loading contributed 39%, and the scheduling algorithm was responsible for the remaining 16%. When all three techniques were applied, the overall performance accelerated the average JCT by 45%.

The system supports 48,000 concurrent agents on a cluster of 1,152 GPUs, configured as 48P:96D, demonstrating linear scaling from 2,000 agents on 2P:4D. The latency of tool calls was not considered in these evaluations.