Towards Load-Aware Prefill Deflection for Disaggregated LLM Serving
Disaggregated large language model (LLM) serving architectures split prefill and decode phases onto separate GPU pools to mitigate scheduling interference. However, under bursty, heavy-tailed workloads, this physical separation introduces a severe load asymmetry: prefill nodes saturate while decode nodes sit underutilized. On a standard production-style A100 cluster utilizing a two-prefill, two-decode (2P2D) configuration, actual prefill compute execution accounts for a mere 2% to 23% of the P95 Time-to-First-Token (TTFT), with the remaining latency dominated by queuing and inter-node GPU-to-GPU Key-Value (KV) cache transfers. To resolve this inefficiency, Shrikara Arun, Anjaly Parayil, Srikant Bharadwaj, Renee St. Amant, and Victor Rühle introduced a proactive prefill-deflecting scheduler in a paper published on arXiv in July 2026. This work is targeted at infrastructure engineers and systems researchers designing high-throughput, low-latency LLM serving frameworks.
The core contribution of this work is a scheduling mechanism that dynamically deflects prefill requests to underutilized decode nodes, executing them as chunked-prefill steps interleaved with active decode batches. The scheduler relies on three main mechanisms. First, it estimates the expected TTFT of a queued request if it were to remain on the prefill node. Second, for each candidate decode node, the scheduler calculates the largest prefill chunk size that can be processed without violating the strict Time-Between-Tokens (TBT) Service Level Objective (SLO) of that node's active decodes. Third, it deflects the prefill only if this local execution improves tail latency, completely eliminating the need for inter-node KV-cache transfer by running the prefill phase in place on the decode node. Implemented on top of vLLM and evaluated using DeepSeek-V2-Lite, this load-aware deflection strategy reduces P95 TTFT by up to 81% and boosts SLO attainment by up to 79% over traditional disaggregated schedulers, with a negligible routing overhead of under one millisecond per request.
Going forward, this research redefines the boundaries of disaggregated serving, demonstrating that rigid physical partitioning of prefill and decode workloads can be dynamically bypassed to maximize hardware utilization and meet strict latency guarantees. It paves the way for hybrid serving paradigms where nodes dynamically adapt their roles based on real-time load, potentially influencing future iterations of popular inference engines like vLLM and TensorRT-LLM. Note that this analysis is based on the published abstract and metadata of the research paper.