Static prefill-decode (P/D) disaggregation assumes phase demand is stable, but demand ratios shift quickly: peak-to-mean ratios reach 4.7x at minute timescales, and a public agentic trace shows median hourly ratio spanning 24.5x within a day, while reassigning a replica takes tens of minutes. This forces operators to strand capacity or accept queueing. Crossflow makes the boundary elastic without changing node roles. Each decode node publishes a short-lived, revocable lease bounding local prefill compute, KV capacity, transfer work, and projected output; the cluster scheduler atomically reserves leases and admits prefill work in bounded units. Implemented in SGLang and evaluated on public and internal traces, Crossflow improves token throughput by 16.2-17.4% geometric mean over static P/D, by up to 43.4% at high load, and reduces mean TTFT at every evaluated point. This is interesting because it directly attacks the static assumption behind P/D disaggregation while preserving node specialization.
Elastic phase sharing improves efficiency in agentic LLM serving
Crossflow lets decode nodes temporarily absorb prefill work through bounded leases, reducing idle capacity without slow replica reassignment.
Industry
Yi Xu · Ehsan K. Ardestani · Wenyin Fu · Martin Schatz · Krishna Malladi · Zhan Shu · +4 more
Meta Platforms
Research Digest··2 min read
In one line
Crossflow makes the prefill-decode boundary elastic by letting decode nodes lend spare capacity to prefill, improving token throughput 16.2-17.4% over static P/D disaggregation.
Before you read this
Three ideas this paper builds onlarge language modelA neural network trained on massive text corpora to generate coherent text and perform diverse language tasks.329 papers →prefill-decode disaggregationSplitting LLM serving into separate prefill and decode phases to optimize GPU resource allocation and batch sizing.7 papers →model routingDirecting each query or subtask to the most suitable model to optimize cost, accuracy, or quality.89 papers →
Read it as
Xu et al. address a mismatch in disaggregated LLM serving, where separate pools handle prompt processing, called prefill, and token generation, called decode, even though demand for each phase changes rapidly.
The authors measured variation in prefill and decode demand using a large internal LLM fleet and a public agentic workload trace.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.