prefill-decode disaggregation
When serving a large language model, there are two main phases: the *prefill* phase, which processes the user's input all at once, and the *decode* phase, which generates the response token by token. Prefill-decode disaggregation means running these two phases on separate groups of computers (workers) so each can be optimized independently. However, this static split assumes demand for each phase stays constant, which is rarely true in practice.
All three papers find that static prefill-decode disaggregation wastes resources or causes slowdowns because the balance between prefill and decode work changes quickly. They each propose ways to make the boundary between the two phases more flexible: one offloads prefill work to decode workers when they are idle, another lets decode nodes offer temporary, revocable capacity for prefill tasks, and a third prunes away unnecessary work from the decode phase only. These approaches improve throughput and meet performance targets without adding more computers, showing that the capacity to handle shifts is often already inside the system.
- Papers
- 7
- Released code
- 0
- First seen
- Sept 2026
- Latest
- Nov 2026
7 papers in the last two months, against 0 in the two before.
Start here
Three papers to read first, in the order they came out- 1Expert pruning in MoE models can be tailored to decode phase, preserving prefill quality while boosting throughput
SlimWise improves MoE decode throughput by pruning experts only during decode while keeping full prefill.
Oct 2026 - 2FluidPD shifts existing GPU capacity to protect LLM latency targets
FluidPD improves SLO attainment by up to 94.6 percentage points over static SGLang without adding GPU workers.
Oct 2026 - 3Stale model memory as draft cuts AI agent delays after context edits
Callo reduces cache-miss latency by speculating from invalidated KV cache while prefill computes the exact cache.
Nov 2026
How fast it is moving
Often studied with
large language modelA neural network trained on massive text corpora to generate coherent text and perform diverse language tasks.2 togethermixture of expertsA neural architecture with many specialized sub-networks, only a few activated per input via a router, balancing capacity and efficiency.2 togetherexpert pruningSelectively removing or deactivating experts in a mixture-of-experts model to improve efficiency while preserving accuracy.1 togetherThe papers
Most central to this idea first, not most recent.
- Independentcs.DC
Stale model memory as draft cuts AI agent delays after context edits
Callo reduces cache-miss latency by speculating from invalidated KV cache while prefill computes the exact cache.
Nov 2026
- Big Techcs.AI
FluidPD shifts existing GPU capacity to protect LLM latency targets
FluidPD improves SLO attainment by up to 94.6 percentage points over static SGLang without adding GPU workers.
University of Illinois Urbana-Champaign, IBM Research · Oct 2026
- Top Universitycs.LG
Expert pruning in MoE models can be tailored to decode phase, preserving prefill quality while boosting throughput
SlimWise improves MoE decode throughput by pruning experts only during decode while keeping full prefill.
a2sys, KAIST · Oct 2026
- Industrycs.DC
Elastic phase sharing improves efficiency in agentic LLM serving
Crossflow makes the prefill-decode boundary elastic by letting decode nodes lend spare capacity to prefill, improving token throughput 16.2-17.4% over static P/D disaggregation.
Meta Platforms · Sept 2026
- Big Techcs.AI
AI services can meet streaming deadlines at twice the demand
A tail-calibrated scheduler that leaves headroom in each decode step to prevent jitters, boosting SLO goodput 2.4–2.9x.
Sprinklr, Microsoft · Nov 2026
- Industrycs.LG
AI models run faster when busy experts share chip resources
Fuse-All MoE fuses expert computation and communication to reduce MoE inference latency by up to 21.3% on TPUs.
Meta SuperIntelligence Lab · Nov 2026
- Academiccs.DC
Decoupling attention from model weights speeds large-language-model serving by up to 2x
Decode Token Parallelism improves LLM throughput by 1.7-2.06x through decoupling attention computation from model weights across nodes.
Texas A&M University, Los Alamos National Laboratory · Nov 2026
Concepts are extracted from each paper and reused across the corpus, so this page grows on its own as the desk reads.