← All concepts

prefill-decode disaggregation

When serving a large language model, there are two main phases: the *prefill* phase, which processes the user's input all at once, and the *decode* phase, which generates the response token by token. Prefill-decode disaggregation means running these two phases on separate groups of computers (workers) so each can be optimized independently. However, this static split assumes demand for each phase stays constant, which is rarely true in practice.

All three papers find that static prefill-decode disaggregation wastes resources or causes slowdowns because the balance between prefill and decode work changes quickly. They each propose ways to make the boundary between the two phases more flexible: one offloads prefill work to decode workers when they are idle, another lets decode nodes offer temporary, revocable capacity for prefill tasks, and a third prunes away unnecessary work from the decode phase only. These approaches improve throughput and meet performance targets without adding more computers, showing that the capacity to handle shifts is often already inside the system.

Papers
7
Released code
0
First seen
Sept 2026
Latest
Nov 2026

7 papers in the last two months, against 0 in the two before.

Start here

Three papers to read first, in the order they came out
  1. 1Expert pruning in MoE models can be tailored to decode phase, preserving prefill quality while boosting throughput

    SlimWise improves MoE decode throughput by pruning experts only during decode while keeping full prefill.

    Oct 2026
  2. 2FluidPD shifts existing GPU capacity to protect LLM latency targets

    FluidPD improves SLO attainment by up to 94.6 percentage points over static SGLang without adding GPU workers.

    Oct 2026
  3. 3Stale model memory as draft cuts AI agent delays after context edits

    Callo reduces cache-miss latency by speculating from invalidated KV cache while prefill computes the exact cache.

    Nov 2026

The papers

Most central to this idea first, not most recent.

Concepts are extracted from each paper and reused across the corpus, so this page grows on its own as the desk reads.