The authors built FluidPD, a serving system that reallocates capacity between the two stages of LLM inference.
FluidPD shifts existing GPU capacity to protect LLM latency targets
The system moves work and worker roles between prefill and decoding as demand changes, without adding GPUs or restarting serving engines.
Big Tech
Kartik Ramesh · Kaidi Fu · Zihan Zheng · Jiahuan Yu · Fabio Oliveira · Carlos Costa · +1 more
University of Illinois Urbana-Champaign · IBM Research
Research Digest··2 min read
Ramesh et al.
Why this paper
From IBM Research and University of Illinois Urbana-Champaign
In one line
FluidPD improves SLO attainment by up to 94.6 percentage points over static SGLang without adding GPU workers.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ✓Reports numbers on named benchmarks
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§