The authors redesigned recurrent state management in state-space models (SSMs).
Replaying cached inputs cuts state memory traffic in SSM decoding
Liou et al. show that representing a state-space model's state as a checkpoint plus a buffer of recent inputs roughly halves state memory traffic and speeds hybrid model serving.
Big Tech
Ze-Wei Liou · Harris Nover · Amey Naik · Andrii Skliar · Igor Shovkun · Ido Shahaf · +1 more
Princeton University · NVIDIA
Research Digest··3 min read
Liou et al.
Why this paper
From NVIDIA and Princeton University
In one line
Representing an SSM state as a checkpoint plus a recent-input buffer cuts state memory traffic, enables cheap speculative rollback, and keeps state quantization error out of later updates.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors
- ✓Reports numbers on named benchmarks (3 benchmarks)
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§