Looped architectures scale quality by passing activations through the same parameter block repeatedly, but each recurrence normally creates its own KV-cache, so inference memory grows linearly with depth.
Shared KV-caches cut looped-model inference memory without quality loss
Prairie et al. show that stochastic depth pretraining plus a post-training correction stage lets looped models reuse the final recurrence's KV-cache across all recurrences, matching unshared perplexity.
AI Startup
Hayden Prairie · Jacob M. Ryan · Taylor Berg-Kirkpatrick · Daniel Y. Fu
University of California, San Diego · Together AI
Research Digest··2 min read
Prairie, Ryan, Berg-Kirkpatrick and Fu investigate why some looped architectures can share key-value caches across recurrences without quality loss, while identically trained models cannot.
§