The framework has three parts: hierarchical streaming memory with fixed budget organization, latent memory evolution with progressively expanding receptive fields, and confidence-guided optimization using group-wise predictive entropy. The authors report state-of-the-art results on existing online and offline video benchmarks. The key novelty is that memory becomes a learned, evolving representation rather than a static external store.
Streaming video memory works better when internalized as evolving latent tokens
LatentStream replaces retrieve-as-context with retrieval-then-internalization, using hierarchical latent memory to set new state-of-the-art results on streaming video benchmarks
Chinese Tech
Hongyu Qu · Guangming Yao · Ling Xing · Xiaobin Hu · Rongxing Ding · Guibin Zhang · +4 more
Nanjing University of Science and Technology · Ant Group · National University of Singapore · The Chinese University of Hong Kong
Research Digest··2 min read
Thread:Memory Management for Agents
In one line
LatentStream converts streaming video memory from store-and-retrieve to retrieve-and-internalize, improving streaming video understanding.
Before you read this
Three ideas this paper builds onagent memoryA repository where an agent stores past interactions and outcomes to inform future decisions.200 papers →context compactionCompressing or summarizing an LLM's past context into a smaller representation to extend memory and improve reasoning within limited token budgets.71 papers →memory consolidationSummarizing or structuring past information to reduce token usage while preserving task accuracy.35 papers →
Read it as
The authors introduce LatentStream, a progressive latent working-memory framework for multimodal LLMs processing streaming video.
LatentStream targets the streaming video setting where an MLLM must continuously ingest visual input and answer queries under strict causality and bounded memory.
The Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.