Streaming video memory works better when internalized as evolving latent tokens

LatentStream replaces retrieve-as-context with retrieval-then-internalization, using hierarchical latent memory to set new state-of-the-art results on streaming video benchmarks

PaperChinese Techcs.CVarXiv:2609.04131v1 ↗

Hongyu Qu · Guangming Yao · Ling Xing · Xiaobin Hu · Rongxing Ding · Guibin Zhang · +4 more

Nanjing University of Science and Technology · Ant Group · National University of Singapore · The Chinese University of Hong Kong

Research Digest·7 September 2026·2 min read

Read the paper →

Thread:Memory Management for Agents

what_they_did

LatentStream targets the streaming video setting where an MLLM must continuously ingest visual input and answer queries under strict causality and bounded memory. The framework has three coordinated components: Query-Agnostic Hierarchical Streaming Memory organizes visual history into short-, mid-, and long-term levels under a fixed memory budget via Jenks-guided adaptive consolidation; Hierarchical Latent Memory Evolution assigns groups of latent memory tokens progressively expanding receptive fields so they can retrieve historical evidence from their corresponding scopes and internalize it into a compact representation; and Progressive Confidence-guided Latent Memory Optimization builds a hierarchical progression reward from group-wise predictive entropy to jointly refine latent tokens and retrieved evidence. The authors evaluate on established online and offline streaming video benchmarks and compare against prior memory-augmented MLLM baselines.

key_findings

LatentStream achieves new state-of-the-art results on existing online and offline streaming video benchmarks, surpassing prior store-and-retrieve memory methods.
The hierarchical latent memory maintains a fixed memory budget while capturing short-, mid-, and long-term visual dependencies through Jenks-guided adaptive consolidation.
Iterative retrieve-and-internalize cycles with expanding memory receptive fields compress historical evidence into compact, fixed-length latent tokens, improving reasoning efficiency.
Confidence-guided optimization reduces group-wise predictive entropy, leading to more confident and consistent streaming predictions.

why_it_matters

This work reframes how streaming video memories are handled: instead of treating history as external context to be fetched at query time, LatentStream internalizes it into latent working memory that continuously guides reasoning. This shift opens the door for more adaptive, long-horizon video understanding systems that operate within strict memory limits while retaining the ability to reference distant evidence.

caveats

The abstract reports state-of-the-art results but does not include specific performance numbers, so the magnitude of the gains cannot be assessed from this summary alone. It is also unclear how the method scales to very long streams or whether the confidence-guided optimization introduces additional training instability. The offline benchmark evaluations may not fully capture real-time latency constraints and memory pressure in actual streaming deployments.

LatentStream sits in a broader trend of moving from external, retrieval-based memory to latent, parametric memory in MLLMs. By collapsing visual history into evolving latent tokens, the approach addresses a key bottleneck of store-and-retrieve designs—namely, that retrieved clips remain separate from the model's reasoning state and can bloat the context window. The hierarchical, entropy-guided training signal is an interesting mechanism for enforcing memory structure without dense annotations. A natural next question is whether similar latent internalization can be applied to other modalities or to agentic reasoning over long-horizon tasks, where memory consolidation and selective retrieval are equally critical.

ZOTPAPER