Smarter merging and packing improve LLM memory under tight budgets

RSM-full combines online clustering with grouped context assembly to retain more answer quality within roughly 2,000–5,000 prompt tokens.

PaperIndependentcs.AIarXiv:2609.04915v1
Jiahe Geng · Jinpeng Wang · Kun Yuan
Research Digest··2 min read
Geng, Wang and Yuan developed a compact-memory pipeline that merges related memories online and packs retrieved information into the prompt as grouped units. On two long-horizon memory benchmarks, the authors report better quality–token trade-offs than several streaming and retrieval baselines, with the strongest results in the 2,000–5,000-token range.

What they did

The authors built RSM-full around two components: a cosine-gated max-member merge rule for deciding when new information joins an existing memory cluster, and an atom-aware packer that groups related retrieved content rather than concatenating items flatly.

They evaluated the pipeline primarily on AMA-Bench across prompt budgets of approximately 2,600–5,000 tokens, averaging the main comparisons over four random seeds and ablations over three. They also tested it on RealMem, an independent benchmark for long-horizon persona memory, against retrieval and agent-memory baselines.

Key findings

  • At a 4,000-token budget on AMA-Bench, RSM-full achieved 83% of Full-Context quality while using 32% of its token cost.
  • Across the approximately 2,600–5,000-token range, it exceeded Online K-Means by 3.5–6.0 percentage points under four-seed averaging, with p < .001.
  • Ablations attributed gains of 5.7 percentage points to the max-member merge rule over Online K-Means and threshold-matched DP-means, and 5.0 points to grouped packing over flat concatenation.
  • On RealMem, RSM-full beat Budget-RAG by 0.69 points (p = .006), Streaming-Proto by 2.97 points and A-MEM by 1.65 points (p < .001). Its 0.27-point advantage over BM25-RAG was not statistically significant (p = .47).

Why it matters

The results suggest that compact agent memory depends not only on retrieving relevant records, but also on how memories are consolidated during a stream and assembled into the limited prompt space. This provides practical design guidance for long-running agents constrained by context length, latency or inference cost.

Caveats

The reported advantage is concentrated in a relatively narrow 2,000–5,000-token regime; higher-token baselines remain stronger outside it. The RealMem result against BM25-RAG establishes no significant difference, not statistical equivalence, and the evidence is limited to two benchmarks and the models and configurations tested.

§
newspaper

Research Digest

Articles published under the Zotpaper byline are synthesized from multiple source publications by our AI editor and reviewed by our editorial process. Each story combines reporting from credible outlets to give readers a balanced, comprehensive view.