Current memory-augmented LLMs estimate a memory's utility only from memories that have been retrieved, causing the utility of never-retrieved memories to be unidentifiable. This is a retrieval-level positivity violation that existing diagnostics miss. The paper introduces Causal Memory Policy (CMP), which restores identification by reserving a fixed number of context slots for memories sampled with known propensities, enabling unbiased utility estimation via self-normalized inverse propensity weighting. Experiments on LongMemEval and LoCoMo show identification fails for 54% and 67% of required memories respectively, and CMP improves discrimination between required and non-required memories from 0.54 to 0.66 AUC. However, the paper also finds that identified utility alone is insufficient for retention decisions, as per-query utility does not transfer across unseen queries.
Randomized retrieval reveals which AI memories actually help
Causal Memory Policy deliberately exposes stored memories to the model, making their effects measurable when ordinary retrieval would never select them.
Academic
Arman Behnam · Binghui Wang
Illinois Institute of Technology
Research Digest··3 min read
In one line
CMP uses randomized retrieval to make memory utility identifiable in memory-augmented LLMs.
Before you read this
Three ideas this paper builds oncounterfactual reasoningComparing actual outcomes with imagined alternatives to estimate the effects of different actions or conditions.114 papers →credit assignmentDetermining which actions or steps in a sequence contributed to a final outcome, often for reinforcement learning.190 papers →memory retentionThe ability of a system to preserve and recall information over time or across contexts, often balancing compression against fidelity.32 papers →
Read it as
Behnam and Wang identify a blind spot in utility-based memory management: a stored memory cannot be evaluated if the retrieval system never places it in the model’s context.
The authors formalize retrieval as the mediator between an external memory store and a language model’s output.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.