The paper introduces Memory Gain Policy Optimization (MGPO), a method for training a memory writer in language models to selectively retain and revise persistent textual memory. MGPO uses counterfactual to isolate the marginal contribution of each memory rewrite, addressing the credit assignment problem where a rewrite's utility may only be realized many steps later. The method compares pre- and post-rewrite memories on the same downstream tasks to compute incremental value. Evaluated on document-level information extraction, MGPO improved extraction performance while reducing average memory length by nearly 80% relative to the initial policy. The learned memory policy also transferred across domains and downstream models without additional training.
This work is interesting because it provides a principled approach to learning what to remember in long-horizon interactions, a key challenge for persistent memory in language models. By focusing on incremental value, MGPO learns compact and effective memories that generalize beyond the training setting. This could improve efficiency and performance in applications like document processing, chatbots, and long-form reasoning, where memory capacity is limited and optimal retention is critical.