Large language models can remember specific examples from their training data. This makes it possible to test whether a particular piece of data was used to train the model, a concern for privacy. Previous research focused on whether the model reproduces the data verbatim, but it was unclear how different training methods create these memory signals.
This study introduces three measures, DUG, that separately track how much membership information is stored in the model's internal activations and how much of that stored information actually appears in the model's output. The authors applied these measures to two training interventions: supervised fine-tuning and .
They found that supervised fine-tuning changes both stored information and its readout, while reinforcement learning mainly changes only the readout. In a more realistic reinforcement learning scenario, behavioral changes occurred without any increase in the standard membership inference signal, revealing a gap between behavior and likelihood-based detection.