The paper proposes TRAP, a one-sided penalty on tokenwise target-reference advantage (TRA) that reduces of sensitive data during fine-tuning. TRA compares the model's probability to a reference model trained on the complementary half of the data, requiring no prior knowledge of which spans are sensitive. On student essays and clinical cases, TRAP brings memorization near the level of an untrained model with little utility loss, outperforming and generic regularizers.
Tokenwise reference penalties curb privacy memorization during language-model fine-tuning
TRAP suppresses unusually high probabilities for record-specific tokens without requiring sensitive spans to be identified beforehand.
Big Tech
Muhammed Ustaomeroglu · Ziyue Xu · Hanshen Xiao · Peter Cnudde · Guannan Qu · Holger R. Roth
Carnegie Mellon University · NVIDIA · Purdue University
Research Digest··2 min read
In one line
TRAP penalty reduces memorization of sensitive data in fine-tuned language models with little utility loss.
Before you read this
Two ideas this paper builds onmemorizationMechanisms by which a model stores and reproduces exact training data, especially sensitive information, rather than learning generalizable patterns.4 papers →differential privacyA formal privacy guarantee that limits how much an algorithm’s output reveals about any single individual’s data.20 papers →
Read it as
Ustaomeroglu et al. compare each fine-tuned model with a reference trained on the complementary half of its corpus, producing a per-token measure of record-specific memorization called Target Reference Advantage.
The authors define Target Reference Advantage (TRA) as the excess token log-likelihood assigned by a model trained on a record relative to a model trained with the same procedure on the complementary half of the corpus.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.