Continually updated AI models reveal when data was used in training

A black-box attack, TMIA, uses confidence-score trajectories from successive model updates to reconstruct the entry and exit points of individual training samples.

Academic

Jie Fu · Xiangpeng Deng · Yuan Hong · Wendy Hui Wang

Stevens Institute of Technology · University of Connecticut

Research Digest·5 November 2026·3 min read

In one line

A black-box attack infers whether and when individual training samples entered or left a continually updated model, beating prior methods by reading confidence patterns across snapshots.

Before you read this

Three ideas this paper builds on

continual learningA learning paradigm where a model sequentially acquires knowledge from multiple tasks without significantly forgetting earlier ones.56 papers →differential privacyA formal privacy guarantee that limits how much an algorithm’s output reveals about any single individual’s data.17 papers →machine unlearningRemoving specific knowledge, behaviors, or concepts from a trained model while preserving overall capability.13 papers →

Read it as

When a machine learning model is retrained over and over as data changes, it leaves a trail of versions behind. Existing privacy attacks check whether a piece of data was used in training, but they look at only two versions at a time, so they miss the order of events and cannot handle data being added, removed, and added again.

The team built an attack that watches how a model's confidence in a given sample shifts across a whole sequence of versions, using a bidirectional LSTM (a neural network that reads a sequence in both directions) to spot when the sample was added or removed. They tested it on three language models and three image classifiers.

The attack worked well at identifying both whether and when a sample was in training, clearly beating older methods, and it still worked against a large generative language model. Three standard privacy defenses reduced the leak but hurt model quality, so there is a real trade-off between privacy and usefulness.

How it works

stage by stage

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

membership inference attack temporal membership continual learning bidirectional lstm differential privacy machine unlearning