The paper tackles estimating probabilities of catastrophic rare events (e.g., as low as 10^-9) occurring along stochastic output trajectories of large agents like language models. It introduces iterative unalignment, a new method that perturbs the model's weights to create a proposal distribution. This proposal is differentiably parameterized, allowing gradient-based search to amplify event probabilities while maintaining estimator stability through adaptive regularization. The method demonstrates over 800x compute-weighted efficiency gains over naive Monte Carlo on models up to 2.6B parameters across 300+ events.
Weight perturbations efficiently estimate extremely rare failures in language-model agents
The authors construct importance-sampling proposals by modifying model weights, allowing probabilities as low as 10^-9 to be estimated without prohibitive naive sampling.
Independent
Hanming Yang · Daksh Mittal · Jing Dong · Hongseok Namkoong
Research Digest··2 min read
Thread:Agent Security & Attacks
In one line
Iterative unalignment estimates rare event probabilities in stochastic agent trajectories via gradient-based importance sampling.
Before you read this
Two ideas this paper builds onRead it as
Yang et al. develop a method for estimating catastrophic events caused by stochastic variation across an agent’s sequence of actions.
The authors treat an agent’s output as a trajectory of context-dependent actions and estimate the probability that this trajectory produces a specified rare event.
The Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.