The authors added two sets of neurons to every multilayer perceptron block: a retain adapter for desired capabilities and a forget adapter intended to absorb unintended behavior.
Routing gradients can isolate reward hacking for removal after training
GRAFT confines learning from monitor-flagged trajectories to removable parameters, reducing unintended behavior without directly rewarding monitor evasion.
Big Tech
Jake Ward · Shawn Hu · Aria Wong · Nathan Hu · Bryce Woodworth · Alexander Matt Turner · +1 more
MATS · Stanford · FAR AI · Anthropic
Research Digest··3 min read
Ward and colleagues introduce Gradient Routing Adapter Fine Tuning, or GRAFT, for reinforcement learning with misspecified rewards and imperfect behavior monitors.
Why this paper
From Anthropic and 3 others
In one line
Routing gradients from flagged samples into sacrificial parameters, then ablating those parameters, removes reward hacking without teaching the model to exploit the monitor.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§