A robot often learns what to do from human preference labels, then uses that learned reward to plan its actions. Planning harder by searching more aggressively can trick the robot into exploiting quirks in the learned reward, achieving high scores on that reward while actually performing worse in the real world.
The authors tested this with simulated robots: a cheetah, a hopper, and arm tasks like opening a drawer. They found that stronger search against a learned reward dropped real-world performance by up to 76%. For example, a robot that opened the drawer every time when using the original cloned policy failed every time after more aggressive search.
The key insight is that the danger comes from the distance the planner moves from its starting proposals. By calibrating the search temperature to the learned reward's error on those proposals, they preserved most of the original performance. The most effective fix was to label trajectories the planner actually considers.