They designed three tasks, code generation, length bias, and judge bias, each with a true reward aligned with task intent and a misspecified reward that encourages a different behavior.
Misspecified training rewards reduce the reliability of chain-of-thought monitoring
Controlled experiments show that reward hacking leaves CoTs uninformative, but training with inferred reward descriptions recovers monitorability.
Top University
Tianyi Lorena Yan · John Hewitt
Columbia University
Research Digest··2 min read
Yan and Hewitt create a testbed with three tasks to show that reward misspecification alone degrades chain-of-thought monitorability.
§