The authors formulate multi-reward post-training as a one-step stochastic decision problem.
Equal reward weighting can steer language models toward easier shortcuts
The authors characterize how auxiliary rewards can displace task performance and propose a gating rule that prioritizes a more trusted reward channel.
Big Tech
Yushu Li · Jiin Woo · Xiaoxiao Li · Youngsuk Park
University of British Columbia · Amazon Annapurna Labs
Research Digest··3 min read
Li and colleagues show that combining a primary reward, such as correctness, with an easier auxiliary reward, such as brevity, can raise the aggregate training objective while reducing task success.
Why this paper
From Amazon Annapurna Labs and University of British Columbia
In one line
Symmetric reward aggregation in LLM post-training degrades primary task performance by favoring easier auxiliary rewards.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors (3 noted)
- ✓Reports numbers on named benchmarks
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§