Madiraju used a controlled 2×2 design on GSM8K, a mathematical word-problem benchmark, to separate a premise’s usability from whether the relevant entity appeared.
Models flag missing facts more readily than usable unsupported premises
Controlled tests found that making an unsupported premise usable reduced models’ warnings, although accepting question-supplied premises can also reflect conditional reasoning.
Big Tech
Naveen Madiraju
Research Digest··2 min read
Naveen Madiraju tested whether language models respond differently to missing information and to fabricated premises they can use to produce an answer.
The Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.