The authors constructed a pattern-controlled logical reasoning benchmark based on linguistic and pragmatic reasoning phenomena, designed so that everyday acceptable conclusions sometimes diverge from textbook formal logic.
Reasoning models favor formal logic over everyday reasoning, lagging simpler models
A pattern-controlled benchmark shows reasoning models systematically prioritize formal rules on everyday tasks, while non-reasoning models with chain-of-thought prompting sometimes align better with everyday judgments.
Academic
Kentaro Ozeki · Risako Ando · Takanobu Morishita · Hirohiko Abe · Koji Mineshima · Mitsuhiro Okada
Keio University
Research Digest··2 min read
Ozeki et al.
§