
Utility metric reveals that accuracy doesn't guarantee responsible AI decisions
The authors present UtilityBench, a 15,000-instance benchmark that assigns sample-specific costs to errors using Best-Worst Scaling, and UtilityAUC, a metric that measures how well model confidence aligns with the utility of outcomes. Evaluating 20 state-of-the-art language models, they find that greater accuracy and more advanced architectures do not reliably translate to higher UtilityAUC, revealing a gap between problem-solving ability and responsible decision-making.













































