The authors constructed UtilityBench by combining answerable queries from 17 datasets with refusal-required queries from AbstentionBench.
Utility metric reveals that accuracy doesn't guarantee responsible AI decisions
UtilityBench and UtilityAUC evaluate language models by the real-world consequences of their errors, showing a persistent gap between problem-solving ability and responsible decision-making.
Independent
Qifeng Lei · Zhiyong Yang · Zitai Wang · Shilong Bao · Linchao Zhang · Qianqian Xu · +1 more
Research Digest··2 min read
The authors present UtilityBench, a 15,000-instance benchmark that assigns sample-specific costs to errors using Best-Worst Scaling, and UtilityAUC, a metric that measures how well model confidence aligns with the utility of outcomes.
§