The authors propose a two-step measurement framework.
Foundational LLM capabilities only partially predict agentic execution gaps
A new operation-level measurement framework reveals systematic departures between benchmark profiles and interactive behaviors across 15 model services.
Big Tech
Wei Guo · Yuanjian Xu · Jianing Hao · Yongyao Duan · Ran Chen · Zhong Li · +1 more
Nankai University · Hong Kong University of Science and Technology (Guangzhou) · Ant Group · OpenCSG · Microsoft Research Asia
Research Digest··3 min read
The authors develop a framework that links foundational LLM benchmarks to agentic operations by measuring capability realization gaps, deviations of observed execution behavior from predictions based on offline capability scores.
§