The authors built an executable framework with nine transformation operators that rewrite tool schemas while preserving the underlying tasks, executable actions and reachable states.
Equivalent tool interfaces can make the same LLM agent fail
Across functionally identical tool schemas, model success varied from zero to 97 percent despite unchanged tasks, actions and reachable states.
Chinese Tech
Yinhong Liu · Zhili Tan · Zilin Wang · Zhijiang Guo
University of Cambridge · Yinwang · Huawei · LARK, HKUST (GZ) · HKUST
Research Digest··2 min read
Liu et al.
Why this paper
From Huawei and 4 others · Released code
In one line
LLM agents exhibit large performance differences across functionally equivalent tool schemas, with success rates ranging from 0% to 97%.
What it released
Code
What we could check
- ✓Code link in the paper (github.com)
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ✓Reports numbers on named benchmarks
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§