The authors constructed 638 multi-turn trajectories containing 3,766 turns across WebShop and TravelPlanner, environments for product shopping and travel planning.
Language-model agents struggle to track and act on changing requirements
A human-constructed benchmark tests eleven agents on shopping and travel tasks, finding weaknesses in both recovering revised intentions and executing them.
Big Tech
Jieyu Zhou · Zijian Wang · Yun Wang
Georgia Institute of Technology · Microsoft Research
Research Digest··2 min read
Zhou and colleagues introduce IntentDelta, a benchmark that tests whether language-model agents can follow users who change requirements during a task.
The Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.