News without the noise
A 181-task benchmark shows agents often miss constraints after additive, overriding, or conditional revisions, and that preserving browser state while rewriting instructions helps.
City University of Hong Kong · Shanghai Jiao Tong University
Web agents often satisfy only part of task requirements when goals are revised dynamically.
Current web agents follow fixed instructions, but users often change requirements mid-task. Most ignore this.
DynaRev includes 181 tasks where goals change at key stages. Revisions can add new constraints, override old ones, or depend on the web state. Agents are evaluated on correctness, responsiveness, and efficiency.
Tests of five models show agents often only partially satisfy revised goals. This matters because real users need adaptable agents, and DynaRev reveals gaps static tests miss.
The authors built DynaRev from Online-Mind2Web and Mind2Web 2, converting 181 tasks into dynamic versions with revisions delivered at semantic milestones rather than fixed trajectory positions.
The Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.