Findings show that high single-turn scores do not guarantee successful sessions, and early exits are driven more by state-tracking failures than by poor clarification. The proposed agent, with explicit session state management, shopping-flow control, catalog-grounded retrieval, and runtime guards, consistently outperforms strong baselines. This matters because real-world shopping requires handling ambiguous intent, bundles, and multiple goals simultaneously.
Shopping agents falter when user needs change across a conversation
A benchmark spanning 3.28 million products finds that explicit session control improves agents facing evolving constraints, ambiguous requests and multi-item purchases.
Top University
Xinwei Yang · Kelong Mao · Yudong Guo · Sulong Xu · Simiu Gu · Chen Huang · +1 more
JD.com · Sichuan University · Engineering Research Center of Machine Learning and Industry Intelligence · National University of Singapore
Research Digest··2 min read
Thread:Process Agent Benchmarks
In one line
Conversational shopping agents struggle with state tracking, constraint updates, and grounded convergence; an executable framework with explicit state management and runtime guards outperforms baselines.
Before you read this
Three ideas this paper builds onbenchmarkA standard set of tasks or data used to evaluate and compare the performance of AI systems.366 papers →agent memoryA repository where an agent stores past interactions and outcomes to inform future decisions.241 papers →runtime monitoringAudits an AI agent's live actions, internal states, or effects against constraints or policies to catch errors and attacks.134 papers →
Read it as
Yang et al. introduce RealWorldShop, a benchmark designed to evaluate the full decision process of conversational shopping rather than isolated replies or final purchases.
The authors built RealWorldShop around a catalog of 3.28 million grounded products.
How it works
stage by stageThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.