When you ask a smart assistant to find a nearby place, it must check if it is open, quiet, or affordable. Current systems struggle to separate must-haves from nice-to-haves, and facts from subjective reviews.
Researchers built a standard test using hundreds of real queries annotated with strict requirements. A sandbox provided the same tools and place database to every agent. Many different large language models were run against these tests.
The top model satisfied about seven in ten hard constraints when it gave a recommendation. Another model found a valid option for about eight in ten queries. Good coverage can come at the cost of reliably meeting every specific need.