AI assistants that click through computer screens must know where to click. Standard tests mark a click wrong unless it lands inside the labeled rectangle around the intended item. But real screens often have several clickable spots for the same action, like a product name in a menu and its title elsewhere.
The researchers built a probe that clicks in a live browser and records what changes. Mapping many screen positions shows which pixels achieve the target's outcome. They also audited screenshots by hand.
On live websites, about one in five and one in three rejected clicks produced the intended outcome. The effect was largest for the strongest models. Tests undercount correct clicks, so reported gaps may be misleading.