Edge vision-language models are as physically attackable as classic object detectors

A unified occlusion-controlled benchmark finds that grounding VLM failures track output format more than architecture, and some failures are invisible to downstream systems.

Big Tech
Eric Liang · Alan Luo · Kaiwen Yuan

Saratoga High School · Google · University of Illinois Urbana-Champaign · Mecka.ai

Research Digest··2 min read
The authors benchmark eleven detection and grounding models against physical adversarial patches worn on a person, introducing a conditional attack success rate that only scores images where a model still localizes under a matched gray occluder.

Using one shared predicate, whether a bounding box lands on a person, the authors trained white-box physical adversarial patches for each of eleven models spanning classic object detectors and sub-4B grounding VLMs.

§