SWE-PolyVision introduces a benchmark of 92 real repository-level software tasks, each paired with at least two visual inputs (402 static images and 6 videos total). The tasks come from 36 open-source organizations and include both public and private holdouts. The benchmark evaluates coding models under three access modes: text-only, native vision, and tool-mediated vision.
Testing eleven coding models, the authors found that adding visual access shifts which tasks get solved, but the effect varies by model and task. Two detailed case studies show that when models successfully combine visual and textual clues they can produce verified repairs, but controlled input interventions reveal that this ability is fragile. The key takeaway is that simply providing multiple images does not mean an agent will actually reason across them; patch success alone is not evidence of cross-image reasoning.