They designed RefCraft, a set of 300 tasks where agents receive a user instruction and visual references, and must selectively apply those references to create a new webpage.
Benchmark reveals AI agents struggle with visual-reference web development
RefCraft's 300 tasks show frontier models score 52 out of 100 despite access to reference screenshots and iterative refinement.
Chinese Tech
Yuchi Wang · Rongyao Fang · Weihao Yu · Xilin Wei · Tiancheng Gu · Dingkang Yang · +4 more
Alibaba Token Foundry · MMLab, CUHK · The Chinese University of Hong Kong · Fudan University · The University of Sydney
Research Digest··2 min read
The authors introduce RefCraft, a benchmark of 300 realistic web development tasks that combine natural language instructions with screenshots from one to five reference webpages.
§