AI agents increasingly create visual materials like slides, graphics, animations, and web pages. However, existing tests often focus on whether the task is completed, missing how well the artifact suits its intended audience and purpose. For example, a presentation might include all required facts but be poorly organized for a technical audience.
The authors introduce VisualArtifactSpec 1.0, a of 211 tasks across presentations, graphics, animations, and web interfaces. Each task specifies the audience, purpose, and medium. The evaluation separates Task Fulfillment (checking explicit requirements) from Artifact Performance (rating Content, Organization, Visuals, and Delivery using task-specific criteria).
Testing nine AI model configurations showed that high task fulfillment can coexist with poor artifact performance. Artifact Performance accounted for 73.4% of the gap between the best and worst systems. This means that simply completing a task is insufficient; the quality of the artifact for its intended use matters significantly.
