What they did

The authors developed VISA (Visual Instruction Synthesis Agent), a multi-round agentic pipeline for generating multimodal instruction-following training data. At each round, VISA analyzes an image to filter incompatible constraints and discover new verifiable ones, samples diversity- and difficulty-aware constraint sets from persistent memory, generates candidate instructions, and verifies samples with executable tools and structured LLM judges. Failed samples trigger diagnostic-guided recovery; accepted samples are probed against the target model to estimate difficulty. Verifier signals and target-model failure profiles are written back to memory, enabling adaptive expansion of the constraint space, reduction of template repetition, and focus on unresolved model weaknesses. The same verifier contracts also provide reward signals for reinforcement learning without a separate reward model.

Key findings

• VISA consistently improves multimodal instruction following on MM-IFEval compared to strong baselines, including one-pass generate-and-filter pipelines.
• The method preserves general multimodal capability across seven public benchmarks, showing no degradation.
• The self-evolving loop reduces template repetition and expands the constraint space over rounds, as shown by analysis of generated instructions.
• Using verifier signals as reward for RL further improves performance without requiring a separately trained reward model.

Why it matters

This work challenges the one-pass generate-and-filter paradigm for synthetic data, showing that recycling feedback from failed samples, verifier outcomes, and target-model errors can yield higher-quality training data. The approach offers a way to automatically generate diverse and challenging instruction-following data, which is critical for advancing multimodal models.

Caveats

The experiments are limited to the MM-IFEval benchmark for instruction following; generalizability to other instruction-following datasets is not tested. The framework relies on executable tools and structured LLM judges, which may introduce biases. The cost of multiple rounds of generation and verification is not reported. The authors do not compare against human-curated data or other sophisticated synthesis methods beyond simple baselines.