Conversational image editing agent learns interpretable tool use

IEA combines supervised fine-tuning, reinforcement learning, and synthetic data to let amateur users edit images through dialogue while exposing every step.

PaperChinese Techcs.CVarXiv:2606.08016v1
Zichen Zhu · Yuheng Sun · Mingxuan Zhu · Wenjie Ma · Situo Zhang · Zhexiang Wang · +8 more

Shanghai Innovation Institution · Huawei Technologies Ltd. · X-LANCE Lab, School of Computer Science, Shanghai Jiao Tong University · Nanyang Technological University · Jiangsu Key Lab of Language Computing

Research Digest··1 min read
The authors present IEA, a conversational image editing agent that manipulates 16 parameterized tools step by step instead of generating pixels directly. Trained via a three-stage multitask pipeline, IEA achieves lower pixel distance on editing tasks and higher ROUGE-L on intent summaries than strong baselines, and outperforms both tool-calling and generative methods in user studies.

What they did

The authors constructed IEA as a vision-language model that selects and applies image editing tools (e.g., brightness, crop, color balance) from a predefined set of 16. Training proceeded in three stages: (1) supervised fine-tuning on expert demonstrations distilled from a larger model, (2) GRPO (GrouP Relative Policy Optimization) with rewards for image-likeness improvement, tool usefulness, and intent summarization, and (3) large-scale synthetic fine-tuning to jointly master editing, refinement, and user intent summarization.

Evaluation used a dataset of edit instructions paired with original and target images. The authors compared IEA against both tool-calling baselines and generative models (including text-to-image and instruction-following diffusion models). Metrics included pixel-level distance (mean squared error), ROUGE-L for summary quality, and human ratings for instruction following and perceptual quality.

§
newspaper

Research Digest

Articles published under the Zotpaper byline are synthesized from multiple source publications by our AI editor and reviewed by our editorial process. Each story combines reporting from credible outlets to give readers a balanced, comprehensive view.