what_they_did

The authors developed COMFYCLAW, an agent system that interacts with ComfyUI—a node-based image generation tool—by treating workflow construction as editing a typed graph. The system provides tools organized by construction stage (e.g., adding nodes, connecting edges), automatically reverts invalid edits, and uses a region-level VLM verifier (based on a pretrained vision-language model) to detect visual defects and translate them into specific repair suggestions. Over multiple runs, COMFYCLAW distills successful trajectories, execution errors, and verifier feedback into a reusable 'Agent Skill' library that is progressively disclosed to the agent. The framework was evaluated on four benchmark splits (each containing different image generation tasks), using three different agent models (e.g., GPT-4, Llama, etc.) and two image generation backbones (presumably Stable Diffusion variants). The baseline was a verifier-only version without skill evolution.

key_findings

• COMFYCLAW achieved the highest average evaluation score across all six agent configurations (3 agents × 2 backbones), outperforming the verifier-only baseline without skill evolution.
• Human annotators preferred the outputs of COMFYCLAW over the baseline in pairwise comparisons, as measured by annotation studies.
• The skill library grew over time, with the agent successfully reusing skills from prior runs to reduce errors and improve image quality, as shown by ablation studies.
• The region-level VLM verifier provided specific, actionable repair suggestions (e.g., 'adjust saturation in region X') that were more effective than global image-level feedback.

why_it_matters

This work demonstrates that agent memory—in the form of reusable skills distilled from past experience—can significantly improve the reliability and performance of agents in complex, recurring visual workflow tasks. The approach is domain-agnostic in principle and could be applied to other workflow-based creative tools, suggesting a path toward more autonomous and adaptive agents for content creation.

caveats

The evaluation is limited to image generation workflows in ComfyUI; it is unclear how well the approach generalizes to other domains or tools. The skill library is specific to the workflow structure and may not transfer across different user preferences or task types. The region-level VLM verifier requires a pretrained vision-language model and may inherit its biases or failure modes. The paper does not report the computational cost of skill evolution or the number of runs needed before benefits appear.