Self-play with evolving skills boosts LLM benchmarks

Co-evolution of proposer, solver, and skill controller bridges verification and exploration

PaperChinese Techcs.CLarXiv:2607.22529v1 ↗

Siyuan Huang · Pengyu Cheng · Haotian Liu · Tao Chen · Yihao Liu · Jingwei Ni · +7 more

Alibaba · The Chinese University of Hong Kong · Renmin University of China · Sun Yat-sen University · Peking University

Research Digest·25 July 2026·2 min read

The authors introduce Skill Self-Play (Skill-SP), a framework where LLMs generate and solve challenging tasks by dynamically routing through learned skills. In a reinforcement learning loop, the system co-evolves a proposer, solver, and skill controller, showing consistent improvements on tool-use and reasoning benchmarks, even turning around initially misaligned models.

What they did

The authors propose Skill Self-Play (Skill-SP), a co-evolutionary framework comprising three components: a proposer that generates tasks conditioned on skills sampled from a dynamic library, a solver that attempts to solve those tasks, and a skill controller that updates the library based on execution feedback. The components are orchestrated via a reinforcement learning loop, allowing continuous self-play. The framework is evaluated on tool-use and reasoning benchmarks using both strong and weak LLM backbones.

Key findings

Skill-SP consistently improved performance on tool-use and reasoning benchmarks compared to baseline models.
The framework catalyzed striking turnarounds for initially misaligned models, substantially narrowing the gap with stronger backbones.
The dynamic skill library enabled both deep verification in specific scenarios and open-ended task variety through routing.
The co-evolution loop allowed the system to autonomously push its own capability ceiling without human annotation.

Why it matters

This work addresses a fundamental tension in self-evolutionary LLM training: the trade-off between task diversity and verification reliability. By using agent skills as a middle ground, Skill-SP provides a scalable method for LLMs to improve without manual data curation, potentially reducing the need for human feedback in continual learning pipelines.

Caveats

The method assumes a well-initialized skill library and may require careful tuning of the co-evolution loop. The computational cost of running the proposer, solver, and controller in tandem could be high. Generalization to domains beyond tool-use and reasoning (e.g., creative writing, long-horizon planning) has not yet been demonstrated.

Analysis

This work fits into the broader trend of self-play and self-improvement for LLMs, offering a structured alternative to open-ended generation methods like STaR or self-rewarding models. By explicitly managing skills, Skill-SP provides a form of curriculum learning that is both diverse and verifiable. The framework could be combined with other techniques such as process reward models or human feedback to further improve alignment. One open question is how the skill library scales with the number of skills and whether it can be automatically learned from scratch.