The authors propose Stackelberg Alignment, a framework where an EXP3 bandit acts as a leader, adaptively selecting which instructions a pool of language models should duel on during collaborative training. This replaces uniform instruction sampling, which wastes compute on instructions where models already agree or all fail. The leader's reward combines instruction difficulty and response discriminability, focusing training on the frontier of disagreement. Experiments across three model pools and 12 benchmarks show consistent improvements over baselines, with gains up to 7.4% over training-time baselines and 12-25% over static inference baselines.
Adaptive instruction selection helps language models learn from peers
A bandit-guided training curriculum improved collaborative alignment by directing model competitions toward instructions that produced useful preference signals.
Big Tech
Christina Hahn · Shangbin Feng · Dean Light · Swastik Roy · Hila Gonen · Yulia Tsvetkov
University of Washington · Amazon · University of British Columbia
Research Digest··3 min read
In one line
Adaptive instruction selection via a Stackelberg game improves multi-LLM collaborative alignment across benchmarks.
Before you read this
Three ideas this paper builds onmulti-agent collaborationMultiple autonomous agents coordinate and combine their specialized capabilities to solve complex tasks beyond individual ability.22 papers →adaptive curriculumA training strategy that gradually increases task difficulty based on the learner's current performance, often using success rates to select the next challenge.26 papers →preference optimizationA post-training method that adjusts model behavior using comparisons or self-generated preferences to improve alignment.69 papers →
Read it as
Hahn et al. introduce Stackelberg Alignment, a training framework in which language models answer, compare and learn from one another while an adaptive controller decides which instructions they should tackle.
The authors cast collaborative model training as a leader-follower game.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.