Automated red teaming uses AI to find weaknesses in other AI systems. A diverse set of attacks is important because it exposes a wide range of potential failures. However, when the attacker and defender are trained together, the attacker may start producing fewer types of attacks.
This study reproduced that narrowing effect using specific models and training methods. The attacker was a Qwen2.5-14B large language model, and the defender was a Phi-4-mini model. They were co-trained with a technique that stabilizes updates, and the attacker's diversity was measured with a metric called Attack Diversity Retention (ADR).
After 12 rounds of co-training, the attacker's diversity dropped by 37% to 39% across three separate runs, while attack success actually improved. This means co-training can produce effective but less varied attacks, which is a trade-off for red-teaming exercises that aim to comprehensively test a system.
