AI agents that learn from their own past actions store lessons as reusable skills, similar to how a person might keep notes on how to fix common problems. This persistence creates a security risk: an attacker could poison the agent's learning process so that it builds a hidden, malicious rule into its own skills. The agent would then behave normally on most tasks but follow the attacker's instructions when a secret trigger appears.
The paper introduces a framework called BadEvolve that attacks this learning process. The attacker only provides poisoned practice tasks; the agent's own learning system then encodes the malicious rule into its skills. BadEvolve works by analyzing failures at each step of this process and adjusting the poisoned tasks to ensure the malicious rule is successfully stored and later executed.
The key finding is that BadEvolve reliably implants backdoors while preserving the agent's normal performance. This matters because it reveals a new vulnerability in self-improving AI systems, showing that attackers need not directly modify an agent's skills, only the tasks it learns from. Defenders must now consider the security of the entire learning pipeline.