agents learn by trial and error, but the training tasks they face can keep changing as they improve. In Unsupervised Environment Design (UED), a system automatically creates or selects new tasks that are just right for the agent's current skill level. This keeps the training challenging but risks pushing the agent into situations it can no longer learn from.
The researchers trained agents using three different UED methods and monitored their internal health over long periods. They found that agents' internal parameters grew too large, their value predictions lost correlation with actual outcomes, and their neural representations became less diverse. These problems made learning stall or get worse over time.
Simple fixes like capping parameter size and using a different way to predict value reduced these problems. Agents with these fixes kept learning longer and could handle new tasks they had never seen before, without changing how tasks were selected. The lesson is that a good curriculum is useless if the student cannot learn from it.