Large language models (LLMs) must be updated with new knowledge over time, like fixing a wrong fact or adding a new behavior. But when you try to make many different kinds of updates one after another, say, correcting facts, then changing how it answers questions, then adjusting its safety rules, the model often forgets earlier changes. This happens because each type of update pushes the model's internal settings in directions that block later updates.
The researchers found that the problem stems from each update task having its own preferred “direction” in the model’s parameter space. They invented a measure called the Conflict Index to quantify how much two tasks interfere. Using this measure, they built Mu-Edit, which first arranges tasks in the order that causes the least interference, and then dynamically relaxes some constraints to create more room for later updates.
Across tests with five different kinds of tasks and three different LLMs, Mu-Edit preserved the benefits of earlier edits far better than existing methods. This means models can be patched repeatedly with diverse knowledge without degrading performance, making them more reliable for real-world use.