When training a large AI model, you might temporarily add a penalty that shrinks the model's weights (weight decay) and then remove it. Normally you'd evaluate the model right after the penalty ends, but this can be misleading.
The researchers tested this by adding temporary weight decay to transformer models (a common AI architecture) trained with the (a method for updating model weights). They varied when the decay started and how long it lasted, for models from 162 million to 3.4 billion parameters.
They found that a temporary decay schedule could make validation loss 0.26 nats worse at removal, but then gradually improve and finish 0.028 nats better than no decay. This means you must evaluate temporary changes at the final training budget, not just when they end.
