In , a neural network learns to reverse a corruption process. There are different ways to set up what the network predicts: the clean image, the noise, or a velocity. The loss (error measure) can also be defined in different spaces. These choices seem equivalent in theory but give different results in practice.
The authors separate the effects of prediction target and loss space. They show that under an architectural bottleneck, a narrow layer that compresses information, noise and velocity prediction suffer an irreducible error floor. Clean-image prediction can circumvent this by exploiting low-dimensional structure in natural images.
Under strong compression, noise prediction errors matched the theoretical lower bound. Clean-image prediction performs better with narrow bottlenecks, guiding design for efficient generative models.
