Representation autoencoders (RAEs) reuse features from a pretrained visual encoder for both reconstruction and generation, but the choice of which encoder layers to fuse involves a trade-off: shallower layers preserve pixel details, deeper layers improve generation metrics. A fixed heuristic fusion couples these two stages, leading to a reconstruction-generation gap. FuseReg replaces this heuristic with training over random subsets of encoder layers, forcing the decoder and generator to work with varying combinations of layers.
A single FuseReg decoder reconstructs from full, sparse, and single-layer fusions without retraining, achieving higher PSNR than decoders specialized to fixed fusions. This flexibility benefits generation: decoder replacement alone reduces unguided gFID by 27% on a DiT-XL generator, and joint regularization of both stages reduces unguided gFID by 29% on DiT-Base. The approach narrows the reconstruction-generation gap without modifying the pretrained encoder or adding architectural changes.