Speech synthesis, or generating spoken audio from text, usually requires models trained from scratch on speech data. This paper asks whether a powerful image-generation model, which has never seen speech, can be reused to make speech.
The authors convert speech into visual spectrogram images, feed them into a frozen image model's encoder, and only fine-tune the model's main transformer backbone on speech data. This lets them leverage the image model's learned representations and generative abilities for a new task.
The frozen image encoder preserves speech intelligibility (0.99 on a standard metric) and speaker identity (0.95 similarity). Fine-tuning the image-pretrained model reduced word error rate from 4.2% to 2.7% compared to training from scratch, showing that image-generation priors help speech synthesis.