VoiceNet is a new human-annotated for understanding vocal performances in real-world speech. It goes beyond the typical six to nine emotion categories by using a 40-emotion taxonomy and also evaluates 57 talking-style attributes like speaking rate and breathiness. The benchmark consists of two subsets: VoiceNet-Emo and VoiceNet-Ext.
The authors release VoiceCLAP, two voice-text contrastive models (110M and 7B parameters) trained on their data. On VoiceNet-Emo, existing CLAP models perform near chance, while VoiceCLAP models significantly outperform them. VoiceCLAP-Large aligns with expert consensus better than individual experts agree with each other.
This work is interesting because it fills a gap in speech perception evaluation, providing fine-grained benchmarks and models that can be used for filtering and clustering large speech corpora by emotional and stylistic content.