Spatial audio, sound that carries location information, is usually handled by separate models for understanding what sounds are present, generating new sounds, and editing existing ones. These tasks are often treated independently, and editing has been limited to simple left-right panning.
The authors built a single model that does all three. They started with a pretrained model that understands spatial scenes, and then added generation and editing abilities by routing different tasks through separate low-rank adapters (LoRA) while keeping the original model frozen. They used a technique called flow matching to generate audio from compressed representations.
The combined model not only kept its original understanding skills but also performed better at generating and editing spatial audio than a fully retrained model. This makes it possible to create and modify immersive sound scenes using either structured commands or natural language.