Music relies on repeating patterns, from short motifs to entire sections. Generating long, coherent pieces requires a model to remember and reference distant past events, which is hard for standard neural networks.
The Transformer, a model using self-attention to look at any previous step, can capture long-range dependencies. However, its existing method for representing relative timing was too memory-hungry for long music sequences. The authors developed an algorithm that dramatically cuts memory use, from scaling with the square of sequence length to scaling linearly.
This enabled the Transformer to generate minute-long piano performances with compelling structure, surpassing previous models. The work demonstrates that efficient relative attention is key for modeling complex, long-form music and suggests broader applications in other sequential domains.