This paper introduces a self-play pretraining method that generates its own training data using a universal Turing machine. A generator proposes programs which are executed to produce byte sequences, and a learner is trained to predict these sequences via next-token prediction. The generator is trained with to produce sequences that are just beyond the learner's current capability, creating an .
The experiments show that zero-shot loss on natural datasets scales predictably with compute, even though neither the generator nor the learner ever sees natural data during training. The model also exhibits in-context learning on held-out tasks without gradient updates, and it discovers recognizable mathematical sequences during training.
This work is significant because it suggests a path toward pretraining without relying on curated human-generated data, potentially enabling models to improve indefinitely as compute scales, limited only by computational resources rather than data availability.