Creating large language models requires vast amounts of high-quality text, but such text is limited. Training on the same text multiple times (epochs) helps less each time.
Researchers tested latent feedback, a technique where the model's internal state from one word feeds into the next word's processing. They compared it to standard training with more epochs, using the same computation. They tested multiple model sizes and unique data amounts.
Latent feedback almost always used computation or data more efficiently. At most, it saved 48% of computation or 47% of unique data to reach the same performance. This matters because when high-quality text is scarce, latent feedback is a better use of resources.
