Training language models to reason step-by-step often uses a fixed set of example answers. Not every answer or every word in an answer is equally useful for learning, so choosing which ones to focus on could make training more efficient.
The authors developed GIST, a method that picks which whole answers to train on and which words inside them to supervise. For correct answers it focuses on words the model finds unlikely, and for incorrect answers it focuses on words the model finds likely, using only the sampling temperature and the model’s own word probabilities.
Across several reasoning benchmarks and model sizes, GIST improved accuracy compared to training on all examples. In one controlled test it outperformed seven other selection methods while supervising 66.6% fewer tokens.