When a large language model is trained by reinforcement learning, each word it generates is treated as an action. But the environment only cares about the final command, not the reasoning or formatting tokens that preceded it. Those extra words add noise to the training signal, making learning slower and less efficient.
This work proposes a different approach: instead of generating text token by token, the model gets a pre-compiled menu of executable actions and simply picks one in a single forward pass. This removes all the extra text and ensures every chosen action is valid, saving computation and reducing training noise.
On several tasks, the new method matched or exceeded the success rate of standard token-level training while using far fewer GPU hours. For example, on the ALFWorld benchmark it reached 99.3% success in 12.7 GPU hours versus 86.6% in 17.3 GPU hours for the usual approach.