The authors studied reinforcement learning in which trajectories are generated using FP4 arithmetic, while policy updates follow a separate quantized training path.
Aligning FP4 training and rollout stabilizes reinforcement learning
TRACE guides training-side quantization with rollout-side rounding outcomes, enabling low-precision reinforcement learning of mixture-of-experts language models.
Chinese Tech
Xin Wang · Hao Yu · Zhengyang Zhuge · Bochao Mao · Zheng Li · Junda Feng · +6 more
Alibaba Token Hub, Alibaba Group · Ohio State University
Research Digest··2 min read
Wang and colleagues introduce TRACE, a quantization-aware training framework designed to reduce numerical disagreement between the training and rollout versions of an RL policy.
Why this paper
From Alibaba Token Hub, Alibaba Group and Ohio State University
In one line
TRACE aligns FP4 training and rollout quantization for MoE RL, matching BF16 rollout performance while accelerating rollout by up to 5.4x.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§