The authors first note that long-context RL needs training data that are long, diverse, challenging, and verifiable, but such data are scarce and costly.
LongZero trains LLMs on self-written long-context tasks, no external data
A new reinforcement learning method has models generate their own reasoning and retrieval exercises with structured answers before expanding them into long documents, outperforming external-data baselines on five benchmarks.
Top University
Zhennan Wan · Jianfei Chen
Tsinghua University · Tsinghua-Bosch Joint ML Center
Research Digest··1 min read
Wan and Chen propose LongZero, in which a language model synthesizes question, evidence, and answer as a compact structured specification, then expands it into a long multi-document context, and trains with GRPO.
§