The authors froze a base LLM and learned input-invariant steering vectors at selected layers to replicate the output of an RL-tuned model. This technique tests whether RL gains are low-dimensional. On 5 LLMs and 6 verifiable-reward tasks, a single vector per layer recovered over 85% of the full fine-tuning gain, but capacity is not infinitely compressible: at very low capacity, intervention dimensionality and input-dependent expressiveness become limiting. Effective control directions lie primarily in the low-variance complement of the activation principal subspace, and training collapse coincides with intrusion of updates into this principal subspace. Based on these geometric insights, the authors built Alpha-Stabler, a plug-and-play framework that monitors principal-subspace intrusion and projects activation gradients away from it, stabilizing training for 2,000 steps and consistently improving RL gains. The finding reveals that RL post-training modifies LLM behavior through a structured, low-dimensional geometry, which can be exploited for more robust training.
RL reasoning gains live in a low-dimensional, low-variance activation subspace
By steering activations with trainable vectors, the authors map where RLVR improvements reside and use that geometry to stabilize training for 2,000 steps.
Chinese Tech
Yuchen Cai · Ding Cao · Qixiang Yin · Xin Xu · Kai Yang · Siye Wu · +7 more
USTC · Tencent Hunyuan · BUPT
Research Digest··3 min read
In one line
RLVR-induced gains in LLMs are explained by a low-dimensional activation manifold where effective control directions lie in the low-variance complement of the principal subspace.
Before you read this
Two ideas this paper builds onactivation steeringActivation steering modifies a model's internal vector representations during inference to shift behavior, such as editing outputs, removing capabilities, or controlling trust.83 papers →reinforcement learning with verifiable rewardsReinforcement learning that uses rewards from objectively checkable signals like correct answers or verification feedback.163 papers →
Read it as
Cai et al. use vector steering to identify a low-dimensional manifold in activation space that captures most RL-induced reasoning gains in large language models.
The authors investigated (RLVR), where models are trained to produce outputs that can be automatically checked for correctness.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.
What it released
Code