Chain-of-thought reasoning traces are legible but not interpretable

Authors quantify step importance via advantage and show that LLM judges cannot fully recover it from text.

PaperAI Startupcs.CLarXiv:2609.04194v1 ↗

Kevin Du · Alexander Hoyle · Laura Ruis · Acyr Locatelli

ETH Zurich · MIT · Cohere

Research Digest·7 September 2026·2 min read

The authors operationalize the importance of a chain-of-thought reasoning step as its advantage—the change in expected reward from including that step—estimated via Monte Carlo rollouts. They find that LLM judges asked to identify high-importance steps outperform a prevalence baseline but fall well short of a noise ceiling, and that fine-tuning a step-level critic improves performance for incorrect responses but remains far from ceiling for correct ones.

what_they_did

The authors define step importance in chain-of-thought reasoning as the advantage: the difference in expected final reward (e.g., correct answer) between including and excluding a step. They estimate this via Monte Carlo rollouts that sample completions with and without the step. On a set of reasoning traces from LLMs, they ask capable LLM judges (e.g., GPT-4) to rate step importance and compare these judgments to the estimated ground truth. They also fine-tune a model as a step-level critic and evaluate its performance separately on correct and incorrect final answers.

key_findings

LLM judges outperform a prevalence baseline (always predicting the majority class) but achieve accuracy far below a noise ceiling derived from Monte Carlo variance.
Fine-tuning a step-level critic yields substantial gains on incorrect responses but leaves performance on correct responses still distant from the noise ceiling.
The results suggest that step importance is only partially recoverable from the text of the reasoning trace, challenging the assumption that legibility implies interpretability.

why_it_matters

This work provides rigorous empirical evidence that chain-of-thought reasoning traces are not a transparent window into model internals. It has direct implications for process reward modeling, where step-level supervision from LLM judges or critics is used to guide or evaluate reasoning—the authors show that such supervision is unreliable for important steps, especially when the final answer is correct.

caveats

The advantage-based definition of importance, while principled, depends on the quality of Monte Carlo rollouts and the reward signal (e.g., binary correctness). The evaluation is limited to specific tasks and models, and the noise ceiling may be unattainable due to inherent stochasticity. The fine-tuning experiment uses a single architecture and may not generalize to other setups.

Analysis

This paper fits into a growing line of work questioning the faithfulness of chain-of-thought explanations. It provides a rigorous causal framework for defining step importance, moving beyond correlation-based evaluations. The finding that text carries only partial information about functional role raises open questions about what additional signals (e.g., hidden states, model internals) are needed for faithful interpretability. It also underscores the risk of relying on chain-of-thought traces for supervision in process reward models.