The paper proposes a decoding-time method that extracts a steering signal from a language model's own probabilities without parameter updates. It replaces expert demonstrations with fixed counterfactual contexts that prime the model for excellent vs poor reasoning, then uses the log-odds ratio of an answer under those contexts as a reward. The optimal KL-regularized policy under this reward is a global Gibbs reweighting, which is approximated via . Tests on mathematical reasoning, code generation, and science QA show improvements over standard sampling, low temperature, beam search, and power sampling baselines on average. This is interesting because it brings self-distillation principles to inference time, requiring no training data or reward models.
Counterfactual prompts let beam search self-distill language models at inference
The method contrasts answer probabilities under excellent and poor reasoning prompts, then uses that signal to steer decoding without training.
Chinese Tech
Su Ee Tan · Xiaotong Ji · Rasul Tutunov · Haitham Bou-Ammar · Matthieu Zimmer
Huawei Noah’s Ark Lab · UCL Centre for AI
Research Digest··3 min read
In one line
Test-time self-distillation using counterfactual contexts and beam search improves language model generation without retraining.
Before you read this
Three ideas this paper builds ontest-time adaptationAdapting a deployed model during inference using new data or interactions, without a separate training phase.86 papers →counterfactual reasoningComparing actual outcomes with imagined alternatives to estimate the effects of different actions or conditions.117 papers →beam searchA search algorithm that expands multiple promising candidates simultaneously to balance generation quality and computational cost.7 papers →
Read it as
Tan and colleagues translate self-distillation from a training procedure into an inference-time decoding method.
The authors introduce test-time self-distillation, which uses two fixed counterfactual prompts rather than expert demonstrations.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.