Tuesday 15 September 2026
Evening EditionRSS

ZOTPAPER

News without the noise

ZOTPAPER
Memory & Planning

Sliding reasoning windows make long test-time scaling 3x faster

Muennighoff et al. show that discarding stale intermediate tokens caps memory use while preserving performance, and RL training with the method extends reasoning beyond 100k tokens.

PaperTop Universitycs.CLarXiv:2608.26070v1 ↗
Niklas Muennighoff · Zhengyang Wang · Zeyi Chen · Weijia Shi · Binyuan Hui · John Yang · +12 more

Stanford University · University of California, Santa Barbara · Prime Intellect · University of Washington

Research Digest··2 min read
Read the paper →
Thread:Context Engineering for Agents
Muennighoff et al. observe that during long reasoning traces, most intermediate tokens lose importance as the model continues. They propose Prefix Sliding, which keeps only the instruction prefix and a recent window of tokens, discarding the rest to cap memory usage. Applied without training to existing models, it achieves up to 3x speedup with no performance loss; training with it enables reasoning traces beyond 100,000 tokens.

What they did

Muennighoff et al. analyzed attention patterns during long reasoning and found that intermediate tokens contribute progressively less to the model's current computation compared with the initial prompt/prefix and the most recent tokens. Based on this, they introduced Prefix Sliding: an inference-time method that keeps the fixed instruction/tool prefix plus a sliding window of the last few thousand tokens, discarding all other reasoning tokens. The authors evaluated the method both as a drop-in inference trick for existing models and as part of reinforcement learning training, comparing against baselines such as summarizing intermediate tokens and plain sliding-window attention.

Key findings

  • Applying Prefix Sliding to existing models without any training makes them up to 3x faster while maintaining performance.
  • RL training with Prefix Sliding allows test-time scaling to reasoning traces beyond 100,000 tokens, achieving better performance than standard RL training.
  • Ablations show Prefix Sliding outperforms both summarizing intermediate tokens and vanilla sliding-window attention.
  • The core empirical finding — that most intermediate tokens lose importance as reasoning progresses — directly motivates the method.

Why it matters

Capping memory during reasoning removes a major bottleneck for test-time scaling. Instead of quadratic memory growth with trace length, Prefix Sliding makes long-horizon thinking practical on existing hardware, potentially enabling frontier models to solve harder problems by reasoning much longer.

Caveats

The empirical observation that intermediate tokens lose importance may not hold universally across all tasks and model generations; the window size remains a hyperparameter that likely needs tuning. The reported gains are measured on specific benchmarks and settings, and the method's effectiveness on tasks requiring retrieval or re-examination of early reasoning steps is not fully addressed.

§

Analysis

Prefix Sliding sits at the intersection of memory compression and test-time scaling. It echoes classical sliding-window attention work but is applied at the level of complete reasoning traces, where the cost of full attention is especially acute. The RL-training result suggests sparse reasoning is learnable, which may encourage more work on training models to reason with bounded context rather than assuming full attention is necessary.

newspaper

Research Digest

Articles published under the Zotpaper byline are synthesized from multiple source publications by our AI editor and reviewed by our editorial process. Each story combines reporting from credible outlets to give readers a balanced, comprehensive view.

Editorial ProcessSubscribe