Language models

Pretraining, architectures, post-training and instruction tuning, reasoning in language models, tokenisation, long context

339 articles · page 1 of 8

Language models

AI forecasters can reason themselves into logical consistency

The authors show that LLM-based forecasting systems often produce logically inconsistent probability estimates for related events, such as assigning a higher probability to a narrow lead than to a wider lead for the same candidate. They develop a method that finetunes a supervisory language model to use the forecaster's reasoning to jointly resolve these inconsistencies and improve accuracy, achieving a 9.3% better Brier score and an 82-90% reduction in inconsistency on the ForecastBench benchmark.

·3 min
Language models

Multimodal training can make language models rely on vision-shaped reasoning for text-only spatial problems

The authors identified a vision-language axis in the hidden-state space of multimodal language models and removed its projection during inference on text-only tasks. This intervention selectively and substantially degraded performance on spatial-imagery tasks such as grid navigation and mental rotation, while leaving symbol-manipulation tasks like boolean logic nearly unaffected. The results provide causal evidence that multimodal training introduces or amplifies vision-shaped representations that the model recruits even when no visual input is present.

·3 min
Language models

AI teacher adapts its hints as a language model learns, boosting performance across four domains

Zhang et al. introduce AutoScaffold, a framework where a frozen language model teacher uses a student's training experience to design adaptive scaffolds, strategies for task classes and hints for hard tasks, that are removed at evaluation. Across ALFWorld, search-based QA, mathematical reasoning, and GPU kernel generation, students trained with AutoScaffold outperform those trained with standard RL baselines, with gains of 7.2 points in ALFWorld success and 4.0 points in average QA exact match, and on math, the student surpasses QuestA's reported accuracy using about one-third as many training steps.

·3 min
Language models

Visual encoding lets language models answer questions with 16 tokens instead of thousands

The authors present OpticalRAG, a framework that replaces text-based retrieval and compression in RAG with pixel-space encoding. Documents are rendered as images, encoded by a frozen vision encoder, and retrieved visual tokens are distilled into as few as 16 query-conditioned tokens. On three benchmarks, it improves the backbone LLM by over 5 points while using orders of magnitude fewer tokens than textual baselines.

·3 min
Language models

Encoder loop adds compute without growing memory in language models

The authors present a looped causal encoder-decoder (CED) that adds recurrent depth without increasing the KV cache memory footprint. By placing recurrence in the encoder and training with stochastic depth, they force representations to converge to a fixed point where only the last round's cache is needed. The method matches the quality of a fully looped dense Transformer while using one sixth of its global KV, and at 552B scale enables serving 38 parallel requests on hardware that could not fit a single naive loop.

·2 min
Language models

Particle decoding entropy reduces language model compute by a third

The authors show that Sequential Monte Carlo (SMC) decoding, used for power sampling in LLMs, naturally yields an estimate of sequence-level Rényi entropy at no extra cost. This entropy, particularly the order-1/2 variant, outperforms log-probability for selective prediction across four models. Moreover, a cheap 8-particle probe using this entropy identifies problems that benefit from more particles, enabling a two-pass policy that matches 64-particle accuracy with 34.2% fewer particles on average.

·3 min
Language models

Video reasoning models improve by auditing their own use of visual clues

The authors introduce VIOS, a post-training framework that lets a video reasoning model audit its own visual reliance by comparing next-token distributions with and without the video. This reveals tokens where the model under-uses visual evidence, allowing targeted training. Across 12 benchmarks, VIOS consistently improves three model backbones while requiring over 9x fewer training steps, 60x fewer FLOPs, and 111x fewer generated tokens than current methods.

·3 min
Language models

Sparse attention for long-context AI verifies its own quality in one pass

Li and Lee present Coverage-Verified Sparse Attention (CVSA), a closed-loop decode method that uses a scalar already computed inside the sparse-attention kernel, the fraction of softmax mass on selected tokens, as a quality signal. Without sampling, draft models, or speculative decoding, the method accepts high-coverage outputs and recovers only the heads that fall short, achieving median speedups of 6.21× at 128K context while coming within 1.13 points of dense attention on LongBench at 7B and statistically matching it at 70B.

·3 min
Language models

AI agents can improve long-context reasoning without costly retraining

The authors propose LongCapHarness, a framework that enhances long-context reasoning in large language models without requiring data synthesis or post-training. By recursively building and refining capability harnesses on a development set, LongCapHarness enables smaller models to outperform larger, trained baselines on HELMET and RULER benchmarks, with development costs estimated to be less than half those of existing methods.

·2 min
Language models

Directly optimizing acceptance length boosts speculative decoding throughput by up to 9.4%

Speculative decoding speeds up large language models by having a small draft model propose tokens that a larger model verifies in parallel. The speedup depends on how many draft tokens are accepted consecutively (acceptance length), but standard training objectives don't directly optimize this. D-REAL introduces a post-training loss that directly maximizes expected acceptance length using an unbiased gradient estimator, yielding consistent gains of up to 0.23 tokens and up to 9.4% higher serving throughput.

·2 min
Language models

Merging specialist AI models saves 42% of training compute compared to joint training

The authors systematically compared on-policy distillation (OPD) and group relative policy optimization (GRPO) across math and code tasks, measuring accuracy against training FLOPs. They found that while external-teacher OPD generally underperforms GRPO, OPD using the student's own GRPO checkpoint as teacher recovers that checkpoint's accuracy with 50 times fewer distillation FLOPs. The key practical result is multi-teacher OPD: consolidating per-domain GRPO specialists into one student adds only 5.2-8.0% extra training cost, and the full pipeline uses 42% fewer total FLOPs than joint GRPO for two domains.

·3 min
Language models

Same answer from different prompts does not mean same belief updating in language models

The authors tested three open-weight language models on a coin-flip bias inference task, preparing beliefs through four textual routes (observed sequences, count summaries, source assertions, instructions) and two activation interventions (steering, state transplantation). By appending identical new evidence and measuring how strongly the initial belief persisted, they found that the format of input decides which model output (continuation vs. explicit probability) reflects the evidence, and that different routes give the belief vastly different effective weights, strong enough that a matched immediate prediction does not imply matched revision under future observations.

·3 min
Language models

Irrelevant context degrades language models via predictable attention heads

The authors trace how irrelevant context, coherent text from unrelated domains, reduces language model accuracy. They find that the degradation arises from a sparse set of attention heads that distribute distractor content and weaken direct support for the correct answer, separate from heads that process helpful context. A training-free intervention that masks these heads recovers over half of the lost performance.

·3 min
Language models

Distilling AI reasoning works better when teacher rewards student's own learned math steps, not style

The authors identify that in on-policy distillation for math reasoning, teacher feedback often focuses on style (e.g., formatting, wording) rather than mathematical correctness, with style tokens getting 3.5 times larger error signals. They propose SD-OPD, which uses the difference between the student and its pretrained base to weight updates, with the teacher only deciding which directions to reinforce. Across eight billion-parameter models, this improves math benchmark averages by 7.76 to 10.37 percentage points over standard on-policy distillation.

·3 min
Language models

Data coverage beats difficulty for choosing language model fine-tuning data

Pandia et al. show that popular difficulty-based data selection methods consistently fail to beat random sampling when fine-tuning large language models, contradicting their success in smaller classification tasks. The reason, the authors argue, is that random selection inadvertently provides better coverage of the training data distribution, and coverage matters more than difficulty for LLM fine-tuning.

·2 min
Language models

Decoupling direction and distance improves on-policy distillation for language models

The authors propose D2-OPD, a method that separates the extrapolation direction from the distance in on-policy distillation for language models. This prevents the target from over-amplifying weak signals and incorporates both the teacher's post-training shift and the initial student–teacher gap. Evaluated on three settings with Qwen3 teacher–student pairs, D2-OPD achieves the highest average scores on AIME24, AIME25, and AMC23, outperforming the strongest baseline by up to 9.45 points.

·2 min