Language models

Pretraining, architectures, post-training and instruction tuning, reasoning in language models, tokenisation, long context

339 articles · page 4 of 8

Language models

Restricting weight updates to non-principal subspace preserves old skills while learning new ones

Chavan et al. introduce Subspace Tuning (SST), a method that restricts each weight update to the bottom singular subspace of the weight matrix, preventing changes along the principal directions. Across SFT and self-distillation, SST preserves prior capabilities (e.g., reasoning, tool use) while matching or exceeding the target-task accuracy of unconstrained fine-tuning, with 2–4× less KL divergence from the base model.

Language models

Token-level teacher selection improves on-policy distillation for LLMs

The authors propose a framework for token-level teacher selection during on-policy distillation (OPD), where different teachers can supervise different tokens of a student rollout. They prove conditions for teacher-student compatibility and introduce representation-based routing, which outperforms confidence-based methods in producing diverse, high-quality student responses. Experiments across math, code, kernel generation, and visual reasoning show consistent improvements of up to 10 percentage points in pass@k over vanilla OPD and individual teacher experts.

Language models

Asymmetric compute allocation improves looped Mixture-of-Experts models

The authors probe the contributions of different loop visits in a looped Mixture-of-Experts (MoE) transformer, finding that the first visit benefits from a reduced residual scale while the second does not. They propose LatentLoop, which compresses the sequence in the first visit and reallocates the saved computation to increase the model's hidden dimension. Across 15 matched budget configurations spanning 1.3B to 7B parameters, LatentLoop consistently achieves lower validation negative log-likelihood than both vanilla MoE and the looped baseline, and improves average scores on nine downstream benchmarks.

Language models

Adaptive distillation triggers teacher takeover when student diverges

The authors propose Divergence-Triggered Dynamic Distillation (DTDD), which monitors student-teacher divergence during training and, when a segment diverges, rewinds to that segment and lets the teacher generate a recovery suffix that the student learns via behavior cloning. In long-horizon math, base-model distillation, and multi-turn agent settings, DTDD reduces training instability, accelerates learning, and improves task performance.

Language models

Top-k distillation's renormalized gradient ignores tail mass

Khanbayov and Kurban rigorously show that the renormalized top-k distillation objective has zero gradient along directions controlling tail-mass variation, so a student can match the teacher on shown tokens while its tail mass drifts arbitrarily. On seven open-weight teachers, recovery of the full teacher distribution becomes exact only when k equals or exceeds the teacher's hidden dimension plus one, a threshold far above the k values commonly cached in practice.

Language models

Condition-aware distillation reduces invalid reasoning in smaller language models

Luo et al. identify a failure mode in on-policy distillation: when a teacher evaluates a student-generated reasoning prefix, it can reward a continuation that follows the prefix but violates the original question. Their Condition-Contrastive Extrapolated On-Policy Distillation method reduced such erroneous preferences in a diagnostic set and improved mathematical reasoning across two student-model sizes.

Language models

Long-context midtraining improves coding agents after identical supervised tuning

Singhal and colleagues insert a 256K-context training stage between conventional pretraining and supervised fine-tuning, or SFT, using hundreds of billions of tokens from software repositories, development histories and agent interactions. After identical SFT, models receiving this midtraining generally solved more long-horizon software tasks, with the largest reported gain being 7 percentage points on SWE-bench Multilingual.

Language models

Reasoning models encode the future value of alternative next tokens

Yang et al. introduce relative long-horizon candidate value to quantify how different plausible next tokens at a reasoning prefix differ in their chances of eventual success. Using the RCTV dataset of mathematical reasoning prefixes paired with candidate tokens and empirical success rates, they train sparse linear probes on internal representations. The probes decode relative candidate value from hidden states, MLP outputs, and attention outputs across multiple layers, and outperform token probability, rank, and embedding baselines. In online intervention experiments, using probe signals to guide token selection improves continuation accuracy and prefix-conditioned Pass@k.

Language models

Computational graphs replace free-form reasoning for verifiable LLM outputs

Senatorov et al. introduce Structural Chain-of-Thought, a framework where large language models produce executable computational graphs instead of free-form reasoning traces. Trained via graph-based reinforcement learning (Graph-Based GRPO), the method consistently improves accuracy over supervised graph training across five benchmarks and four model families, while reducing generation length by up to 6.1× compared with standard GRPO. Faithfulness experiments confirm that answers derive from the generated graphs rather than post-hoc rationalizations.

Language models

Self-improving data synthesis adapts embedding models to new tasks with minimal human input

The authors propose SYNTRA, a method that generates task-specific training data for embedding models by iteratively improving synthesis instructions using feedback on trial outputs. Evaluated on 24 retrieval datasets across 15 domains, SYNTRA-generated data improved retrieval performance over an MS MARCO-trained baseline on all tasks, and automatic self-improvement outperformed human calibration on four of six tasks.

Language models

Entropy signals guide efficient reasoning repair in LLMs

The authors find that incorrect reasoning traces in large language models have mean token entropy roughly 112% higher than correct ones, a signal that can be used to localize errors. They propose REWIND, a controller that identifies high-entropy peaks, backtracks to the largest entropy gradient increase, and regenerates the suffix from that point. On five benchmarks and three model families, REWIND outperforms random-proposal MCMC methods by 0.51–10.00 percentage points without additional training.

Language models

Vision scales multimodal learning without altering pure-text loss

The authors trained native multimodal models with either real image embeddings or content-free padding, holding text supervision and training configuration constant across model sizes and compute budgets. They found that visual content leaves the pure-text loss frontier and compute-optimal allocations nearly unchanged, but reduces multimodal loss with a relative benefit that grows approximately as a power law of compute. Text-only spatial reasoning also improves with scale, with the most sustained visual gains in localization, navigation, and symbolic planning.

Language models

Reward-guided token weighting makes language-model distillation more effective

Chen et al. introduce Reward-Aligned Distillation (RAD), which tests whether token-level guidance from a stronger model is likely to improve the student’s eventual reward. At matched training-response budgets, RAD outperformed standard distillation and token-selection baselines on competition mathematics and coding benchmarks, although it requires additional optimization and reward-gradient computation.

Language models

Multimodal training needs different scaling recipes for text, vision and audio

Khandelwal et al. fit scaling laws to 268 multimodal pretraining experiments, spanning 457 million to 8.3 billion parameters and as many as 150 billion tokens per run. They find that text, vision and audio respond differently to additional model capacity and data, while interactions between modalities depend strongly on direction. Their framework also predicts losses for several larger, unusual-mixture and continual-pretraining runs with low error.

Language models

Iterative reasoning states help multimodal models refine their own answers

Yu and colleagues recast multimodal chain-of-thought reasoning as an iterative state-update process rather than a sequence of separately trained stages. Their ReState framework learns from empty, correct, erroneous and model-generated reasoning states, then repeatedly feeds its current reasoning back into the model at inference. Across several model backbones and public benchmarks, the authors report improvements ranging from 0.18 to 13.07 percentage points.

Language models

UltraBERT balances masking strategies to improve classification and retrieval

Park, Lee and Kang investigate bidirectional encoder pretraining at scales of up to 8.8 trillion tokens, focusing on masked language modeling, where models reconstruct hidden tokens. They report that downstream gains level off after roughly 1 to 2 trillion tokens, while the proportion of masked tokens determines whether an encoder is better suited to classification or retrieval. Their UltraBERT recipe combines both regimes in one model.

Language models

Smaller batches and adapted recipes improve repeated-data pretraining

This paper studies language model pretraining when unique data is finite and models must train repeatedly on the same data. The authors find that smaller-than-typical batches provide implicit regularization against overfitting, and that adapting training recipes such as repacking documents each epoch and tying embeddings further improves loss. They also show that under fixed time budgets, using fewer GPUs with smaller batches can achieve lower loss than maximizing parallelism.

Language models

Diffusion language models are cheaper to train than the widely cited 16x compute gap suggests

The authors revisit the scaling laws of masked diffusion models by training about 1,700 models across ten model variants. They find that the previously reported 16x training compute gap to autoregressive language models is substantially overstated; under a protocol that reflects current practice, the gap drops to 12.8x with exact likelihood, further to 7.8x at block size 32, and to 4.6x at block size 8 and 3.0x at block size 4.

Language models

Sparse attention mechanism achieves zero-shot extrapolation to longer sequences

Zekun Shi and Alastair James Arthur Price introduce FreeSolo, a sparse attention mechanism that uses mean summaries of learned token keys for hierarchical search and maximum token-key norms for pruning. Its linear-time variant FS-L and balanced-tree variant FS-B demonstrate zero-shot extrapolation from 4k to 16k context, with single-needle retrieval at 256k, outperforming baselines without long-context fine-tuning.

Language models

New scaling law better predicts losses from low-precision pretraining

Kim and colleagues develop a scaling law for predicting language-model pretraining loss when numerical precision varies between computational passes or Transformer layers. In experiments on Qwen3 and published Llama-style models, their formulation achieved up to 11 times lower prediction mean squared error than prior precision-aware laws and indicated that the compute-optimal precision rises with larger budgets.

Language models

Long reasoning distillation may teach behavior more than factual knowledge

Yu and colleagues test whether long chain-of-thought distillation depends on correct answers and broad problem coverage, two assumptions behind common rejection-sampling pipelines. They find that much of the improvement persists with flawed trajectories, repeated prompts, or already-mastered problems, supporting their hypothesis that distillation primarily aligns models with sustained, step-by-step reasoning behavior.

Language models

A model can become its own stronger teacher by looping selected layers

The authors introduce LoOPSD, an on-policy self-distillation method that constructs a stronger teacher from the same model by repeating selected layer blocks during inference. Alternating route search with distillation, they show that the student progressively absorbs the gains from this looped execution. On GSM8K, LoOPSD raises Qwen3-1.7B accuracy from 42.30% to 79.61% under a 400-token thinking budget.

Language models

On-policy distillation fails to improve models on problems they already get wrong

The authors tested on-policy distillation (OPD) on competition mathematics problems, restricting training to only those problems the student model currently failed. All eight configurations showed a decline or no improvement in accuracy, despite continued decreases in training KL divergence. In contrast, the same pipeline improved accuracy by 4.6 percentage points when training was not concentrated on failures.

Language models

Attention design cuts KV cache and compute while improving LLM quality.

The authors introduce Virtual Head Attention (VHA), which combines Q Premix and Linear Postmix to recover attention diversity from halved physical query heads. At inference both operations fold into standard projections, so VHA reduces to a GQA-2 model. Across pretraining runs from 0.6B to 30B-A3B MoE, VHA reduces KV cache and training FLOPs while improving downstream quality over existing attention mechanisms.

Language models

One-draw posterior updates achieve multi-sample reasoning gains at reduced cost

The authors formulate finite-K posterior learning to directly optimize candidate coverage in multi-sample reasoning. They propose a Fenchel-Bregman (FB) approach that replaces the K posterior draws required by direct Monte Carlo optimization with a single draw per update, using an exact reformulation and exponential running average of failures. Instantiating FB with the IVON variational optimizer yields FBI, which improves Pass@K across recursive and autoregressive reasoning benchmarks while substantially cutting training cost.

Language models

Training only the root token enables efficient adaptive reasoning

The authors introduce Root‑token Policy Optimization (RPO), a framework that allows large reasoning models (LRMs) to decide adaptively whether to engage in explicit reasoning by optimizing only the first token (the root token) that gates the reasoning process. This reduces training VRAM and time drastically while maintaining the model's core capabilities. Across multiple model families and scales, RPO learns difficulty‑aware adaptive thinking and improves the accuracy‑routing trade‑off under most scenarios.

Language models

A simple diagnostic predicts whether two AI models will work well together before costly combination

The authors propose COMPASS, a theoretically motivated score that estimates model-pair compatibility using two cheap-to-compute axes: ease (k-NN overlap) and ceiling (maximum overlap allowed by training paradigms). Tested on 72 cross-modal alignment experiments and 392 task-arithmetic merges, COMPASS achieves Pearson correlations up to 0.91 with downstream success, significantly outperforming prior metrics like CKA.

Language models

Slow oscillatory modes bottleneck recurrent reasoning; a rotational symmetry sidesteps them

Park and Rusch analyze looped reasoning as convergence in a learned dynamical landscape and identify a mechanism: a slowly decaying oscillatory mode can impose its convergence rate on coupled latent dynamics, delaying settlement near solution roots. They propose QRR (Quotient Recurrent Reasoner), a gauge-equivariant architecture in which per-token phase rotations leave the reasoning-relevant quotient state unchanged by construction. QRR converges to solution roots in quotient space faster and more stably than prior recurrent reasoners while matching or exceeding their accuracy on Sudoku, Maze and Mini-ARC.

Language models

A training recipe for looped language models that matches non-recurrent models at one loop and improves with more compute.

The authors present Elastic Latent Recurrence (ELR), a recipe for training looped language models that avoids the one-loop tax—the performance gap at a single loop compared to a non-recurrent model of the same size. At one loop, ELR models match or exceed their non-recurrent baselines; each additional loop then buys further accuracy. On 4B dense and 30B-A3B MoE models, four loops add 7.25 and 4.75 points respectively to the one-loop ten-task average.

Language models

Uncertainty-weighted credit improves cross-tokenizer distillation for LLMs

Jingru Yang and colleagues propose CAST, a shift-or-stay method for cross-tokenizer on-policy distillation. CAST solves credit alignment by distributing each aligned chunk's advantage according to student predictive uncertainty and local teacher-signal smoothness, while penalizing drift at tokens without teacher correspondence. On six benchmarks, it raises average scores over the strongest baseline by 2.45 points for Qwen3-8B to Mistral-7B transfer and 2.10 under shared tokenization.

Language models

Branching student trajectories improves on-policy distillation efficiency and accuracy

The authors propose BranchOPD, an on-policy distillation method that selects informative fork points using student uncertainty and teacher-student disagreement, then resamples student continuations from shared prefixes. Across mathematics and agent benchmarks, BranchOPD outperforms vanilla OPD and seven-rollout OPD while generating roughly 19% fewer student tokens on ALFWorld and 31% fewer on WebShop.

Language models

Raw video alone, with no captions, can mid-train language models for better vision.

The authors test whether a pretrained language model can improve its visual understanding by mid-training on raw video frames with no captions or text loss. They encode frames as continuous visual tokens and train the model to predict the next visual token, then apply identical instruction tuning to this model and a baseline. The mid-trained model outperforms the baseline on video and image benchmarks while matching text performance.

Language models

Composed matrix normalizations unify Muon optimizers and cut pretraining token use

The authors unify recent Muon optimizer variants as chained linear minimization oracles (LMOs), compositions of norm-ball projection steps such as the matrix sign. They show that many practical chains lack standard LMO guarantees and can diverge on convex objectives, yet in linear associative memory chaining provides a provable learning advantage. Their proposed TensorChain optimizer stacks compatible weight matrices across layers into a 3D tensor and normalizes across axes, achieving 9.6% average token savings over Muon at matched validation loss in Qwen3 0.6B and 1.7B pretraining.