Language models

Pretraining, architectures, post-training and instruction tuning, reasoning in language models, tokenisation, long context

339 articles · page 6 of 8

Language models

Continuous diffusion language models match discrete peers while enabling smoother control

Yang et al. introduce Sigma, a blockwise language model that generates through continuous diffusion rather than predicting tokens autoregressively or denoising categorical states. Across math and coding evaluations, the authors report performance comparable to similarly sized discrete diffusion and autoregressive models, while continuous trajectories provide direct control over output quality, diversity and inference steps.

Language models

LLM layers follow a three-stage functional segmentation, fine-tuning only the critical bottleneck layer boosts performance

The authors hypothesize that LLM layers are organized into three contiguous stages—conceptualization, reasoning (with active and idle sub-stages), and textualization. They validate this by tracking hidden state similarity across semantically equivalent cross-lingual queries. Building on this, they develop a sensitivity analysis using quantization to identify the bottleneck stage for a specific task, and introduce Layer-Informed Fine-Tuning (LIFT), which updates only those critical layers. Experiments show that LIFT accelerates training and improves model performance on reasoning tasks compared to full fine-tuning.

Language models

Decoding as optimisation on the probability simplex unifies and composes sampling strategies

The authors reframe large language model decoding as an optimisation problem on the probability simplex, where the goal is to maximise expected model score subject to regularisers that encode distributional preferences. They show that existing decoding algorithms (greedy, softmax, Top-K, Top-P, sparsemax) are special cases of this template, and that composing regularisers via a weighted sum yields new decoders that balance quality, diversity, and coverage without external rewards or model updates. They release CompoSimplex, a library for building and evaluating such compositional decoders.

Language models

Projecting wayward reasoning steps back into a focused latent region cuts token use and boosts accuracy

The authors observe that in large reasoning models, efficient reasoning steps occupy a concentrated region in the latent space while verbose steps deviate. They propose a training-free method that, at each step, projects the hidden state onto the nearest point in that region via quadratic programming. Across four models and six benchmarks, the method reduces generated tokens by 11.8–52.8% and improves accuracy by up to 12.1%.

Language models

A recalibration method stabilizes LLM self-distillation by treating correct and incorrect outputs differently

The authors propose FIRE, a dual-branch framework for fine-tuning large language models (LLMs) using their own outputs under external feedback. It distinguishes correct from incorrect responses, applying re-weighted supervised fine-tuning to correct ones and recalibrating feedback targets for incorrect ones using a token-level radius derived from the softmax Fisher trace. The method stabilizes training where standard feedback-conditioned self-distillation becomes unstable, while maintaining competitive downstream performance.

Language models

Contextualization principle: informative words are sources, less informative are sinks

Cho et al. propose a general principle of contextualization in transformer language models: words carry varying amounts of static information, and during contextualization, less informative words absorb more information from more informative words, with finer-grained selectivity for matched token pairs. They validate this principle through observations of model outputs and hidden states, then explain its mechanism by reinterpreting attention heads as channels whose singular vectors control information routing and enable automatic head interpretation.

Language models

Continual learning's core challenges stem from a single evolving update-behavior interaction

Ren et al. develop a structured decomposition of how learning from one token changes a model's prediction on another, separating softmax forces, shared readout geometry, and residual connections. They show this local interaction governs data selection, interference mechanisms, and long-term learnability across models and training regimes, yielding practical diagnostics and interventions.

Language models

Routing-signature diversity prevents expert collapse and boosts MoE specialization

The paper proposes Distributional Orthogonalization Loss (DO-loss), a regularization that records each expert's token assignment history as a binary signature and penalizes overlap, directly encouraging functional specialization and self-correcting expert collapse. The authors also introduce Replica Expert Mechanism (REM), which replicates hot experts onto underloaded ranks and routes tokens via shortest-queue dispatch, restoring load balance without constraining specialization. On 4.8B and 30B MoE models, the combined method outperforms existing routing algorithms on six benchmarks while maintaining comparable training efficiency.

Language models

Coupling continuous latent with token feedback improves diffusion language models

Ren et al. propose Hierarchical Continuous Diffusion Language Models (HC-DLM), a generative framework that combines a continuous latent trajectory with discrete token readouts at every denoising step. The tokens are re-noised and fed back as conditioning for the next latent update, yielding a single variational bound on the token likelihood. On Sudoku puzzle solving, Countdown arithmetic planning, and LM1B language modeling, HC-DLM outperforms both discrete and continuous diffusion baselines at matched model size, with accuracy gains of up to 10 percentage points on reasoning tasks and lower perplexity on language modeling.

Language models

Single queries can predict useful weight update distributions for LLM adaptation

The authors introduce distributional hypernetworks, which predict a distribution over LoRA weight updates conditioned on a single input query. They show that even the mean of the learned distribution outperforms deterministic hypernetworks by 4.6–11.7% across nine benchmarks. Crucially, sampling multiple weight updates from the distribution provides a new form of test-time scaling that competes with or exceeds traditional token-sampling methods.

Language models

Semi-autoregressive continuous diffusion models match discrete baselines and enable KV caching.

The authors introduce Clock Diffusion, a semi-autoregressive framework for continuous diffusion language models that denoises token embeddings in blocks or sliding windows according to position-dependent noise schedules. ClockDLMs achieve state-of-the-art perplexity among continuous diffusion language models on OpenWebText, surpassing even block discrete diffusion baselines, and outperform continuous baselines on GSM8K. Their Cache Grab samplers accelerate inference by committing high-confidence tokens and applying self-speculative decoding.

Language models

Mixture of experts unifies time-series forecasting and reasoning in one model

The authors introduce OpenTSLM TeeMoE, a generalist time-series language model that combines numerical forecasting, context-conditioned prediction, and temporal reasoning within a single Qwen3.6-27B backbone. Three separate LoRA experts are trained independently with different objectives and data, then a mixture-of-experts controller learns to weight their contributions per request. The model ranks among the top three on the GIFT-Eval, Context is Key, and TimeSeriesExam benchmarks, demonstrating that specialist performance can be preserved while gaining broad capability coverage.

Language models

Diffusion models beat autoregressive LLMs on globally constrained problems

The authors show that masked diffusion language models can outperform autoregressive LLMs on tasks requiring global constraints. Working with a fixed LLaDA-8B-Instruct checkpoint, they use mean confidence to guide iterative revision and search over a fixed canvas. The approach reaches 90.4% accuracy on ZebraLogic-Hard and beats frontier models such as GPT-5.4 and Gemini 2.5 Pro on two of three benchmarks.

Language models

Average generator yields 16x faster discrete diffusion language model sampling

Ouyang and colleagues introduce the Discrete Average Generator, a MeanFlow-style construction for discrete state spaces. The method defines an average transition generator over time intervals, uses a self-consistency identity, and projects the training objective onto per-coordinate marginals. It gives the lowest perplexity at 8 to 64 sampling steps on OpenWebText with 16x fewer steps, and reduces sampler total variation by up to 67% in simulations.

Language models

Supervising token-level loss improves routing in sparse mixture-of-experts models

The authors introduce token-error supervision (TES) and affinity-concentration supervision (ACS), two methods that directly align sparse routing decisions in mixture-of-experts language models with the realized next-token cross-entropy loss. On the Granite MoE backbone, ACS improves accuracy by 2.1 percentage points over standard fine-tuning, while TES outperforms a parameter-matched baseline by 2.3 points, with gains on ARC-Challenge reaching 2.94 points.

Language models

Gradient decomposition supplements LoRA updates to match full fine-tuning performance

Ouyang et al. characterize the gradient space accessible by LoRA's low-rank factors and identify a complementary "normal gradient" component that is orthogonal to it. They propose GDLoRA, which reconstructs the full gradient for LoRA-injected layers, extracts this normal component, and applies it directly to the base weights while retaining standard AdamW optimization for the LoRA factors. Experiments on natural language understanding, mathematical reasoning, commonsense reasoning, and image classification show that GDLoRA consistently outperforms LoRA by 2–4% and approaches the performance of full fine-tuning without additional optimizer-state memory.

Language models

Synthetic warm-up training scales, but does not teach language grammar

Yamaguchi and colleagues tested whether pre-pretraining language models on synthetic, non-natural sequences remains useful at substantially larger scales and training budgets than previously studied. They report persistent downstream and token-efficiency gains, including savings of at least 21 billion pre-training tokens for 3-billion-parameter models, but find no consistent support for the proposed explanation that synthetic training supplies a transferable grammatical prior.

Language models

Self-distillation choices govern learning gains and preservation of prior abilities

Zuin and colleagues separate self-distillation with privileged context into three design choices that prior studies often bundle together: who generates training rollouts, how closely the teacher follows the student, and which direction of Kullback-Leibler divergence is minimized. Testing these choices independently shows that no single configuration universally maximizes acquisition of a new task while retaining pretrained capabilities.

Language models

A synthetic-trained model predicts across tabular datasets without retraining

Kong et al. introduce TabFM, a 400 million-parameter model that treats supervised tabular prediction as an in-context learning problem. Trained only on synthetic tables generated from structural causal models, it uses labeled examples supplied at inference time to predict new rows in one forward pass, without updating its weights. The authors report leading aggregate results across all 51 datasets in the TabArena benchmark.

Language models

Training on generation trajectories makes masked diffusion language models more efficient

Madeira and colleagues address a mismatch in masked diffusion language models: training uses independently sampled masks, while generation follows sequences of states shaped by the model’s own predictions. Their PUMBA framework trains across these trajectories, carries continuous information between denoising steps and propagates gradients through multiple steps, reducing the number of model evaluations needed at matched performance.

Language models

Teacher alignment method closes capacity gap in reasoning distillation without discarding data

The authors propose Teacher Alignment to address the Gap Curse in reasoning distillation, where larger teacher models produce distributions too complex for smaller students to approximate. They develop TeacherGRPO, built on Group Relative Policy Optimization, which adapts the teacher via curriculum selective alignment and importance-adaptive length regularization. Experiments show consistent performance gains over existing baselines across diverse reasoning benchmarks.

Language models

Language models can suddenly and repeatedly switch between pattern-matching and genuine generalization during pre-training

The authors developed a behavioral test suite to distinguish whether language models rely on shallow patterns or true generalization. Tracking models across pre-training checkpoints, they found that models frequently and abruptly alternate between these modes, often within tens of billions of tokens. This mode-hopping is not an artifact of noise or metric selection and cannot be smoothed by checkpoint averaging.

Language models

Small models can steer stronger ones through shared reasoning traces

Liang and colleagues introduce Allspark, a framework for transferring reasoning behavior from a reinforcement-learned small model to stronger models through alternating text-based chains of thought. In experiments using Qwen models and ARC-AGI-2, the trained weak teacher improved some within-family and cross-family students, although the benefit depended on the inference configuration and required additional tokens.

Language models

Geometry-aware LoRA optimization improves convergence across supervised and reinforcement learning

Ding, Zazo and Hensman introduce Rotated Manifold Optimization, or RoM, an optimizer designed around a symmetry in low-rank adaptation: many pairs of LoRA factors represent exactly the same weight update. Across supervised fine-tuning and reinforcement learning experiments, RoM converged faster and reached lower held-out loss than the compared LoRA optimizers, while producing better or comparable downstream performance.

Language models

Future-aware distillation corrects teacher-student information mismatch in block diffusion language models

The authors introduce d-OPD, a future-aware on-policy distillation method for converting autoregressive LLMs into block diffusion language models. It addresses a fundamental mismatch: the block-diffusion student sees future context within a block, but the standard AR teacher distribution does not. By aligning the teacher distribution with the student's visible state, d-OPD improves average benchmark scores by up to 4.0 points and reduces training time by about 1.35-1.58x on Qwen3 models from 0.6B to 8B.

Language models

Early answer confidence reveals when language models take reasoning shortcuts

Zhaohan Zhang and colleagues propose ConfLens, a framework that monitors how an LLM's confidence in its final answer evolves during chain-of-thought reasoning. They find that shortcut reasoning consistently shows a pattern of premature confidence, where the model commits to an answer early. Their new metric, the Distributional Answer Commitment Score (DACS), detects this reliably across math and code tasks, improving shortcut detection F1 by over 4.3% over strong baselines.