Language models

Pretraining, architectures, post-training and instruction tuning, reasoning in language models, tokenisation, long context

339 articles · page 2 of 8

Language models

Giving harder inputs more attempts helps little on hard LLM workloads

The authors analyze test-time compute allocation for LLMs, showing that the common practice of giving harder inputs more attempts is near-optimal only on easy workloads. On hard workloads, it captures at most 13% of the gain possible with an oracle allocation. With a verifier and a prior from a related model's past pass rates, Thompson Sampling raises the solved share on the hardest workload from 14% to 27%; without a good prior, no policy beats equal allocation.

·3 min
Language models

Language models reason better when they think aloud only at key steps

The authors propose VITAL, a hybrid reasoning framework that adds a lightweight gating head to a language model, deciding at each step whether to generate an explicit token or feed the hidden state back directly. This lets the model verbalize only a few informative anchor tokens while keeping most computation silent. Across three model sizes, VITAL improved accuracy over fully latent methods at comparable reasoning lengths, and beat hybrid baselines with substantially shorter trajectories.

·2 min
Language models

A new loss function can keep fine-tuned chatbots from losing general abilities

The authors trace fine-tuning's tendency to erode a language model's general skills to geometric interference between features, and propose a training loss that controls that interference directly. Across 3 model families and 14 tasks, it beats representative methods from all existing mitigation paradigms on both target and general capabilities, at about 1.3 times the compute of standard fine-tuning.

·2 min
Language models

Near-zero token embeddings silently stall training in gated AI models

The authors identify a failure mode in gated sequence models (e.g., Mamba, Gated DeltaNet, xLSTM) where input embeddings with norms below 1e-6 can cause training to silently stall. These near-zero tokens, common in untied gated models, lead to gradient amplification of over 30 orders of magnitude, overflowing FP32 global-norm clipping and zeroing out all gradients in a batch. The paper proposes two simple fixes: GroupClip for released checkpoints and GateFloor for pretraining.

·3 min
Language models

Merging reasoning models layer by layer cuts answers by up to 93%

Yu, Zhang and colleagues propose Low-Rank Hierarchical Merging, a way to fuse a long chain-of-thought reasoning model with its base model so that answers get much shorter without retraining. Instead of setting per-layer blend weights from activation statistics, they estimate each layer's Hessian, a measure of how sharply the loss curves in that layer, and use it to decide which layers tolerate aggressive merging. On 1.5B and 7B models across six maths benchmarks they report gains over ACM-TIES and ORCA, including a 4.2% accuracy improvement over ACM-TIES at 1.5B and up to 93.1% shorter outputs than the long-reasoning model.

·3 min
Language models

Masking wrong turns in AI reasoning traces raises accuracy and shortens answers

Researchers at Baidu and the Chinese Academy of Sciences propose MicroMouse, a pre-training filter for supervised fine-tuning on long chains of thought. The insight is that even a trace ending in the right answer can contain bad teaching material: wrong turns and reasoning that continues after the answer is already within reach. Masking those steps from the training loss, while leaving them visible as context, improved accuracy and cut output length across three model sizes compared with training on the full trace.

·3 min
Language models

Distillation under-weights the tokens a model is most confident about

On-policy distillation trains a smaller language model on its own generated reasoning, with a stronger teacher scoring each token it produced. Cao et al. show that the usual sampled reverse-KL objective gives almost no gradient to the tokens the student is already confident about, even when the teacher still disagrees on them. Their replacement objective, which matches teacher and student probabilities directly for each sampled token, recovers that signal and improves reasoning scores over the standard method in four separate teacher-student configurations.

·4 min
Language models

A single prefilled token can substantially improve control over AI reasoning traces

The authors trace the inability of language models to control their chain-of-thought reasoning to an entropy collapse in the first few tokens after the opening thinking tag. By prefilling even a single token or banning common openers, they recover average compliance by 34–41 percentage points, but mainly for addition and style tasks. Suppression control shows limited improvement and correlates differently with monitorability, indicating distinct capabilities.

·2 min
Language models

Language model that indexes 200 million web pages outperforms embedding search for navigation

The authors introduce Octen-WebGR, a generative retrieval framework that stores web page identifiers in a language model's parameters. After indexing 201 million URLs and training on past clicks, the model outperforms embedding-based retrievers on overall navigational traffic and especially on unseen queries for pages with rich interaction histories. The 4-billion-parameter model runs on a single GPU with 10 GB memory, in contrast with the 412 GB required for exact dense retrieval.

·3 min
Language models

Several routers pick experts in language models, boosting accuracy without extra compute

The authors propose Multi-View MoE, a modification to mixture-of-experts language models that uses several independently initialized routers to score experts and reuses their relative preferences to mix learned input views for the selected experts. On a 10-billion-parameter model trained on 708 billion tokens, the full method raises the average over nine downstream tasks by 1.18 percentage points over the baseline, while preserving the active-expert count, expert FLOPs, and communication volume.

·3 min
Language models

AI mathematical reasoning improves more with textbook structure than with isolated problems

The authors built TOME-Math, a corpus that converts 2,118 university-level mathematics textbooks into roughly 380,000 proof-oriented question-answer pairs while preserving chapter order, proposition-proof links, and dependency references. Fine-tuning a 30B language model on this corpus lifted its mean score across six mathematical benchmarks from 29.17% to 45.83%, a gain of 16.7 percentage points, and beat three public instruction-data baselines in aggregate. Controlled experiments show that the structural signals, not just the extra data, drive much of the gain.

·2 min
Language models

Teacher rewrites of drifting student reasoning stabilize training and improve accuracy

The authors train a student language model on its own responses using token-level feedback from a stronger teacher, a setup known as on-policy distillation. To stop the student's reasoning from drifting as responses grow long, they replace each response's earliest high-disagreement segment with a short teacher-written segment, then let the student continue. This stabilizes training and improves student-only accuracy across five mathematical reasoning benchmarks and two teacher-student pairs.

·2 min
Language models

Structured tags help AI chatbots ignore hidden instructions in data

Shbita et al. present LLMON, a tagging framework that separates executable instructions from passive data in AI language models. After post-training Granite-4.1 base models at 3B and 8B scale on a 4.6 billion token structured corpus with supervised fine-tuning and reinforcement learning, indirect prompt-injection attacks succeeded in only 0.02-5.6% of cases, down from 53-100%, with general capabilities mostly intact.

·2 min
Language models

Language models’ training data rankings shift with irrelevant wording and formatting

The authors show that existing gradient-based data attribution methods for instruction-tuning large language models are sensitive to minor changes in wording or formatting, even when the content is unchanged. They propose Content-Invariant Gradient Attribution (CIGA), which selects an influential and stable layer for gradient features and fuses them with semantic representations, improving selection consistency and downstream performance.

·3 min
Language models

Quantization error doesn't determine semantic fidelity of discrete visual tokens; bit budget does

The authors challenge the widespread assumption that quantizing continuous visual features into discrete tokens inevitably causes semantic loss. Using multimodal understanding as a proxy for semantic fidelity, they show that quantization error is not a reliable measure, and that the effective bit budget of discrete indices is what drives downstream performance. They introduce Factorized Vector Quantization (FVQ), which keeps improving at larger bit budgets and ultimately matches continuous visual representations.

·3 min
Language models

Data curation advances LLM training only if a train-target gap exists

Wang et al. formalize two effects of data curation: advancement, which improves the attainable optimum, and acceleration, which reaches the same performance with less data. They show that true advancement requires a correctable train–target gap and propose the Δ-test, a diagnostic computable from a single baseline run. Their experiments on LLM training confirm that curation yields persistent gains only when this gap is nonzero.

·2 min
Language models

Language models’ reasoning steps reveal mistakes better than final answers alone

Chenyang Yu et al. introduce STEP, a training-free framework that scores each reasoning step of a large language model's output by re-encoding the partial trajectory and reading soft probability distributions over candidate answers. Across four closed-set benchmarks and two models, the resulting trajectory features improve AUROC for correctness prediction over output-based baselines and hard-indicator trajectory methods. The largest gains come from a temporally weighted consistency metric that emphasizes later steps.

·2 min
Language models

Constrained decoding does not inherently degrade LLM reasoning, contrary to common assumption

The authors revisit the assumption that constrained decoding inherently harms LLM reasoning. A theoretical proof and causal ablations across eleven open-source models show that apparent performance drops stem from prompt-level differences, not decoder-level masking. Validity constraints improve accuracy by up to 17.4%, while reasoning constraints help only when they enforce beneficial trajectories.

·2 min
Language models

Open-weights German-English model claims Pareto frontier for quality and serving cost

Aleph Alpha presents Kolibri, a 78.1B-parameter mixture-of-experts transformer with 3.46B active parameters per token, trained on 24T tokens including a German-heavy data mix and released under Apache 2.0. The authors report that among the open models they evaluated, Kolibri sits on the Pareto frontier of quality versus serving cost in both English and German, while covering regulation-sensitive domains such as public administration, industry and aerospace.

·3 min