News without the noise
Pretraining, architectures, post-training and instruction tuning, reasoning in language models, tokenisation, long context
66 articles · page 2 of 2
The authors pair a generator that writes programs with a learner that predicts the resulting byte sequences. Although neither model sees natural data during training, the learner’s zero-shot loss on several natural datasets improves with self-play compute and it develops in-context learning behavior.
Hu and colleagues introduce a neuro-symbolic framework for detecting logically invalid steps even when a model reaches the correct final answer. Across eight reasoning benchmarks and seven language models, they report improvements in both reasoning-chain verifiability and final-answer pass rates.
The authors adapted pretrained Qwen3.5 models at four parameter scales into diffusion language models, which generate by iteratively filling masked positions rather than strictly proceeding left to right. Despite the backbone’s structurally causal recurrent layers, the resulting dQwen3.5 models retained any-order and parallel decoding behavior while adapting more efficiently than a full-attention baseline.
Suleymanzade et al. propose looped flows, a method that trains recurrent neural networks for many-step inference using local denoising losses. This overcomes the difficulty of training early updates to support later ones when backpropagation is truncated. The authors report state-of-the-art accuracy among looped models on six reasoning benchmarks, including 58.8% on ARC-AGI-1 and 12.2% on ARC-AGI-2.
Jha et al. trained dense Transformers and mixture-of-experts models under varying data-repetition rates, domain mixtures, expert counts, and expert granularities. They find that MoE models lose performance more rapidly than dense models as examples are reused, with susceptibility tracking total parameter count rather than the number of parameters activated per token. Strong masking-based regularization preserves an MoE advantage beyond 64 repetitions, but does not match training on unique data.
Gao et al. developed VikingRAG, a retrieval-augmented generation system that navigates both the meaning and structure of documents while retrieving additional evidence only when needed. Across real-world datasets, the authors report token consumption of 11.6%–51.9% of state-of-the-art baselines, falling to 5.1%–32.5% when retrieval traces are reused and multi-round search is invoked selectively.
Zhang et al. introduce ConvMem, a training-free framework that uses prompted language models as query-dependent summarization kernels over segments of long documents. On two synthetic long-context, multi-hop question-answering benchmarks, the authors report better performance than training-free baselines and stronger out-of-distribution behavior than reinforcement-learning-trained memory methods.
Sahoo et al. introduce Uno, a model class that retains standard next-token training and its resulting autoregressive distribution but uses discrete diffusion to generate several tokens concurrently. Their Ψ-Spec samplers provide lossless acceleration without a separate draft model, outperforming tested speculative-decoding systems across evaluated batch sizes.
Li and colleagues compare ways to combine on-policy distillation (OPD), which supplies dense teacher feedback at the token level, with reinforcement learning from verifiable rewards (RLVR), which uses sparse outcome-based signals. Across logic and mathematics benchmarks, they find that completing OPD before switching to RLVR works consistently better than pure OPD, pure RLVR, or joint optimization.
The authors introduce the Graph Machine (GM), an architecture that maintains an O(n)-sized state and accesses it via sparse, dynamic routing using edges—pointer-like objects updated by a referral mechanism. They replace 75% of the dense Transformer layers in Qwen3-0.6B with GM sparse layers and pretrain from scratch on 15.7B tokens. With only 2 of 4,096 tokens retrieved per KV head per sparse layer, loss degrades only slightly; with 4 tokens, the best model marginally improves loss over the dense baseline.
Baek tests whether large language models can solve individual operations-research problems and design algorithms for entire problem classes. With one untuned prompt and access to a compute-limited Python sandbox, the strongest tested model matched or exceeded established methods on almost all evaluated instances.
The authors present VISA, an agentic framework that reformulates multimodal instruction synthesis as a self-evolving loop. VISA iteratively generates diverse and challenging instruction-following samples, using verifier signals and target-model failure profiles to guide subsequent rounds. The method consistently improves multimodal instruction following on MM-IFEval while maintaining general multimodal capability across seven benchmarks.
Muennighoff et al. observe that during long reasoning traces, most intermediate tokens lose importance as the model continues. They propose Prefix Sliding, which keeps only the instruction prefix and a recent window of tokens, discarding the rest to cap memory usage. Applied without training to existing models, it achieves up to 3x speedup with no performance loss; training with it enables reasoning traces beyond 100,000 tokens.
The authors distinguish two sources of parallelism in long reasoning traces: decomposing a problem into independent subtasks and exploring multiple speculative trials at once. They restructure sequential traces with a formal grammar, train models to balance accuracy and latency, and report about 1.7× average wall-clock acceleration on AIME24 and AIME25 while retaining competitive accuracy.
Liu et al. study effective learning rate (ELR), defined as the learning rate relative to parameter norm, as a common coordinate for language model pretraining. They report that runs with matched ELR follow closely aligned loss curves across optimizers, architectures, datasets, and model scales, with typical discrepancies smaller than measured seed-to-seed variation.
Tam et al. jointly trained a 4-billion-parameter Qwen3 model on tool creation and tool use across 13 procedural reasoning tasks. The resulting model achieved 79.8% macro-average accuracy on held-out tasks and transferred to tabular and visual question answering without training on either data type.
Kimi Team introduces Kimi K3, a 2.8T parameter Mixture-of-Experts model with 104B activated parameters, native vision, and a 1M-token context window. The authors demonstrate that innovations in attention (Kimi Delta Attention, Attention Residuals) and routing (Stable LatentMoE) yield roughly 2.5× scaling efficiency improvement over Kimi K2, enabling frontier-level performance across long-horizon coding, agentic, reasoning, and vision tasks.
The authors introduce LLM-as-a-Verifier, a general-purpose verification framework that extracts continuous scores from LLMs by taking expectations over scoring token logits. This approach achieves state-of-the-art results on four benchmarks—Terminal-Bench V2 (86.5%), SWE-Bench Verified (78.2%), RoboRewardBench (87.4%), and MedAgentBench (73.3%)—and provides fine-grained feedback that can be used for reinforcement learning and progress monitoring.