Efficiency & systems

Quantisation, inference serving, kernels, distributed training, compression, efficient architectures

240 articles · page 3 of 5

Efficiency & systems

Shared KV-caches cut looped-model inference memory without quality loss

Prairie, Ryan, Berg-Kirkpatrick and Fu investigate why some looped architectures can share key-value caches across recurrences without quality loss, while identically trained models cannot. They propose stochastic depth pretraining followed by BeNice!, a post-training step that makes KV-sharing reliable, and demonstrate gains in end-to-end quality, compression compatibility and inference efficiency.

Efficiency & systems

Language models can be trained for compression nearly 13 times faster

The authors introduce SQuAT, a quantization-aware training method for lattice-based vector quantization of LLM weights, decomposing each weight block into a normalized direction and a scalar gain. Training with separate straight-through estimators for shape and gain, they scale exact-codebook VQ-QAT to models up to 4B parameters and 10B tokens, outperforming standard joint-STE VQ-QAT and matching or beating scalar INT-QAT at 2.25 bits per weight.

Efficiency & systems

Language models can give identical answers up to 2.78 times faster

The authors propose ValuSpec, a candidate valuation mechanism that filters low-quality branches from a candidate tree before target-side verification. It introduces transfer-based supervision to calibrate a filtering threshold using the target model's greedy trajectories, requiring no parameter training. The main finding is up to 2.38x end-to-end speedup in retrieval-based tree speculative decoding settings.

Efficiency & systems

Rewriting softmax calculations fixes two rounding errors that distort training

Jonathan T. Barron, Dan Gnanapragasam and Michael Tschannen identify two floating-point errors in standard softmax and cross-entropy implementations, functions that turn model scores into probabilities and training losses. Their replacements preserve small gradient signals without higher-precision arithmetic and, in the reported experiments, remove training instabilities and unwanted drift during knowledge distillation.

Efficiency & systems

Lower precision beats structured pruning until GPU quantization limits are reached

Xiuyi Qin compares two ways to accelerate large language models: reducing numerical precision and imposing 2:4 sparsity, which retains at most two values in each hardware-defined group of four. In the tested configurations, lower precision was both faster and more accurate whenever another precision step was available. At the hardware’s lowest precision, sparsity delivered additional speed, but preserving model quality required retraining.

Efficiency & systems

Reusing target probabilities in speculative decoding can improve tree selection but requires careful trade-offs.

The authors investigate when next-token probabilities from a target model's verification pass can be reused to select the next tree of draft tokens. They find that while exact reuse is only guaranteed under restrictive conditions that cost accepted tokens, using a separately computed probability distribution for the first token (root row) improves acceptance rates and decoding speed. In experiments across six language models, both root selection and root-draw selection outperform draft-based ranking.

Efficiency & systems

Vector codebooks preserve useful language models at roughly two bits

Sukthanker and colleagues tested post-training quantization across six dense and mixture-of-experts language models ranging from 3.8 billion to 32 billion parameters. They find that the weight representation is decisive at two bits: vector codebooks preserve usable models, while every tested scalar pipeline falls to chance-level accuracy. Calibration and recovery datasets also need to be selected jointly, especially for generative tasks.

Efficiency & systems

Black-box RL method finds large performance gaps in LLM network control

Alizadeh et al. introduce LENS, a black-box performance analyzer that uses reinforcement learning to search for inputs that cause LLM-based network control systems to perform poorly. They demonstrate that LENS can find traffic matrices that induce a performance gap of over 50% for LMTE, a traffic engineering system, and a 28.5 times larger gap than any seen in test sets for a GPU allocation solver.

Efficiency & systems

Selective high-precision layers close the accuracy gap in four-bit training

Agrusa and colleagues systematically compare recipes for training language models with NVFP4, a hardware-supported four-bit floating-point format, against conventional BF16 training. Across dense 8-billion-parameter and 30B-A3B mixture-of-experts models trained for as many as one trillion tokens, they find that combining several error-control methods narrows the loss gap, but selected layers must remain at higher precision for reliable accuracy recovery at scale.

Efficiency & systems

Per-layer KV-cache budgets learned via multi-objective search improve LLM inference efficiency

The authors introduce MOSAIC, a multi-objective search method that learns per-layer key-value cache budgets for LLM inference. On two models and four eviction methods, MOSAIC beats uniform allocation in 146 of 154 (task, method, budget) settings, achieving gains of up to +48 points on RULER and +11.9 on LongBench. It reaches the same score as uniform allocation with up to 4x less cache.

Efficiency & systems

Jointly co-compiling draft and target models boosts speculative decoding speed

The authors introduce Conjugate Compilation, a method that reframes speculative decoding as a cross-model compilation problem by jointly optimizing the small draft model and the larger target model within their shared execution on a single GPU. A self-improving agentic system couples an LLM with Monte Carlo tree search, proposing and evaluating cross-model plans based on actual co-located GPU performance, achieving speedups up to 1.59x and improving GPU utilization by up to 1.50x.

Efficiency & systems

Speculative decoding throughput gains are conditional on workload and serving conditions

The authors systematically evaluate when additional acceptance length in speculative decoding is worth the computational cost. By remeasuring released checkpoints and retraining six drafter recipes under shared conditions on two target models, they find that published method gaps shrink substantially. They identify two routes to improve acceptance—nearly free context/objective changes and more expensive token-to-token conditioning—and show that throughput gains depend on task entropy and serving concurrency.

Efficiency & systems

KV-cache eviction method is provably near-optimal, not just heuristic

DropKV addresses the KV-cache memory bottleneck in long-context LLMs by turning the NP-hard problem of minimizing output perturbation from cache eviction into independent per-token scores. The authors prove this decoupled proxy is within a constant factor of the optimal combinatorial solution under a cone condition they verify empirically, and implement it in fused Triton kernels. Across RULER and LongBench at 5% and 10% budgets, DropKV beats existing eviction baselines in accuracy and latency.

Efficiency & systems

Differentiable soft pruning makes transformers elastic across computational budgets without retraining

Simoncini et al. introduce a retraining-free method that makes pre-trained transformer encoders elastic by learning individual importance scores for every MLP neuron and attention head via differentiable soft pruning across many sparsity targets simultaneously. The method produces a single model that can be pruned to any sparsity level up to the deepest training budget, outperforming one-shot baselines. For example, pruning DINOv3 ViT-H+/16 to 50% sparsity removes 427M parameters with only a 0.8% drop in ImageNet-1k linear probing accuracy.

Efficiency & systems

Hybrid drafters split depth and breadth to cut mobile-cloud GPU costs

Bae et al. (Samsung Research) propose HyCo, a hybrid drafting scheme for device-cloud speculative decoding. By splitting drafting between the server LLM's native multi-token prediction (MTP) head, run on the device, and one pass of a small language model for alternative branches, they cut server GPU time per output token by 38.6% on Spec-Bench and 29.2% on long summarization versus server-side MTP self-speculation, while keeping user latency competitive.

Efficiency & systems

Functional supervision improves cross-model KV cache transport over reconstruction alone

The authors study zero-prefill cross-model context handoff, where a sender's KV cache is translated for direct receiver use. They find reconstruction fidelity is a poor proxy for receiver utility, as similar K/V errors can produce different attention and predictions. They introduce CacheLink, which trains the transported cache through functional supervision from the frozen receiver, consistently outperforming prior methods across four model pairs and seven datasets while achieving up to 7.27x latency reduction.

Efficiency & systems

E8 lattice vector quantization enables 2-bit keys for LLM KV-cache compression

The authors apply E8 lattice vector quantization (VQ) to compress key vectors in LLM KV caches to 2 bits per dimension, achieving roughly 6x total compression when combined with 2-bit values. Across five LLM families, the scheme yields lower long-context perplexity than scalar 3-bit keys in 80% of evaluated settings and matches the decode throughput of scalar baselines through a table-based asymmetric scoring mechanism.

Efficiency & systems

Recurrent-state quantization fails at long decode horizons due to scale-state feedback

The authors study training-free INT4 quantization of the recurrent state in linear-attention (Gated DeltaNet) and state-space (Mamba-2) language models. They find that while recall remains perfect at short horizons, perplexity rises eightfold after a thousand decode steps due to a feedback loop between the quantizer scale and the state norm. Yoking the scale to a full-precision reference eliminates the numerical instability but recovers only 1.8 of the 5.4 nats lost, showing that numerical and functional failures are separable. The results show that group-wise or persistent scaling outperforms all outlier-aware schemes, and that per-layer bit allocation collapses to protecting the first few layers.

Efficiency & systems

Echo enables fast 4-bit tensor-parallel all-reduce with controlled error accumulation

The authors present Echo, a 4-bit all-reduce for tensor parallelism that bounds quantization error accumulation across layers. Echo moves data in 4 bits but sums in high precision, and feeds back each rounding error to cancel in later calls. On A100 GPUs, it achieves up to 2.09x speedup over 16-bit NCCL all-reduce, with average accuracy within 0.4 points of 16-bit inference on Qwen3-8B and final pretraining loss within 0.9% of 16-bit.

Efficiency & systems

TempoKV compresses video generation KV caches by exploiting key-value predictability asymmetry

The authors propose TempoKV, a training-free quantization framework that compresses the KV cache, the growing store of attention keys and values, in autoregressive video diffusion models. They exploit an asymmetry: keys are predictable from their temporal history, while values are better predicted from the aligned current key plus a short history. Storing only low-precision residuals reaches up to 6.71x compression while improving generation quality over existing quantization methods.

Efficiency & systems

Near-square experts and depth-aware allocation improve MoE models

The authors study expert shape in LatentMixture-of-Experts (MoE) models, showing that when a fixed parameter budget is split between an expert's input dimension and intermediate width, a near-square balance maximizes the rank of the expert's Jacobian. Guided by layerwise intrinsic dimension (ID) estimates, they further allocate per-expert capacity across depths while keeping per-layer parameters and compute constant, achieving lower validation loss and stronger downstream performance than homogeneous baselines.

Efficiency & systems

Two-bit scalar quantization better preserves language model predictions

Chen and colleagues present FidelityQ, a framework for compressing language-model weights to two bits, giving each weight only four possible reconstruction values. The method reshapes weight distributions, chooses reconstruction levels row by row, and calibrates on sequences where the quantized model most strongly disagrees with its full-precision teacher. On LLaMA-2-7B, it approaches stronger codebook methods while using a regular scalar representation.

Efficiency & systems

Dynamic GPU Shifting Speeds Language Model Reinforcement Learning Post-Training

Zhao et al. present PivotRL, a system for reinforcement learning post-training in which GPUs can switch dynamically among policy rollout, generative reward evaluation and policy updates. In experiments on 128 GPUs across four dense and mixture-of-experts model groups, the authors report substantially higher end-to-end throughput than systems using fixed allocations, without changing observed training convergence.

Efficiency & systems

Reusing Ranking Blocks Cuts Communication While Preserving Short-Video Quality

Jiang and colleagues tested parameter-shared iterative computation in a production, multi-task short-video ranking system. Their compressed model remained within 0.03% of the baseline’s offline quality while using 26.6% fewer trainable parameters and delivering 6.4% higher training throughput. The gains primarily came from reducing distributed communication and optimizer work, rather than from reducing computation.

Efficiency & systems

Method converts LLMs to efficient attention while handling QKNorm and MoE

The authors present MLAfication, an extension of the MHA2MLA framework that adapts pretrained LLMs using multi-head or grouped-query attention to the more efficient Multi-Head Latent Attention (MLA) architecture. Crucially, it handles QKNorm and Mixture-of-Experts (MoE) components, which prior methods could not support without quality loss. Across four model families, MLAfication preserves accuracy on commonsense and long-context tasks while enabling substantial KV-cache compression and up to 5.35 times the serving throughput of the original BF16 GQA baseline.

Efficiency & systems

Local alignment enables predictive coding for error-bounded KV cache compression

The authors present AlignPred, a codec for persistent KV caches in long-running LLM agents. By applying local orthogonal transforms to align coordinate frames of related attention heads or layers, the method exposes cross-block predictive redundancy that can be exploited with fixed arithmetic, followed by error-bounded lattice quantization and entropy coding. Across three attention mechanisms, AlignPred requires less storage than evaluated baselines at comparable reconstruction error, decodes at least 1.6× faster in CPU implementations, and reduces persistent KV storage by 42–72% in an SGLang HiCache integration with comparable latency.

Efficiency & systems

Low-rank KV cache compression aligned to attention distributions

The authors propose BELT, a low-rank key-value cache compression framework for large language models that preserves attention behavior rather than raw key or pre-softmax logit fidelity. A differentiable rank allocator distributes a global component budget across heads, and experiments on three LLMs show up to 70% compression while improving LongBench scores by up to 23.3 points over the strongest baseline.

Efficiency & systems

Curvature-aware pruning preserves influence functions efficiently.

Makni et al. introduce CACTUS, a curvature-aware pruning algorithm that jointly optimizes layerwise reconstruction and gradient alignment via an ADMM solver. On Qwen3-4B at 50% pruning, CACTUS reduces the C4 perplexity gap to the dense model by 45%, improves per-sample gradient cosine similarity from 0.670 to 0.755, and runs over 28 times faster than LLM-Surgeon on substantially weaker hardware. The preserved influence functions enable effective data selection: fine-tuning Llama-3.2-3B on 5% of data selected by CACTUS achieves 49.44% on BIG-Bench Hard, nearly matching dense-model selection.

Efficiency & systems

Kronecker product factorization compresses key-value caches up to 10x without accuracy loss.

The authors propose Kronecker Product Attention (KPA), a structured attention mechanism that caches keys and values as compact Kronecker product factors. Trained from 124M to 1.5B parameters, KPA matches or beats prior KV compression methods while shrinking the cache by 6.4x to 10x. It also introduces a Kronecker-separable rotary embedding that preserves relative positions without a decoupled RoPE branch.

Efficiency & systems

Two-rate precision allocation cuts MoE memory footprint and decode traffic

The authors identify a store-read gap in Mixture-of-Experts (MoE) models, where a fixed-precision checkpoint stores weights at one rate but decode reads them at a much higher rate (2.95× for BitsMoE). They propose DuoRate, which stores one uniform tier of a prefix-readable weight stream offline and then, for each workload, selects per-expert read tiers via a Lagrangian rate-distortion optimization that cancels routing frequency. On Qwen3-30B-A3B, DuoRate reading 2.3 bpw outperforms every 2-bit-class baseline by 6.3–47.0 average accuracy points, and at 2.9 bpw it matches 4-bit formats that read 1.4–1.6 more bpw.