Efficiency & systems

Quantisation, inference serving, kernels, distributed training, compression, efficient architectures

240 articles · page 4 of 5

Efficiency & systems

Lower reconstruction loss can degrade quantized LLM performance; robust refinement fixes it

The authors show that minimizing reconstruction loss on calibration data during weight-only post-training quantization (PTQ) can hurt model performance, even on the very same data. To address this, they propose Distributionally Robust Quantization (DRQ), a post-hoc refinement that minimizes worst-case reconstruction loss over a constrained set of input activation distributions, and demonstrate gains across six representative PTQ methods on dense and mixture-of-experts LLMs.

·2 min
Efficiency & systems

Recovery-aware calibration improves distillation recovery for pruned reasoning models

The authors propose ReCal, a plug-and-play calibration method that adjusts pruning criteria based on forward KL divergence between an unpruned teacher and a pruned probe. Applied before pruning, ReCal helps preserve predictions vulnerable to pruning, enabling better recovery via on-policy distillation. Across Qwen3 models and multiple pruning criteria, ReCal consistently improves mathematical reasoning and code generation.

·3 min
Efficiency & systems

H-SVDQuant improves 4-bit quantization of diffusion transformers with low-rank branches

Wang et al. present H-SVDQuant, a framework for post-training quantization of diffusion transformers to 4-bit weights and activations (W4A4). By formulating a joint optimization problem that accounts for low-rank compensation, residual quantization, and activation noise, they derive a deflated Hessian and an activation-noise surrogate. In experiments, rank-4 H-SVDQuant outperforms rank-32 SVDQuant on SANA-1.6B and FLUX models, while reducing quantization calibration time by up to 6.25 times.

·2 min
Efficiency & systems

SparseDecoding aligns pruning calibration with autoregressive decoding activations for faster LLM inference.

Wang et al. introduce SparseDecoding, a pruning framework for LLM inference that addresses two problems: the distribution shift between teacher-forced calibration and autoregressive decoding, and the lack of efficient sparse matrix-vector (SpMV) kernels for N:M sparsity. By using activation statistics from the model's own decoding (excluding prefill) to guide pruning, and by implementing a bitmask-indexed N:M SpMV kernel, the authors achieve consistent quality improvements over standard fixed-text calibration on long-form generation benchmarks and deliver practical decoding speedups on A100 GPUs.

·3 min
Efficiency & systems

EDiS caches edge-disjoint subgraphs to cheaply vary sparse GNN training graphs

The authors introduce EDiS, a framework that performs a one-time decomposition of a graph into edge-disjoint subgraphs, then efficiently recombines these subgraphs per epoch to produce training graphs with varying edge budgets. On 19 node classification benchmarks, EDiS achieves the highest mean accuracy/ROC-AUC and lowest average rank among 17 baselines under a 30% edge budget, with the largest gains at tight budgets.

·3 min
Efficiency & systems

Random Fourier features make Gaussian process attention linear-time and well calibrated

The authors propose RFF-GPA, a plug-and-play Transformer attention module that treats attention as a Gaussian process posterior with a kernel approximated by random Fourier features. The low-rank formulation computes predictive mean and covariance in time linear in sequence length, versus cubic for exact GP attention. On image and text classification, including CIFAR-10-C under distribution shift, it improves calibration while maintaining accuracy.

·2 min
Efficiency & systems

Centering activations enables calibration-free 4-bit diffusion transformer quantization

Jovanović, Salzmann and Javed identify why data-free Hadamard rotation performs poorly when quantizing diffusion transformers: adaptive layer normalization introduces a token-specific mean that rotation concentrates into a few range-dominating coordinates. Their CentriQ method removes this mean before quantization and restores it through a full-precision rank-1 computation, enabling calibration-free quantization of both weights and activations.

·3 min
Efficiency & systems

Multi-precision LLM weight format keeps a single prefix-readable stream

The authors present StagQ, a multi-precision weight format for LLMs that stores weights in a single prefix-readable stream: a 2-bit group-wise affine base followed by configurable 1-bit refinement planes. A sparse side record holds out extreme weights via two guard stages. On Llama-3.1-8B, Phi-4, and OLMo-2-7B, StagQ-XXS leads the strongest multi-precision baseline on MMLU by 3.1 to 7.0 points at 2 bits, with faster batch-one matrix-vector products on an A100 GPU in most cases.

·3 min
Efficiency & systems

Adaptive attention pruning compresses language-model caches without sacrificing accuracy

Wertheimer and colleagues introduce a transformer attention layer whose learned decay mechanism doubles as an input-dependent rule for pruning the key-value cache used during generation. In experiments with a 1-billion-parameter model trained on 50 billion tokens, the authors report more than 10-fold cache compression without degrading downstream performance, rising to 25-fold at a 16,000-token context length.

·2 min
Efficiency & systems

Random Outputs Can Replace Model Updates in Collaborative Learning

Fenoglio and colleagues introduce Distributed Subliminal Learning (DSL), in which participants probe locally adapted models with random, task-unrelated inputs and send the resulting outputs instead of parameters. Across language-model composition and federated image classification, DSL preserved useful local behavior while reducing communication, although its effectiveness depended on the model interface and carrier coverage.

·2 min
Efficiency & systems

Search framework finds better layer-wise settings for efficient LLM inference

Chen and colleagues formulate inference configuration as constrained multi-objective black-box optimization, seeking Pareto-efficient settings that balance model quality against memory or computation. On Qwen2.5-7B and DeepSeek-V2-Lite, InferOpt found layer-specific configurations that outperformed uniform allocations and three general-purpose search baselines under matched resource budgets.

·2 min
Efficiency & systems

Nyström sketching enables second-order optimizer with linear memory cost

The authors introduce Clean, a second-order optimizer that uses randomized Nyström sketching to approximate the preconditioners in SOAP, reducing memory complexity from quadratic to linear in model dimensions. They also propose Q-Clean, a quantized variant. Clean matches SOAP's performance while using less memory than AdamW, converges 26% faster than AdamW, and enables pre-training a 13B parameter model on a single 80GB GPU.

·2 min
Efficiency & systems

Zephon data loader provides deterministic training across GPU topology changes

The authors present Zephon, a data loader for foundation models that delivers elastic determinism—identical global batches despite changes in GPU topology—for online, stateful data pipelines. They show that Zephon eliminates loss curve variability across topology changes while maintaining competitive throughput on text and vision-language workloads, a combination of guarantees not offered by existing loaders.

·3 min
Efficiency & systems

Tailored vector quantization space enables accurate 1-bit KV cache compression

The authors introduce TaSQ, a method that tailors the vector quantization (VQ) target space for 1-bit KV cache compression in large language models. By addressing how quantization errors affect attention logits and exploiting inter-channel dependencies, TaSQ consistently outperforms existing low-bit VQ baselines across general, reasoning, and long-context retrieval benchmarks, while enabling up to 14× larger batch sizes and 1.87× higher throughput compared to BF16 baselines.

·3 min
Efficiency & systems

Commercial LLM routers fail to beat carefully chosen random routing

Wang et al. test whether dynamic routers can reduce inference costs by assigning each query to an appropriately capable language model. On a benchmark spanning eight task categories, every commercial router performed no better than a cost-matched random baseline using two carefully selected models, while some trailed it by more than 10 percentage points. The authors argue that both prevailing evaluation objectives and poorly chosen model rosters help explain the failures.

·3 min
Efficiency & systems

Learned controller adapts speculative decoding to maximize speed and reduce waste

The authors introduce APEX, a learned controller that dynamically selects the speculative decoding mechanism (EAGLE-3, n-gram, or a draft model) and adapts draft depth at each verification block to balance throughput and wasted computation. Evaluated with Qwen3-8B across six workloads, APEX achieves up to 5.24x wall-clock speedup over autoregressive decoding, and its depth-optimizing variant reduces wasted-token percentage by 41% relative to fixed n-gram speculation at depth 16.

·3 min
Efficiency & systems

Leto speeds up LLM training recovery by reusing state on surviving hardware

Leto is a fault-tolerant training system that leverages surviving hardware after hardware-operable failures (HOFs) to recover quickly. It retains model and reusable process state on the same node and preinitializes a shadow trainer, eliminating checkpoint loading and recomputation of completed iterations. On 6- and 72-GPU A100 clusters, Leto recovers 3.6-6.5x faster than checkpointing baselines and improves productive training time by up to 13.7 percentage points.

·2 min
Efficiency & systems

Recalibrating activations helps quantized vision transformers withstand distribution shifts

Cha, Kwon and Lee identify a quantization-specific weakness under distribution shift: channel-level activation statistics drift away from the ranges used to calibrate frozen quantizers, distorting how activations map to discrete codes. Their Quantizer-Aligned Recalibration method, or QuAR, corrects these statistics during inference and outperforms other backpropagation-free adaptation methods across several quantization levels and image benchmarks.

·3 min
Efficiency & systems

Sparse, exact updates cut trillion-parameter policy refits to minutes

Jiang et al. address a bottleneck in disaggregated reinforcement learning, where updated model weights must repeatedly move from training clusters to rollout clusters. Their system exploits the observation that only about 0.6 to 1.2% of stored BF16 values changed per training step across six measured models, reducing a reported trillion-parameter cross-region refit from 87.5 minutes to 150 seconds at a 3% change rate.

·3 min
Efficiency & systems

Adaptive mixed-precision attention matches full precision with selective 16-bit recomputation

The authors introduce LampAttention, a mixed-precision FlashAttention variant that uses 8-bit arithmetic for most attention computations and selectively recomputes a minority of sub-blocks in 16-bit. Simulated inference on Qwen3 and Gemma 3 shows that this adaptive recomputation recovers baseline accuracy while reducing high-precision arithmetic, and they propose a dedicated accelerator to realize the pipeline.

·2 min
Efficiency & systems

Compressed looped models settle, not drift, so precise final loops recover them

Kolawole et al. test the dominant explanation for collapse in compressed looped models—accumulating rounding error—across more than 30 models from five families. They find that when a loop settles, the error is fixed and shifts the equilibrium point like tilting a bowl, not a slippery slope. This view lets them predict which models will fail from a single label-free measurement and recover them with a few 8-bit finishing loops, halving weight traffic on hard puzzles.

·3 min
Efficiency & systems

Training models for deployment preserves quality under four-bit quantization

Xu and colleagues present QATFactory, an open-source framework that trains large language models to tolerate the numerical errors introduced by deployment-grade quantization. Across models from 8 billion to 230 billion parameters, quantization-aware distillation produced more accurate deployed models than post-training quantization, with the best training recipe depending on the target four-bit format.

·2 min
Efficiency & systems

Three-step hybrid pipeline validates GPU kernels by combining concurrency testing with formal proof

The authors present RESOLVE, a GPU-kernel validation framework that separates concurrency correctness from functional correctness. In a three-step pipeline, it tests for nondeterminism via timing perturbation, rewrites kernels into reduced-concurrency versions that preserve bitwise behavior, and then formally proves equivalence using the F*/Pulse framework. RESOLVE validated kernels from KernelBench and fused GEMMs across CUTLASS, Triton, and Gluon, and found four previously unreported issues in mega-kernels, including two clear bugs.

·3 min
Efficiency & systems

A library that makes gradient-based training data attribution practical for large language models

The authors introduce dattri-LLM, a library for training data attribution (TDA) at LLM scale. It uses a FLOP-aware cost model to efficiently represent per-example gradients, and non-invasive hooks to capture gradients from existing training loops. On benchmarks, it achieves on average 3.2x the throughput of the fastest competing library and scales to 110B-parameter models across four H200 GPUs.

·2 min
Efficiency & systems

Joint sparsity and quantization enable efficient trillion-parameter MoE serving

The authors propose moesq, an end-to-end framework that jointly optimizes semi-structured sparsity and low-precision quantization for Mixture-of-Experts models, leveraging hardware Sparse Tensor Cores. Across models from 30 billion to one trillion parameters, moesq preserves 96.09% of the original model's accuracy while enabling up to 1.65x faster matrix multiplication and 1.18x higher serving throughput on NVIDIA B200 GPUs.

·3 min
Efficiency & systems

Sensitivity-aware residual-stream pruning improves LLM compression without retraining

The authors introduce a sensitivity-aware approach to residual-stream pruning for large language models. Instead of minimizing activation reconstruction error, they derive a tractable upper bound on the KL divergence between original and pruned model outputs, which balances activation covariance with output sensitivity. Across multiple instruction-tuned LLMs, their method consistently outperforms activation-only pruning, especially at higher compression rates.

·3 min