Efficiency & systems

Quantisation, inference serving, kernels, distributed training, compression, efficient architectures

240 articles · page 5 of 5

Efficiency & systems

Joint token-feature KV cache compression speeds long-context LLMs

Zihan Teng, Jiayu Zhao, Wentao Ren, Minhao Fan, Tianrui Ma, Song Chen, and Weichen Liu propose SlimKV, a KV-cache compression method that jointly reduces the number of cached states and their per-token dimensions. Using low-rank-aware training, SlimKV compresses long contexts into compact beacon memory states with layer-adaptive rank allocation, and exploits an asymmetric dependence on key-side RoPE to remove its application from beacon keys during decoding. This achieves up to a 7.34x attention speedup and 3.38x end-to-end decoding speedup over the uncompressed model at 128K context length.

Efficiency & systems

Unified retention and compensation for KV cache eviction via coverage calibration

Liu et al. derive an exact factorization of KV cache eviction error into evicted attention mass and directional gap, motivating a unified approach to retention and compensation. They propose CORE, which distills offline set-level coverage into a lightweight cache-aware indexer that simultaneously drives retention ordering and parameterizes latent-memory writes. Across three backbones, CORE exceeds the strongest RULER baseline by up to 3.78 points at 90% compression on long-context benchmarks.

Efficiency & systems

Low-rank KV cache compression preserves all tokens during long reasoning

The authors propose iS-KV, an online low-rank KV-cache compression method for long chain-of-thought reasoning. It keeps recent tokens at full precision and folds older tokens into bounded-rank representations using block-incremental SVD, updating both the basis and historical coefficients to prevent drift. On two 8B models, iS-KV achieves accuracy within 1.0–6.6 points of the uncompressed model at 4–6x persistent-KV compression, outperforming token-eviction baselines at matched memory budgets.

Efficiency & systems

Expert pruning in MoE models can be tailored to decode phase, preserving prefill quality while boosting throughput

The authors present SlimWise, a serving framework for mixture-of-experts (MoE) models that decouples expert pruning across prefill and decode phases. By retaining the full expert pool during compute-bound prefill and pruning only during memory-bound decode, and directly reusing the prefill-generated KV cache, they achieve significant decode throughput improvements with minimal accuracy loss. They also introduce a low-cost distillation stage that further recovers accuracy and controls generation length distortions.

Efficiency & systems

Pruning experts by replacement damage better preserves MoE reasoning performance

Song and Zheng introduce RAZOR, a training-free method for removing stored experts from mixture-of-experts language models while retaining their reasoning performance. Across two primary models, two pruning budgets and nine tasks, RAZOR produced the highest average score among the evaluated methods, although substantial pruning still changed generation behavior and reduced performance relative to unpruned models.

Efficiency & systems

Post-training multi-token prediction heads match joint pretraining with far less data

The authors show that post-training multi-token prediction (MTP) heads on approximately 2.5 billion tokens of target-generated chain-of-thought data matches or exceeds the expected speedup of jointly pretrained MTP heads on math, coding, and knowledge benchmarks, using 10^3 to 10^4 times fewer training tokens. They further propose a chain-aware relaxation of the verification rule that increases expected speedup by 12–16% without degrading task accuracy, and an adaptive controller that recovers 11–14% of the speedup lost by using a fixed draft length.

Efficiency & systems

HAWK boosts speculative decoding for LVLMs via target hidden states and prediction-shift training

The authors propose HAWK, a speculative decoding framework for large vision-language models. It improves drafting by feeding the drafter compressed visual hidden states from the target model instead of raw tokens, and by training it to predict how the target's logits shift when the drafter's own tokens are used as context. On SmolVLM-256M across ten benchmarks, HAWK increases average acceptance length from 3.32 to 4.08 and speedup from 2.19x to 2.60x under greedy decoding versus EAGLE-3.

Efficiency & systems

Adaptive verification speeds block-diffusion decoding under heavy concurrent demand

Chen and colleagues introduce DScale, a runtime system for speculative decoding that keeps DFlash’s existing block-diffusion drafter and verifies selected candidate prefixes using half the native token capacity. Across two Qwen3 models, four datasets and concurrency levels from 8 to 32, it increased throughput by 43.9% to 48.8% over DFlash while reducing mean end-to-end latency by 31.4% to 33.1%.

Efficiency & systems

End-to-end subspace learning preserves transformers under aggressive low-rank compression

Bini et al. introduce Learnable Subspace Projections (LSP), a compression method that learns low-rank subspaces against the behavior of the complete network rather than approximating each layer independently. Across language and vision transformers, the authors report that this global optimization preserves substantially more performance at high compression, while producing standard dense low-rank factors suitable for existing hardware.

Efficiency & systems

Training-free inference framework trims redundant computation and memory in looped Transformers

FlashLoop is a training-free inference framework that reduces cross-loop redundancy in looped Transformers through token-sparse updates, sparse attention, and KV-residual quantization. The authors show that as loops progress, state changes concentrate on few tokens, attention-output differences concentrate on few key columns, and KV residuals become quantization-friendly. Across several looped models, FlashLoop achieves lossless accuracy with up to 1.64x speedup and 6x KV-cache memory reduction.

Efficiency & systems

Masked co-adaptive fine-tuning cuts expert fetches by over 23% for MoE models.

Wu et al. propose MaskCoFT, a fine-tuning method that uses learnable binary masks to restrict each layer's Top-K routing to a subset of experts while allowing those experts to adapt to the redirected tokens. For Mixtral-8x7B and DeepSeek-V2-Lite, the method reduces expert fetches per token by 23.7% and 10.1%, respectively, and lowers time per output token by up to 16.4% and 5.5% in a real offloading system. Average accuracy over nine benchmarks remains slightly above the base model.

Efficiency & systems

Compressed LLM memories can preserve skills beyond their source documents

Laguna and colleagues test whether compressed key-value caches remain safe to reuse when a user asks questions unrelated to the cached document. They find that standard Cartridges preserve document-specific information well but can contaminate unrelated answers, weaken general knowledge and disrupt instruction following. Two simple modifications largely preserve these off-context capabilities with little reported loss of document utility.

Efficiency & systems

Training-free feature folding cuts memory and improves accuracy for wide-table tabular models.

Zhou et al. introduce Support-Compiled Feature Folding (SCFF), a training-free inference method for tabular foundation models that avoids full-width feature mixing by organizing columns into a core and a tail. On an 18-dataset wide-table benchmark, SCFF improved accuracy and negative log-likelihood across all six evaluated frozen backbones, while cutting median peak GPU memory by 2.09-2.36x. The method also used saved memory to retain more evidence, yielding accuracy gains of 4.06 and 3.72 points over widest feasible single-leaf baselines.

Efficiency & systems

Elastic phase sharing improves efficiency in agentic LLM serving

Xu et al. address a mismatch in disaggregated LLM serving, where separate pools handle prompt processing, called prefill, and token generation, called decode, even though demand for each phase changes rapidly. Across public and internal traffic traces, Crossflow increased token throughput by 16.2 to 17.4% on geometric mean over static partitioning, while reducing average time to first token at every evaluated load.

Efficiency & systems

Adaptive prefill-decode execution and elastic GPU reuse boost agentic RL throughput

The authors introduce PEARL, a system that coordinates external elastic GPU resources, temporary reuse of idle training GPUs, and adaptive prefill-decode (PD) configuration for asynchronous agentic reinforcement learning. In experiments with Qwen3-8B and Qwen3-30B-A3B models, PEARL achieves 2.17 to 2.79 times the throughput of a fixed-resource baseline (ROLL) and up to 26.9% and 36.3% improvement over RLBoost+ for the respective models.

Efficiency & systems

Dispatch-aware agents safely optimize compiler-generated GPU kernels within full models

Poddar et al. present a five-agent optimization system that respects compiler dispatch decisions, leaving cuBLAS and cuDNN calls intact while targeting generated Triton kernels. On 250 KernelBench problems, KernelOPT reports geometric-mean speedups over torch.compile ranging from 1.07× to 1.40×, with a verification cascade that retains the baseline whenever no candidate passes.

Efficiency & systems

Engineering agents need evidence-bound authorization before their outputs trigger action

Koch proposes a cross-domain assurance architecture for deciding when agent-produced software, firmware, or PCB artifacts may advance to consequential actions. Rather than treating tests, attestations, or human approval as sufficient in isolation, the model requires evidence to remain artifact-specific, policy-bound, current, independent where risk demands it, and valid at the moment of execution.

Efficiency & systems

Selective attention calls, guided by model's own state, speed long-context inference

The authors introduce On-Demand Attention (ODA), a decoding method that uses a trained recall head to decide when to invoke global attention based on the model's decoding state. ODA trains only the recall head, leaving the pretrained model untouched, and achieves substantial speedups in long-context inference by selectively reading the full history. Experiments show ODA recovers near-full-attention performance while using far fewer global attention calls.