Efficiency & systems

Quantisation, inference serving, kernels, distributed training, compression, efficient architectures

34 articles

All Agents Language models Reinforcement learning Vision Generative media Speech & audio Robotics & embodied AI Safety & security Evaluation & benchmarks Efficiency & systems Code & software AI for science & health Theory

Viewgrid list compact

Pruning experts by replaceability, not magnitude, preserves reasoning in MoE models

The authors introduce RAZOR, a training-free pruning method for mixture-of-experts models that scores experts by how well surviving experts can compensate for their removal. Across four MoE models (GLM-4.7-Flash, Qwen3.6-35B-A3B, DeepSeek-V4-Flash-0731, Hy3) at 25% and 50% pruning budgets, RAZOR achieved the highest macro average over nine reasoning tasks, outperforming the REAP baseline by up to 5.59 points. However, pruned models still exhibited shifts in response diversity, formatting, and termination.

3 Oct·2 min

Efficiency & systems

Training-free inference framework trims redundant computation and memory in looped Transformers

FlashLoop is a training-free inference framework that reduces cross-loop redundancy in looped Transformers through token-sparse updates, sparse attention, and KV-residual quantization. The authors show that as loops progress, state changes concentrate on few tokens, attention-output differences concentrate on few key columns, and KV residuals become quantization-friendly. Across several looped models, FlashLoop achieves lossless accuracy with up to 1.64x speedup and 6x KV-cache memory reduction.

3 Oct·1 min

Efficiency & systems

Mentored decoding can trade small distribution shifts for better predictions

Tran-Thien and Nock formalize mentored decoding, in which a fast draft model’s tokens are accepted while the resulting distribution remains within a chosen distance of a larger target model. Their analysis shows how this controlled deviation can increase draft acceptance and, under boosting-style conditions, produce predictions better than the target’s rather than merely approximate it.

3 Oct·3 min

Efficiency & systems

Sparse feed-forward activations cut language-model inference costs substantially

Lomeli and colleagues introduce model casting, a mid-training procedure that makes feed-forward network activations sparse enough for dedicated inference kernels to skip most inactive computation. They also propose LoPA Gating, which reduces the cost of identifying active units and surpasses the theoretical threefold speedup limit of conventional gated feed-forward layers.

3 Oct·2 min

Efficiency & systems

Masked co-adaptive fine-tuning cuts expert fetches by over 23% for MoE models.

Wu et al. propose MaskCoFT, a fine-tuning method that uses learnable binary masks to restrict each layer's Top-K routing to a subset of experts while allowing those experts to adapt to the redirected tokens. For Mixtral-8x7B and DeepSeek-V2-Lite, the method reduces expert fetches per token by 23.7% and 10.1%, respectively, and lowers time per output token by up to 16.4% and 5.5% in a real offloading system. Average accuracy over nine benchmarks remains slightly above the base model.

3 Oct·3 min

Efficiency & systems

Compressed LLM memories can preserve skills beyond their source documents

Laguna and colleagues test whether compressed key-value caches remain safe to reuse when a user asks questions unrelated to the cached document. They find that standard Cartridges preserve document-specific information well but can contaminate unrelated answers, weaken general knowledge and disrupt instruction following. Two simple modifications largely preserve these off-context capabilities with little reported loss of document utility.

3 Oct·2 min

Efficiency & systems

Restoring softmax gradient symmetry prevents late BF16 attention instability

Chen and colleagues trace a late-training failure in BF16 FlashAttention-3 to two numerical errors, including a previously overlooked violation of a softmax conservation law. Their correction, GProj, reduced median query and key gradient errors to below 0.4 percent and trained a 450 million-parameter transformer to the same loss as FP32 attention, with a 4.7 percent increase in step time.

3 Oct·3 min

Efficiency & systems

EntroPack compresses neural weights at finely adjustable storage rates

Zhang, Duan and Chen present a weight compressor that can target rates between the coarse options offered by fixed-width formats. On the Z-Image-Turbo generator, EntroPack produced lower weight and denoiser-output errors than comparable fixed-width representations, including roughly 24% lower relative L2 weight error than NF4 when targeting 4 bits per parameter.

3 Oct·2 min

Efficiency & systems

Approximate scores recover context lost by sparse long-context attention

Shao et al. address a weakness of sparse attention: tokens excluded from retrieval normally receive zero weight, even when useful information is distributed across the context. Their method, PQ-HSA, retains approximate contributions from those tokens and, at 128K context, improves accuracy while making the decode attention call faster than full attention on the tested hardware.

3 Oct·3 min

Efficiency & systems

Input-conditioned fine-tuning preserves language model behavior on unrelated tasks

Lin et al. introduce ATLAS, a method for fine-tuning language models that preserves behavior on retained domains while learning new tasks. On Qwen3-8B, ATLAS achieved lower mean retained-output KL divergence than all seven published baselines at matched coding performance, with fewer rewritten answers across mathematical and commonsense domains.

3 Oct·2 min

Efficiency & systems

Simple clustering preserves video understanding with far fewer visual tokens

Zhang et al. introduce SimpleCluster, a baseline that compresses video representations by grouping similar visual tokens across frames and replacing each group with its mean feature. Their experiments suggest that preserving broad coverage of the original feature distribution can matter more than sophisticated token selection, particularly when the model must retain very few tokens.

3 Oct·3 min

Efficiency & systems

Simple LRU rivals sophisticated eviction for LLM prefix caching

The authors analyzed production traces from two companies and evaluated 14 eviction algorithms for LLM prefix caching. They found that simple Least Recently Used (LRU) performs competitively with sophisticated policies, despite substantial headroom to Belady’s optimal. The reason is that prefix reuse is dominated by the regular pacing of active sessions, making recency a strong predictor.

3 Oct·2 min

Efficiency & systems

Training-free feature folding cuts memory and improves accuracy for wide-table tabular models.

Zhou et al. introduce Support-Compiled Feature Folding (SCFF), a training-free inference method for tabular foundation models that avoids full-width feature mixing by organizing columns into a core and a tail. On an 18-dataset wide-table benchmark, SCFF improved accuracy and negative log-likelihood across all six evaluated frozen backbones, while cutting median peak GPU memory by 2.09-2.36x. The method also used saved memory to retain more evidence, yielding accuracy gains of 4.06 and 3.72 points over widest feasible single-leaf baselines.

3 Oct·3 min

Efficiency & systems

Exact quantiles and gradient corrections improve expert load balancing

Neitemeier et al. address two distinct forms of imbalance in mixture-of-experts models: persistent skew across an optimizer step and uneven expert use within individual microbatches. In experiments on 7.5-billion-parameter models trained for up to 500 billion tokens, their methods improved global and local balance while maintaining, and in some settings improving, downstream performance.

3 Oct·2 min

Efficiency & systems

Training-free MoE expert pruning via consensus residuals preserves reasoning performance

The authors introduce Razor, a training-free pruning method for mixture-of-experts LLMs that selects experts to remove by aggregating consensus residuals—exact deviations of expert outputs from the weighted mixture. On GLM-4.7-Flash and Qwen3.6-35B-A3B at 25% and 50% expert removal, Razor achieves the highest macro average on nine reasoning tasks, gaining 2.12 to 5.59 points over the REAP baseline while also reducing reverse KL divergence.

3 Oct·2 min

Efficiency & systems

Looped Transformer inference sped up by exploiting sparse cross-loop redundancy

The authors demonstrate that much of the computation in looped Transformers is redundant: as recurrence progresses, changes become concentrated on a small subset of tokens, attention-output differences are dominated by sparse key columns, and key-value (KV) residuals between adjacent loops become amenable to low-bit quantization. They introduce FlashLoop, a training-free inference framework that uses token-sparse updates, loop-aware sparse attention, and KV residual quantization, achieving up to 1.64× end-to-end speedup and up to 6× KV-cache memory reduction without loss of accuracy.

3 Oct·3 min

Efficiency & systems

Query-side adapters recover full-context quality in KV cache reuse

The authors propose Attuner, a method that learns to read frozen key-value (KV) caches of reusable content artifacts via query-side adaptation, eliminating the need for cache recomputation. On Qwen3-4B and Qwen3-8B across seven benchmarks, Attuner matches full-context prefill quality while providing up to 3.73x speedup, and substantially outperforms prior position-independent caching baselines.

30 Sept·3 min

Efficiency & systems

LLM-generated PTX rivals autotuned Triton across three GPU generations

Costa et al. test “AI lowering,” in which a large language model bypasses Triton’s optimizing compiler and generates GPU assembly directly from Triton source. Across 22 kernels, the resulting PTX was sometimes slower than autotuned Triton but achieved substantial gains when the agent found hardware-specific transformations absent from Triton’s lowering pipeline.

30 Sept·2 min

Efficiency & systems

Elastic phase sharing improves efficiency in agentic LLM serving

Xu et al. address a mismatch in disaggregated LLM serving, where separate pools handle prompt processing, called prefill, and token generation, called decode, even though demand for each phase changes rapidly. Across public and internal traffic traces, Crossflow increased token throughput by 16.2 to 17.4% on geometric mean over static partitioning, while reducing average time to first token at every evaluated load.

30 Sept·2 min

Efficiency & systems

KV cache offloading works when host memory fits agent reuse

The authors study why moving language-model key-value states from GPU to host memory can either accelerate or slow concurrent agents. On SWE-bench Verified coding-agent workloads, sizing the host cache around the agents’ reuse working set reduced recomputed prompt tokens by 93% and end-to-end time by 39%.

30 Sept·2 min

Efficiency & systems

Adaptive prefill-decode execution and elastic GPU reuse boost agentic RL throughput

The authors introduce PEARL, a system that coordinates external elastic GPU resources, temporary reuse of idle training GPUs, and adaptive prefill-decode (PD) configuration for asynchronous agentic reinforcement learning. In experiments with Qwen3-8B and Qwen3-30B-A3B models, PEARL achieves 2.17 to 2.79 times the throughput of a fixed-resource baseline (ROLL) and up to 26.9% and 36.3% improvement over RLBoost+ for the respective models.

30 Sept·2 min

Efficiency & systems

Streaming KV caches makes compacted agent training up to five times faster

The authors modify agent training so that key-value caches, the model’s internal attention state, continue across context-compaction events instead of being discarded and rebuilt. On text-based games and software-engineering tasks, KV-streams reduced training wall time by 2.6 to 5 times, while reaching similar final scores to conventional re-prefill compaction.

29 Sept·2 min

Efficiency & systems

Read-only LoRA coupling preserves old skills while adding new ones

Li and colleagues introduce Read-only Expansion of Adapter Deltas, or READ, for incrementally combining LoRA task adapters into one model. Across four benchmark suites and two model families, the method outperformed competing approaches built from the same adapters, including gains exceeding 20 points on SuperGLUE and seven points on a domain suite.

29 Sept·2 min

Efficiency & systems

Agent workloads expose blind spots in standard LLM inference benchmarks

Lau and colleagues built an inference benchmark from traces of agentic tasks, including SWE-Bench and TerminalBench, where requests span multiple turns and accumulate context. They find that benchmarks using fixed sequence lengths or a single unsaturated load point can misrepresent serving performance and hardware constraints.

29 Sept·2 min

Efficiency & systems

Prioritizing action-relevant context cuts LLM agent cache costs

Wang and colleagues introduce ActKV, a cache-compression system designed around the actions generated by LLM agents rather than the quality of every generated token. It retains cache entries likely to affect future actions, adjusts memory budgets using model confidence, and integrates compression with paged memory, producing substantial reported gains in memory use and serving throughput.

29 Sept·2 min

Efficiency & systems

DynBranch starts and reuses agent work before branches resolve

Shen et al. target the branch-resolution barrier, where an agent must wait for a model output or user choice before beginning downstream work. Across four agentic workloads, DynBranch reduced mean latency by up to 32% relative to each workload's strongest prior system while preserving workflow results.

29 Sept·2 min

Efficiency & systems

Action-aware cache compression makes long-running LLM agents more efficient

Wang et al. designed ActKV to reduce the key-value cache accumulated during repeated observation, reasoning and action loops. On long-trace tasks, the system preserved 98.53% of uncompressed-cache accuracy on average while using 25.98% of the peak cache memory, and it substantially increased serving throughput.

29 Sept·2 min

Efficiency & systems

Dispatch-aware agents safely optimize compiler-generated GPU kernels within full models

Poddar et al. present a five-agent optimization system that respects compiler dispatch decisions, leaving cuBLAS and cuDNN calls intact while targeting generated Triton kernels. On 250 KernelBench problems, KernelOPT reports geometric-mean speedups over torch.compile ranging from 1.07× to 1.40×, with a verification cascade that retains the baseline whenever no candidate passes.

27 Sept·2 min

Efficiency & systems

Engineering agents need evidence-bound authorization before their outputs trigger action

Koch proposes a cross-domain assurance architecture for deciding when agent-produced software, firmware, or PCB artifacts may advance to consequential actions. Rather than treating tests, attestations, or human approval as sufficient in isolation, the model requires evidence to remain artifact-specific, policy-bound, current, independent where risk demands it, and valid at the moment of execution.

25 Sept·2 min

Efficiency & systems

Selective attention calls, guided by model's own state, speed long-context inference

The authors introduce On-Demand Attention (ODA), a decoding method that uses a trained recall head to decide when to invoke global attention based on the model's decoding state. ODA trains only the recall head, leaving the pretrained model untouched, and achieves substantial speedups in long-context inference by selectively reading the full history. Experiments show ODA recovers near-full-attention performance while using far fewer global attention calls.

21 Sept·2 min

Efficiency & systems

Similarity-aware context windows improve model routing across multi-turn conversations

Wang et al. introduce SWRouter, a method for routing each turn of a multi-turn conversation to a suitable large language model. Across dialogue benchmarks, the authors report higher evaluation accuracy than both the best individual model and a strong conversation-context routing baseline.

12 Sept·2 min

Efficiency & systems

A Vestigial Attention Branch Signals Which Cache Entries to Keep

Fan introduces VestigeKV, a training-free method for compressing the actively attended KV cache in a NoPE multi-head latent attention model. On needle-retrieval tests spanning 8,000 to 65,000 tokens, it preserves perfect retrieval at 8× compression and scores 0.92 at 32×, while moving rather than deleting excluded cache rows.

7 Sept·2 min

Efficiency & systems

Full-context drafters help compressed agent models retain accuracy

Liang et al. modify speculative decoding so its drafting and verification models can see different versions of an agent’s accumulated context. Across four agentic capabilities and two end-to-end benchmarks, the method retained about 90% of full-context accuracy on average while reporting substantial throughput and compute gains on isolated text tasks.

28 Aug·2 min

Efficiency & systems

Cost-aware model routing matches exhaustive estimation with fewer expensive checks

Fisch et al. model AI routing as a search problem in which inspecting each candidate model’s likely value is itself costly. Across multi-LLM routing, retrieval-augmented specialists, and variable reasoning budgets, their centralized policy reportedly preserves the routing quality of exhaustive estimation while invoking expensive estimators much less often.

23 Aug·2 min