Efficiency & systems

Quantisation, inference serving, kernels, distributed training, compression, efficient architectures

240 articles · page 2 of 5

Efficiency & systems

New system halves compute waste during AI response generation training

Researchers developed FlexFloor, a system that cuts wasted compute time in reinforcement-learning post-training for mixture-of-experts language models. By releasing workers that have finished generating their responses while keeping their experts available for ongoing responses, FlexFloor boosts response tokens per occupied compute time by 65% and overall throughput by 53% with no loss in output quality.

Efficiency & systems

Language models can cut cache memory nearly 27-fold, retaining most performance

Language models store information about preceding tokens in a growing memory cache, which can limit how many requests they process simultaneously. Delgado and colleagues show that much of one component, the stored values, can be predicted from another, the keys. Their combined compression pipeline stays close to baseline performance at moderate compression and supports more aggressive reductions when paired with existing methods.

Efficiency & systems

Dynamic operator-level partitioning speeds up AI inference across diverse hardware by up to 3.5 times

The authors develop a profile-guided dynamic programming method that discovers latency-minimizing partitions of LLM computational graphs across heterogeneous accelerators. In systems combining H100 and B200 GPUs with emerging accelerators, HELIX reduces multi-query decoding latency by up to 3× for dense and sparse transformers and up to 3.5× for hybrid state-space models compared to fixed attention-FFN disaggregation.

Efficiency & systems

Chunked prefill, not cache architecture, drives LLM serving engine speed differences

The authors compared two popular open-source LLM serving engines, vLLM and SGLang, under default settings and with chunked prefill matched. Under defaults, vLLM's median time to first token was up to 64 times that of SGLang. Matching the chunked prefill setting removed 78% of the capacity gap and 30–67% of the latency gap, showing that the perceived advantage of SGLang's cache architecture is largely due to configuration differences.

Efficiency & systems

Recycling high-scored unexposed candidates boosts feed discovery without engagement loss

The authors report that URED reuses candidates the ranking model scored highly but never showed to users due to position constraints, a pool that is both relevant and reliably predictable, unlike boosted cold items. Online A/B tests on Taobao feed recommendation showed a 3.47 percent increase in discovered categories and 8.47 percent increase in content richness, with engagement preserved.

Efficiency & systems

Optical circuits cut LLM inference latency by 65% and energy use by 119%

The authors propose workload-static optical circuit switching (OCS) for LLM inference decode, where circuits are configured once per deployment rather than per token. In a simulated 576-accelerator system running 5-trillion-parameter mixture-of-experts models, all-OCS fabrics improve interactivity by up to 65% and energy efficiency by up to 119% over oversubscribed electronic packet-switched fabrics.

Efficiency & systems

Fusing adapters into low-precision GEMMs cuts latency for AI serving and training

The authors introduce EpiLoRA, a vLLM plugin that fuses low-rank adapter (LoRA) computations into the base model's low-precision matrix multiplication (GEMM). On NVIDIA Hopper GPUs, this reduces time per token by up to 24% for single-adapter serving, and with many adapters it increases throughput on most GPU-model pairs tested. The same approach enables LoRA reinforcement learning on FP8 weights, allowing a 27B model to train on a single 48 GB GPU.

Efficiency & systems

Overlapping indexer scan with attention speeds up long-context decoding by up to 14%

The authors introduce Multi-layer Indexer (MLI), a drop-in schedule for DeepSeek Sparse Attention that overlaps the per-layer indexer scan with attention. Using the selection from the adjacent indexer layer removes the dependency, allowing concurrent execution. On DeepSeek-V3.2 and GLM-5, MLI achieves up to 14.5% faster decoding at equal retention and can halve the indexer budget at no speed cost, while accuracy stays within a few points.

Efficiency & systems

Fewer draft tokens verified per round makes hybrid models a third faster

Speculative decoding has a cost problem that depends on architecture: when the target model mixes linear-attention layers with mixture-of-experts layers, checking a long block of drafted tokens gets expensive fast. Wang, Zhang, Liu and Li measure this and build HybridSpec, which chooses a verification length per round on the GPU and adds one extra token only up to the first expert-routing layer, where a cheap correction can veto it. Across three such models in single-request greedy decoding, it is 1.75x to 1.88x faster than ordinary autoregressive decoding and 1.30x to 1.33x faster than the parallel drafter DFlash, which verifies every drafted token.

Efficiency & systems

Million-token prompts served at 2.3 times the throughput without retraining

Long-context inference is expensive because attention keeps rereading a growing key-value cache. The authors replace token-level selection with fixed-size chunk selection, deriving a compact summary per 16-token chunk from the model's own keys, and share one selection across neighbouring prefill queries. On million-token prompts with 32 concurrent requests, the modified models sustained 2.29 times the end-to-end throughput of dense attention on four NVIDIA B300 GPUs, with benchmark scores close to dense.

Efficiency & systems

Mid-depth layers judge KV-cache evictions better than the owning layer

The authors show that the layer used to score tokens for KV-cache eviction critically determines accuracy. Across 18 transformer models, the best scoring layer is consistently at mid-depth (40-75% of layers), while using a model's own per-layer cache or its last layer degrades performance substantially. A predictive model based on prefill attention reproduces this curve and selects the scoring layer without extra forward passes.

Efficiency & systems

Backward pass for long-context AI models runs up to 1.8 times faster

Researchers at Tianjin University have reformulated the backward pass of delta-rule linear attention, the recurrent state mechanism used in models such as Kimi Linear and Qwen3-Next. Rather than differentiating each auxiliary object in the standard chunkwise algorithm separately, they treat the chunk correction as one triangular residual system, which removes an entire adjoint chain. The reduced computation runs 1.15 to 1.81 times faster as a complete backward operator against the reference Flash Linear Attention implementation on NVIDIA GPUs, with gradients checked to match.

Efficiency & systems

Precomputing attention indices from earlier layers speeds long-context AI without accuracy loss

The authors propose EarlySelectKV, a training-free transformation for query-conditioned sparse KV selection. It uses proxy queries from preceding hidden states to precompute which historical tokens to attend to, while keeping the real target-layer query for the actual attention computation. Evaluated on three benchmarks, it matches the baseline within 0.02 points, and in a TensorRT-LLM deployment it improves throughput by up to 12% and reduces GPU memory by 11%.

Efficiency & systems

Combining model compression techniques can collapse small language models

The authors systematically evaluated 72 compression configurations across four small language models (2.7B to 7B parameters) to understand how quantization, pruning, and fine-tuning interact. They found that INT8 quantization is effectively lossless, while NF4 quantization combined with L1 magnitude pruning causes model collapse in all tested combinations. LoRA-based fine-tuning recovered up to 97% of degraded perplexity after aggressive pruning, and the recovery did not require a teacher signal.

Efficiency & systems

Pruned language models retain problem-solving coverage despite lower one-shot accuracy

The authors compare pruned and original large language models on mathematical and coding tasks using repeated sampling (pass@k) to disentangle reasoning coverage from sampling efficiency. They find that even after substantial pruning, models can solve as many or more distinct problems than the original despite lower pass@1, but the cost in tokens and attempts needed to reach that coverage can increase sharply.

Efficiency & systems

4-bit language models run as integer math on Apple's M5 chips without extra accuracy loss

Kang et al. demonstrate that MXFP4 quantized weights can be converted to INT8 operands for matrix multiplication on Apple M5 GPUs, enabling exact W4A8 inference. They derive a condition, block exponents spanning at most three, that allows per-row scale folding, which holds for 97.9–99.97% of rows across seven models, including a 125B-parameter mixture-of-experts model. This yields speedups of 1.10–1.74× in prefill compared to MLX on dense and MoE models up to 125B parameters.

Efficiency & systems

AI agents’ average agreement can mask declines in later browsing actions

Zhao and colleagues examine speculative decoding, where a small model proposes text that a larger model checks, across browsing and software-engineering agents. They find that overall agreement can improve with longer histories even while agreement worsens for comparable late browsing actions. Using these differences to guide draft-model training improves throughput when replaying saved agent histories.

Efficiency & systems

Long AI prompts use disproportionately more energy than output length predicts.

The authors profiled compute time and energy consumption for LLM inference across many input and output token lengths. They found that output length scales linearly with cost, but input length has a quadratic component, making long contexts disproportionately expensive. They also show that compute time is a reliable proxy for energy, enabling estimates for proprietary APIs without direct measurements.

Efficiency & systems

AI chatbots can answer sooner without permanently discarding visual context

Huang et al. investigate whether chatbots can start answering questions about images and videos before processing all the visual input, without permanently losing the skipped information. Their system, Raincheck, processes that information in the background, recovering much of the answer-quality loss caused by discarding it while reducing first-response latency relative to processing everything upfront.

Efficiency & systems

Scalar two-bit weight quantization can preserve language model reasoning accuracy

The authors present ThinkBit, a five-stage framework for quantizing reasoning language models to two bits per weight in a strictly scalar format, where each weight is an integer codeword times a bfloat16 scale shared by 128 weights. On Qwen3-1.7B, their 0.52 GB checkpoint scores up to 73.0 on MATH-500 and is smaller than every published 2-bit GGUF of the model; at 14B, a 3.74 GB model scores 94.8 versus 97.5 for the 29.5 GB bfloat16 baseline.

Efficiency & systems

Language models can learn nearly as well with four-bit numbers

The authors trained the 20-billion-parameter mixture-of-experts model GPT-OSS-20B under the MXFP4 4-bit microscaling format on the MLPerf Small MoE benchmark. By composing existing techniques with a simplified weight de-oscillation scheme adapted from TetraJet-v2, they achieved a final validation loss of 3.3350 on C4, only 0.007 above the BF16 baseline (3.3283). De-oscillation halved the remaining loss gap relative to MXFP4 without it (which gave 3.3418) and reduced the steps-to-target overhead from +20% to +5%.

Efficiency & systems

Mixed-precision LLM accelerator turns lower bit weights into proportionally faster compute

The authors present FIGFLEX, a co-designed accelerator and quantizer for LLM inference that keeps activations in FP16 and weights as low-bit integers, without dequantizing. Its radix-4 bit-parallel datapath halves partial products and scales throughput as precision falls. On LLaMA-2-7B, it beats five prior quantizers in perplexity at 2 to 3 bit average precision and delivers up to 1.56x higher throughput per area than AxCore.