Efficiency & systems

Quantisation, inference serving, kernels, distributed training, compression, efficient architectures

240 articles · page 1 of 5

Efficiency & systems

Live GPU hidden states reveal what an AI model did for a single request

Pan et al. demonstrate that the transient hidden states of a large language model during live inference can be acquired from GPU memory using a non-modifying debugger, without prior knowledge of the model's architecture or weights. They validate byte-exact recovery across 33 decoder-only models and show that linear probes on these states can distinguish refusal from compliance with an AUROC of 0.88 on held-out prompts, offering a new forensic capability for analyzing deployed AI systems.

Efficiency & systems

One program generates fast neural network kernels for most quantized formats

Quantized neural networks need a different kernel for every combination of weight format, activation format, and instruction set, a combinatorial burden that hand-written libraries like GGML only partly meet. Reinking extends the Halide compiler to treat approximation as a separate, schedulable concern, letting one algorithm and one schedule generate kernels for 31 formats on Arm CPUs and Apple GPUs. The single codebase produces 347 kernels, covers 38 weight-activation pairs that GGML omits, and on an Apple M3 Max reaches at least 95% of the speed of GGML's hand-written kernels in 91.0% of cases.

Efficiency & systems

AI video generators could produce nearly 75 times more emissions by 2030

The authors combined measurements of commercial video platforms with provider disclosures to model the environmental footprint of AI video generation from 2026 to 2030. Their scenarios suggest that efficiency improvements alone cannot offset growing demand and increasingly resource-intensive videos: reference-scenario emissions rise 74-fold, while holding output constant cuts emissions by about two-fifths.

Efficiency & systems

New method makes quantized AI models up to 31% smaller with no quality loss

The authors present a technique that losslessly compresses the already-quantized weights of large language models beyond standard quantization. By tuning the quantizer to produce more compressible codes and placing a parallel Huffman decoder between DRAM and L2 cache, they reduce model size by 5.7–31.4% at the same perplexity compared to the best existing format. On a Jetson Orin Nano Super, latency per token drops up to 36% and energy per token up to 38% (more with prompt-lookup speculation); on a Jetson Nano, a 7B model that otherwise does not fit runs with 58–64% less time and 53–57% less energy per token.

Efficiency & systems

Parallel decoding answers many questions on one document 3-6x faster

Zhao and Flanigan introduce Parallel Prompting, which decodes many independent questions about a shared document as parallel streams within a single LLM inference request, reading the document's key-value cache once per step instead of once per question. On an A100 GPU, this yields 3.7x speedups over standard prefix caching for CodeLlama-7B with long documents and many questions, and up to 6x on an RTX A6000.

Efficiency & systems

New method cuts secure AI inference time by over half

The authors present MOAI, a framework for performing transformer inference on encrypted inputs using fully homomorphic encryption (FHE) without any interaction between client and server. By redesigning the packing of encrypted matrices and devising rotation-free algorithms for Softmax and LayerNorm, they reduce the number of costly FHE rotations by at least 1.7x compared to prior work, cutting total evaluation time by 52.8% relative to the previous best non-interactive system THOR.

Efficiency & systems

Decoupling attention from model weights speeds large-language-model serving by up to 2x

The authors present Decode Token Parallelism (TKNP), a parallelization strategy for serving large language models that separates where model weights are stored from where attention is computed. In tests with Llama 3.1 405B and Nemotron-4 340B on up to 64 H100 GPUs, TKNP improves completion throughput by up to 1.7× over decode context parallelism and up to 2.06× over data parallelism.

Efficiency & systems

Language models generate 5–8% faster by keeping draft alternatives separate

Huang and colleagues identify a structural inefficiency in speculative decoding, where a smaller model proposes text for a larger model to check: plausible continuations get mixed into combinations the larger model would not produce. Their Multi-Future Drafting method keeps alternatives coherent within one drafting pass, delivering 5–8% faster generation than the strongest identically trained single-mode comparison without changing the target model’s output distribution.

Efficiency & systems

LLM serving becomes batch-invariant with near-stock speed

The authors present Detent, a system that makes large language model (LLM) serving engines batch-invariant, meaning a request's output bits do not change when other requests in the batch change, while preserving near-stock latency. Detent achieves this by expressing kernel selection as a program, identifying the minimal set of batch-dependent values that must be fixed, and tuning them for each workload. On multi-GPU deployments of Llama-3.1-70B and Qwen3-30B-A3B in vLLM and SGLang, Detent serves at 1.04–1.17x the latency of standard settings, compared to 1.67–10.3x for existing deterministic modes.

Efficiency & systems

Flow matching speeds up speculative decoding by 39% over existing drafters

The authors introduce ChainFlow, a speculative decoding drafter that uses flow matching on a compressed latent space of the target model's hidden states to generate draft tokens in a single forward pass. Compared to autoregressive drafters like EAGLE-3 and parallel drafters like DFlash, ChainFlow achieves 39% and 11% higher throughput on English tasks with Qwen3.5 models, and shows improved adaptability to new languages.

Efficiency & systems

Sparse GPU grids gain dynamic editing and faster stencil operations

The authors introduce two advances to the NanoVDB data structure: a suite of dynamic topology operators (dilation, merging, pruning, coarsening and refinement) that run directly on the GPU, and a voxel-sequential access paradigm that optimizes stencil and streaming operations. Operators can reconstruct grids with hundreds of millions of voxels in milliseconds, and voxel-sequential processing achieves compute density approaching peak memory bandwidth. This removes the read-only constraint on NanoVDB, establishing it as a primary representation for dynamic GPU simulation and geometry manipulation.

Efficiency & systems

Affine folding cuts KV cache decoding latency by up to 3.4x on Hopper and Blackwell GPUs

Zhuang et al. present FOSCAR, a kernel library that accelerates decode attention for low-bit key-value caches on NVIDIA H100 and B200 GPUs. By folding quantization scale and offset into matrix-product inputs or partial results, FOSCAR reduces instruction demand by orders of magnitude, achieving geometric-mean speedups of 3.36× and 2.83× over FlashInfer BF16 for INT2 and INT4 kernels on H100, and up to 170% peak aggregate throughput increase in long-context serving with SGLang.

Efficiency & systems

Co-scheduling CPU preprocessing and GPU execution slashes training stalls for multimodal models

The authors present DataHelix, a CPU-GPU co-scheduling data plane for multimodal model training. By jointly reasoning about preprocessing demand and GPU progress, it treats each GPU compute window as a budget for preparing the next batch, reducing mean deadline-miss rate by 67.8% and improving throughput 2.72x over rank-local DataLoaders on 11 video and multimodal workloads.

Efficiency & systems

Moving softmax to matrix operations speeds up AI inference by over 2x

The authors introduce Fused-Softmax Attention, a training-free algorithm that reformulates the softmax operation in attention so that its dominant computations are moved from vector units to matrix cores. Implemented on AMD's CDNA 4 architecture, their kernels achieve 2.1x to 3.2x speedups over BF16 attention on an MI355X GPU, with additional error below the quantization noise of the matrix operands. On a full video generation model, this yields up to 2.05x end-to-end speedup with no visible quality loss.

Efficiency & systems

Language model libraries often fail outside their core calculation code

The specialized libraries that run language models can fail because of their surrounding software stack, not just faulty calculation code. Gao and colleagues manually examined 300 GitHub issue reports across seven libraries, finding that compiler, build-system and runtime-environment causes accounted for a majority of the sample. They also identified hidden input and hardware constraints that can make otherwise supported operations return incorrect results or fail to launch.

Efficiency & systems

Small vision-language AI models can be compressed fourfold with only 1% accuracy loss

A new quantization-aware training framework called ProQuant enables low-bit (W4A8) compression of small multimodal language models with only about 1% accuracy drop. The method introduces a self-relabeling scheme that re-annotates open-source multimodal data to better match the model distribution, and a progressive block activation mechanism that first quantizes vision-sensitive layers before language layers. The approach achieves state-of-the-art results, outperforming existing post-training quantization and quantization-aware training methods on models from 2B to 8B parameters.

Efficiency & systems

Weight mapping algorithm for 3D AI chips boosts throughput 2x

The authors propose CAMP, a context-aware weight mapping algorithm for 3D In-Memory Computing (3D-IMC) arrays used in large language model inference. CAMP models potential resource conflicts between matrix-vector multiplications via a conflict matrix and uses a profiling-driven feedback loop to find near-optimal weight placements. The resulting throughput is up to 2x higher for the 3D-IMC chip and 1.74x higher end-to-end compared to state-of-the-art mapping, with compute utilization rising from 42.4% to 60.3%.

Efficiency & systems

Training a gate to drop AI memory defaults to dropping nothing

The authors investigate learnable gating for KV cache eviction in large language models. They find that naively training the gating function on the language-modeling loss leads to it learning a constant score, effectively no-oping because the frozen backbone's loss prefers dense attention. They identify two structural obstacles: the softmax normalizer absorbs constant gates, and the gradient also charges the gate for errors the model already makes.

Efficiency & systems

On-device AI drafters can accept more tokens with less memory by separating context from local attention

The authors propose HyFlash, a compact parallel drafter for speculative decoding on memory-constrained devices. By separating historical context modeling (via recurrent memory) from recent and intra-block interactions (via local attention), HyFlash achieves higher token acceptance than full-context attention while using substantially less KV storage, especially with shallow drafters and long contexts.

Efficiency & systems

Three-axis compression lets medical AI agents run 8x larger batches on a single GPU

The authors propose TACT-KV, a compression method that reshapes the KV cache from 3D medical scan VLMs into a 3D tensor and applies a separable 3D DCT to exploit spatial correlations along all axes. They prove that the distortion gain of adaptive bit allocation depends on the spectral concentration of the transformed cache. In tests on six medical VLMs and two CT benchmarks, TACT-KV achieved the lowest prediction-flip rate among compression methods and enabled up to 8x larger decoding batch sizes at the same memory budget.