LLM serving becomes batch-invariant with near-stock speed

A new system, Detent, pins the few batch-dependent values in kernel selection to ensure identical output bits across different batch compositions.

Big Tech

Peixu Xin (Independent Researcher, Boston University) · Runchuan Feng (Independent Researcher, Boston University) · Jiazhen Guo (Google) · Kan Wu (RadixArk) · Ming Yin (Princeton University, Georgia Institute of Technology)

Research Digest·10 November 2026·3 min read

Thread:Inference Cache Optimization

Illustration: LLM serving becomes batch-invariant with near-stock speed

In one line

Detent achieves batch-invariant LLM serving and RL rollouts at near-stock speed by pinning only the batch-dependent values that affect kernel selection.

Before you read this

Three ideas this paper builds on

group relative policy optimizationA reinforcement learning method that optimizes policy by computing advantages based on group-level reward statistics.180 papers →large language modelA neural network trained on massive text corpora to generate coherent text and perform diverse language tasks.301 papers →mixture of expertsA neural architecture with many specialized sub-networks, only a few activated per input via a router, balancing capacity and efficiency.82 papers →

Read it as

In (LLM) serving, the output a request gets can change depending on which other requests are sharing the same batch. This makes evaluations unrepeatable and can cause reinforcement learning (RL) training to fail because the training data (rollouts) don't match bit for bit.

Detent identifies the few batch-dependent values that influence which fast computation kernels the engine picks. It fixes those values for a given workload while keeping the fast kernels themselves, so the same request always gets the same output regardless of batch composition.

Detent serves at 1.04 to 1.17 times the speed of the standard engine, far faster than fully deterministic modes. It also prevents RL training collapse by ensuring bit-level reproducibility.

How it works

stage by stage

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

batch invariance kernel selection rollout verification group relative policy optimization large language model mixture of experts