Language models can gain accuracy by adjusting precision during generation

The authors report accuracy approaching a six-bit baseline while retaining approximately 90–98% of four-bit decoding throughput on GB300 GPUs.

Independent

Akshat Ramachandran · Reena Elangovan · Rangharajan Venkatesan · Charbel Sakr · Tijmen Blankevoort · Brucek Khailany · +1 more

Research Digest·10 November 2026·3 min read

Thread:Low-Bit Quantisation

Illustration: Language models can gain accuracy by adjusting precision during generation

In one line

AutoEMatic dynamically adjusts LLM inference precision at each step, improving accuracy by up to 21.7% over static NVFP4 with minimal throughput loss.

Before you read this

Three ideas this paper builds on

post-training quantizationQuantizing pre-trained neural network weights and activations to49 papers →test-time adaptationAdapting a deployed model during inference using new data or interactions, without a separate training phase.82 papers →kv cache compressionReducing the size of the key-value cache in transformers via selective retention, low-rank updates, or adaptive compression.78 papers →

Read it as

Large language models are expensive to run. Typically their numbers are compressed to use fewer bits, but the compression level is set before runtime and never changes, wasting resources because different parts need different precision.

This paper describes a system that adapts the number of bits for each layer on the fly, using signals from the model's internal behavior, like prediction uncertainty, to pick the right precision. A new number format makes it possible to switch between 2, 4, and 6 bits without extra storage.

The result: up to 21.7% better accuracy than a fixed 4-bit format, with an average of 4.9 bits per weight and virtually no speed loss. This shows that dynamic precision can make AI inference both more accurate and efficient.

How it works

before and after

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

post-training quantization runtime adaptation multi-scale quantization test-time adaptation kv cache compression