Inference Reproducibility & Serving Robustness
Investigates how serving context factors (batch size, batch composition, padded length, precision, quantization, device, inference backend) affect the outputs of trained models, and methods to ensure stable predictions.
7 papers
Where this stands
The written synthesis of this thread is for subscribers. Subscribe.
How this thread developed
Oct 2026 · Huawei Technologies Co., Ltd.
Search framework finds better layer-wise settings for efficient LLM inference
Proposes a multi-objective search framework for Pareto-efficient layer-wise inference settings balancing quality and cost.
Oct 2026 · Nanjing University of Science and Technology, Anhui University of Technology
Watermarking LLM outputs can induce factual errors even with correct context
Shows that watermarking LLM outputs can induce factual hallucinations even when correct context is present, revealing a serving robustness failure.
Oct 2026 · Amazon
Text classifier predictions shift with serving context, even with fixed model and input
Shows that text classifier predictions shift significantly with serving context factors, highlighting the need for output stability in production.
1 further paper
Oct 2026 · National University of Singapore, Ant Group
Cross-model KV cache reuse introduces safety incompatibility in language models
Shows cross-model KV cache value reuse causes output deviations and safety incompatibility across model pairs and reuse configurations.
released code
1 further paper
Oct 2026
Echo enables fast 4-bit tensor-parallel all-reduce with controlled error accumulation
Introduces a 4-bit all-reduce for tensor parallelism that bounds error accumulation across layers, improving serving efficiency with controlled precision.
5 of 7 papers shown