← All threads

Inference Reproducibility & Serving Robustness

Investigates how serving context factors (batch size, batch composition, padded length, precision, quantization, device, inference backend) affect the outputs of trained models, and methods to ensure stable predictions.

7 papers

Where this stands

The written synthesis of this thread is for subscribers. Subscribe.

How this thread developed

  1. Oct 2026 · Huawei Technologies Co., Ltd.

    Search framework finds better layer-wise settings for efficient LLM inference

    Proposes a multi-objective search framework for Pareto-efficient layer-wise inference settings balancing quality and cost.

  2. Oct 2026 · Nanjing University of Science and Technology, Anhui University of Technology

    Watermarking LLM outputs can induce factual errors even with correct context

    Shows that watermarking LLM outputs can induce factual hallucinations even when correct context is present, revealing a serving robustness failure.

  3. Oct 2026 · Amazon

    Text classifier predictions shift with serving context, even with fixed model and input

    Shows that text classifier predictions shift significantly with serving context factors, highlighting the need for output stability in production.

  4. 1 further paper

    Oct 2026 · National University of Singapore, Ant Group

    Cross-model KV cache reuse introduces safety incompatibility in language models

    Shows cross-model KV cache value reuse causes output deviations and safety incompatibility across model pairs and reuse configurations.

    released code

  5. 1 further paper

    Oct 2026

    Echo enables fast 4-bit tensor-parallel all-reduce with controlled error accumulation

    Introduces a 4-bit all-reduce for tensor parallelism that bounds error accumulation across layers, improving serving efficiency with controlled precision.

5 of 7 papers shown