The authors built InferOpt around three inputs: bounded configuration variables, a deterministic resource-cost function, and an evaluation hook.
Search framework finds better layer-wise settings for efficient LLM inference
InferOpt jointly searches model quality and resource use across continuous KV-cache retention and discrete mixture-of-experts routing configurations.
Chinese Tech
Qi Chen · Yingying Cheng · Zhaoyi Sun · Li Zhou · Fan Zhang · Jie Sun
Huawei Technologies Co., Ltd.
Research Digest··2 min read
Chen and colleagues formulate inference configuration as constrained multi-objective black-box optimization, seeking Pareto-efficient settings that balance model quality against memory or computation.
Why this paper
From Huawei Technologies Co., Ltd.
In one line
InferOpt replaces mechanism-specific heuristics with a unified search framework for LLM inference configurations.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ✓Reports numbers on named benchmarks (2 benchmarks)
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§