LLM serving requires tuning many inference knobs like KV-cache retention and MoE expert counts per layer. Current practice uses mechanism-specific heuristics that return a single operating point and do not scale to layer-wise search. InferOpt reformulates inference configuration as constrained multi-objective black-box optimization, providing a reusable search framework that only needs variable bounds, a resource-cost function, and an evaluation hook. It searches on a frozen proxy set, rejects over-budget candidates before model calls, adaptively tightens the budget, and re-validates Pareto representatives on the full dataset.
On a 28-dimensional KV-cache space for Qwen2.5-7B, InferOpt's searched allocation cuts the 16K cache by 64.4% and token-generation time by 22.9–48.5%, beating a matched uniform budget by 7.3% and 14.0% of the Full KV reference. On a 26-dimensional MoE space for DeepSeek-V2-Lite, it removes 43.0% of routed token–expert pairs while staying within 0.59 points of the default validation score, closer than uniform baselines. InferOpt also outperforms Random Search, NSGA-II, and MOTPE on both proxy hypervolume and final retention.
The work is interesting because it shows that a single, budget-aware optimization framework can replace ad-hoc tuning heuristics across very different inference mechanisms, without mechanism-specific redesign. This allows practitioners to explore high-dimensional configuration spaces efficiently and obtain full Pareto frontiers instead of a single operating point.