Koi plans each job’s accelerator, parallelism, serving-engine settings and replica count, then combines those choices into a cluster-wide allocation subject to capacity, cost and service-level constraints.
Koi learns from deployments to plan heterogeneous LLM inference fleets
The system combines language-model-guided search, deterministic performance tools and accumulated deployment evidence to allocate shared accelerators across jobs.
Big Tech
Mankeerat Sidhu · Gangmuk Lim · Hetarth Chopra · Maximillian Eisenberg · Quan Hao Ng · Aryan Gosaliya · +7 more
Tandemn · University of Illinois Urbana-Champaign · UC Berkeley · New York University · ByteDance
Research Digest··3 min read
Thread:Agent Harness Optimization
Sidhu et al.
§