Running large language models (LLMs) efficiently is difficult because different parts of the model have different bottlenecks. For example, the first stage, which processes the input, mostly uses compute power, while the later stage, which generates each new word, mostly reads memory. These differences mean no single type of accelerator is optimal for all parts. Previous attempts split the work into fixed chunks, like separating the compute-heavy and memory-heavy stages, but this lacks flexibility for different models and hardware.
HELIX is a software system that automatically finds the best way to split an LLM's work across a mix of accelerators, such as different GPUs and specialized chips. It does this by profiling how fast each part of the model runs on each type of accelerator, then uses a planning algorithm to decide which piece of the model goes where to minimize total response time. It can even run just a few operations on a less capable accelerator if that helps.
The main finding is that this fine-grained, automatic partitioning reduces the time to generate many answers by up to 3.5 times compared to the best fixed splitting methods. This matters because it allows data centers to mix cheaper or specialized accelerators with powerful GPUs to serve LLMs faster and more cost-effectively, without manually redesigning the system for each new model or hardware combination.