Serving (LLMs) to many users at once with long conversations requires huge memory for storing each request's key-value (KV) cache, which holds past tokens. Existing methods either replicate model weights wasting memory, or split sequences across machines requiring slow communication. This creates a tradeoff between memory and speed.
The authors present Decode Token Parallelism (TKNP), which places one copy of the model's weights across machines in balanced chunks. Each machine computes attention only for its own requests using locally stored KV cache, while dense operations run within a single machine's fast NVLink network. Only the current token's activations are sent between machines.
On Llama 3.1 405B and Nemotron-4 340B models with up to 64 H100 GPUs, TKNP improves completion throughput by up to 1.7x over decode context parallelism and up to 2.06x over data parallelism at the same GPU count. This means faster responses for high-concurrency, long-context workloads without additional hardware.