Generating text with large language models happens word by word (decode). This step is very sensitive to delays. Large models must be spread across many chips, and today they are connected by electronic packet switches designed for training. These add latency.
The authors propose static optical circuits set up once per deployment, matching the model's fixed communication pattern. They co-designed this with model parallelism and evaluated it on a large simulated cluster.
For a very large model, interactivity improved by up to two-thirds while energy efficiency roughly doubled compared to the electronic baseline. This suggests a simpler, faster path for future inference clusters.