Generating each new word (token) with a large language model uses a fixed amount of math. The energy for that math sets a lower limit, but today's general-purpose chips waste hundreds to thousands times more energy because they are flexible, run software, and must fetch data from memory.
The authors designed a special-purpose chip that holds the model's weights and all conversation history on the chip itself, avoiding slow off-chip memory. They then co-designed the model and the circuits together, tuning both for the exact task, instead of separating them.
Their design for the Qwen3-1.7B model reaches just 1.8 times the physical energy limit, uses 0.52 millijoules per token, and handles twelve users at once. Compared to a measured NVIDIA B300 serving the same twelve users, it is 110 times faster and uses 690 times less energy.
