Large language models generate responses using a cache of prior computations. When an AI agent edits its conversation history, this cache becomes outdated, forcing a slow recomputation before the model can respond. This delay makes the agent feel unresponsive.
Callo reduces this delay by using the outdated cache to produce a guess while the correct cache is recomputed. Once the correct cache is ready, Callo checks the guess and keeps the tokens that match, without harming output quality.
In tests, Callo sped up all requests by 1.05 to 1.19 times compared to vLLM, a standard serving system. This improvement matters because it makes interactive AI agents faster and more responsive.
