Agentic AI powered by large cloud models faces issues with latency, privacy, and cost. This work introduces on-device operation caches that transform the natural language to action problem from a generative task into a classification task. A semantic matcher identifies recurring operations and handles them locally.
The authors tested this on generating Excel formulas from natural language. The on-device cache achieved a 55% hit rate with over 96% correctness. This led to a 56% reduction in total inference cost compared to cloud-only inference and a 5x speedup in response time on cache hits.
This approach is interesting because it directly tackles the biggest practical barriers to deploying agentic features broadly: cost and user experience. By handling common cases locally, it dramatically cuts expenses and latency without relying on constant cloud connectivity.