PhoneCLI addresses the inefficiency of mobile GUI agents that repeatedly rely on slow, costly (VLM) calls for every navigation step. It offline explores an app from the outside, without any internal API or training, building a map of screens and interactive elements. Each screen becomes a deterministic command that can be replayed in sub-second time at zero VLM cost.
In online use, the agent selects a command for a task, verifies it, then executes it deterministically. Only when a command fails or open-ended interaction is needed does it fall back to the VLM, so performance never degrades. On AndroidLab, PhoneCLI improved task success rate while reducing steps and token consumption. It also transferred to AndroidWorld's official M3A agent with consistent efficiency gains.
This is interesting because it removes the most repetitive part of mobile app interaction—navigation—from the VLM loop, without requiring model updates or app-specific training. The compiled knowledge supports new tasks, not just repeated ones, and integrating a new app takes only minutes.