Phone assistants that operate apps usually work by generating the next instruction word by word with a huge neural network. This is slow and needs lots of memory, like typing a sentence letter by letter every time the assistant chooses where to tap.
The authors built a system that takes the screen image and creates a compact representation of it using a (JEPA), a model trained on screenshots. For each possible action, such as tapping a button or swiping, it transforms that representation slightly and scores all candidates against the requested task in one pass, without generating any words.
On a phone-control benchmark, this approach was several times faster and used far fewer resources than a much larger language-vision model, while still reaching most of its accuracy. That points toward faster, lighter phone agents that decide from representations instead of generating text.