In unfamiliar homes, robots cannot see everything at once. They must decide where to look and how to act despite uncertainty about object locations and whether a move is physically possible. Directly calling tools can produce actions that fail those physical constraints.
TAMPAgent combines a large , which reasons from images and language, with task and motion planning (TAMP), which checks whether actions are feasible. The robot keeps a “belief,” or best current guess about the world, observes and updates it, and gets precise grasp and movement values certified before acting.
On simulated kitchen tasks, TAMPAgent achieved 77.2% success and outperformed direct tool-calling approaches. This shows how checking physical feasibility within a plan can help robots act reliably when their surroundings are uncertain.