HumanoidToolBench is a new that tests humanoid robots on using tools, from selecting the correct tool to actually performing a task with it, including locomotion. It includes 18 tasks with different scenarios, execution levels, and tool sets, and provides 3.1k demonstrations for training and evaluation.
The benchmark revealed that current humanoid policies often fail to complete tasks even after selecting the right tool. When presented with unfamiliar tools or given instructions that are irrelevant to the tool, selection accuracy drops and policies continue executing unrelated actions.
This is interesting because it shows that tool use in humanoids is not just about manipulation but also about robust decision-making. The benchmark provides a standardized way to measure these capabilities and highlights specific weaknesses that need to be addressed.