Agents that turn instructions into actions need examples linking words, what the agent sees, and the keys and mouse moves. Collecting such examples in the real world is slow and expensive, so video games are a practical alternative. Earlier game datasets were limited to few games, sparse language labels, or stage boundaries based on time rather than actual changes of objective.
AceAction collects human play across many 3D games and labels each episode with an overall task, ordered subgoals at each stage change, and fine-grained skill instructions aligned frame by frame to keyboard-and-mouse input. A vision-language model pipeline detects when goals or interaction targets change, and automated and human checks verify the annotations.
In an inspection of 500 sampled trajectories, reviewers accepted 95.8 percent of annotations. By making subgoal transitions explicit, the dataset gives agents a clear signal for when to switch skills, which is essential for following complex instructions in changing game environments.