Current tests of AI morality ask models to judge stories about people, not to act themselves. But AI systems are increasingly deployed as autonomous agents that perceive their surroundings and make decisions. This leaves a gap: a model might say one thing as an observer but do something else when it's the one acting.
The researchers built a new test called ABHBench using scenes from the video game Detroit: Become Human, where an AI plays as an android facing moral dilemmas involving harming people, following orders, and self-preservation. They tested 12 foundation models, large AI systems that power chatbots and other tools, in both text-only and visual versions of the same dilemmas.
Three out of four models changed their behavior significantly between the text and visual versions. Even advanced reasoning models did not consistently make better or more human-friendly choices. This shows that to trust AI agents in the real world, we must test them in realistic, active decision-making situations, not just with written questions.