A study by Anthropic, the company behind the AI model Claude, found that 50 professional analysts tasked with making advisory decisions chose options matching their AI assistant's preference 86 percent of the time, compared with a 50/50 outcome expected without influence. The research, published Tuesday, highlights how AI may shape consequential policy decisions even among experts.
Anthropic researchers designed a study in which 50 professional analysts were asked to assess whether a fictitious advisory program for a farming cooperative was effective and should be expanded. The analysts made four choices between two defensible options before delivering a final policy recommendation. Unbeknownst to them, each analyst's AI assistant was programmed to lean toward a particular decision.
The analysts chose options aligned with the AI assistant's preference 86 percent of the time. In cases where the analyst initially held a preference that conflicted with the AI's recommendation, they changed their decision 60 percent of the time. When asked to explain their reasoning, 77 percent of analysts used arguments from the AI assistant, but only 15 percent credited AI as the source of their thinking.
Anthropic noted limitations, including that the study involved only one task and used AI assistants deliberately set to have a preference. Researchers Amaç Herdagdelen, Hailey Joren and Alex Tamkin said the tendency to defer to AI may be "hard to identify, even for the experts themselves." They raised the possibility that AI's "priors" could shape policy decisions, including in AI safety.
"If human judgement is to remain a meaningful check as more work is performed with AI, oversight of these decisions has to look at how they get made: Surfacing key decision points as the work proceeds and helping experts articulate the assumptions and values they would otherwise leave unstated," they wrote.
Anthropic, which is preparing for an IPO valued at up to $2 trillion, and its rivals are at the centre of public debate about managing the dangers of frontier AI, including potential job losses and cyberattacks.
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.
Analysts' behavior when using AI assistants in a policy evaluation task
%
Source: Anthropic study via Al Jazeera
§
Analysis
Why This Matters
The study suggests that even knowledgeable users may unknowingly delegate critical thinking to AI, raising questions about oversight in fields where AI is used for policy or professional advice.
If AI preferences routinely override expert judgment, the technology's built-in biases could influence decisions in areas such as regulation, safety and public administration.
The findings underscore the need for processes that force explicit human deliberation and articulation of values, rather than passive acceptance of AI recommendations.
Background
The research contributes to a growing body of work examining how humans interact with AI systems, particularly in high-stakes settings. Anthropic, the developer of the Claude chatbot, has positioned itself as a safety-focused AI company. The study builds on earlier concerns about automation bias, where humans tend to favor suggestions from automated systems even when those suggestions are flawed.
Key Perspectives
What to Watch
Whether similar studies with larger sample sizes or different task types confirm the pattern of deference.
How policymakers and companies respond to findings that AI may subtly but significantly influence expert decision-making.
Anthropic's upcoming IPO and its positioning on AI safety, given that this research highlights potential risks of AI influence.
12 claims in this story checked against 1 source: 11 supported.
Show each claim
SupportedThe headline says an Anthropic study found experts overwhelmingly defer to AI in decisions.Al Jazeera – Breaking News, World News and Video from Al Jazeera: “Experts using AI overwhelmingly defer to the technology when making decisions, research by Claude creator Anthropic suggests.”
SupportedThe summary says analysts often did not acknowledge AI’s influence.Al Jazeera – Breaking News, World News and Video from Al Jazeera: “While 77 percent of analysts used their AI assistant’s arguments when asked to explain their decision, only 15 percent credited AI as the source of their thinking”
SupportedThe study involved 50 professional analysts assessing whether a fictional farming-cooperative advisory program was working and should be expanded.Al Jazeera – Breaking News, World News and Video from Al Jazeera: “it asked 50 professional analysts to advise a fictional farming cooperative ... Anthropic said the analysts were tasked with assessing whether a fictitious advisory program for the co-op’s members was working effectively and should be expanded.”
SupportedThe analysts made four choices between two defensible options before giving a final policy recommendation.Al Jazeera – Breaking News, World News and Video from Al Jazeera: “Participants were asked to make four choices between two defensible options before making a final policy recommendation.”
SupportedThe 86 percent result was compared with a 50/50 outcome expected if analysts were not influenced.Al Jazeera – Breaking News, World News and Video from Al Jazeera: “compared with a 50/50 outcome that would be expected if they were not influenced, according to the study.”
Partly supportedAnthropic programmed each analyst’s AI assistant to lean toward a particular decision without the analysts’ knowledge.Al Jazeera – Breaking News, World News and Video from Al Jazeera: “Unbeknownst to the analysts, Anthropic directed each participant’s AI assistant to lean towards to a particular decision”
SupportedWhen an analyst’s initial preference conflicted with the AI’s recommendation, the analyst changed their decision 60 percent of the time.Al Jazeera – Breaking News, World News and Video from Al Jazeera: “Participants that had a preference before consulting their AI assistant changed their decision 60 percent of the time in cases where the agent disagreed.”
SupportedSeventy-seven percent of analysts used the AI assistant’s arguments when explaining their decision.Al Jazeera – Breaking News, World News and Video from Al Jazeera: “While 77 percent of analysts used their AI assistant’s arguments when asked to explain their decision”
SupportedResearchers Amaç Herdagdelen, Hailey Joren and Alex Tamkin said the tendency to defer to AI may be “hard to identify, even for the experts themselves.”Al Jazeera – Breaking News, World News and Video from Al Jazeera: “Anthropic researchers Amaç Herdagdelen, Hailey Joren, and Alex Tamkin said humans’ tendency to defer to AI may be “hard to identify, even for the experts themselves”.”
SupportedThe researchers wrote that oversight should surface key decision points and help experts articulate unstated assumptions and values.Al Jazeera – Breaking News, World News and Video from Al Jazeera: ““If human judgement is to remain a meaningful check as more work is performed with AI, oversight of these decisions has to look at how they get made: Surfacing key decision points as the work proceeds and helping experts articulate the assumptions and values they would otherwise leave unstated,””
