A cluster of new academic studies has raised significant questions about the reliability of large language models (LLMs) in legal contexts, finding that AI systems can be unduly swayed by the quality of legal advocates, exhibit unpredictable refusal behaviour when given institutional prompts, and fall prey to coherence illusions similar to those observed in human readers — all findings with serious implications for proposals to deploy AI as legal decision-making tools.
AI in the Courtroom: New Research Highlights Risks
As governments, courts, and legal technology companies explore using large language models as judicial assistants or even first-instance decision-makers, three newly published studies are sounding notes of caution about the readiness of current AI systems for such high-stakes roles.
Persuadability: When Advocacy Skills Distort AI Judgments
A study by Oisín Suttle and David Lillis, published on arXiv, examined how frontier LLMs — both open- and closed-weights models — respond to legal arguments when the perceived quality of the advocate presenting those arguments is varied. Their findings reveal that LLMs are measurably more likely to agree with a legal position when it is presented by an advocate framed as more skilled or authoritative, regardless of the underlying merits of the argument.
In legal settings, this is a critical flaw. A judge or administrative decision-maker is expected to rule on the strength of the law and evidence, not on the rhetorical skill of the lawyers involved. If an AI assistant can be tilted by advocacy quality, it risks systematically disadvantaging parties with less sophisticated legal representation — a concern with profound implications for access to justice.
Over-Refusal: When Safety Features Become Barriers
A separate study by Anastasiia Kucherenko and colleagues investigated small, on-premises LLMs — the kind a law firm or public defender's office might run locally for confidentiality reasons — and found a troubling pattern of over-refusal in legal contexts.
Counter-intuitively, adding authority-style prefixes such as "you are acting as an assistant of the national supreme court" or "you are assisting a defence lawyer" caused refusal rates to increase by between two and twenty times compared to a neutral baseline. This means that the very contextual framing a legal professional would naturally introduce to obtain relevant assistance could cause the AI to become less helpful, not more.
The researchers warn that selective refusal rates could introduce systemic bias by processing some legal queries faster than others — effectively creating a two-tier system depending on which legal topics an AI is willing to engage with.
Coherence Illusions: When AI Mistakes Plausibility for Truth
A third study, led by Ece Takmaz and colleagues, examined a phenomenon borrowed from psycholinguistics: coherence illusions. Human readers sometimes perceive incoherent text as coherent when a nearby word or phrase matches what they expect to come next. The researchers found that Dutch-language LLMs display the same vulnerability.
Using measures of surprisal, attention entropy, and associative-memory energy, the team demonstrated that a contextually matching distractor word could reduce a model's sensitivity to logical incoherence in surrounding text. While this study did not focus exclusively on legal applications, its implications are clear: an LLM reviewing a legal document or argument may be fooled into accepting a flawed chain of reasoning if it superficially resembles a coherent one.
A Pattern of Concern
Taken together, the three studies paint a picture of AI systems that remain fundamentally unreliable for legal decision-making in their current form. They can be persuaded by style over substance, paralysed by the institutional framing they most need to operate within, and deceived by surface-level textual plausibility. Legal scholars and AI researchers are likely to point to these findings as evidence that much more rigorous testing and regulatory scrutiny is required before LLMs are deployed in any judicial or administrative capacity.
Analysis
Why This Matters
- Access to justice is at stake. If AI legal tools respond more favourably to well-resourced advocates, they could entrench existing inequalities in the legal system, disadvantaging self-represented litigants and public defenders.
- Regulatory proposals are already in motion. Courts and governments in multiple jurisdictions are actively considering or trialling AI legal tools; these findings arrive at a critical moment when deployment decisions are being made.
- The flaws are systemic, not incidental. All three studies point to issues baked into current LLM architectures — persuadability, refusal instability, and coherence illusions — not easily fixed with simple fine-tuning.
Background
Interest in AI-assisted legal decision-making has grown rapidly since the public release of powerful LLMs beginning in 2022. Proponents argue that AI could reduce court backlogs, lower the cost of legal services, and improve consistency in administrative decisions. Early pilots have been conducted in jurisdictions including Singapore, Estonia, and several US states, typically for lower-stakes administrative tasks such as benefits determinations or case triage.
However, concerns have mounted alongside enthusiasm. A 2023 incident in which a US lawyer submitted ChatGPT-generated case citations — which turned out to be fabricated — drew widespread attention to AI hallucination risks. Since then, the academic literature has steadily accumulated evidence of additional failure modes: bias in outputs, inconsistency across prompts, and susceptibility to adversarial inputs.
The three studies published this month represent a more systematic attempt to probe specifically legal failure modes, moving beyond anecdote to controlled experimental results. They build on a growing body of work in AI alignment and robustness that questions whether current models can reliably perform high-stakes reasoning tasks.
Key Perspectives
Legal Technology Proponents: Advocates for AI in legal settings argue that the technology need not be perfect to be useful — only better than the status quo of under-resourced courts and inaccessible legal services. They contend that human decision-makers also exhibit biases and are persuaded by advocacy quality, and that AI at least offers the possibility of auditable, consistent reasoning.
Academic Researchers: The authors of these studies do not necessarily oppose AI in legal contexts, but argue that deployment must be preceded by rigorous, domain-specific testing. They highlight that the failure modes they identify are not random errors but systematic vulnerabilities that could be exploited or could consistently disadvantage particular groups.
Critics and Civil Liberties Advocates: Critics argue that the stakes in legal decisions — liberty, property, family — are too high to tolerate the kinds of instability these studies document. They point out that the burden of proof for AI reliability in legal settings should be exceptionally high, and that commercial and efficiency pressures may push deployment before that bar is met.
What to Watch
- Regulatory developments in the EU and US: The EU AI Act classifies AI used in judicial decisions as high-risk; watch for implementing regulations that specify what testing standards must be met before deployment.
- Court and tribunal pilot programmes: Several jurisdictions are expected to announce expanded AI pilots in 2025–2026; the scope and oversight mechanisms of these programmes will indicate how seriously policymakers are taking findings like these.
- Industry response: Whether major LLM providers — OpenAI, Anthropic, Google DeepMind — respond to these studies with targeted research or updated model cards will signal the degree to which legal reliability is being treated as a priority.