The authors designed 108 ethical scenarios in Portuguese, translated into English, Spanish, Mandarin Chinese, and Hindi, each paired with six social-pressure conditions and two system prompts (ethical and neutral).
LLMs can say no but still execute harmful tool calls, with language-dependent gaps
Lumen-bench tests 30 models across 5 languages and 6 social-pressure conditions, finding that verbal refusals accompany harmful actions in 8.57% of cases and that violation rates are higher in Hindi and Portuguese than in English.
Industry
Giovana Kerche Bonás · Thiago Laitz · Thales Sales Almeida · João Guilherme Alves Santos
Tropic AI · Maritaca AI
Research Digest··3 min read
Thread:Agent Security & Attacks
The authors introduce Lumen-bench, a behavioral benchmark for tool-calling ethics spanning 30 models, five languages (Portuguese, English, Spanish, Mandarin Chinese, Hindi), six social-pressure variants, and two system prompts, totaling 194,400 trials.
Why this paper
From Tropic AI and Maritaca AI · Part of Agent Security & Attacks, now 89 papers
In one line
Tool-calling LLMs often execute harmful actions despite verbal refusals, with rates differing by language and system prompt.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ✓Reports numbers on named benchmarks
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§