The authors applied the twelve bias tests from the CALM benchmark to eight TDMs, including Jev and Perplexity Decider, and compared them with seven LLM judges (from 2024 and 2026) on identical prompts.
Typed decision models show bias but offer cheaper, consistent judgment
The CALM benchmark reveals all tested TDMs are swayed by emotional tone and answer order, with accuracy dropping from 99% to 42%, yet they cost 37 times less than 2026 LLM judges.
Academic
Fabian Lukassen · Christoph Weisser · Alexander Silbersdorff
University of Göttingen · Bielefeld University of Applied Sciences and Arts (HSBI)
Research Digest··2 min read
Lukassen, Weisser, and Silbersdorff systematically tested eight typed decision models (TDMs) on the CALM bias benchmark for LLM judges.
§