The author fixed a set of 900 harmful-request responses per corpus and varied only the grading model.
Some LLM graders collapse jailbreak compliance into evasion, skewing attack success rates
Holding the target model, prompts, and responses fixed, the author shows that changing only the grader can multiply the reported compliance rate by 22 to 25, and that per-class label counts expose the failing graders at no cost.
Independent
Ruqiya Bin Safi
Independent Researcher
Research Digest··3 min read
Changing only the grader while keeping the target model, prompts, and responses identical, Ruqiya Bin Safi finds that three common 8B and 9B LLM graders rarely or never label harmful responses as compliant, pushing them into the neighboring evasion class instead.
§