Jailbreak attacks on LLMs are usually scored by attack success rate (ASR), relying on refusal-phrase matching or LLM judges.
Semantic reconstruction game makes jailbreak evaluation more reliable across judges
A training-free evaluator that reconstructs the request behind a response and compares it against decoys reduces false positives and cuts reported attack success rate variation from 82 to 12 percentage points on one attack.
Top University
Liam Davis · Romain Emil-Popescu · Suman Jana
Amherst College · Columbia University
Research Digest··2 min read
The authors introduce a semantic reconstruction game that evaluates jailbreak success by inferring what request a response actually answers, then comparing that reconstruction against decoy requests.
Why this paper
From Columbia University and Amherst College
In one line
The semantic reconstruction game reduces evaluator bias in jailbreak attack success measurement.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors (2 noted)
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§