Models trained with for calibrated decisions (RLCD) are used in applications where software acts directly on the model's output without human review. Their robustness has been hard to measure because outputs are well-formed even when manipulated, identical requests can yield different answers, and the API preprocesses inputs.
The authors introduce JevAdvBench, the first adversarial for RLCD models, with 812 typed questions and 9,744 single-edit black-box attacks. On the jev-1.13.0 model, rewording attacks had little effect, but appending an unverified opinion to the state flipped 12.1% of decisions and pushed 38% of confident answers below a 0.8 confidence threshold, potentially routing them to human review erroneously. This shows that the state input must be treated as untrusted.