PASTABench is a of 1,139 multi-turn agent trajectories with hazards across 5 categories and 13 subcategories. Each trajectory is annotated with Earliest-Signal and Trigger turns, defining an Optimal Intervention Window (OIW) so a monitor can be scored on whether to intervene, when to intervene, and what the risk is. Evaluation of 16 LLMs finds the best model intervenes with optimal timing only 40.74% of the time.
A fine-grained diagnosis shows that competitive safety scores on smaller models hide lexical overfitting: they are hypersensitive to hazard keywords rather than genuinely understanding risk. When hazard vocabulary is neutralized, their proactive capability largely collapses. This suggests current safety evaluations may overestimate agent risk comprehension.