The paper introduces NAQD-Env, a synthetic that tests a language agent's ability to selectively withdraw actions when evidence, permissions, or constraints change. It uses a deterministic reference policy and measures policy agreement, task value, withdrawal, resumption, and event reporting across eleven dependency families. Three open-weight models were evaluated under three prompt conditions on 350 frozen scenarios. The results show that withdrawal recall is at most 0.06, no valid resumption occurred at eligible opportunities, and only one episode matched the full reference policy. Under the NAQD prompt, larger models had fewer unsafe attempts but also completed less useful work and preserved unaffected actions less accurately. Supervised fine-tuning probes improved decision accuracy from 0.45-0.54 to 0.83-0.92, but introduced issues like inappropriate withdrawal after curriculum omissions and loss of event reporting. This work highlights selective withdrawal as an underexplored but critical component of agent reliability.
Language agents rarely withdraw selectively when instructions or permissions change
In a synthetic benchmark, three open-weight models largely failed to suspend affected actions, preserve valid work, and resume after repair.
In one line
Language agents exhibit very low withdrawal recall and no valid resumption in the NAQD-Env benchmark.
Before you read this
Three ideas this paper builds onagent safetyRisks from AI agents including timing errors, shutdown avoidance, collusion, overclaiming, and unsafe execution, mitigated by constrained decoding, safety harnesses, and swarm governance.188 papers →agentic evaluationMeasuring AI agents' performance in complex tasks, highlighting failures in planning, monitoring, and real304 papers →verificationChecking outputs for correctness before accepting them, often to catch errors or reward hacking.341 papers →
Read it as
Abouzahra introduces NAQD-Env, a benchmark for testing whether language agents revise plans selectively when evidence, authorization, or constraints change.
NAQD-Env represents plans using explicit dependencies among evidence, permissions, constraints, and actions.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.