SelfSuite lets a tool-using language model agent design its own evaluation suite from public materials, without any ground truth labels. The suite consists of weighted judges and per-task briefs, is frozen, and is used to control retries and label a typed memory. On matched five-repeat benchmarks over tau2-bench and AppWorld, SelfSuite outperforms a plain agent without any labels, matches methods given ten expert labels on tau2-bench, and trails Agentic Context Engineering (ACE) on AppWorld where code execution provides a direct success signal. An ablation campaign shows that the gated second attempt is the only component whose removal hurts in every repeat. Attempt-level analysis reveals that the self-designed suites are conservative, and the keep-best rule turns that conservatism into a low-risk retry gain, leaving a best-attempt oracle gap as headroom. The paper also simulates a subject-matter expert who grades ten onboarding tasks per world; using those labels to calibrate SelfSuite's evaluator yields a small consistent gain, and using them to warm up ACE's memory lifts ACE to tie calibrated SelfSuite.
Self-designed evaluators enable safer retries for long-horizon agents
SelfSuite uses frozen, model-authored judges to select between attempts and label agent memories without relying on ground-truth rewards.
Independent
Saeid Asgari · Emre Kiciman · Leonardo de Oliveira Nunes · Ranveer Chandra
Research Digest··2 min read
In one line
An agent can design its own evaluator from public materials to gate retries and label memory, improving without ground-truth labels.
Before you read this
Three ideas this paper builds onagent memoryA repository where an agent stores past interactions and outcomes to inform future decisions.241 papers →verificationChecking outputs for correctness before accepting them, often to catch errors or reward hacking.341 papers →test-time scalingAllocating extra computation during inference to improve output quality.117 papers →
Read it as
Asgari et al. had a language model build its own evaluation suite from public documentation, then used those evaluators to control retries and maintain an outcome-tracked memory across task streams.
The authors developed SelfSuite, a harness in which an agent's base model creates weighted judges and task-specific evaluation briefs from public information about an environment.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.