What they did

The authors studied outcome-blind, long-horizon agent training, where no ground-truth success signal is available. DRACO generates multi-criteria rubrics dynamically as the policy changes, scores each rubric once after a completed trajectory, and identifies the steps associated with individual rubric criteria.

It then uses a closed-form procedure to redistribute the trajectory-level judgment into differentiated per-step advantages for Group Relative Policy Optimization (GRPO). This avoids training a separate attribution model. The method was evaluated on AppWorld and transferred to the out-of-domain τ-bench.

Key findings

• On AppWorld, DRACO improved performance by 15.9 points over the base model.
• It exceeded GRPO trained with a sparse ground-truth reward by 5.3 points on AppWorld, despite using no verifier.
• On out-of-domain τ-bench, DRACO gained 5.3 points over the base model without relying on a frontier judge.
• On τ-bench, it also outperformed ground-truth-reward training and the other tested rubric-based training settings.

Why it matters

Long-horizon agents often receive only a final evaluation, making it difficult to determine which actions deserve credit or blame. DRACO suggests that evolving natural-language rubrics can supply useful, fine-grained training signals even when verifiable rewards are unavailable, potentially broadening reinforcement learning to tasks without programmatic checkers.

Caveats

The reported evidence comes from two agent benchmarks, and the abstract does not specify the statistical variability, judge reliability, computational cost, or sensitivity to rubric quality. The method still depends on rubric generation, scoring, and step annotations being accurate enough to guide training; its robustness in noisier real-world settings remains an open question.