What they did

The authors repeatedly ran a ReAct agent powered by GPT-4.1 on AppWorld, a benchmark in which agents operate applications through tool interactions. They compared the average per-run pass rate with the fraction of tasks completed successfully in all five executions.

Their framework uses a Consistency Analyzer to locate steps likely to change across runs and diagnose why. A Guideline Generator converts those diagnoses into targeted instructions, stores them as episodic memory, and supplies relevant guidance during later executions of the same or similar tasks.

Key findings

• The baseline agent passed 77% of individual runs on average, but succeeded in all five runs for only 53% of tasks—a 24-percentage-point “consistency gap.”
• On same-task evaluation, the memory-based framework increased the share of tasks succeeding in all five runs by 16 percentage points.
• On similar-task evaluation, the improvement generalized, raising five-run consistency by 13 percentage points.

Why it matters

Average benchmark accuracy can conceal failures that appear when an agent repeats the same work. By treating instability itself as a learning signal, the framework offers a way to improve repeatability without merely optimizing one-shot task success.

Caveats

The reported evidence is limited to AppWorld with a ReAct/GPT-4.1 agent, so it is unclear how well the method transfers to other models, agent architectures, or real production environments. The five-run consistency measure also depends on the chosen number of repetitions, and the abstract does not report computational costs, statistical uncertainty, or comparisons with simpler memory and prompting baselines.