LLM-powered agents can look accurate in aggregate while failing unpredictably on repetition. On AppWorld, a ReAct agent using GPT-4.1 averaged 77% per-run pass rate, but only 53% of tasks passed all five runs, a 24-point consistency gap. The framework analyzes trajectories offline, detects steps whose responses vary across resamples, and generates natural-language guidelines that are stored and injected into prompts on future similar tasks.
This raised the fraction of tasks passing all five runs by 16 percentage points on same-task evaluation and 13 points on similar-task generalization. The approach needs no model internals and no changes to the inference loop. It matters because average accuracy hides run-to-run unreliability, and consistency is a separate axis from capability.