The paper reports on a twelve-week deployment of Karpathy's AutoResearch paradigm in Amazon's book recommendation pipeline. The agent autonomously modifies training scripts to improve retrieval embeddings, with experiments spanning two systems of vastly different computational costs (from minutes to hours per iteration). The key finding is the identification of five failure modes that emerge at production scale: infrastructure fragility, decay, search-direction stagnation, iteration-cost asymmetry, and metric fixation. The authors propose a three-principle scaffolding design—prevent, persist, redirect—that structurally addresses each failure mode. The framework achieved a 1.82x Recall@6 lift and a 2.1x coherence lift over hand-tuned baselines, and the agent autonomously designed a text-only fallback that expanded catalog coverage by 5.8x. The fact that the same five failure modes appeared across two systems with nearly three orders of magnitude difference in iteration cost suggests these are structural properties of production-scale autonomous research, not application-specific artifacts.
Production-scale AutoResearch exposes five failure modes and a three-principle fix
Twelve weeks of running Karpathy's iterative LLM search across two Amazon book recommendation systems yielded 1.82x and 2.1x metric lifts, while revealing structural failure modes.
Big Tech
Amazon
Research Digest··3 min read
In one line
AutoResearch at production scale reveals five failure modes fixed by a prevent-persist-redirect multi-agent framework.
Before you read this
Three ideas this paper builds onmulti-agent architectureA system where multiple AI agents collaborate, each handling specialized roles, communicating and coordinating to complete complex tasks.117 papers →deep researchA multi-step AI system that autonomously gathers, evaluates, and synthesizes evidence from diverse sources to produce comprehensive reports.25 papers →agent memoryA repository where an agent stores past interactions and outcomes to inform future decisions.241 papers →
Read it as
The authors applied AutoResearch, an LLM that iteratively edits training scripts and keeps changes that improve a held-out metric, to two production embedding systems at Amazon.
The authors ran Andrej Karpathy's AutoResearch paradigm in two independently developed representation-learning systems for Amazon's book recommendation pipeline.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.