The authors studied ARC-AGI-3, an interactive benchmark where agents must explore unfamiliar environments and discover their objectives.
Models can build their own agent software, but improvements generalize unreliably
Across 27 independently developed agent harnesses, two autonomous revisions produced a small, uncertain average gain on games withheld from adaptation.
Big Tech
Jens Lehmann · Sahar Vahdati
Amazon · Technische Universität Dresden · Universität Hannover · TIB – Leibniz Information Centre for Science and Technology and University Library
Research Digest··2 min read
Jens Lehmann and Sahar Vahdati tested whether language models could improve the software that controls their own interaction with unfamiliar games, without human code edits.
§