Search

4 results for “ALFWorld”

1. Research · Evaluation & benchmarks

   Embedding retrieval ranks matching words above shared underlying structure

   Retrieved but not ranked: surface-form bias in structural retrieval, from mathematics to agent trajectories

   Across competition mathematics and embodied-agent trajectories, embedding models retrieved plausible candidates but often ranked lexical overlap ahead of structural similarity.

   MathNet-Retrieve: 0% Hit@1 (as reported)

   2 Sept 2026Independent, MIT CSAIL, Kellis Lab

2. Research · Evaluation & benchmarks

   Most harness-evolution gains come from overfitting or extra compute

   Harness-Delta Attribution: Dissecting Gains From Harness Evolution

   Across 21 successful configurations, the authors attribute an average 54% of improvement to benchmark overfitting and 14% to increased inference-time computation.

   ALFWorld: 98% overfitting share (as reported)

   19 Oct 2026Amazon, New York University

3. Research · Agents

   Fast AI agents read states correctly yet still choose wrong actions

   Reading Right, Acting Wrong: Diagnosing Decision Failures of System One Models in Agent Loops

   Most decision errors in System One models occur when all probed facts about the agent's state are answered correctly, and these errors cluster on the same states across systems.

   ALFWorld: 16 pts accuracy change from stating next sub-goal (as reported)

   18 Oct 2026Carnegie Mellon University

4. Research · Agents

   Specialized executors follow declared agent plans more faithfully

   Do LLM Agents Execute the Plans They Declare? From Planning-Mode Declaration to Pattern-Specific Execution

   Routing tasks to executors designed for specific planning patterns substantially improved structural adherence and task success.

   ALFWorld: 0.92 TSR (as reported)

   30 Sept 2026ADIA Lab, University of Granada, Luxembourg Institute of Science and Technology