The authors grounded 1,800 natural language planning tasks in hidden PDDL models, which mechanically decide action applicability, goal satisfaction and plan existence.
Formal plan validation shows LLM agents rarely plan as claimed
Testing 1,800 episodes against hidden PDDL models, the authors find that online agents solve at most 40% of solvable tasks and rarely detect unsolvable ones.
Top University
Oriol Miro-Lopez-Feliu · Oscar Miró López-Feliu · Victor Gimenez-Abalos · Javier Vazquez-Salceda · Victor Carballo Araruna · Sergio Alvarez-Napagao
Barcelona Supercomputing Center · University of Amsterdam · Safe AI Netherlands · Universitat Politècnica de Catalunya
Research Digest··2 min read
Miro-Lopez-Feliu et al.
§