Building a complete game requires coordinating rules, interactions, states, and visuals over a long process. A single score hides which parts an AI system fails to implement. OCTAVE provides a diagnostic for HTML5 game generation.
The benchmark includes 180 tasks with 1,800 requirements covering eight key game development capabilities, from core rules to feedback. Requirements are split into public (completeness) and hidden (richness) to test both explicit and deeper implementation.
When many AI systems were evaluated, the best achieved only about a third on both public and hidden requirements. AI/NPC behavior and game feel were particularly weak. This diagnosis helps developers focus on the most challenging aspects of game generation.
