Software engineering agents automatically navigate code to fix bugs, but current evaluations often leak future code information or hints, making it unclear if agents truly understand the codebase.
The researchers created a strict test that prevents such leakage by locking code snapshots and removing hints from prompts. They compared three ways of giving agents code knowledge: no extra structure, a flat syntax graph, and a hierarchical graph that organizes code into packages and modules across large codebases and multiple foundation models.
The hierarchical graph improved localization accuracy (the ability to find the right files to modify) by at least 15.6 percent relative to the others. Importantly, equipping a smaller model with the graph allowed it to match or beat a larger model that lacked it, showing that investing in structured knowledge representation can reduce the need for the largest models.
