CABRA constructs programs as directed acyclic call graphs, then transforms them into coding tasks whose size can be varied while holding other properties constant.
Code understanding, not edit volume, limits coding agents
Controlled synthetic tasks and SWE-bench traces indicate that the work required to comprehend code predicts agent difficulty better than the amount edited.
Big Tech
Nishant Balepur · Kiran Tomlinson · Tobias Schnabel
Microsoft Research · University of Maryland
Research Digest··3 min read
Balepur, Tomlinson and Schnabel introduce CABRA, a framework that generates controlled coding tasks from call graphs and varies their difficulty along specific dimensions.
Why this paper
From Microsoft Research and University of Maryland
In one line
Code understanding is the bottleneck for coding agents: synthetic tasks show tools mask other LLM weaknesses, and understanding effort predicts errors better than edit count.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§