The authors recorded agent trajectories across 37 coding tasks and extracted task-state facts at repeated checkpoints.
Coding agents’ reports omit much of their recoverable task state
Across 37 coding tasks, report-only reviewers recovered 52.0% of task state, compared with 92.7% for reviewers given full execution context.
Chinese Tech
Guoyang Xu · Rui Zhang · Xiuyu Wu · Hao Chen
Tencent
Research Digest··3 min read
Xu and colleagues introduce CTXGAP, an evaluation that tests how much information about a coding agent’s execution remains recoverable from its accumulated user-facing reports.
Why this paper
From Tencent
In one line
Coding agents' routine reports recover only 52% of task-state information, a 40.7-point gap from full execution context.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors
- ✓Reports numbers on named benchmarks
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§