LoLBench is a multilingual of 100 modular development tasks drawn from 29 large software systems in five domains. Each task provides a human-written enhancement proposal with about 5,000 words of user intent and high-level design, with target systems averaging 2.4 million lines of code and implementation pull requests changing about 5,500 lines.
Across 28 agents, the best resolved only 14% of tasks and reached a 52.7% fail-to-pass pass rate. Failure analysis shows incomplete code localization is a major bottleneck; providing reference-derived file trees and API specifications improves resolved rates by 16 to 22 percentage points, up to 34%.
This matters because it isolates the perception step, deriving a specification from intent, as a distinct and limiting capability for on real modular software, not just the ability to write correct edits.