VidHarness automates the design of video agents (harness programs) for cost-efficient long video understanding. Instead of hand-crafting harnesses through slow trial-and-error, it uses an iterative evolution process with Monte Carlo tree search and uncertainty-aware multi-fidelity validation to find effective strategies under a strict frame budget.
The method achieves new state-of-the-art results on LongVideoBench, Video-MME, and Video-Holmes, outperforming the strongest hand-crafted video agent by up to 11.2 points. It also generalizes to out-of-domain benchmarks while using fewer than half the frames of uniform sampling, making it interesting for any application needing efficient long video question answering.