What they did

The authors built VideoHarness-RSI, a controlled system that searches over executable programs for converting long videos into evidence that fits within a vision-language model’s limited context. The answering model and interface remain fixed, separating gains from context construction from gains due to model training.

An outer-loop proposer generates candidate harnesses using previous programs, evaluation results, and execution traces. Candidates are run and evaluated end to end, with successful variants retained as starting points for further recursive search.

Key findings

• Starting from uniform video sampling, recursive search found context-construction programs that outperformed several weaker hand-crafted baselines.
• Seeding the search with a stronger hand-crafted harness produced an additional improvement, indicating that the method can refine an already capable design.
• The selected harness transferred to other long-video benchmarks without further search.
• Because the vision-language model stayed frozen, the reported gains are attributable to changes in the executable context-construction process rather than model updates.

Why it matters

The results support treating context construction as a distinct optimization layer around frozen multimodal models. This creates a controlled way to study automated harness design and may reduce reliance on manually assembling sampling, retrieval, compression, and evidence-acquisition pipelines for each task.

Caveats

The supplied abstract does not report benchmark names, model identities, score changes, search costs, or comparisons with the strongest long-video systems, so the magnitude and efficiency of the gains cannot be assessed here. Transfer is reported only to unspecified additional benchmarks, leaving robustness across models, video domains, and context budgets open.