Long-video agent adapts retrieval to each question’s evidence needs

VESTA routes questions among focused, recall-oriented, and contrastive retrieval while tracking evidence coverage, conflicts, and verification.

PaperIndependentcs.CVarXiv:2608.31005v1
Can Zhang · Baofeng Zhang · Xiaotian Han · Junyuan Shang · Yuchen Ding · Shuohuan Wang · +2 more
Research Digest··2 min read
The authors developed a training-free agent that searches long videos differently depending on whether a question requires localized evidence, broad coverage, or comparison between hypotheses. Across four benchmarks, this policy-steered approach generally outperformed a uniform-retrieval baseline, including gains of 2.7 accuracy points on Video-MME-v2 and 6.9 points on LongVideoBench’s long-video subset.

What they did

The authors built VESTA around an acquire–verify–consolidate loop. An intent router selects focused, recall, or contrastive retrieval over a shared index of visual scenes and speech, while leaving the reasoning model free to verify results, issue follow-up searches, or inspect video regions outside the retrieved set.

A temporal evidence ledger compresses observations while retaining their timestamps, provenance, coverage, conflicts, verification status, and support for competing hypotheses. The system requires no additional training and was evaluated on Video-MME-v2, LongVideoBench, EgoSchema, and LVBench using shared query-time models for comparison.

Key findings

  • On Video-MME-v2, VESTA improved average accuracy by 2.7 percentage points over VideoARM and gained across all six reported metrics.
  • On the LongVideoBench long-video subset, it improved accuracy by 6.9 points under shared query-time models.
  • On LVBench, VESTA gained 1.5 points; on EgoSchema, it matched VideoARM rather than surpassing it.
  • The results indicate that selecting retrieval behavior by evidence type can improve performance without prescribing a complete question-specific solution procedure.

Why it matters

Long-video question answering can fail because relevant evidence was never gathered, not because the final reasoning was inadequate. VESTA shows that retrieval policy and explicit evidence accounting can be useful parts of an agent’s control system, particularly when questions differ in whether they demand precision, coverage, or hypothesis discrimination.

Caveats

The reported improvements vary substantially by benchmark and disappear on EgoSchema. The abstract does not provide inference-cost, latency, ablation, or statistical-significance results, so it remains unclear how much each routing or ledger component contributes and whether the additional retrieval and verification steps are computationally efficient.

§
newspaper

Research Digest

Articles published under the Zotpaper byline are synthesized from multiple source publications by our AI editor and reviewed by our editorial process. Each story combines reporting from credible outlets to give readers a balanced, comprehensive view.