The authors examined video evaluation pipelines across five models and seven benchmarks, varying factors including frame count and sampling, video decoder, frame resolution, prompt, input representation, and inference backend.
Video benchmark scores shift sharply with hidden evaluation choices
The authors show that configuration differences can outweigh claimed model gains, then propose a compact benchmark and standardized protocol.
Big Tech
Soumya Shamarao Jahagirdar · Anna Kukleva · Vishaal Udandarao · Samuel Thomas · Brian Kingsbury · Rogerio Feris · +2 more
University of Tübingen · Max Planck Institute for Informatics, SIC · IBM Research · MIT · MIT-IBM Watson AI Lab
Research Digest··2 min read
5 percentage points for the same checkpoint.
Why this paper
From IBM Research and 5 others
In one line
Unreported evaluation choices can shift video benchmark scores by up to 19.5 percentage points, and the curated Meta-Video benchmark preserves rankings with 5x less compute.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ✓Reports numbers on named benchmarks (2 benchmarks)
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§