The authors propose the attack pyramid, which organizes five levels of shortcut attacks with increasing access to a benchmark item: option level (only option layout), text level (adds question text), pool level (adds other items in the benchmark), frame level (adds one frame or caption), and order level (adds shuffled or truncated video).
Most video benchmarks fail to actually test video understanding
A systematic audit of 115 benchmarks reveals that many can be solved without watching the video, and the authors release a curated meta-benchmark of the hardest items.
Independent
Enxin Song · Yinuo Xu · Shusheng Yang · Wenhao Chai · Jiatao Gu
Research Digest··3 min read
Song et al.
Why this paper
Independent
In one line
Video benchmarks are vulnerable to shortcut attacks; Video-Index keeps only the 840 items that no attacker solves.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ✓Reports numbers on named benchmarks
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§