Understanding a movie requires following who spoke, what they said, and why it matters. Most AI tests check these abilities separately, missing that a model can answer a plot question without knowing who actually delivered the line.
CineMME is a new built from hundreds of clips with expert annotations. Each clip has timestamps for every line of speech, the text of the line, the locations of speakers' faces, and plot questions that require connecting all this evidence together.
The main finding is a "who-what-when" gap: the best large language model identifies speech in time and transcribes it well, but correctly pinpoints the active speaker only in a small fraction of cases. Surprisingly, adding visual information helps models answer questions about scenes but actively reduces their accuracy on dialogue-focused questions.