Video models normally learn to find events in time by training on videos paired with written descriptions, which is expensive to label and store. This work asks whether a language model, the kind that only reads and writes text, can learn the task from written descriptions alone and then handle real video at test time.
They trained a text-only model on many short written sequences, each one describing what happens at a series of time steps, and gave it written word meanings turned into number vectors. They also compared two ways of turning text into those vectors, a common one limited to short inputs and a newer one built for longer, richer text, which did better.
Handed video frames instead of words, the model mostly failed, showing that words and images sit in different regions of the same number space. Their fix, replacing each frame with the average of similar text concepts, raised accuracy to 14.45 average overlap with the correct answer on one benchmark and 20.72 on another. It is a partial bridge, not a solution.