How Well Do Vision-Language Models Understand Movies?

The gist: MF2 introduces a new benchmark for evaluating whether models can comprehend, consolidate, and recall key narrative information from full-length movies (50-170 minutes long) by using manually constructed sets of claim pairs targeting core narrative elements like character motivations and causal chains

The Benchmark MF2

MF2 is a new benchmark for evaluating whether models can comprehend, consolidate, and recall key narrative information from full-length movies (50-170 minutes long) It includes over 50 full-length, open-licensed movies, each paired with manually constructed sets of claim pairs—one true (fact) and one plausible but false (fib), totalling over 850 pairs These claims target core narrative elements such as character motivations and emotions, causal chains, and event order Instead of multiple-choice formats, the protocol adopts a binary claim evaluation protocol where models must correctly identify both the true and false claims This approach reduces biases like answer ordering and enables a more precise assessment of reasoning

Dataset Construction

The dataset comprises 53 full-length, open-licensed movies with an average duration of 88.33 minutes For each movie, annotators construct a set of contrastive claim pairs, each consisting of one factually correct statement (fact) and one minimally altered, false counterpart (fib) Claim granularity is labeled as single-scene, multi-scene, or global to test different levels of reasoning Comprehension dimensions are also labeled to indicate the specific aspects of narrative understanding being tested, including Event/Entity Understanding, Temporal Perception, Emotion Understanding, and Causal Reasoning The annotation process involved 26 annotators who watched the full movies and constructed pairs of contrastive claims following guidelines that minimize lexical differences between the fact and the fib A quality control stage involved two annotators reviewing comments left during annotation to identify problematic claims, resulting in a cleaner set of 868 high-quality pairs

Experimental Setup and Results

Experiments evaluate state-of-the-art open and closed models across vision-language setups, testing both video-only and video with subtitles as input modalities The evaluation protocol reports two metrics: pairwise accuracy, which measures how often models correctly classify both the true and the false claim in a pair, and standard accuracy Results demonstrate that both open-weight and closed models fall significantly short of human performance, underscoring the relative ease of the task for humans Among closed models, Gemini 2.5 Pro achieves the highest scores with a pairwise accuracy of 60.6%, followed by InternVL3-72B as an open-weight model

Ablation and Analysis

Ablation studies show that models perform substantially better when subtitles are available compared to relying on video alone The analysis across comprehension dimensions reveals that models handle temporal perception more effectively than other comprehension aspects across all modalities, likely because time-related information is often directly observable in visual and textual inputs Event and entity understanding is notably weaker under visual-only conditions, likely due to the need for linguistic disambiguation, a limitation that becomes evident when subtitles are introduced Furthermore, reasoning performance under visual-only inputs remains relatively consistent across reasoning types, while global reasoning becomes more challenging than single- and multi-scene reasoning under the presence of textual cues The analysis on reasoning granularity suggests that Qwen2.5-VL shows improved accuracy on claims requiring global reasoning compared to the other granularities, which may suggest that global narrative information is more frequently represented in pretraining corpora

Conclusion

MF2 provides a rigorous framework for diagnosing true narrative understanding in video-language models by focusing on long-form, manually annotated movie narratives and a binary claim evaluation protocol The extensive evaluation reveals a significant performance gap between models and humans, showing that commercial models such as Gemini 2.5 Pro outperform others but still fall short of human-level performance Incorporating transcripts significantly boosts model accuracy, and the framework effectively targets the harder challenge of global narrative understanding which current models continue to struggle with The work aims to advance the evaluation of VLMs by focusing on narrative understanding, a key component of human-level reasoning

Limitations. Despite careful design and validation, minor issues such as typos may remain in the claims, and annotators cannot fully control for memory effects when labeling claims Future work could involve splitting claims into disjoint sets and rating them by different annotators to better isolate such effects The paper publicly releases all data and code to facilitate reproducibility and support future research on long movie understanding The dataset is available at https://huggingface.co/datasets/sardinelab/MF2 and https://github.com/deep-spin/MF2, respectively

Acknowledgments. We gratefully acknowledge Miguel Graça, Evan Paces-Wiles, Maya Nachesa, Daniil Larionov, Bryan Sukidi, and José Pombal for their participation in the human evaluation process This work was supported by the Portuguese Recovery and Resilience Plan through project C645008882-00000055 (Center for Responsible AI), by EU’s Horizon Europe Research and Innovation Actions (UTTER, contract 101070631), by the project DECOLLAGE (ERC-2022-CoG 101088763), by a research grant (VIL53122) from VILLUM FONDEN, by the Pioneer Center for AI DNRF grant number P1, by FCT/MECI through national funds and when applicable co-funded EU funds under UID/50008: Instituto de Telecomunicações, by DARPA ECOLE Program No. HR00112390060, NSF-AI Engage Institute DRL-2112635, ARO Award W911NF2110220, ONR Grant N00014-23-1-2356, and the Microsoft Accelerate Foundation Models Research (AFMR) grant program, by the project PID2023- 151351NB-I00 funded by MCIN/ AEI /10.13039/501100011033 and by ERDF This collaboration resulted from an ELLIS workshop at MFO, the Oberwolfach Research Institute for Mathematics in the German Black Forest

References. [2] Kirolos Ataallah, Chenhui Gou, Eslam Abdelrahman, Khushbu Pahwa, Jian Ding, and Mohamed Elhoseiny. Infinibench: A comprehensive benchmark for large multimodal models in very long video understanding [3] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li [4] Ivana Balazevic et al. Memory consolidation enables long-context video understanding [5] Anna Bavaresco et al. Llms instead of human judges? a large scale empirical study across 20 nlp evaluation tasks [6] Keshigeyan Chandrasegaran et al. Hourvideo: 1-hour videolanguage understanding [7] Guo Chen et al. Cg-bench: Clue-grounded question answering benchmark for long video understanding [8] Zhe Chen et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks [9] Matt Deitke et al. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models [10] Xinyu Fang et al. Mmbench-video: A long-form multi-shot benchmark for holistic video understanding [11] Chaoyou Fu et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis [12] Kairui Hu et al. Video-mmmu: Evaluating knowledge acquisition from multi-discipline professional videos [13] Kairui Hu et al. Video-mmmu: Evaluating knowledge acquisition from multi-discipline professional videos 2025 [14] Qingqiu Huang et al. Movienet: A holistic dataset for movie understanding [15] Alon Jacovi et al. Stop uploading test data in plain text: Practical strategies for mitigating data contamination by evaluation benchmarks [16] Greg Kamradt. Needle in a haystack - pressure testing LLMs [17] Marzena Karpinska et al.

1. Improved Narrative Comprehension through Contrastive Claim Evaluation: The system can be trained to distinguish between a true and a plausible but false claim based on minimal lexical differences, as described in Section 2.2: annotators were instructed to minimize lexical differences between the fact and the fib, changing only the parts needed to flip the truth value. This forces models to perform precise reasoning rather than superficial retrieval.

2. Enhanced Reasoning Granularity Assessment: The system can be explicitly trained on reasoning across different levels of complexity, such as single-scene, multi-scene, and global, by utilizing the granularity labels introduced in Section 2.2: (i) single-scene: answerable using information from one scene; (ii) multi-scene: requiring integration across multiple scenes; and (iii) global.

3. Fine-Grained Comprehension Profiling: The system can be evaluated on specific narrative understanding dimensions, including Event/Entity Understanding, Temporal Perception, Emotion Understanding, and Causal Reasoning, by aligning its outputs with the definitions provided in Table 2: Comprehension Dimension Definition.

4. Robust Input Modality Adaptation: The improved system can dynamically adjust its reliance on different inputs (video-only, subtitles, synopsis) based on performance analysis from Table 4, learning when to leverage textual context versus visual input for accurate inference.

5. Mitigation of Answer Selection Bias: By adopting the binary claim evaluation protocol instead of multiple-choice formats, the system is trained to avoid answer selection biases based on superficial cues or poorly constructed distractors, as noted in Section 1 and supported by reference [30].

Vision-language models (VLMs) can increasingly handle extended contexts and answer questions about long videos--but can they understand the narrative structure of movies as humans do? To investigate this question, we introduce MF squared (Movie Facts and Fibs), a fully manually curated benchmark designed to evaluate whether models can identify, recall, and comprehend information central to the movie's storyline, using both the visual and textual modalities. MF squared includes over 50 open-licensed movies, each paired with manually constructed sets of claim pairs: one true claim (fact) and one plausible but false (fib), totaling over 850 pairs. Our analysis across 20 open-source and 2 proprietary models reveals a substantial gap relative to human performance. While subtitles provide a strong signal for narrative understanding, other textual sources (e.g., synopses) are less informative. In contrast, when visual information is distributed across the entire movie and jointly embedded with subtitles, models struggle to identify and effectively use the relevant content. However, when the visual input is restricted to claim-relevant visual content, models improve significantly, indicating substantial room for improvement in processing large multimodal contexts and distinguishing relevant from irrelevant information. Overall, our findings highlight persistent challenges in multimodal narrative understanding for current VLMs. We release our data and code to support further research in this direction.