What the paper's results show on simple visual tasks
A sharp contradiction defines current public perception of multimodal artificial intelligence. Models such as GPT-5.4 and Gemini-3.1 Pro can synthesize complex medical papers, write complete production applications in Rust, and solve graduate-level mathematics proofs.
Yet when shown a five-second animation in which a sliding disc temporarily occludes several small crosses or a cup swaps places with another, model performance falls well below the human level reported in the paper. Across the entire Video-MME-Logical test set, GPT-5.4 achieved 22.7% accuracy, while Gemini-3.1 Pro scored 28.6%. The paper mixes multiple-choice and exact structured-answer formats, so this page does not convert those scores into a single random-guessing baseline.
The authors show that aggregating multiple frames does not by itself amount to logical reasoning over time: in the paper's tasks, tracking a hidden ball through occlusion remains an operation the evaluated models often fail (Kwan et al., Video-MME-Logical, EMNLP 2026). The child example in the paper's introduction is the authors' motivational illustration; the paper reports no child sample.
Official Video-MME-Logical leaderboard results
The authors evaluated leading proprietary and open-source models across 25 procedurally generated visual reasoning categories. Task difficulty is mathematically controlled by varying temporal duration, occlusion count, and intermediate state transitions.
| Model / Evaluator | Overall (%) | Easy (%) | Medium (%) | Hard (%) |
|---|---|---|---|---|
| Human Level (independent annotators) | 95.9 | 98.4 | 95.9 | 93.4 |
| Gemini-3.1 Pro (Google DeepMind) | 28.6 | 33.1 | 24.1 | 20.6 |
| GPT-5.4 (OpenAI) | 22.7 | 31.7 | 20.3 | 16.1 |
| Qwen2.5-VL-72B-Instruct | 12.5 | 15.2 | 13.1 | 9.1 |
| InternVL3.5-8B-Instruct | 12.1 | 13.8 | 13.5 | 8.9 |
| Qwen3-VL-8B-Instruct | 11.9 | 13.4 | 12.8 | 9.6 |
| KimiVL-16B-A3B-Think | 7.6 | 10.2 | 6.8 | 5.8 |
| LLaVA-Video-72B-Qwen2 | 2.4 | 4.7 | 1.9 | 0.7 |
What five task types require and what was observed in the evaluated models
The paper describes five operations its tasks require. For each, the text below states what the task requires and what was observed in the evaluated models' results, without turning an observation into an impossibility holding for every architecture:
1. Tracking changes between sampled frames
The task requires tracking changes between frames. In the paper's protocol, models received frames sampled at 2 FPS; a swap or turn between two sampled frames is missing from the input, and the reported answers fail at exactly those transitions.
2. Counting objects after they are covered
The task requires counting objects after a disc covers them. In the reported results, the evaluated models miscounted the crosses after occlusion, as if the register of covered items had not been kept during the task.
3. Deducing a solid's structure from its rotation
The task requires deducing a solid's structure from its rotation (here, 3 × 2 × 1 cubes). In the reported results, the evaluated models answered the cube count incorrectly as faces became visible in turn; the observation concerns these models and this protocol.
4. Reconstructing a shape from rebounds, with no drawn outline
The task requires reconstructing an enclosure's shape from impact points and reflection angles, with no drawn outline. In the reported results, the evaluated models identified the shape incorrectly under those conditions.
5. Establishing the order of discrete events
The task requires the exact order of discrete events (here, the first lit key: M, then I, W, R). In the reported results, the evaluated models named the lit keys but established incorrectly which came first.
Human evaluation protocol
To establish a reliable performance baseline, the researchers sampled 3,750 items from the Video-MME-Logical dataset and evaluated them with three paid human annotators. Annotators were compensated at 50 USD per hour and evaluated items under the exact same sensory conditions as the models, without access to program metadata. The paper does not describe a child sample and does not report annotator ages; the comparison on this page concerns the paper's evaluators.
Human accuracy reached 95.9% across the full benchmark (98.4% on Easy, 95.9% on Medium, and 93.4% on Hard). This demonstrates that the benchmark tasks are solvable and that model failure reflects an authentic temporal reasoning deficit.
Methodology and citation
The Video-MME-Logical benchmark and its accompanying 500K training corpus are publicly available under the CC BY 4.0 license. For technical reproduction details and model evaluation scripts, refer to the published study:
Hohin Kwan, Hongyu Li, Ray Zhang, Manyuan Zhang, Xianghao Kong, Anyi Rao, Jiahao Xie, Si Liu. "Video-MME-Logical: A Controlled Diagnostic Benchmark for Video Temporal-Logical Reasoning", Findings of EMNLP 2026 / arXiv:2606.27828 (HTML version). Table results are those reported in the paper; models received frames sampled at 2 FPS.
The ten clips in the interactive test are exercises created for this page, inspired by the benchmark's categories. They are not the paper's official evaluated items, and your score out of 10 does not measure the same protocol as the table percentages.