# Watch carefully: why multimodal AI fails simple video reasoning

*Marius Comper · 14 September 2026*
*Source: Video-MME-Logical (EMNLP 2026 Findings, Hohin Kwan et al.)*
*Full interactive version with all 10 video puzzles: https://mariuscomper.uk/watch-carefully/en/*

A six-year-old child can track a ball placed under a cup as it is moved three times across a tabletop. For the world's most capable multimodal artificial intelligence models, the exact same task exposes an unresolved cognitive failure.

On 12 September 2026, a research team led by Hohin Kwan released the 500,000-example training corpus for the Video-MME-Logical benchmark (accepted to Findings of EMNLP 2026). Their findings document a striking divergence: human evaluators achieve 95.9% overall accuracy, while OpenAI's GPT-5.4 scores 22.7% and Google's Gemini-3.1 Pro scores 28.6% on the full benchmark dataset.

## Official Video-MME-Logical leaderboard results

The authors evaluated leading proprietary and open-source models across 25 procedurally generated visual reasoning categories. Task difficulty is mathematically controlled by varying temporal duration, occlusion count, and intermediate state transitions:

| Model / Evaluator | Overall (%) | Easy (%) | Medium (%) | Hard (%) |
|---|---:|---:|---:|---:|
| **Human Level (independent annotators)** | **95.9** | **98.4** | **95.9** | **93.4** |
| Gemini-3.1 Pro (Google DeepMind) | 28.6 | 33.1 | 24.1 | 20.6 |
| GPT-5.4 (OpenAI) | 22.7 | 31.7 | 20.3 | 16.1 |
| Qwen2.5-VL-72B-Instruct | 12.5 | 15.2 | 13.1 | 9.1 |
| InternVL3.5-8B-Instruct | 12.1 | 13.8 | 13.5 | 8.9 |
| Qwen3-VL-8B-Instruct | 11.9 | 13.4 | 12.8 | 9.6 |
| KimiVL-16B-A3B-Think | 7.6 | 10.2 | 6.8 | 5.8 |
| LLaVA-Video-72B-Qwen2 | 2.4 | 4.7 | 1.9 | 0.7 |

## The ten video puzzles in the interactive test

1. **Cup Trick (State Tracking)**: Tracking a blue ball placed under Cup 1 across multiple swaps. The ball finishes at Cup 2. Humans achieve 96%, while GPT-5.4 fails as lateral occlusions break object identity.
2. **Rotating Voxel Block (Structural Composition)**: Counting the unit cubes of a solid rotating 3D shape. The block measures 3 × 2 × 1 = 6 cubes. Vision models hallucinate phantom cubes or missed voids upon rotation.
3. **Hidden Container (Dynamic Spatiality)**: Deducing the invisible enclosure shape from particle rebound paths. The geometry is a rounded capsule (stadium). Models fail without explicit boundary outlines.
4. **Maze Trace (Dynamic Spatiality)**: Following an orange dot to the blue exit circle. The dot completes all 8 turns without getting trapped. Large sampling strides cause models to lose the marker.
5. **Occlusion Count (Structural Composition)**: Counting golden crosses partially hidden behind a moving red disk. There are 9 fixed crosses in total. Models suffer from object permanence collapse, only counting concurrent tokens.
6. **Trajectory Intersection (Dynamic Spatiality)**: Determining if the paths of a blue and green moving dot cross. Because the blue dot remains along the top corridor and green in the lower lanes, their paths cross 0 times. Models confuse temporal concurrency with spatial intersection.
7. **Keyboard Sequence (Temporal Ordering)**: Identifying the first key activated on a virtual keyboard (Key M, followed by I, W, R). Models treating video frames without strict temporal order lose chronological precedence.
8. **Falling Shapes (Structural Composition)**: Spotting the absent shape in a shower of falling objects. The animation includes triangles, rectangles, squares, and diamonds, but no circles. Models fall back on linguistic priors rather than visual inspection.
9. **Rotation Center (Dynamic Spatiality)**: Identifying the orbit center of a rotating piece of furniture (the center of the frame). Models confuse intrinsic spin with external orbit.
10. **3D Tunnel Exit (Dynamic Spatiality)**: Identifying the color of the final exit chamber (light blue). While obvious to human eyes, models miss terminal states lasting only a few frames.

## Why multimodal AI fails

1. **Frame decimation**: Video models do not process continuous video at 30 or 60 fps; they sample 8 to 32 static frames. Intervening motions are lost.
2. **Object permanence collapse**: Feedforward vision encoders possess no persistent memory buffer; occluded objects vanish from latent state.
3. **Absence of 3D spatial voxel modeling**: Models treat space as flat 2D projection planes rather than 3D mental solids.
4. **Blindness to implicit physics deduction**: Models recognize drawn patterns but cannot infer invisible walls from reflection vectors.
5. **Chronological blurring**: Transformer attention frequently treats video frames as unordered sets, scrambling temporal sequence.

## Human evaluation protocol

The authors evaluated 3,750 benchmark items with three independent human annotators compensated at 50 USD/hour, under identical sensory conditions without metadata access. Human accuracy reached 95.9% overall (98.4% on Easy, 95.9% on Medium, and 93.4% on Hard).

## Citation

Hohin Kwan, Hongyu Li, Ray Zhang, Manyuan Zhang, Xianghao Kong, Anyi Rao, Jiahao Xie, Si Liu. "Video-MME-Logical: A Controlled Diagnostic Benchmark for Video Temporal-Logical Reasoning", Findings of EMNLP 2026 / arXiv:2606.27828.