Why state-of-the-art vision models collapse on elementary temporal reasoning
A sharp contradiction defines current public perception of multimodal artificial intelligence. Models such as GPT-5.4 and Gemini-3.1 Pro can synthesize complex medical papers, write complete production applications in Rust, and solve graduate-level mathematics proofs.
Yet when shown a five-second animation in which a sliding disc temporarily occludes several small crosses or a cup swaps places with another, model performance deteriorates to near-random guessing. Across the entire Video-MME-Logical test set, GPT-5.4 achieved 22.7% accuracy, while Gemini-3.1 Pro scored 28.6%.
"Merely aggregating multi-frame inputs is not equivalent to executing logical reasoning over time. While a young child can effortlessly deduce the position of a hidden ball in a shell game by tracking motion through occlusion, contemporary multimodal models struggle with this foundational logic."
— Kwan et al., Video-MME-Logical (EMNLP 2026)
Official Video-MME-Logical leaderboard results
The authors evaluated leading proprietary and open-source models across 25 procedurally generated visual reasoning categories. Task difficulty is mathematically controlled by varying temporal duration, occlusion count, and intermediate state transitions.
| Model / Evaluator | Overall (%) | Easy (%) | Medium (%) | Hard (%) |
|---|---|---|---|---|
| Human Level (independent annotators) | 95.9 | 98.4 | 95.9 | 93.4 |
| Gemini-3.1 Pro (Google DeepMind) | 28.6 | 33.1 | 24.1 | 20.6 |
| GPT-5.4 (OpenAI) | 22.7 | 31.7 | 20.3 | 16.1 |
| Qwen2.5-VL-72B-Instruct | 12.5 | 15.2 | 13.1 | 9.1 |
| InternVL3.5-8B-Instruct | 12.1 | 13.8 | 13.5 | 8.9 |
| Qwen3-VL-8B-Instruct | 11.9 | 13.4 | 12.8 | 9.6 |
| KimiVL-16B-A3B-Think | 7.6 | 10.2 | 6.8 | 5.8 |
| LLaVA-Video-72B-Qwen2 | 2.4 | 4.7 | 1.9 | 0.7 |
Five structural failure modes in artificial vision
Why do neural networks break down on tasks that human perception resolves automatically? The paper identifies five architectural causes:
1. Frame decimation and temporal sampling gaps
Video models do not ingest continuous video streams at 30 or 60 frames per second. To contain computational complexity, video encoders decimate inputs into 8, 16, or 32 static keyframes. If a swap, bounce, or direction change occurs between two sampled frames, the model has no sensory record of the event.
2. The collapse of object permanence
In developmental psychology, object permanence describes the understanding that objects continue to exist even when unobserved. Contemporary vision encoders are memoryless feedforward networks: when a sliding red disc covers background crosses, the tokens corresponding to those objects disappear from the latent representation.
3. Absence of an internal 3D spatial voxel representation
The human visual cortex constructs an internal 3D model of observed objects, mentally rotating solids to track volume. Multimodal models treat 3D rotation as a sequence of flat 2D pixel matrices, misinterpreting newly visible faces as newly created physical matter.
4. Blindness to implicit boundaries and kinematic deduction
In the bouncing particles test, a human viewer reconstructs the geometry of an invisible container by connecting collision points and reflection angles. Because multimodal models are trained predominantly on explicitly annotated objects, they fail to infer geometry from kinematic laws alone.
5. Diffuse temporal attention and chronological blurring
Self-attention layers across video tokens frequently treat frames as concurrent candidates rather than an immutable, unidirectional timeline. In sequential activation tasks such as the virtual keyboard, models recognize which keys were pressed but fail to establish strict temporal precedence.
Human evaluation protocol
To establish a reliable performance baseline, the researchers sampled 3,750 items from the Video-MME-Logical dataset and evaluated them with three independent human annotators. Annotators were compensated at 50 USD per hour and evaluated items under the exact same sensory conditions as the models, without access to program metadata.
Human accuracy reached 95.9% across the full benchmark (98.4% on Easy, 95.9% on Medium, and 93.4% on Hard). This demonstrates that the benchmark tasks are solvable and that model failure reflects an authentic temporal reasoning deficit.
Methodology and citation
The Video-MME-Logical benchmark and its accompanying 500K training corpus are publicly available under the CC BY 4.0 license. For technical reproduction details and model evaluation scripts, refer to the published study:
Hohin Kwan, Hongyu Li, Ray Zhang, Manyuan Zhang, Xianghao Kong, Anyi Rao, Jiahao Xie, Si Liu. "Video-MME-Logical: A Controlled Diagnostic Benchmark for Video Temporal-Logical Reasoning", Findings of EMNLP 2026 / arXiv:2606.27828.