Modeling of Human Motion Perception Mechanism: A Simulation based on Deep Neural Network and Attention Transformer
Zitang Sun, Yen‐Ju Chen, Yung‐Hao Yang, Shin’ya Nishida · Journal of Vision · 2023
The motion perception system has been widely explored as an essential function for creatures to perceive and interact with the world. Nevertheless, the current motion models based on the V1-MT structure struggle to derive informative motion flows from local motion energy in complex natural scenes. This study takes advantage of the flexibility of deep neural networks (DNNs) to simulate the motion perception process. To tackle this challenging problem while trading off neurophysiological plausibility, we developed a two-stage process that merged the classical motion energy constraint with a well-developed transformer structure. The first stage consists of a group of cells with trainable spatiotemporal frequencies tunning to capture different preferences of local motion energy. The subsequent stage implemented the attention mechanism from deep learning. It enables recurrent integration of the local motion signals via pairwise correlation across each spatial location to resolve the aperture problem. The supervised training was applied to fit the motion ground truth in digital videos, which consist of a sizeable multi-frame training set, including the self-designed non-texture dataset, MPI-Sintel dataset, natural image sequences with pseudo-labels, etc. Based on drifting gratings and plaids, we used virtual neurophysiology to measure model's activation and found some similar representations between humans' V1-MT and the model's two processing stages. The model generalizes well from non-texture stimuli used in psychophysics to complex natural scenes, demonstrating similarity to human interpretation. In more complex scenarios, the model demonstrates a high-level function of integrating local energy to infer global motion flow. In summary, the proposed model could finely replicate the performance from simple drifting Gabor to the complex natural scene and introduce some characteristics similar to the known human visual system's properties.