Assessing attentiveness and cognitive engagement across tasks using video-based action understanding in non-human primates
Sin-Man Cheung, Adam R. Neumann, Thilo Womelsdorf · Journal of Neuroscience Methods · 2025
BACKGROUND: Distractibility and attentiveness are cognitive states that are expressed through observable behavior, but how behavioral features can be used to quantify these cognitive states has remained poorly understood. Video-based analysis promises to be a versatile tool to quantify the behavioral features that reflect subject-specific distractibility and attentiveness and are diagnostic of cognitive states. NEW METHOD: We describe an analysis pipeline that classifies cognitive states using a 2-camera set-up for video-based estimation of attentiveness and screen engagement in nonhuman primates performing cognitive tasks. The procedure reconstructs 3D poses from 2D labeled DeepLabCut videos, reconstructs the head/yaw orientation relative to a task screen, and arm/hand/wrist engagements with task objects, to segment behavior into an attentiveness and engagement score. RESULTS: Performance of different cognitive tasks was robustly classified from video within a few frames, reaching > 90 % decoding accuracy with ≤ 3 min long time segments. The analysis procedure allows adjusting thresholds for segmenting subject-specific movements for a time-resolved scoring of attentiveness and screen engagement. COMPARISON WITH EXISTING METHODS: Current methods also extract poses and segment action units; however, they haven't been combined into a framework that enables subject-adjusted thresholding for specific task contexts. This integration is needed for inferring cognitive state variables and differentiating performance across various tasks. CONCLUSION: The proposed method integrates video segmentation, scoring of attentiveness and screen engagement, and classification of task performance at high temporal resolution. This integrated framework provides a tool for assessing attention functions from video.