Light in the Dark: Cooperating Prior Knowledge and Prompt Learning for Low-Light Action Recognition
Qi Liu, Shu Ye · 2024
Low-light environments present a significant challenge for action recognition. Due to insufficient lighting, video frames exhibit reduced detail and contrast, lacking the rich illumination information characteristic of standard RGB videos. Additionally, the dynamic range of cameras in low-light conditions is limited, making it difficult to capture changes in the video scenes. Thus, capturing motion information from low-light videos presents a significant challenge. We introduce a low-light action recognition model named Prior-knowledge Enhancement Prompt-learning Network (PEPN), which integrates video features based on illumination-invariant priors with large model motion description prompts. Unlike conventional methods, our method builds a feature bridge between low-light and standard videos through the use of illumination-invariant priors, addressing the issue of capturing lighting features in traditionally challenging low-light environments. Moreover, we enhance motion information using action descriptions. Specifically, we employ a pre-trained large-scale language model as a knowledge engine to enhance lighting information with illumination-invariant priors and propose a multimodal training scheme. Furthermore, we align action descriptions with action features through joint understanding of video and text. We introduce an action-specific video enhancement module for priors with RGB features, which demonstrates superior performance on DARK-48 and INFAR datasets compared to previous models.