Cross-Modal Hierarchical Knowledge Distillation for Image Aesthetics Assessment
Hangwei Chen, Feng Shao, Weiyi Jing, Huizhi Wang, Qiuping Jiang · IEEE Transactions on Multimedia · 2024
The field of image aesthetics assessment (IAA) is rapidly advancing due to its wide applications. However, relying solely on single-modal information for aesthetic evaluation presents inherent limitations. While multimodal IAA models incorporating user comments have achieved significant advancements, these comments are often unavailable due to privacy concerns and practical considerations, and they also introduce additional computational overhead during inference. To address this issue, we propose a cross-modal hierarchical knowledge distillation method, termed HKD-IAA, to enhance the performance of unimodal image models effectively. Specifically, HKD-IAA comprises four components: feature extraction, feature decomposition, hierarchical knowledge distillation, and dynamic decay. During training, we first decompose the extracted features into a weighted sum of basic aesthetic elements and their corresponding weights, thereby reducing the learning difficulty for the student model. Building on this, we design a new hierarchical knowledge distillation framework, which aligns features at the feature, relation, and response levels to effectively transfer the knowledge from the teacher model. Finally, we introduce a dynamic decay strategy to adjust the weight of the distillation loss, thereby enhancing the student model's learning effectiveness during training. Extensive experiments on two benchmark datasets validate that the proposed method achieves state-of-the-art performance using only visual modal data. Our code is available athttps://github.com/Hangwei-Chen/HKD-IAA.