Interpretive self-supervised pre-training
Siladittya Manna, Saumik Bhattacharya, Umapada Pal · 2021
Self-supervised learning algorithms have become one of the best tools for unsupervised representation learning. Although self- supervised algorithms have achieved state-of-the-art performance for classification tasks in the case of natural image data, their application on medical data has been limited. In this work, we have proposed a novel loss function and derive it's asymptotic lower bound. We have also shown that self-supervised pre-training with the proposed loss function helps in surpassing the supervised baseline on the downstream task. We have also shown that the self-supervised pre-training helps a model in learning better representation in general to achieve better performance compared to supervised baselines. We have mathematically derived that the contrastive loss function asymptotically treats each sample as a separate class and works by maximizing the distance between any two samples and this helps to get better performance. Finally, through exhaustive experiments, we demonstrate that self-supervised pre-training helps to surpass the performance of fully supervised models on downstream tasks.