Pseudo Random Masked AutoEncoder for Self-supervised Learning
Li Tian, Yuan Cheng, Zhibin Li · 2022
Masked AutoEncoder (MAE) has recently been proposed and has shown its effectiveness as a vision learner by an elegant asymmetric encoder-decoder design, which significantly optimizes both the pre-training efficiency and fine-tuning accuracy. In this paper, we propose a Pseudo Random Masking (PRM) strategy based on local gradient information instead of using random masking strategy in MAE. Our Pseudo Random Masked AutoEncoder (PRMAE) can preserve significant batches in the images and successfully improves the performances of MAE. Experimental results show that our method performs better than the previous work.