Rethinking Crowdsourcing Annotation: Partial Annotation With Salient Labels for Multilabel Aerial Image Classification
Jianzhe Lin, Tianze Yu, Z. Jane Wang · IEEE Transactions on Geoscience and Remote Sensing · 2022
Annotated images are required for supervised model training and evaluation in aerial image classification. Manually annotating images is arduous and expensive, especially for aerial images, which often cover a large land area with multiple labels. A recent trend for conducting such annotation tasks is through crowdsourcing, where images are annotated by volunteers or paid workers (e.g., annotation volunteers for Open Street Map) online from scratch. However, for crowdsourcing image annotations, the quality cannot be guaranteed, and incompleteness and incorrectness are two major concerns. To address such concerns, we have a rethinking of crowdsourcing annotations: Our simple hypothesis is that if annotators only partially annotate multi-label images with salient labels they are confident in, there will be fewer annotation errors and annotators will spend less time on uncertain labels. As a pleasant surprise, with the same annotation budget, we show that a multi-label aerial image classifier supervised by images with salient annotations can outperform models supervised by fully annotated images. Our contributions are 2-fold: An active learning way is proposed to acquire salient labels for multi-label aerial images; and a novel Adaptive Temperature Associated Model (ATAM) specifically using partial annotations is proposed for multi-label aerial image classification. When tested on practical crowdsourcing aerial data, the Open Street Map (OSM) dataset, the proposed ATAM can achieve higher accuracy than state-of-the-art classification methods trained on fully annotated images. The proposed idea is promising for crowdsourcing aerial image annotation. Our code will be publicly available.