Active learning from relative queries
Buyue Qian, Xiang Wang, Fei Wang, Hongfei Li, Jieping Ye, Ian Davidson · 2013
Active learning has been extensively studied and shown to be useful in solving real problems. The typical setting of traditional active learning meth-ods is querying labels from an oracle. This is only possible if an expert exists, which may not be the case in many real world applications. In this pa-per, we focus on designing easier questions that can be answered by a non-expert. These questions poll relative information as opposed to absolute in-formation and can be even generated from side-information. We propose an active learning ap-proach that queries the ordering of the importance of an instance’s neighbors rather than its label. We explore our approach on real datasets and make several interesting discoveries including that query-ing neighborhood information can be an effective question to ask and sometimes can even yield bet-ter performance than querying labels. 1