Budget efficient online active learning and its applications
Shuji Hao · 2017
Online Active Learning (OAL) has been an important research area in machine learning, which aims to minimize the number of labeled instances and maximize the predictive performance meanwhile.OAL has both the efficiency and effectiveness of online learning and the labeling frugality of active learning.Due to these advantages, OAL has been widely used in real-world large-scale applications, such as information retrieval, data mining, recommendation system, and so on.However, there are still several problems existing in current OAL designs.First, in the online learning with expert advice setting, most of the exiting OAL algorithms assume that all the experts are comparably reliable, which is usually not true in reality.For example, noisy workers are quite common in the crowdsourcing platforms.To correct this weak assumption, this study proposes two robust online active learning algorithms, which not only consider the predictions of experts on current instance, but also consider the cumulative performance of experts on past instances.To validate the proposed algorithms, a series of experiments are conducted, in which the results show that the proposed algorithms greatly outperform the state-of-the-art existing algorithms and can achieve robust performance both in the normal and noisy scenarios.Second, to obtain reliable labels in crowdsourcing, most of the algorithms either require a set of golden questions to filter out the noisy workers, or require several labels for each instance.These types of requirements are labeling costly both in terms of money and time.To save costs, a framework of Active Crowdsourcing for Annotation (ACA) is proposed based on the online learning with expert advice.The proposed framework consists of two main components: "Who to label" and "When to query".The first component actively allocates the instance to reliable workers to gain labels, and the second component actively decides which instance is worthy to be a golden question.The empirical studies both on i First, I would send my great gratitude to my thesis committee members Dr.