Active learning of neural network from weak and strong oracles
Björn Persson Mattsson · Chalmers Publication Library (Chalmers University of Technology) · 2017
When implementing deep learning techniques in real-world applications acquiring labeled data can be the most difficult and expensive part.Sometimes in this process there is both a weak (and cheap to ask) oracle as well as a strong (but expensive to ask) oracle available to which one can ask for labels to examples.The weak oracle can for example be non-expert mechanical turks or a rule-based system, whereas the strong oracle may be human experts that provide very reliable labels.We propose an algorithm to do active learning in the presence of a weak and a strong oracle.A central part of the algorithm is an agreement classifier which predicts the probability of the weak oracle knowing the correct label for an example.However, at prediction time the agreement classifier is only assumed to be able to sort the examples after how much the weak oracle can be trusted, which is believed to be a key reason behind why the algorithm works in practice.A second key idea in the algorithm is that the agreement classifier does not only condition the classification on the information from the input space, but also on the label proposed by the weak oracle.A third idea that is examined is to leverage the probabilities supplied by the agreement classifier directly in the cost function of the main classifier.To test the algorithm we built a test environment based on binary sentence classification as well as three types of synthetic weak oracles.The algorithm performs well with these three synthetic oracles.It manages to decrease the cross entropy on the learning task more per example queried to the strong oracle than what standard active learning do.