Optimizing Features in Active Machine Learning for Complex Qualitative Content Analysis
Jasy Suet Yan Liew, Nancy J. McCracken, Shichun Zhou, Kevin Crowston · 2014
We propose a semi-automatic approach for content analysis that leverages machine learn-ing (ML) being initially trained on a small set of hand-coded data to perform a first pass in coding, and then have human annotators cor-rect machine annotations in order to produce more examples to retrain the existing model incrementally for better performance. In this “active learning ” approach, it is equally im-portant to optimize the creation of the initial ML model given less training data so that the model is able to capture most if not all posi-tive examples, and filter out as many negative examples as possible for human annotators to correct. This paper reports our attempt to op-timize the initial ML model through feature exploration in a complex content analysis project that uses a multidimensional coding scheme, and contains codes with sparse posi-tive examples. While different codes respond optimally to different combinations of fea-tures, we show that it is possible to create an optimal initial ML model using only a single combination of features for codes with at least 100 positive examples in the gold stand-ard corpus. 1