Multi-label classification of open-ended questions with BERT
Matthias Schonlau, Julia Weiß, Jan E. Marquardt · 2023
Open-ended questions in surveys are valuable because they do not constrain the respondent’s answer, thereby avoiding biases. However, answers to open-ended questions are text data which are harder to analyze. Traditionally, answers were manually classified as specified in the coding manual. In the last 10 years, researchers have tried to automate coding. Most of the effort has gone into the easier problem of single label prediction, where answers are classified into a single code. However, open-ends that require multi-label classification, i.e., that are assigned multiple codes, occur frequently. This paper focuses on multi-label classification of text answers to open-ended survey questions in social science surveys. Here, open-ends are frequently mildly multi-label, where the average number of labels per answer text is relatively low. We evaluate the performance of the transformer-based architecture BERT for the German language in comparison to traditional multi-label algorithms (Binary Relevance, Label Powerset, ECC) in a German social science survey, the GLES Panel (N=17,584, 55 labels). We evaluate the algorithms on 0/1 loss. We find that classification with BERT (forcing at least one label) has the smallest 0/1 loss (13.1%) among methods considered (18.9%-21.6%). Our work has important implications for social scientists: 1) Multi-label classification with BERT works in the German language for open-ends. 2) For mildly multi-label classification tasks, the loss now appears small enough to allow for fully automatic classification. Previously, the loss was more substantial, usually requiring semiautomatic approaches. 3) Unlike the nearest competitor, ECC, multi-label classification with BERT requires only a single model for all labels.