Quality Control of Crowd Labeling for Improving the Quality of Peer Assessments
Banpreet Singh Chhabra, Edward F. Gehringer · 2023
The effectiveness of machine learning depends on the quality of the training data. In some fields, the training data is generated from crowdsourced labels. Ensuring the reliability and accuracy of crowd-generated labels is a critical challenge. To address this, we present comprehensive research to develop and implement robust quality control strategies in crowd labeling. We focus on enhancing the effectiveness of feedback and suggestions by evaluating the taggers and the quality of tags they assign by using natural language processing techniques and machine learning. Our approach aims to assign reliability metrics to each tagger and tag, enabling researchers to filter and create machine-learning training datasets from the most reliable annotations. This research addresses this issue by implementing four quality control strategies in the domain of peer assessment and comparing their performance with manual grade scores assigned by professors. The four key quality control strategies encompass identifying taggers who tag too quickly, detecting taggers providing inconsistent labels, uncovering unreliable taggers employing pattern-based tagging, and performing agreement/disagreement analysis for tags. By individually implementing and assessing the impact of each strategy on data alignment with manual grade scores, we ascertain their effectiveness in enhancing the reliability of crowd -generated labels.