It is better to Verify: Semi-Supervised Learning with a human in the loop for large-scale NLU models
Verena Weber, Enrico Piovano, Melanie Bradford · 2021
When a NLU model is updated, new utterances must be annotated to be included for training.However, manual annotation is very costly.We evaluate a semi-supervised learning workflow with a human in the loop in a production environment.The previous NLU model predicts the annotation of the new utterances, a human then reviews the predicted annotation.Only when the NLU prediction is assessed as incorrect the utterance is sent for human annotation.Experimental results show that the proposed workflow boosts the performance of the NLU model while significantly reducing the annotation volume.Specifically, in our setup, we see improvements of up to 14.16% for a recall-based metric and up to 9.57% for a F1score based metric, while reducing the annotation volume by 97% and overall cost by 60% for each iteration.