Towards Boosting the Open-Domain Chatbot with Human Feedback
Hua Cai Lu, Siqi Bao, Huang He, Fan Wang, Hua Wu, Haifeng Wang · 2023
Many open-domain dialogue models pretrained with social media comments can generate coherent replies but have difficulties producing engaging responses.This phenomenon might mainly result from the deficiency of annotated human-human conversations and the misalignment with human preference.In this paper, we propose a novel and efficient framework Diamante to boost the open-domain chatbot, where two kinds of human feedback (including explicit demonstration and implicit preference) are collected and leveraged.By asking annotators to select or amend the modelgenerated candidate responses, Diamante efficiently collects the human demonstrated responses and constructs a Chinese chit-chat dataset.To enhance the alignment with human preference, Diamante leverages the implicit preference in the data collection process and introduces the generation-evaluation joint training.Comprehensive experiments indicate that the Diamante dataset and joint training paradigm can significantly boost the performance of pre-trained dialogue models.The overall engagingness of the previous state-ofthe-art model has been improved remarkably by 50% in Chinese open-domain conversations.