A Survey on Evaluation Methods for Chatbots
Wari Maroengsit, Thanarath Piyakulpinyo, Korawat Phonyiam, Suporn Pongnumkul, Pimwadee Chaovalit, Thanaruk Theeramunkong · 2019
Nowadays chatbots have been widely adopted in many industries to automatically answer users' questions and requests via chat interfaces. While it has become much easier to develop a chatbot system, the system itself is a complex system in nature. It is a challenge to evaluate and compare various chatbot systems in terms of effectiveness, efficiency, goal achievability, and the ability to satisfy users. This paper presents a survey, starting from literature review, chatbot architecture, evaluation methods/criteria, and comparison of evaluation methods. Focused on the three subprocesses in the chatbot architecture: text processing, semantic understanding, and response generation. Moreover, the survey is conducted with classification of chatbot evaluation methods and their analysis according to chatbot types and three main evaluation schemes; content evaluation, user satisfaction, and chat function.