DOES CONFIDENCE ANNOTATION MEET THE DIALOG GOAL?: A QUANTITATIVE ANALYSIS
Kadri Hacıoğlu, Wayne Ward · 2015
In this paper, we quantify the impact of using confi-dence annotation on the performance of a dialog system for a given dialog strategy. Although the ultimate goal in a di-alog system is to optimize the user satisfaction, it is very difficult to quantify it. Instead, we consider the probability of success and the number of expected turns of a transaction as our dialog performance metrics. In fact, they are fairly strong predictors of user satisfaction. The desired dialog goal in terms of these metrics is mapped onto an operating region on the receiver operating characteristics (ROC) plane of the confidence annotation method. In doing so, one can not only figure out whether the confidence annotation would meet the desired goal at the dialog level but also understand how to tune the system in order to reach that goal. Such an analysis therefore allows the system designer to optimize the use of confidence annotation and tune the system with-out doing any time consuming online experiments. 1.