Comparative Error Analysis of Dialog State Tracking
Ronnie W. Smith · 2014
A primary motivation of the Dialog State Tracking Challenge (DSTC) is to allow for direct comparisons between alterna-tive approaches to dialog state tracking. While results from DSTC 1 mention per-formance limitations, an examination of the errors made by dialog state trackers was not discussed in depth. For the new challenge, DSTC 2, this paper describes several techniques for examining the er-rors made by the dialog state trackers in or-der to refine our understanding of the lim-itations of various approaches to the track-ing process. The results indicate that no one approach is universally superior, and that different approaches yield different er-ror type distributions. Furthermore, the results show that a pairwise comparative analysis of tracker performance is a useful tool for identifying dialogs where differ-ential behavior is observed. These dialogs can provide a data source for a more care-ful analysis of the source of errors.