Getting to Know Users: Accounting for the Variability in User Ratings
Nina Dethlefs, Heriberto Cuayáhuitl, Helen Hastie, Verena Rieser, Oliver Lemon · Edinburgh Research Explorer (University of Edinburgh) · 2014
Evaluations of dialogue systems and language generators often rely on subjective user ratings to assess output quality and performance. Humans however vary in their preferences so that estimating an accurate prediction model is difficult. Using a method that clusters utterances based on their linguistic features and ratings (Dethlefs et al., 2014), we discuss the possibility of obtaining user feedback implicitly during an interaction. This approach promises better predictions of user preferences through continuous re-estimation.