A Questionnaire for Subjective Evaluation ofthe Intelligent Speech System
Yang Chen, Xiaowei Chen, Qing Xiang Xu, Lifen Hu, Chenxia Jin, Cheng Wang · 2018
This research aims to construct a questionnaire to subjectively evaluate the intelligent speech assistant (ISA). In a pilot interview, keywords extracted from users' descriptions and expectations of an ISA. Based on these keywords, 43 items was formed and rated online by 1462 users of ISA across China. Through exploratory factor analysis, 32 items was selected and compose the six dimensions of the final questionnaire. These dimensions were described as: (1) semantic intelligence, (2) timbre & sound quality, (3) long-term comfort, (4) character set, (5) habit dependence, and (6) interactive feedback. The questionnaire has a good validity (six factors accounted for 60.78 % of the total variance) and reliability (Cronbach's a coefficient: 0.944, p <; 0.01), meeting the requirements of psychometrics. The questionnaire can be used in the subjective evaluation of ISA.