A smart speaker performance measurement tool
Hyunsu Mun, Hyungjin Lee, Soo-Hyun Kim, Youngseok Lee · 2020
Recently voice-controlled virtual assistants (VA) in smart speakers or smartphones have been popular. As VA provides interactive services by executing complicated processes such as speech recognition, natural language understanding, service invocation, and TTS generation jobs, its functions are performed in the cloud. However, we do not know why the response time of voice commands is slow and what is the performance bottleneck of the VA service. In this paper, we present a comprehensive VA performance measurement framework that analyzes the timing events and the response time by processing audio, video and packets. From experiments of 414 voice commands with five smart speakers and 178 commands for two VAs in smartphones, we observed that 24.9% of voice commands are completed within two seconds and 63.2% within three seconds and 36.8% of voice commands over three seconds result in poor user experiences. In particular, 96.2% of music commands and 66.7% of IoT control commands show the slow response time longer than three seconds. We found that our performance measurement tool is useful for finding the slow service such as music and news with the overhead of extracting the user intent from the voice command, the content app startup delay, and the initial playback time. Our tool shows that IoT control with a smart speaker produces the slow response time.