Evaluation of a WFST-based ASR System for Train Timetable Information
Josef R. Novak, Edward W. D. Whittaker, Sadaoki Furui · Hokkaido University Collection of Scholarly and Academic Papers (Hokkaido University) · 2009
This paper focuses on the description and evaluation of our new prototype Weighted Finite State Transducer (WFST) based ASR system for accessing Japanese train timetable information from a mobile device. The system employs a constrained, WFST grammar which permits users to make natural language queries related to the train timetables domain, including queries related to supplementary concepts such as "last train", "arrival time", etc. We report preliminary results for four related WFST networks, which include weighted and un-weighted variants of an Zenkoku network which covers 9199 stations across Japan, and weighted and un-weighted variants of a Tokyo network which contains 1098 stations from the greater Tokyo area. For weights we employ unigram station probabilities which were estimated from search engine log data. We compare the performance of the un-weighted networks to their weighted counterparts, using a test set which includes 474 utterances produced by 6 different speakers, and which was recorded in a variety of different acoustic environments. In particular, we focus on task completion rates, including accuracy on at-least-one-station, both-stations, and both-stations-plus-time. In the best case, for the 10-best, weighted Zenkoku network we report a both-stations accuracy of 92.2%.