Multimodal Interaction with Internet of Things and Augmented Reality : Foundations, Systems and Challenges
Joo Chan Kim · Diva portal (Dalarna University Library) · 2020
The development of technology has enabled diverse modalities that can be used by humans or machines to interact with computer systems. In particular, the Internet of Things (IoT) and Augmented Reality (AR) are explored in this report due to the new modalities offered by these two innovations which could be used to build multimodal interaction systems. Researchers have utilized multiple modalities in interaction systems for providing better usability. However, the employment of multiple modalities introduces some challenges that need to be considered in the development of multimodal interaction systems to achieve high usability. In order to identify a number of remaining challenges in the research area of multimodal interaction systems with IoT and AR, we analyzed a body of literature on multimodal interaction systems from the perspectives of system architecture, input and output modalities, data processing methodology and use cases. The identified challenges are regarding of (i) multidisciplinary knowledge, (ii) reusability, scalability and security of multimodal interaction system architecture, (iii) usability of multimodal interaction interface, (iv) adaptivity of multimodal interface design, (v) limitation of current technology, and (vi) advent of new modalities. We are expecting that the findings of this report and future research can be used to nurture the multimodal interaction system research area, which is still in its infancy.