Multimodal BCIs - the hands-free Tetris paradigm
Elisavet Chatzilari, Georgios Liaros, Spiros Nikolopoulos, Ioannis Yiannis Kompatsiaris · 2020
In this chapter, we will explore ways to integrate three natural sensory modalities, i.e. vision, brain commands and stress levels into a single visceral experience that allows simultaneous control of various interface options. In this direction, we present MM-Tetris, the multimodal reinvention of the popular Tetris game, modified to be controlled with the user's eye-movements, mental commands and bio-measurements. MM-Tetris is intended for use by motor -impaired people who are not able to operate computing devices through the regular controllers (i.e. mouse and keyboard). In the proposed version of the game, the use of eye -movements and mental commands works in a complementary fashion, by facilitating two different controls, the horizontal movement of the tiles (i.e. tetriminos) through the coordinates of the gaze and the tile rotation through sensorimotor rhythm (SMR) signals detection, respectively. Additionally, bio-measurements provide the stress levels of the player, which in turn determines the speed of the tiles' drop. In this way, the three modalities smoothly collaborate to facilitate playing a game like Tetris. Eventually, the design of the game provides a natural gamified interface for user training in generating more discriminative SMR signals for better detection of imaginary movements.