Exploring Motion-Based and Mid-Air Gestures in Mobile Web Contexts
Linda Di Geronimo · Repository for Publications and Research Data (ETH Zurich) · 2018
In the last decade, the use of mobile devices such as smartphones and tablets has increased dramatically.Portable devices are now capable of achieving a multitude of tasks and, therefore, have become an important tool in our everyday life.Moreover, users often own more than one device and, for this reason, cross-device applications have raised interest among researchers and companies.In this multi-device era, improving the overall user experience on mobile devices in single and cross-device scenarios is particularly critical.Despite this need, the form of interaction currently used on these devices relies mainly on touch.While touch gestures offer an intuitive way of interacting with smartphones and tablets, they can fail in certain situations.For instance, touch often requires the user to change their holding position to reach some locations on the screen.Moreover, to perform touch gestures users need to look at the screen and they are often required to held the device with both hands.To overcome these issues, researchers have proposed alternative forms of interactions such as tilting and mid-air gestures.Tilting interactions are movements of the device in some direction and, given their nature, they are one-hand and eyes-free interactions.Alternatively, researchers have also exploited the use of the phones' cameras to perform mid-air poses to trigger actions on the device and, therefore, enlarge the interaction zone of the phone or tablet.However, this research has been applied mainly to native applications rather than on mobile web browsers.On the web, the primary focus has been on the adaptation of content to the screen real estate of the clients rather than its modes of interactions.While responsive design solved some of the issues concerning the usability of web applications on mobile devices, it does not tackle any alternative forms of gestures.In this thesis, we bridge this gap via a series of contributions aimed at improving the mobile user experience by enriching the set of possible interactions available on single and cross-device web applications.We start with an in-depth analysis of the literature on tilting gestures and, based on these, derived a set of API as Tilt-and-Tap (TAT), a JavaScript framework for the rapid development of motion gestures on the web.TAT offers a catalogue of previously proposed tilting interactions and allows developers to customise gestures with a number of different parameters.TAT has the goal of encouraging developers to use tilting interactions in their web applications by hiding implementation details and supporting coherent interactions across different platforms and browsers.While TAT targeted developers, we also wanted to allow end-users with little or no experience in web technologies to experiment with motion gestures.To reach this goal, we used the drag and drop UI paradigm to add gestures to a web application.To test our approach, we developed WP-TAT, a WordPress extension that allows users to map tilting gestures to global actions via drag and drop interactions.Global actions are i ii functions which affect the entire website.For instance, a rapid movement of the device to the left or to the right could redirect the user to the previous or next posts.Users can define these mappings by using drag and drop interactions on our visual interface and, therefore, they are not required to write any code.Concerning cross-device scenarios, a preliminary user study output a series of requirements to improve the user experience, as well as the development process, of cross-device applications that exploit tilting interactions.Inferred by these requirements we developed Cross-Tilt-and-Tap (CTAT), a set of high-level APIs for the rapid prototyping of motion interactions in cross-device environments.After the developer has attached to the desired gesture (i.e.tilt right), the sender of the message and its receivers, the framework will be responsible for recognising the devices involved in the communication, detect the motion interaction and sending the messages.Finally, we continued our work on cross-device applications by employing the use of mid-air gestures to share data across devices.Via an elicitation study, we asked users to suggest a series of gestures to target co-located and remote devices.The proposed gestures, as well as the feedback received from participants, informed the design of MyoShare, a system that allows users to copy web data from and to their desktop computers and mobile devices.To evaluate our frameworks and tools, we have carried out a number of user and developer studies as well as proposing various example applications.