Evaluating new requirements to pen-centric intelligent user interface based on end-to-end mathematical expressions recognition
Dmytro Zhelezniakov, Anastasiia Cherneha, Viktor Zaytsev, Tetiana Ignatova, Olga Radyvonenko, Oleg Yakovchuk · 2020
Advances in deep learning end-to-end recognition systems allow moving forward intelligent user interfaces. At the same time, they lead to new implications and restrictions in the UI design. In this paper, we discuss new requirements for the pen-centric intelligent user interface for operating with mathematical expressions. We argue that the following features are necessary for such an interface: use of variables (both automatically assigned and defined by the user), quick editing, and connected writing support. We demonstrate and evaluate features in SMath - a solution for iterative input and automatic calculation of handwritten mathematical expressions. It allows pen-based input and editing of two-dimensional mathematical expressions in various scenarios with a possibility to reuse the results of previous calculations. Through the evaluation study, we justify the proposed requirements from the point of user needs and preferences. In addition to the automatic metrics that were collected during test tasks execution using the application, we gathered and summarized users' feedback, which can be further used to formulate functional requirements and design the user interface. We also provide the evaluation of the recognition performance of the system on an open-source dataset of handwritten mathematical expressions.