User-Centered Design for Multimodal Interaction in Image Retrieval: Optimizing Sketch and Semantic Input
Fan Yang, Nor Azman Ismail, Yee Yong Pang, Che Soh Said, Farkhana Muchtar, Layla Hasan, Sim Hiew Moi, Jianbo Fan · 2025
Image retrieval systems based on sketches and natural language input have attracted much attention due to their intuitiveness and multimodal features. However, the user interface (UI) design of such systems still faces challenges, requiring a balance between usability, efficiency, and user satisfaction. This study adopts a user-centered design (UCD) approach to explore the design process of a sketch and semantic-based image retrieval interface. Through a literature review and semi-structured interviews, we analyze key challenges and user needs in existing systems and examine how user feedback informs the iterative refinement of low-fidelity prototypes. The results demonstrate that the UCD approach effectively enhances the iterative refinement of the interface, ensuring better alignment with users’ interaction patterns and requirements. This study provides insights into the UI design of multimodal interaction image retrieval systems and summarizes the best practices for future development.