Enhancing Accessibility: A Natural Human-Computer Interaction System Using Hand Gestures and Voice for Disabled Populations
Siddharth Singh, Mohammed Bakhtiyar Siddiqui, Michelle Zhu · 2025
This paper presents an innovative application that enables control of multimedia content through voice commands and static hand gestures, offering a transformative solution for individuals with disabilities. The application provides an intuitive, natural method for interacting with slides, images, videos, and audio, bypassing traditional input devices like keyboards and mice. It enhances accessibility and user experience by accommodating users with mobility impairments or low vision. Another key feature is the virtual writing capability, allowing users to draw or erase in mid-air and project the writing and drawing onto the screen, providing greater flexibility. The software leverages advanced technologies such as MediaPipe for gesture recognition, SpeechRecognition for voice command processing, and PyAutoGUI for automation, making it suitable for various environments, including personal entertainment, classroom, and conference settings. Additionally, users can customize hand gestures for specific commands, with newly captured gestures used to retrain the model. This paper outlines the system architecture, implementation, and performance evaluation, and discusses future developments.