Allie – AI Voice and Sign Assistant
Sayyada Kaneez Sogra Fatima, M. Ali · International jounal of information technology and computer engineering. · 2025
Effective communication is a vital aspect of humaninteraction, yet individuals with speech and hearingdisabilities frequently encounter barriers in conveyingtheir thoughts or understanding others. Conventionalsolutions such as sign language interpreters or visualaids are not always accessible, which can result insocial isolation and reduced opportunities forparticipation. To address this issue, Allie – AI Voiceand Sign Assistant has been developed as a smartAndroid-based mobile application that leveragesartificial intelligence (AI) and mobile technology toenable seamless two-way communication.Allie offers two key features: (1) Voice/Text to SignTranslation, where spoken or typed input is translatedinto sign language using animated GIFs for commonlyused words or phrases (e.g., “Hello”, “Thank you”)and static alphabet images (A–Z) for letter-by-letterexpression; and (2) Sign to Text/Voice Recognition,which utilizes MediaPipe’s HandLandmarker todetect hand landmarks and classifies them through acustom-trained TensorFlow Lite (TFLite) model. Thisenables users to see the recognized gesture as text andperform context-based actions such as launching apps(e.g., WhatsApp, YouTube, Spotify) through specificsigns.The app is designed for flexibility and usability in bothonline and offline environments, with sign mediastored in the res/drawable directory and optionally incloud storage using Firebase or Supabase. The customgesture classification model is built using landmarkdata collected through a dedicated in-app feature thatrecords hand movements into CSV format. The app’sinterface is built using Kotlin, incorporating JetpackCompose and ViewBinding to ensure a responsive andaccessible user experience.This paper explores the app’s architecture,development methodology, technological choices, andimplementation details. It highlights how machinelearning and computer vision contribute to buildinginclusive communication tools for differently-abledindividuals. The target users include not only thosewith speech or hearing impairments but also theirfamilies, educators, and learners of sign language.Future enhancements include integration of text-tospeechfunctionality for gesture outputs, animatedavatars for real-time sign rendering, and a built-inchatbot for conversational assistance. By combiningAI and mobile innovation, Allie represents a stepforward in bridging communication gaps andfostering inclusivity through technology.