Unvoiced: Designing an LLM-assisted Unvoiced User Interface using Earables
Tanmay Srivastava, Prerna Khanna, Shijia Pan, Phuc Le Hoang Nguyen, Shubham Kumar Jain · 2024
We present Unvoiced, a novel unvoiced user interface that leverages jaw motion to enable users to silently interact with their devices using earables. The core idea is to translate low-frequency jaw motion signals into high-frequency information-rich mel spectrograms. Our proposed cross-modal translation incorporates phonetic, contextual, and syntactic information, while the specialized loss function optimizes for these linguistic features. This ensures that the generated spectrograms capture nuanced speech characteristics. Evaluated for 19 users across four tasks, Unvoiced demonstrates >94% task completion rate and 90% task completion rate in noisy conditions.