Activity analysis of sign language video for mobile telecommunication
Richard E. Ladner, E.A. Riskin, Nera Cherniavsky · 2009
The goal of enabling access for the Deaf to the current U.S. mobile phone network by compressing and transmitting sign language video in real-time on an off-the-shelf mobile phone gives rise to challenging research questions. Encoding and transmission of real-time video over mobile phones is a power-intensive task that can quickly drain the battery, rendering the cell phone useless. Properties of conversational sign language can help save power and bits: namely, lower frame rates are possible when one person is not signing due to turn-taking, and the grammar of sign language is found primarily in the face. Thus the focus can be on the important parts of the video, saving resources without degrading intelligibility. In this dissertation, I describe my algorithms for determining in real-time the activity in the video and encoding a dynamic skin-based region-of-interest. I use features available “for free” from the encoder, and implement my techniques on an off-the-shelf mobile phone. I evaluate my sign language sensitive methods in a user study, with positive results. The algorithms can save considerable resources without sacrificing intelligibility, helping make real-time video communication on mobile phones both feasible and practical.