Accessible American Sign Language Recognition with the Leap Motion Controller
Shm Garanganao Almeda · 2019
Communities that use visual languages like American Sign Language (ASL) to communicate are underrepresented in the domain of translation and language learning tools. Translating between a visual, gestural language like ASL and a spoken, written language like English presents multiple unique challenges. The visual elements of hand shape, location, orientation, and motion used in ASL can be challenging to detect, interpret, and represent. We present an algorithm to recognize the 24 static signs in the ASL fingerspelling alphabet using the Leap Motion Controller. This algorithm can be used in ASL teaching applications, as well as to translate finger-spelled words. Existing Sign Language Recognition systems use cameras to extract low resolution images of hands where positions of individual fingers are difficult to detect. Our program utilizes the Leap Motion device, a motion sensor that provides skeletal hand tracking functionality. Based on joint coordinate and orientation information provided by the device, we extract a number of features including finger configuration (extended, bent, or curled), distance between fingers, and palm orientation. We then use a decision tree to classify signs as individual letters. This method does not require large amounts of training data like other machine learning based approaches. Preliminary results indicate that our method is successful in recognizing most of the static signs in the ASL fingerspelling alphabet.