MSigns: 3D Solution in Virtual Sign-Language Translation

MSigns program is a revolutionary invention aims to develop technology that would "listen" to speech and translate it into sign language that a person could see - in 3D in real time - on a handheld computer. Jason Gilbert and Judy Yu, Graduate students in Michigan Engineering’s Department of Atmospheric, Oceanic and Space Sciences (AOSS), developed this to make life easier for people who rely on sign language to communicate.

It was a highly complementary team effort from Gilbert and Yu that led to the invention of MSigns program, said Thomas Zurbuchen, the project advisor and AOSS and Aerospace Associate Professor. Gilbert’s experience as a sign interpreter, and Judy’s skills in creating software proved crucial to the success of the program.

Translation of voice to text was the first step in the program. The language used for translation of voice to text is American English because its grammar is less complex than American Sign Language grammar, and the software to translate voice to American English text was easily available.

The next step was to render the text. Gilbert and Yu selected a method in which the motion-capture equipment records signs that translate into individual words, which can be pieced together to form sentences.

A special pair of gloves, a shirt, and a cap with markers at strategic locations is also part of the outfit. The motion-capture equipment records the movement of the markers and the MotionBuilder software processes motion-capture data. Web and database technologies are used to write a program that converts the output from MotionBuilder into data to animate the skeleton.

The assistive technology MSigns program is still under the process of optimization. Using a standard pose at the beginning of each movement may help in making transitions between movements more comprehensible, ensuring flexibility in a 3D virtual environment.

Judy Yu is using web and database technologies to write a program that converts raw output into data that animates a virtual skeleton. Yu plans to create fluid movements in a 3D virtual environment by building a virtual interpreter. The resultant advantage is that since everything plays on a web interface, there is no need to download special software to see what is on the handheld screen.

The team aims to develop a assistive technology system that can translate about 100 words into sign language. For words missing in the current database, finger spelling will be used to fill in the gaps.

Source: Michigan Engineer