Japanese JSL Bi-Directional Translating System

A paper on a bi-directional translating system between Japanese and Japanese Sign Language was presented by the staff of the Kyoto Institute of Technology, Japan, at the 10th International Conference on Computers Helping People with Special Needs (ICCHP) held July 12-14 at the University of Linz, Austria.

Assistive technology solutions, such as computer-synthesized sign animation, are promising communication tools for the disabled. These systems facilitate the ability to translate between different sign languages, translation between sign and verbal languages and news programs. In this paper a bi-directional translational between Japanese and Japanese Sign Language (JSL) has been proposed. Typically, such systems employ animation featuring a human model. However, a lot of problems have been encountered by the users. The Japanese JSL translating system addresses these issues and is working towards incorporating the improvised version by means of two experiments.

The system translates JSL into Japanese and vice versa and it aims to facilitate mutual communication between a person with a hearing disability and an able person who does not understand JSL. The system employs a case frame as an intermediate expression between Japanese and JSL in both directions of translation and then translates Japanese into JSL. It is represented by a case frame that describes its semantic structure with case relations between verbs and noun phrases, tense, aspects and modalities. The case frame is then turned into a sign expression via a Japanese-JSL dictionary. At this stage, the sign expression is visualized by means of the synthesis-by-rule method and the translation is presented in the form of the animated person model.

Experiment 1, called image matching, was conducted to determine the display conditions necessary for people to judge the identities of the two pieces of animation. In experiment 2, the sign understanding, the users read JSL signs from animation displayed under different circumstances, as the signs had clues to decipher their meanings. These conditions were expected to be looser in experiment 2 than 1.

For the experiments, ten hearing-impaired students, without any previous experience watching animation were used. In Experiment 1, the subjects were put through a series of image-matching trials. Two pieces of animation were presented at short intervals and the subjects were asked to differentiate between them. In Experiment 2, the subjects were made to see 300 animations. These were typically the same as the foregoing one or differed from it in one or two items. It was presented under one of 27 display conditions combining three display sizes, three different resolutions and three different frame rates.

The paired pieces of animation in the experiments were statistically gauged through the ANOVA test taking factors like size, resolution and frame rate into consideration. The results suggested that hearing impaired people and non hear-impaired people had different characteristics in attention; while the hearing set displayed a tendency to watch the whole body of the model, the hearing-impaired focused mainly on the hand shape and movement. The mouth signs, one-handed signs and two-signs were determined as the important keys. The results revealed that physical matching of animation needed image quality higher than sign reading. However, the conditions required for image matching and reading signs were found to be very different - signs were correctly read under looser conditions than in image matching.