The system provides a foundation for building an “editing” platform for Sign Language (SL). Text editors (such as Microsoft Word application) are communication tools for natural language speakers. Deaf and hearing-impaired people have to depend on the visual channels (gestures) for their interactions (using SLs). How can these people make use of the latest computer technology to enhance their communications, that is, can we develop an SL “text-editor” for SL users? My PhD work addresses this problem.
Traditionally (and still in the same way), SL users record their signing procedures (like words/sentences/phases/paragraphs in natural languages) in real time and then send the video clips for the exchange of information. These video data is huge in its capacity and cannot be edited again to transfer a “new” meaning. That is to say, SL has its syntax rules and you cannot get a new video clip (i.e., a new SL “sentence”) that follows the syntax rules by simply cutting a video (i.e., a SL “sentence”) into small clips (i.e., SL “words”) and then putting them in a different order, which in fact, is a difficult job.
By using the system I developed, virtual gestures (instead of the gestures recorded by SL signers) can be constructed to simulate the visual signs of SL “words” and even “sentences”:
Lifelike virtual hand and body are constructed on the computer in real time with the use of techniques in the area of computer graphics and software engineering and HCI (human-computer interactions). Any hand gestures and body poses can be simulated and recorded in parameters (a set of numbers, not video sequences).
Research advancements from SL studies (particularly, American Sign Language) has been applied to create SL’s phonological and morphological elements, which are virtual hand configurations—hand orientation and figures’ arrangements (like “w”, “or” and “d” in the English word “word”).
These basic SL elements can be combined into larger SL meaningful parts—SL words (like “word” in English), which can be saved in the system’s SL database to form a SL dictionary.
SL sentences/phrases/paragraphs (i.e., SL virtual signs) can be created by retrieving the SL words from the dictionary (like “I speak English”).
All these operations have been implemented in computer programs.
Major contributions/breakthroughs:
There are very few SL software packages, most of which are still using the traditional video-recordings of SL signs, and all of which are like dictionaries in a particular SL and lack the capability of creating new SL “words” and of providing a “text-editing” environment. The system I developed has provided (proven to be) a good example of an SL digital dictionary and of an SL “text-editor”.
Such software will be very helpful for the under-represented populations.
Other notes:
Please check http://cs.salemstate.edu/~byi/sEditor/index.html for demos of this system.