Kerie2006 Poster Template 01
-
Upload
kkkseld -
Category
Technology
-
view
878 -
download
0
description
Transcript of Kerie2006 Poster Template 01
![Page 1: Kerie2006 Poster Template 01](https://reader038.fdocuments.us/reader038/viewer/2022110302/54813915b4af9fa1348b49b0/html5/thumbnails/1.jpg)
Speech to Sign Language Interpreter System (SSLIS)
Khalid K. El-Darymli, Othman O. Khalifa and Hassan Enmsoah
Department of Computer Engineering, Faculty of EngineeringInternational Islamic University Malaysia, PO BOX 10, Kuala Lumpur, 50728, Malaysia
Phone: 03-6196-4433, Fax: 03-6196-4433, E-mail: [email protected]
Sphinx 3.5 speech recognition
engine
ASL database
* Speech to text* Equivalent SE translation
Recognized textLive input speech
Figure 1: Flowchart depicts basic structure of SSLIS
Input constituents of recognized text separately
Is the input word a basic?
Is the basic word within the ASL video clips vocabulary?
YES
Extract the basic word out of the input word and temporally memorize
the word before extractionNO
ASL pre-recorded video clips
YES
Final Output
The American Manual Alphabet
NO
Equivalent ASL video clip
of input basic word
Fingerspelling of the input word
Is extracted word within ASL db
vocabulary?
NO
Depending on the type of temporally memorized word append a suitable
marker to its equivalent ASL video clip
YES
The temporally memorized word
Recognized text(The speech-recognition
engine's output)
Signal processing (FE)
Continuous input speech
Acoustic ModelP(A1,…,AT|p1,...pk)
DictionaryP(p1,…,pk|W)
Language ModelP(Wn|W1,W2,…,Wn-1)
Knowledge Base
Training
Extracted features vector
Hypothesis evaluation
Decoder
P(X|W)*P(W)
Search feedback
Decoding
Best Hypotheses
WBest
H={W1, W2, …,Wk}
X={x1, x2,…, xT}
Figure 2: Flowchart depicts simplified structure of Sphinx 3.5 speech engine
Regular past verbs:-edtalked, wanted, learned
Regular plural nouns: sbears, houses
3rd person singular: -swalks, eats, sings
Irregular past verbs:(sweep RH open B, tips
out, to the right )saw, heard, blew
Irregular plural nouns:(sign the word twice )
children, sheep, mice
mice
Possessive: -'scat's, daddy's, chair's
verb form: -ingclimbing, playing, running
Adjective: -ySleepy, sunny, cloudy
Adverb: -lyBeautifully, happily, nicely
Participle:Fallen, gone, grown
Comparative: -ersmaller, faster, longer
Superlative: -estSmallest, fastest ,
longest
Opposite of: un-, im-, in-, etc.
(made before the sign word, as a prefix )
unhappy, impatient, inconsiderate
Agent (person):(sign made near the body )
teacher, actor, artist
Agent (thing)(sign made away from the body )
washer, dryer , planter
Figure 3: Signed English markers
1. Research Goal and Objectives:Design and Manipulation of Speech to Sign Language Interpreter System. To fill the gap between deaf and nondeaf people in two senses. Firstly, by using this SW for educational purposes for deaf people and secondly, by facilitating the communication between deaf and nondeaf people.To increase independence and self-confidence of the deaf person.To increase opportunities for advancement and success in education, employment, personal relationships, and public access venues.To improve quality of life.
2. SSLIS Capabilities:Real time speech to text to video sign language.Text to video sign language.Automatic WER calculation.Text to computer generated voice with synchronized lips.Speed control of ASL movies in play.“Minimize to Auto” allows “drag and drop” from any text editor to be signed.Demonstration of SE manual as parallel to English.Demonstration of decoding process of speech.“Live Decode” Program allows real time speech recognition while “Live Pretend” and “Decode” allows speech recognition in batch mode.
3. ConclusionsThe research aim of offering freely available and open source SSLIS is fulfilled.Sphinx 3.5 was manipulated as the SR engine.Signed English manual was followed for translation.
4. Shortcomings and Further Work: Degradation in the speech recognition accuracy [an acoustic model generated from dictation tasks DB needs to be generated and employed],To highly improve the speech recognition accuracy, speech recognition engine can be adapted to different users through introducing enrollment session.
Figure 4: ASL alphabets
Figure 5: Demonstration of ASL in SSLIS
Figure 6: Some Snapshots of SSLIS