Lip Feature Extraction Using Red Exclusion
description
Transcript of Lip Feature Extraction Using Red Exclusion
Lip Feature Extraction Using Red Exclusion
Trent W. Lewis and David M.W. Powers
Flinders University of SA
VIP2000
01/12/2000 Lip Feature Extraction Using Red Exlucsion
2
Overview
• Context
• Lip Feature Extraction– Related Work (greyscale, horizontal edges, red and hue colour spaces)
– Red Exclusion
• AVSR: Results and Issues
• Summary
01/12/2000 Lip Feature Extraction Using Red Exlucsion
3
Context
• Audio Speech Recognition (ASR)
• Psycholinguistic Research
• Audio Visual Speech Recognition (AVSR)
01/12/2000 Lip Feature Extraction Using Red Exlucsion
4
Context - ASR
• Up to 99% word accuracy
• However,– limited context– limited vocabulary– trained on individual– close microphone, cannot handle noise
01/12/2000 Lip Feature Extraction Using Red Exlucsion
5
Context - Psycholinguistic
• McGurk Effect– A[ba] + V[ga] [da]
• Viseme– visual phonemes– form complementary sets
• Demonstrates vision can assist the perception of speech AVSR
01/12/2000 Lip Feature Extraction Using Red Exlucsion
6
Context - AVSR
• Acoustic Features• Visual Features
– width
– height
– oral cavity
• Integration– Early
– Late
01/12/2000 Lip Feature Extraction Using Red Exlucsion
7
Lip Feature Extraction
• Pixel-Based Model
01/12/2000 Lip Feature Extraction Using Red Exlucsion
8
Lip Feature Extraction
• Pixel-Based Model– raw pixels or minimal processing– retain linguistically relevant data– large amounts of data, time– shift and lighting variant– normalisation and PCA
01/12/2000 Lip Feature Extraction Using Red Exlucsion
9
Lip Feature Extraction
• Pixel-Based Model– reduced input to set of hand-crafted features– width, height, average intensity, etc.– less features, time– model fitting, time– lose linguistically relevant features
01/12/2000 Lip Feature Extraction Using Red Exlucsion
10
Lip Feature Extraction
• Pixel-Based Model– feature extraction– Steps
• preprocess to enhance contrast
• locate mouth edges
• identify corners, height, and other key features
• train recogntion engine
Our Approach
01/12/2000 Lip Feature Extraction Using Red Exlucsion
11
Lip Feature Extraction
• Database
1 2 3
01/12/2000 Lip Feature Extraction Using Red Exlucsion
12
Lip Feature Extraction
• Preprocessing Techniques– Grey-scale– Horizontal Edges– Red Analysis– Hue, Saturation, and Value (HSV)– Red Exclusion
01/12/2000 Lip Feature Extraction Using Red Exlucsion
13
Lip Feature Extraction
• Grey-scale– vertical position of mouth
• minimum row sum
– threshold minimum row• average of min and max of row
– search for above threshold pixels
01/12/2000 Lip Feature Extraction Using Red Exlucsion
14
Lip Feature Extraction
• Grey-scale
01/12/2000 Lip Feature Extraction Using Red Exlucsion
15
Lip Feature Extraction
• Grey-scale
01/12/2000 Lip Feature Extraction Using Red Exlucsion
16
Lip Feature Extraction
• Horizontal Edges– high horizontal edge content– 3x3, DY Prewitt operator
111
000
111
DY
01/12/2000 Lip Feature Extraction Using Red Exlucsion
17
Lip Feature Extraction
• Horizontal Edges
“Found” Corners Binary Image
01/12/2000 Lip Feature Extraction Using Red Exlucsion
18
Lip Feature Extraction
• Red Analysis– overcome bearded subjects– used for face location
limlim UG
RL
01/12/2000 Lip Feature Extraction Using Red Exlucsion
19
Lip Feature Extraction
• Red Analysis
01/12/2000 Lip Feature Extraction Using Red Exlucsion
20
Lip Feature Extraction
• HSV– disentangles illumination from colour Illumination > Hue
otherwise
whhw
hhhf o
o
,0
,)(
1)( 2
2
01/12/2000 Lip Feature Extraction Using Red Exlucsion
21
Lip Feature Extraction
• HSV
01/12/2000 Lip Feature Extraction Using Red Exlucsion
22
Lip Feature Extraction
• Red Exclusion– needed extraction method for AVSR– similar to Red Analysis
• face predominantly red
• variations occur in the blue and green colours
B
Glog
01/12/2000 Lip Feature Extraction Using Red Exlucsion
23
Lip Feature Extraction
• Red Exclusion
01/12/2000 Lip Feature Extraction Using Red Exlucsion
24
Lip Feature Extraction
• Corners found using Red Exclusion
01/12/2000 Lip Feature Extraction Using Red Exlucsion
25
Lip Feature Extraction
• ComparisonAlgorithm
Subject 1 female
Subject 2 male bearded
Subject 3 male thin lips
Reliable Corners
Other Features
Grey-scale
Edge
Red Analysis
HSV Red Exclusion
01/12/2000 Lip Feature Extraction Using Red Exlucsion
26
AVSR: Results and Issues
• Application for red exclusion
• Used in finding lip features– Width– Height– Key pixels
01/12/2000 Lip Feature Extraction Using Red Exlucsion
27
AVSR: Results and Issues
• Visual Speech RecognitionStatic (%) Dynamic (%)
Voicing 32.2 30.8
Viseme 54.7 51.3
Phoneme 14.7 13.6
01/12/2000 Lip Feature Extraction Using Red Exlucsion
28
AVSR: Results and Issues
• AVSR - IntegrationEarly
Static
Early
Dynamic
Late
Voice/Vis
Late Error
Voice/Vis
Phoneme20.1 18.0 29.0 19.5
01/12/2000 Lip Feature Extraction Using Red Exlucsion
29
Summary
• Vision can help ASR– AVSR
• Needed good extraction technique– Red Exclusion
• AVSR is difficult when both signals degraded
01/12/2000 Lip Feature Extraction Using Red Exlucsion
30
Questions?
Trent W. Lewis
BSc (Cognitive Science)
Flinders University