Lip Feature Extraction Using Red Exclusion

30
Lip Feature Extraction Using Red Exclusion Trent W. Lewis and David M.W. Powers Flinders University of SA VIP2000

description

Lip Feature Extraction Using Red Exclusion. Trent W. Lewis and David M.W. Powers Flinders University of SA VIP2000. Overview. Context Lip Feature Extraction Related Work (greyscale, horizontal edges, red and hue colour spaces) Red Exclusion AVSR: Results and Issues Summary. Context. - PowerPoint PPT Presentation

Transcript of Lip Feature Extraction Using Red Exclusion

Page 1: Lip Feature Extraction Using Red Exclusion

Lip Feature Extraction Using Red Exclusion

Trent W. Lewis and David M.W. Powers

Flinders University of SA

VIP2000

Page 2: Lip Feature Extraction Using Red Exclusion

01/12/2000 Lip Feature Extraction Using Red Exlucsion

2

Overview

• Context

• Lip Feature Extraction– Related Work (greyscale, horizontal edges, red and hue colour spaces)

– Red Exclusion

• AVSR: Results and Issues

• Summary

Page 3: Lip Feature Extraction Using Red Exclusion

01/12/2000 Lip Feature Extraction Using Red Exlucsion

3

Context

• Audio Speech Recognition (ASR)

• Psycholinguistic Research

• Audio Visual Speech Recognition (AVSR)

Page 4: Lip Feature Extraction Using Red Exclusion

01/12/2000 Lip Feature Extraction Using Red Exlucsion

4

Context - ASR

• Up to 99% word accuracy

• However,– limited context– limited vocabulary– trained on individual– close microphone, cannot handle noise

Page 5: Lip Feature Extraction Using Red Exclusion

01/12/2000 Lip Feature Extraction Using Red Exlucsion

5

Context - Psycholinguistic

• McGurk Effect– A[ba] + V[ga] [da]

• Viseme– visual phonemes– form complementary sets

• Demonstrates vision can assist the perception of speech AVSR

Page 6: Lip Feature Extraction Using Red Exclusion

01/12/2000 Lip Feature Extraction Using Red Exlucsion

6

Context - AVSR

• Acoustic Features• Visual Features

– width

– height

– oral cavity

• Integration– Early

– Late

Page 7: Lip Feature Extraction Using Red Exclusion

01/12/2000 Lip Feature Extraction Using Red Exlucsion

7

Lip Feature Extraction

• Pixel-Based Model

Page 8: Lip Feature Extraction Using Red Exclusion

01/12/2000 Lip Feature Extraction Using Red Exlucsion

8

Lip Feature Extraction

• Pixel-Based Model– raw pixels or minimal processing– retain linguistically relevant data– large amounts of data, time– shift and lighting variant– normalisation and PCA

Page 9: Lip Feature Extraction Using Red Exclusion

01/12/2000 Lip Feature Extraction Using Red Exlucsion

9

Lip Feature Extraction

• Pixel-Based Model– reduced input to set of hand-crafted features– width, height, average intensity, etc.– less features, time– model fitting, time– lose linguistically relevant features

Page 10: Lip Feature Extraction Using Red Exclusion

01/12/2000 Lip Feature Extraction Using Red Exlucsion

10

Lip Feature Extraction

• Pixel-Based Model– feature extraction– Steps

• preprocess to enhance contrast

• locate mouth edges

• identify corners, height, and other key features

• train recogntion engine

Our Approach

Page 11: Lip Feature Extraction Using Red Exclusion

01/12/2000 Lip Feature Extraction Using Red Exlucsion

11

Lip Feature Extraction

• Database

1 2 3

Page 12: Lip Feature Extraction Using Red Exclusion

01/12/2000 Lip Feature Extraction Using Red Exlucsion

12

Lip Feature Extraction

• Preprocessing Techniques– Grey-scale– Horizontal Edges– Red Analysis– Hue, Saturation, and Value (HSV)– Red Exclusion

Page 13: Lip Feature Extraction Using Red Exclusion

01/12/2000 Lip Feature Extraction Using Red Exlucsion

13

Lip Feature Extraction

• Grey-scale– vertical position of mouth

• minimum row sum

– threshold minimum row• average of min and max of row

– search for above threshold pixels

Page 14: Lip Feature Extraction Using Red Exclusion

01/12/2000 Lip Feature Extraction Using Red Exlucsion

14

Lip Feature Extraction

• Grey-scale

Page 15: Lip Feature Extraction Using Red Exclusion

01/12/2000 Lip Feature Extraction Using Red Exlucsion

15

Lip Feature Extraction

• Grey-scale

Page 16: Lip Feature Extraction Using Red Exclusion

01/12/2000 Lip Feature Extraction Using Red Exlucsion

16

Lip Feature Extraction

• Horizontal Edges– high horizontal edge content– 3x3, DY Prewitt operator

111

000

111

DY

Page 17: Lip Feature Extraction Using Red Exclusion

01/12/2000 Lip Feature Extraction Using Red Exlucsion

17

Lip Feature Extraction

• Horizontal Edges

“Found” Corners Binary Image

Page 18: Lip Feature Extraction Using Red Exclusion

01/12/2000 Lip Feature Extraction Using Red Exlucsion

18

Lip Feature Extraction

• Red Analysis– overcome bearded subjects– used for face location

limlim UG

RL

Page 19: Lip Feature Extraction Using Red Exclusion

01/12/2000 Lip Feature Extraction Using Red Exlucsion

19

Lip Feature Extraction

• Red Analysis

Page 20: Lip Feature Extraction Using Red Exclusion

01/12/2000 Lip Feature Extraction Using Red Exlucsion

20

Lip Feature Extraction

• HSV– disentangles illumination from colour Illumination > Hue

otherwise

whhw

hhhf o

o

,0

,)(

1)( 2

2

Page 21: Lip Feature Extraction Using Red Exclusion

01/12/2000 Lip Feature Extraction Using Red Exlucsion

21

Lip Feature Extraction

• HSV

Page 22: Lip Feature Extraction Using Red Exclusion

01/12/2000 Lip Feature Extraction Using Red Exlucsion

22

Lip Feature Extraction

• Red Exclusion– needed extraction method for AVSR– similar to Red Analysis

• face predominantly red

• variations occur in the blue and green colours

B

Glog

Page 23: Lip Feature Extraction Using Red Exclusion

01/12/2000 Lip Feature Extraction Using Red Exlucsion

23

Lip Feature Extraction

• Red Exclusion

Page 24: Lip Feature Extraction Using Red Exclusion

01/12/2000 Lip Feature Extraction Using Red Exlucsion

24

Lip Feature Extraction

• Corners found using Red Exclusion

Page 25: Lip Feature Extraction Using Red Exclusion

01/12/2000 Lip Feature Extraction Using Red Exlucsion

25

Lip Feature Extraction

• ComparisonAlgorithm

Subject 1 female

Subject 2 male bearded

Subject 3 male thin lips

Reliable Corners

Other Features

Grey-scale

Edge

Red Analysis

HSV Red Exclusion

Page 26: Lip Feature Extraction Using Red Exclusion

01/12/2000 Lip Feature Extraction Using Red Exlucsion

26

AVSR: Results and Issues

• Application for red exclusion

• Used in finding lip features– Width– Height– Key pixels

Page 27: Lip Feature Extraction Using Red Exclusion

01/12/2000 Lip Feature Extraction Using Red Exlucsion

27

AVSR: Results and Issues

• Visual Speech RecognitionStatic (%) Dynamic (%)

Voicing 32.2 30.8

Viseme 54.7 51.3

Phoneme 14.7 13.6

Page 28: Lip Feature Extraction Using Red Exclusion

01/12/2000 Lip Feature Extraction Using Red Exlucsion

28

AVSR: Results and Issues

• AVSR - IntegrationEarly

Static

Early

Dynamic

Late

Voice/Vis

Late Error

Voice/Vis

Phoneme20.1 18.0 29.0 19.5

Page 29: Lip Feature Extraction Using Red Exclusion

01/12/2000 Lip Feature Extraction Using Red Exlucsion

29

Summary

• Vision can help ASR– AVSR

• Needed good extraction technique– Red Exclusion

• AVSR is difficult when both signals degraded

Page 30: Lip Feature Extraction Using Red Exclusion

01/12/2000 Lip Feature Extraction Using Red Exlucsion

30

Questions?

Trent W. Lewis

BSc (Cognitive Science)

Flinders University

[email protected]