Lecture: Face Recognitionvision.stanford.edu/teaching/cs131_fall1718/files/13_LDA... ·...

Lecture 12 -Stanford University

Lecture:FaceRecognition

JuanCarlosNiebles andRanjayKrishnaStanfordVisionandLearningLab

07-Nov-171


Whatwewilllearntoday• Introductiontofacerecognition• TheEigenfaces Algorithm• LinearDiscriminantAnalysis(LDA)

07-Nov-172

TurkandPentland,Eigenfaces forRecognition,JournalofCognitiveNeuroscience 3 (1):71–86.

P.Belhumeur,J.Hespanha,andD.Kriegman."Eigenfaces vs.Fisherfaces:RecognitionUsingClassSpecificLinearProjection".IEEETransactionsonpatternanalysisandmachineintelligence 19 (7):711.1997.


CourtesyofJohannesM.Zanker

07-Nov-17

“Faces”inthebrain

3


“Faces”inthebrainfusiform facearea

07-Nov-17

Kanwisher,etal.1997

4


DetectionversusRecognition

07-Nov-175

Detectionfindsthefacesinimages RecognitionrecognizesWHOthepersonis


• Digitalphotography

FaceRecognition

07-Nov-176


FaceRecognition• Digitalphotography• Surveillance

07-Nov-177


• Digitalphotography• Surveillance• Albumorganization

FaceRecognition

07-Nov-178


FaceRecognition• Digitalphotography• Surveillance• Albumorganization• Persontracking/id.

07-Nov-179


• Digitalphotography• Surveillance• Albumorganization• Persontracking/id.• Emotionsandexpressions

FaceRecognition

07-Nov-1710


• Digitalphotography• Surveillance• Albumorganization• Persontracking/id.• Emotionsandexpressions

• Security/warfare• Tele-conferencing• Etc.

FaceRecognition

07-Nov-1711


TheSpaceofFaces

• Animageisapointinahighdimensionalspace– Ifrepresentedingrayscale intensity,

anNxMimageisapointinRNM

– E.g.100x100image=10,000dim

07-Nov-1712

Slidecredit:ChuckDyer,SteveSeitz,Nishino


100x100imagescancontainmanythingsotherthanfaces!

07-Nov-1713


TheSpaceofFaces

• Animageisapointinahighdimensionalspace– Ifrepresentedingrayscale intensity,

anNxMimageisapointinRNM

– E.g.100x100image=10,000dim

• However,relativelyfewhighdimensionalvectorscorrespondtovalidfaceimages

• Wewanttoeffectivelymodelthesubspaceoffaceimages

07-Nov-1714

Slidecredit:ChuckDyer,SteveSeitz,Nishino

ɸ1


Wherehaveweseensomethinglikethisbefore?

07-Nov-1715

ɸ1


Imagespace

Facespace

•Maximizethescatter ofthetrainingimagesinfacespace

• Computen-dimsubspacesuchthattheprojectionofthedatapointsontothesubspacehasthelargestvariance amongalln-dimsubspaces.

07-Nov-1716


• So,compressthemtoalow-dimensionalsubspacethatcaptureskeyappearancecharacteristicsofthevisualDOFs.

KeyIdea

• USEPCAforestimatingthesub-space(dimensionalityreduction)

•ComparetwofacesbyprojectingtheimagesintothesubspaceandmeasuringtheEUCLIDEANdistancebetweenthem.

07-Nov-1717



07-Nov-1718




Eigenfaces:keyidea

• Assumethatmostfaceimageslieonalow-dimensionalsubspacedeterminedbythefirstk(k<<d)directionsofmaximumvariance

• UsePCAtodeterminethevectorsor“eigenfaces”thatspanthatsubspace

• Representallfaceimagesinthedatasetaslinearcombinationsofeigenfaces

07-Nov-1719

M. Turk and A. Pentland, Face Recognition using Eigenfaces, CVPR 1991


Trainingimages:x1,…,xN

07-Nov-1720


Topeigenvectors:ɸ1,…,ɸk

Mean: μ

07-Nov-1721


VisualizationofeigenfacesPrincipal component (eigenvector) ɸk

μ + 3σkɸk

μ – 3σkɸk

07-Nov-1722


Eigenface algorithm• Training

1. Aligntrainingimagesx1,x2,…,xN

2. Computeaverageface

3. Computethedifferenceimage(thecentereddatamatrix)

07-Nov-1723

Notethateachimageisformulatedintoalongvector!

µ =1N

xi∑


Eigenface algorithm4. Computethecovariancematrix

5. ComputetheeigenvectorsofthecovariancematrixΣ6. Computeeachtrainingimagexi ‘sprojectionsas

7. Visualizetheestimatedtrainingfacexi

07-Nov-1724

xi → xic ⋅φ1, xi

c ⋅φ2,..., xic ⋅φK( ) ≡ a1,a2, ... ,aK( )

xi ≈ µ + a1φ1 + a2φ2 +...+ aKφK

Lecture 12 -Stanford University 07-Nov-1725

6. Computeeachtrainingimagexi ‘sprojectionsas

7. Visualizethereconstructedtrainingfacexi

xi → xic ⋅φ1, xi

c ⋅φ2,..., xic ⋅φK( ) ≡ a1,a2, ... ,aK( )

xi ≈ µ + a1φ1 + a2φ2 +...+ aKφK

a1φ1 a2φ2 aKφK...

Eigenface algorithm

Reconstructedtrainingface

𝑥"


Eigenvalues (variancealongeigenvectors)

07-Nov-1726


• OnlyselectingthetopKeigenfacesà reducesthedimensionality.• Fewereigenfaces resultinmoreinformationloss,andhenceless

discriminationbetweenfaces.

ReconstructionandErrors

K =4

K =200

K =400

07-Nov-1727


Eigenface algorithm

• Testing1. Takequeryimaget2. Projectintoeigenface space andcomputeprojection

3. CompareprojectionwwithallNtrainingprojections• Simplecomparisonmetric:Euclidean• Simpledecision:K-NearestNeighbor

(note:this“K”referstothek-NNalgorithm,isdifferentfromthepreviousK’sreferringtothe#ofprincipalcomponents)

07-Nov-1728

t→ (t −µ) ⋅φ1, (t −µ) ⋅φ2,..., (t −µ) ⋅φK( ) ≡ w1,w2,...,wK( )


Eigenfaces looksomewhatlikegenericfaces.

07-Nov-1729

Visualizationofeigenfaces


Shortcomings• Requirescarefullycontrolleddata:

– Allfacescenteredinframe– Samesize– Somesensitivitytoangle

• Alternative:– “Learn”onesetofPCAvectorsforeachangle– Usetheonewithlowesterror

• Methodiscompletelyknowledgefree– (sometimesthisisgood!)– Doesn’tknowthatfacesarewrappedaround3Dobjects(heads)

– Makesnoefforttopreserveclassdistinctions

07-Nov-1730


SummaryforEigenface

Pros• Non-iterative,globallyoptimalsolution

Limitations

• PCAprojectionisoptimalforreconstruction fromalowdimensionalbasis,butmayNOTbeoptimalfordiscrimination…

07-Nov-1731


Besidesfacerecognitions,wecanalsodoFacialexpressionrecognition

07-Nov-1732


Happinesssubspace(methodA)

07-Nov-1733


Disgustsubspace(methodA)

07-Nov-1734


FacialExpressionRecognitionMovies(methodA)

07-Nov-1735



07-Nov-1736




Whichdirectionwillisthefirstprinciplecomponent?

07-Nov-1737


Fischer’sLinearDiscriminantAnalysis

• Goal:findthebestseparationbetweentwoclasses

07-Nov-1738

SlideinspiredbyN.Vasconcelos


DifferencebetweenPCAandLDA

• PCApreservesmaximumvariance

• LDApreservesdiscrimination– Findprojectionthatmaximizesscatterbetweenclassesandminimizesscatterwithinclasses

07-Nov-1739


IllustrationoftheProjection

Poor Projection

x1

x2

x1

x2

• Using two classes as example:

Good

07-Nov-1740


Basicintuition:PCAvs.LDA


07-Nov-17

42

LDAwith2variables• WewanttolearnaprojectionWsuchthattheprojectionconvertsallthe

pointsfromxtoanewspace(Forthisexample,assumem==1):

• Lettheperclassmeansbe:

• Andtheperclasscovariancematricesbe:

• Wewantaprojectionthatmaximizes:

𝑧 = 𝑤&𝑥


Fischer’sLinearDiscriminantAnalysis

07-Nov-1743



07-Nov-17

44

LDAwith2variablesThefollowingobjectivefunction:

Canbewrittenas


LDAwith2variables

• Wecanwritethebetweenclassscatteras:

• Also,thewithinclassscatterbecomes:

07-Nov-1745



• Wecanpluginthesescattervaluestoourobjectivefunction:

• Andourobjectivebecomes:

LDAwith2variables

07-Nov-1746



LDAwith2variables

• Thescattervariables

07-Nov-1747



2S

1S

BS

21 SSSW +=

x1

x2Withinclassscatter

Betweenclassscatter

07-Nov-1748

Visualization


LinearDiscriminantAnalysis(LDA)

• Maximizingtheratio

• Isequivalenttomaximizingthenumeratorwhilekeepingthedenominatorconstant,i.e.

• AndcanbeaccomplishedusingLagrangemultipliers,wherewedefinetheLagrangian as

• Andmaximizewithrespecttobothwandλ

07-Nov-1749



LinearDiscriminantAnalysis(LDA)• Settingthegradientof

Withrespecttowtozerosweget

or

• Thisisageneralizedeigenvalueproblem

• Thesolutioniseasywhen exists

07-Nov-1750



LinearDiscriminantAnalysis(LDA)

• Inthiscase

• AndusingthedefinitionofSB

• Notingthat(μ1-μ0)Tw=αisascalarthiscanbewrittenas

• andsincewedon’tcareaboutthemagnitudeofw

07-Nov-1751



LDAwithNvariablesandCclasses


Variables• N Sample images:

• C classes:

• Average of each class:

• Average of all data:

{ }Nxx ,,1 !

å==

N

kkxN 1

1µ

å=Î ikx

ki

i xN c

µ 1

07-Nov-1753


ScatterMatrices

• Scatter of class i:

å=

=c

iiW SS

1

( )( )å=

--=c

i

TiiiB NS

1

µµµµ

• Within class scatter:

• Between class scatter:

( )( )Tikx

iki xxSik

µµc

--= åÎ

07-Nov-1754


Mathematical Formulation• Recall that we want to learn a projection W

such that the projection converts all the points from x to a new space z:

• After projection:– Between class scatter– Within class scatter

• So, the objective becomes:

WSWS BT

B =~

WSWS WT

W =~

WSW

WSW

SS

WW

TB

T

W

Bopt WW

max arg~~

max arg ==

07-Nov-1755

𝑧 = 𝑤&𝑥


MathematicalFormulation

• Solvegeneralizedeigenvectorproblem:

WSW

WSWW

WT

BT

opt Wmax arg=

miwSwS iWiiB ,,1 !== l

07-Nov-1756


Mathematical Formulation

• Solution: Generalized Eigenvectors

• Rank of Wopt is limited– Rank(SB) <= |C|-1– Rank(SW) <= N-C

miwSwS iWiiB ,,1 !== l

07-Nov-1757


PCAvs.LDA

• Eigenfaces exploitthemaxscatterofthetrainingimagesinfacespace

• Fisherfaces attempttomaximisethebetweenclassscatter,whileminimisingthewithinclassscatter.

07-Nov-1758


Basicintuition:PCAvs.LDA


Results:Eigenface vs.Fisherface

• Variation in Facial Expression, Eyewear, and Lighting

• Input: 160imagesof16people• Train: 159images• Test: 1image

With glasses

Without glasses

3 Lighting conditions

5 expressions

07-Nov-1760


Eigenface vs.Fisherface

07-Nov-1761


Whatwehavelearnedtoday• Introductiontofacerecognition• TheEigenfaces Algorithm• LinearDiscriminantAnalysis(LDA)

07-Nov-1762



Lecture: Face Recognitionvision.stanford.edu/teaching/cs131_fall1718/files/13_LDA... ·...

Documents

Transcript of Lecture: Face Recognitionvision.stanford.edu/teaching/cs131_fall1718/files/13_LDA... ·...