Confirmatory Factor Analysis Part Two
Transcript of Confirmatory Factor Analysis Part Two
ConfirmatoryFactorAnalysisPartTwo
STA2101:Fall/Winter2019
THETRUTH(Well,closertothetruth,anyway)
Regression-likemodels
• Realityismassivelynon-linear.• Wecanlivewithalinearapproximation,asinmultipleregression.
• Allthemodelequationshaveunknownslopesandunknownintercepts.
• LikeD1=λ0,1+λ1F1+e1• Latentvariableshaveunknownexpectedvaluesandvariances.
• Callthisthe“originalmodel.”
Identifiability
• Iftherearelatentvariables,theparametersoftheoriginalmodelarenotidentifiable.
• Inaway,whenwedropinterceptsandignoreexpectedvalues,it’slikeweareassumingallexpectedvalues=0,“centering”themodel.
• OriginalModelCenteredModel
Centeringisare-parameterization
• Notone-to-one.• Itreducesthedimensionoftheparameterspace,helpingwithidentifiability.
• Doesnotaffectslopes,variancesorcovariances.
• Meaningisunaffected.
• WhataboutVar(Fj)=1?
Whyshouldthevarianceofthefactorsequalone?
• Inheritedfromexploratoryfactoranalysis,whichwasmostlyadisaster.
• Thestandardanswerissomethinglikethis:“Becauseit’sarbitrary.Thevariancedependsuponthescaleonwhichthevariableismeasured,butwecan’tseeittomeasureitdirectly.Sosetittooneforconvenience.”
• Butsayingitdoesnotmakeitso.IfFisarandomvariablewithanunknownvariance,then
• Var(F)=ϕisanunknownparameter.
CenteredModel
D1 = �1F + e1
D2 = �2F + e2
D3 = �3F + e3
D4 = �4F + e4
e1, . . . , e4, F all independentV ar(ej) = !j V ar(F ) = ��1,�2,�3 6= 0
CovarianceMatrix
� =
�
⇧⇧⇤
�21⇥ + ⇤1 �1�2⇥ �1�3⇥ �1�4⇥
�1�2⇥ �22⇥ + ⇤2 �2�3⇥ �2�4⇥
�1�3⇥ �2�3⇥ �23⇥ + ⇤3 �3�4⇥
�1�4⇥ �2�4⇥ �3�4⇥ �24⇥ + ⇤4
⇥
⌃⌃⌅
PassestheCountingRuletestwith10equationsin9unknowns
Butforanyc≠0�1 ⇥ �1 �2 �3 �4 ⇤1 ⇤2 ⇤3 ⇤4
�2 ⇥/c2 c�1 c�2 c�3 c�4 ⇤1 ⇤2 ⇤3 ⇤4
Bothyield
� =
�
⇧⇧⇤
�21⇥ + ⇤1 �1�2⇥ �1�3⇥ �1�4⇥
�1�2⇥ �22⇥ + ⇤2 �2�3⇥ �2�4⇥
�1�3⇥ �2�3⇥ �23⇥ + ⇤3 �3�4⇥
�1�4⇥ �2�4⇥ �3�4⇥ �24⇥ + ⇤4
⇥
⌃⌃⌅
The choice � = 1 just sets c =�
�: convenient but seemingly arbitrary.
Youshouldbeconcerned!
• Foranysetoftrueparametervalues,thereareinfinitelymanyuntruesetsofparametervaluesthatyieldexactlythesameSigmaandhenceexactlythesameprobabilitydistributionoftheobservabledata.
• Thereisnowaytoknowthefulltruthbasedonthedata,nomatterhowlargethesamplesize.
• Butthereisawaytoknowthepartialtruth.
Certainfunctionsoftheparametervectorareidentifiable
At points in the parameter space where �1,�2,�3 6= 0,
• �12�13�23
= �1�2��1�3��2�3�
= �21�
• And so if �1 > 0, the function �j�1/2 is identifiablefor j = 1, . . . , 4.
• �11 � �12�13�23
= !1, and so !j is identifiable for j = 1, . . . , 4.
• �13�23
= �1�3��2�3�
= �1�2, so ratios of factor loadings
are identifiable.
Reliability• Reliabilityisthesquaredcorrelationbetweentheobservedscoreandthetruescore.
• Theproportionofvarianceintheobservedscorethatisnoterror.
• ForD1=λ1F+e1it’s
⇢2 =
✓Cov(D1, F )
SD(D1)SD(F )
◆2
=
�1�p
�21�+ !1
p�
!2
=�21�
�21�+ !1
⇢2 =�21�
�21�+ !1
� =
�
⇧⇧⇤
�21⇥ + ⇤1 �1�2⇥ �1�3⇥ �1�4⇥
�1�2⇥ �22⇥ + ⇤2 �2�3⇥ �2�4⇥
�1�3⇥ �2�3⇥ �23⇥ + ⇤3 �3�4⇥
�1�4⇥ �2�4⇥ �3�4⇥ �24⇥ + ⇤4
⇥
⌃⌃⌅
�12�13
�23�11=
�1�2��1�3�
�2�3�(�21�+ !1)
=�21�
�21�+ !1
= ⇢2
Soreliabilitiesareidentifiabletoo.
Whatcanwesuccessfullyestimate?
• Errorvariancesareknowable.• Factorloadingsandvarianceofthefactorarenotknowableseparately.
• Butbothareknowableuptomultiplicationbyanon-zeroconstant,sosignsoffactorloadingsareknowable(ifonesignisknown).
• Relativemagnitudes(ratios)offactorloadingsareknowable.
• Reliabilitiesareknowable.
TestingModelFit• Notethatalltheequalityconstraintsmustinvolveonlythecovariances:σijfori≠j.
• Intheoriginalmodel,thecovariancesareallmultipliedbythesamenon-zeroconstant.
• So,theequalityconstraintsoftheoriginalmodelandthepretendmodelwithϕ=1arethesame.
• Thechi-squaretestforgoodnessoffitappliestotheoriginalmodel.Thisisagreatrelief!
• Likelihoodratiotestscomparingfullandreducedmodelsaremostlyvalidwithoutdeepthought.– Equalityoffactorloadingsistestable.– CouldtestH0:λ4=0,etc.
Re-parameterization• Thechoiceϕ=1isaverysmartre-parameterization.
• Itre-expressesthefactorloadingsasmultiplesofthesquarerootofϕ.
• Itpreserveswhatinformationisaccessibleabouttheparametersoftheoriginalmodel.
• Muchbetterthanexploratoryfactoranalysis,whichlosteventhesignsofthefactorloadings.
• Thisisthesecondmajorre-parameterization.Thefirstwaslosingthethemeansandintercepts.
Re-parameterizations
OriginalmodelSurrogatemodel1Surrogatemodel2
D1 D6D5D4D3D2
F1 F2
Addafactortothecenteredmodel
Addafactortothecenteredmodel
D1 = �1F1 + e1
D2 = �2F1 + e2
D3 = �3F1 + e3
D4 = �4F2 + e4
D4 = �5F2 + e5
D6 = �6F2 + e6
cov
✓F1
F2
◆=
✓�11 �12
�12 �22
◆
e1, . . . , e6 independent of each other and of F1, F2
�1, . . .�6 6= 0V ar(ej) = !j
� =
�
⇧⇧⇧⇧⇧⇧⇤
�21⇥11 + ⇤1 �1�2⇥11 �1�3⇥11 �1�4⇥12 �1�5⇥12 �1�6⇥12
�1�2⇥11 �22⇥11 + ⇤2 �2�3⇥11 �2�4⇥12 �2�5⇥12 �2�6⇥12
�1�3⇥11 �2�3⇥11 �23⇥11 + ⇤3 �3�4⇥12 �3�5⇥12 �3�6⇥12
�1�4⇥12 �2�4⇥12 �3�4⇥12 �24⇥22 + ⇤4 �4�5⇥22 �4�6⇥22
�1�5⇥12 �2�5⇥12 �3�5⇥12 �4�5⇥22 �25⇥22 + ⇤5 �5�6⇥22
�1�6⇥12 �2�6⇥12 �3�6⇥12 �4�6⇥22 �5�6⇥22 �26⇥22 + ⇤6
⇥
⌃⌃⌃⌃⌃⌃⌅
�1 = (�1, . . . ,�6,⇥11,⇥12,⇥22,⇤1, . . . ,⇤6)�2 = (��
1, . . . ,��6,⇥
�11,⇥
�12,⇥
�22,⇤
�1, . . . ,⇤
�6)
��j = �j for j = 1, . . . , 6
Wherec1≠0andc2≠0
��1 = c1�1 ��
2 = c1�2 ��3 = c1�3 ⇥�
11 = ⇥11/c21
��4 = c2�4 ��
5 = c2�5 ��6 = c2�6 ⇥�
22 = ⇥22/c22
⇥�12 = �12
c1c2
Parametersarenotidentifiable
Variancesandcovariancesoffactors
• Areknowableonlyuptomultiplicationbypositiveconstants.
• SincetheparametersofthelatentvariablemodelwillberecoveredfromΦ=cov(F),theyalsowillbeknowableonlyuptomultiplicationbypositiveconstants–atbest.
• Luckily,inmostapplicationstheinterestisintesting(pos-neg-zero)morethanestimation.
Cov(F1,F2)isun-knowable,but• Easytotellifit’szero• Signisknownifonefactorloadingfromeachsetisknown–saylambda1>0,lambda4>0
• And,
• Thecorrelationbetweenfactorsisidentifiable!
⇥14��12�13
�23
��45�46
�56
=�1�4⇤12
�1�
⇤11�4�
⇤22
=⇤12�
⇤11�
⇤22
= Corr(F1, F2)
Thecorrelationbetweenfactorsisidentifiable
• Furthermore,itisthesamefunctionofSigmathatyieldsϕ12underthesurrogatemodelwithVar(F1)=Var(F2)=1.
• Therefore,Corr(F1,F2)=ϕ12underthesurrogatemodelisequivalenttoCorr(F1,F2)undertheoriginalmodel.
• Estimatesandtestsofϕ12underthesurrogatemodelapplytoundertheoriginalmodel.
�12��11�
�22
Settingvariancesoffactorstoone
• Isaverysmartre-parameterization.• Isexcellentwhentheinterestisincorrelationsbetweenfactors.
• Allowsestimationofclassicalpathcoefficientsforthelatentvariablemodel.
• (Thatlastremarkwasjustfortherecord.)
Re-parameterizationasachangeofvariables
• Var(Fj’)=1• ThenewfactorloadingisinunitsofthestandarddeviationofFj.
• ThisappliestoallobservablevariablesconnectedtoFj.
• Putsfactorloadingsfordifferentfactorsonacommonscale.
Dj = �jFj + ej
= (�j
p�jj)
1p�jj
Fj
!+ ej
= �0jF
0j + ej
Covariances
• Covariancesbetweenfactorsinthesurrogatemodelequalcorrelationsintheoriginalmodel.
• Latentvariableparametersarestronglyaffected.• Parametersinthelatentsurrogatemodelaretheoriginalparameterstimespositiveconstants.
Cov(F 0j , F
0k) = E
1p�jj
Fj1p�kk
Fk
!
=E(FjFk)p�jj
p�kk
=�jkp
�jjp�kk
= Corr(Fj , Fk)
Whathappensifthereisalatentvariablemodel?
Yi = �1Xi + ✏i
V ar(Yi) = �21�+
StandardizebothXandY.
p�21�+
✓1p
�21�+
Yi
◆= �1
p�
⇣1p�Xi
⌘+ ✏i
)✓
1p�21�+
Yi
◆=
⇣�1
q�
�21�+
⌘ ⇣1p�Xi
⌘+ ✏ip
�21�+
Y 0i = �0
1 X 0i + ✏0i
Whatdoesitmean?
Y 0i = �0
1X0i + ✏0i
�01 = �1
s�
�21�+
Cov(X 0i, Y
0i ) = �0
1 = Corr(Xi, Yi)
Becausecovariancesunderthesurrogatemodelequalcorrelationsundertheoriginalmodel.
FactorLoadingsareaffectedtoo
Di = �Yi + · · ·+ ei
=
✓�q�21�+
◆ 1p
�21�+
Yi
!+ · · ·+ ei
= �0Y 0i + · · ·+ ei
Cascadingeffects
• Understandthere-parameterizationasachangeofvariables
• Notjustanarbitraryrestrictionoftheparameterspace.
• Itshowstherearewidespreadeffectsthroughoutthemodel.
• Alsoshowshowthemeaningsofothermodelparametersareaffected.
Theotherstandardtrick
• Settingvariancesofallthefactorstooneisanexcellentre-parameterizationindisguise.
• Theotherstandardtrickistosetonefactorloadingequaltooneforeachfactor.
• D=F+eishardtobelieveifyoutakeitliterally.
• It’sactuallyare-parameterization.• Everymodelyou’veseenwithafactorloadingofoneisasurrogatemodel.
Backtoasingle-factormodelwithλ1>0
D1 = �1F + e1
D2 = �2F + e2
D3 = �3F + e3...
Dj =
✓�j
�1
◆(�1F ) + ej
= �0jF
0 + ej
D1 = F 0 + e1
D2 = �02F
0 + e2
D3 = �03F
0 + e3...
� =
�
⇤⇥ + ⇤1 �2⇥ �3⇥
�2⇥ �22⇥ + ⇤2 �2�3⇥
�3⇥ �2�3⇥ �23⇥ + ⇤3
⇥
⌅
Value under model
Function of ⌃ Surrogate Original
�23�13
�2�2�1
�23�12
�3�3�1
�12�13�23
� �21�
Σunderthesurrogatemodel
Underthesurrogatemodel
• It looks like �j is identifiable, but actually it’s �j/�1.
• Estimates of �j for j �= 1 are actually estimates of �j/�1.
• It looks like ⇥ is identifiable, but actually it’s �21⇥.
• ⇥ is being expressed as a multiple of �21.
• Estimates of ⇥ are actually estimates of �21⇥.
Everythingisbeingexpressedintermsofλ1.
MakeD1theclearestrepresentativeofthefactor.
Addanobservablevariable
• Parametersareallidentifiable,evenifthefactorloadingofthenewvariableequalszero.
• EqualityrestrictionsonSigmaarecreated,becauseweareaddingmoreequationsthanunknowns.
• Theseequalityrestrictionsapplytotheoriginalmodel.
• Itisstraightforwardtoseewhattherestrictionsare,thoughthecalculationscanbetimeconsuming.
Findingtheequalityrestrictions
• CalculateΣ(θ).• Solvethecovariancestructureequationsexplicitly,obtainingθasafunctionofΣ.
• SubstitutethesolutionsbackintoΣ(θ).• Simplify.
Example:Adda4thvariable
D1 = F + e1
D2 = �2F + e2
D3 = �3F + e3
D4 = �4F + e4
e1, . . . , e4, F all independentV ar(ej) = !j V ar(F ) = ��1,�2,�3 6= 0
�(�) =
�
⇧⇧⇤
⇥ + ⇤1 �2⇥ �3⇥ �4⇥�2⇥ �2
2⇥ + ⇤2 �2�3⇥ �2�4⇥�3⇥ �2�3⇥ �2
3⇥ + ⇤3 �3�4⇥�4⇥ �2�4⇥ �3�4⇥ �2
4⇥ + ⇤4
⇥
⌃⌃⌅
Solutions
�2 = �23�13
�3 = �23�12
�4 = �24�12
⇥ = �12�13�23
Substitute
⇥12 = �2⇤
=⇥23
⇥13
⇥12⇥13
⇥23= ⇥12
Substitutesolutionsintoexpressionsforthecovariances
�12 = �12
�13 = �13
�14 =�24�13
�23�23 = �23
�24 = �24
�34 =�24�13
�12
EqualityConstraints
�14�23 = �24�13
�12�34 = �24�13
Theseholdregardlessofwhetherfactorloadingsarezero(1234).
�12�34 = �13�24 = �14�23
Addanother3-variablefactor• Identifiabilityismaintained.• Thecovarianceϕ12=σ14• Actuallyσ14=λ1λ4ϕ12undertheoriginalmodel.
• Thecovariancesofthesurrogatemodelarejustthoseofthesurrogatemodel,multipliedbyun-knowablepositiveconstants.
• Asmorevariablesandmorefactorsareadded,allthisremainstrue.
Comparingthesurrogatemodels• Eithersetvariancesoffactorstoone,orsetoneloadingperfactortoone.
• Botharisefromasimilarchangeofvariables.• Fj’=cjFj,wherecj>0.• cjiseitherafactorloadingoroneoverastandarddeviation.
• Interpretationofsurrogatemodelparametersisdifferentexceptforthesign.
• Mathematicallythemodelsareequivalent:
• Thetruemodelandbothsurrogatemodelssharethesameequalityconstraints,andhencethesamegoodnessoffitresultsforanygivendataset.
Exchange �j and 1p�jj
.
Whichre-parameterizationisbetter?• Technically,theyareequivalent.• Theybothinvolvesettingasingleun-knowableparametertoone,
foreachfactor.• Thisseemsarbitrary,butactuallyitresultsinaverygoodre-
parameterizationthatpreserveswhatisknowableaboutthetruemodel.
• Standardizingthefactors(Surrogatemodel2A)ismoreconvenientforestimatingcorrelationsbetweenfactors.
• Settingoneloadingperfactorequaltoone(Surrogatemodel2B)ismoreconvenientforestimatingtherelativesizesoffactorloadings.
• HandcalculationswithSurrogatemodel2Bcanbeeasier.• Ifthereisaseriouslatentvariablemodel,Surrogatemodel2Bis
mucheasiertospecifywithSAS.• MixingSurrogatemodel2Bwithdoublemeasurementisnatural.• Don’tdobothrestrictionsforthesamefactor!
Whyarewedoingthis?• Theparametersoftheoriginalmodelcannotbeestimateddirectly.Forexample,maximumlikelihoodwillfailbecausethemaximumisnotunique.
• Theparametersofthesurrogatemodelsareidentifiable(estimable)functionsoftheparametersofthetruemodel.
• Theyhavethesamesigns(positive,negativeorzero)ofthecorrespondingparametersofthetruemodel.
• Hypothesistestsmeanwhatyouthinktheydo.• Parameterestimatescanbeusefulifyouknowwhatthenewparametersmean.
TheCrossoverRule• Itisunfortunatethatvariablescanonlybecausedbyonefactor.Infact,it’sunbelievablemostofthetime.
• Apatternlikethiswouldbenicer.
Whenyouaddasetofobservablevariablestoameasurementmodelwhoseparametersare
alreadyidentifiable
• Straightarrowswithfactorloadingsonthemmaypointfromeachexistingfactortoeachnewvariable.
• Youdon’tneedtoincludeallsucharrows.• Errortermsforthenewsetofvariablesmayhavenon-zerocovarianceswitheachother,butnotwiththeerrorvariancesorfactorsoftheoriginalmodel.
• Someofthenewerrortermsmayhavezerocovariancewitheachother.It’suptoyou.
• Allparametersofthenewmodelareidentifiable.
Proof• Haveameasurement(factoranalysis)modelwithpfactorsandk1observablevariables.Theparametersareallidentifiable.
• Assumethatforeachfactor,thereisatleastoneobservablevariablewithafactorloadingofone.
• Ifthisisnotthecase,re-parameterize.• Re-orderthevariables,puttingthepvariableswithunitfactorloadingsfirst,intheorderofthecorrespondingfactors.
Thefirsttwoequationsbelongtotheinitialmodel
D1 = F + e1
D2 = �2F + e2
D3 = F + e3�3
�33
cov
0
@e1e3e3
1
A =
0
@⌦11 ⌦12 0
⌦22 0
1
A
cov(F) = �
�33
⌃ =
0
@⌃11 ⌃12 ⌃13
⌃22 ⌃23
⌃33
1
A
=
0
@� + ⌦11 �⇤>
2 �⇤2�⇤>
2 +⌦22 ⇤2�⇤>3
⇤3�⇤>3 +
1
A⇤>
3
⇤3 = ⌃>13�
�1
⌦33 = ⌃33 �⇤3�⇤>3
Solveforitanditbecomesblack
Comments• Therearenorestrictiononthefactorloadingsofthevariablesthatarebeingaddedtothemodel
• Therearenorestrictiononthecovariancesoferrortermsforthenewsetofvariables,exceptthattheymustnotbecorrelatedwitherrortermsalreadyinthemodel.
• Thissuggestsamodelbuildingstrategy.Startsmall,perhapswith3variablesperfactor.Thenaddtheremainingvariables–maximumflexibility.
• Couldevenfittheone-factorsub-modelsoneatatimetomakesuretheyareokay,thencombinefactors,thenaddvariables.
Addanobservedvariabletothefactors• Oftenit’sanobservedexogenousvariable(likesexor
experimentalcondition)youwanttobeinalatentvariablemodel.
• Supposeparametersoftheexisting(surrogate)factoranalysismodel(pfactors)areallidentifiable.
• Xisindependentoftheerrorterms.
• Addarow(andcolumn)toΣ.• Addp+1parameterstothemodel.• SayVar(X)=Φ0,Cov(X,Fj)=Φ0,j• Dk=λkFj+ek,λkisalreadyidentified.• E(XDk)=λkE(XFj)+0=λkΦ0,j
• Solveforthecovariance.• Dothisforeachfactorinthemodel.Done.
Wehavesomeidentificationrules
• DoubleMeasurementrule.• Three-variableruleforstandardizedfactors.• Three-variableruleforunstandardizedfactors.• Cross-overrule.• Error-freerule.
CopyrightInformation
ThisslideshowwaspreparedbyJerryBrunner,Departmentof
Statistics,UniversityofToronto.ItislicensedunderaCreative
CommonsAttribution-ShareAlike3.0UnportedLicense.Use
anypartofitasyoulikeandsharetheresultfreely.These
Powerpointslidesareavailablefromthecoursewebsite:
http://www.utstat.toronto.edu/~brunner/oldclass/2101f19