Misconceptions UStudents’ misconceptions about basic chemical ideas
Common Factors versus Components: Principals and Principles, Errors and Misconceptions Keith F....
-
Upload
jocelyn-martinez -
Category
Documents
-
view
236 -
download
0
Transcript of Common Factors versus Components: Principals and Principles, Errors and Misconceptions Keith F....
Common Factors versus Components:Common Factors versus Components:Principals and Principles,Principals and Principles,
Errors and MisconceptionsErrors and Misconceptions
Keith F. WidamanKeith F. Widaman
University of California at DavisUniversity of California at Davis
Presented at conference “Factor Analysis at 100”
L. L. Thurstone Psychometric Lab, University of North Carolina at Chapel Hill, May 2004
2
Goal of the Talk
• Flip rendition– (With apologies to Will) I come not to praise principal
components, but to bury them
– Thus, we might inter the procedure beside its creator
• More serious– To outline several key assumptions, usually implicit,
of the “simpler” principal components approach
– Compare and contrast common factor analysis and principal component analysis
3
Organization of the Talk
• Principals– Major figures/events
– Important dimensions – factors/components
• Principles– To organize our thinking
– Lead to methods to evaluate procedures
• Errors– Structures of residuals
– Unclear presentations
• Misconceptions
4
Principal Individuals & Contributions
• Spearman (1904)– First conceptualization of the nature of a common
factor – the element in common to two or more indicators (preferably three or more)
– Stressed presence of two classes of factors –
• general (with one member) and
• specific (with a potentially infinite number)
– Key: Based evaluation of empirical evidence on the tetrad difference criterion (i.e., on patterns in correlations among manifest variables) with no consideration of diagonal
5
Principal Individuals & Contributions
• Thomson (1916)– Early recognition of elusiveness of theory – data
connection
– Single common factor implies hierarchical pattern of correlations, but so does an opposite conceptualization
– Key for this talk: Focus was still on the patterns displayed by off-diagonal correlation. Diagonal elements were of no interest or importance
6
Principal Individuals & Contributions
• Thurstone (1931)– First foray into factor analysis
– Devised a “center of gravity” method for estimation of loadings
– Led to centroid method
– Key: Again, diagonal values explicitly disregarded
7
Principal Individuals & Contributions
• Hotelling (1933)– Proposed method of principal components
– Method of estimation
• Least squares
• Decomposition of all of the variance of the manifest variables into dimensions that are:
(a) orthogonal
(b) conditionally variance maximized
– Key 1: Left unities on diagonal
– Key 2: Interpreted unrotated solution
8
Principal Individuals & Contributions
• Thurstone (1935) – The Vectors of Mind– “It is a fundamental criterion for a valid method of
isolating primary abilities that the weights of the primary abilities for a test must remain invariant when it is moved from one test battery to another test battery.”
– “If this criterion is not fulfilled, the psychological description of a test will evidently be as variable as the arbitrarily chosen batteries into which the test may be placed. Under such conditions no stable identification of primary mental abilities can be expected.”
9
Principal Individuals & Contributions
• Thurstone (1935)– This implies invariant factorial description of a test
(a) across batteries and (b) across populations
– Again, diagonal values explicitly disregarded
– Developed rationale for necessity for rotation
– Contra Hotelling:
• Unities on diagonal – imply manifest variables are perfectly reliably
• Need for # dimensions = # manifest variables
• No rotation! This appears, to me, to be the most important criticism of Hotelling by Thurstone.
10
Principal Individuals & Contributions
• McCloy, Metheny, & Knott (1938)– Published in Psychometrika
– Sought to compare Common FA (Thurstone’s method) vs. Principal Components Analysis (Hotelling’s method)
– Perhaps the first comparison of the two methods
11
Principal Individuals & Contributions
• Thomson (1939)– Clear statement of the differing aims of
• Common factor analysis – to explain the off-diagonal correlations among manifest variables
• Principal component analysis – to re-represent the manifest variables in a mathematically efficient manner
12
Principal Individuals & Contributions
• Guttman (1955, 1958)– Developed lower bounds for the number of factors
– Weakest lower bound was number of “factors” with eigenvalues greater than or equal to unity
• With unities on diagonal
• With population data
– Other bounds used other diagonal elements (e.g., strongest lower bound used SMCs), but these did not work as well
13
Principal Individuals & Contributions
• Kaiser (1960, 1971)– Described the origin of the Little Jiffy
• Principal components
• Retain components with eigenvalues >= 1.0
• Rotate using varimax
– Later modifications – Little Jiffy Mark IV – offered important improvements, but were not followed
14
Principles – Mislaid or Forgotten
• Principle 1: Common factor analysis and principal component analysis have different goals – à la Thomson (1939)– Common factor analysis – to explain the off-diagonal
correlations among manifest variables
– Principal component analysis – to re-represent the original variables in a mathematically efficient manner
• (a) in reduced dimensionality, or
• (b) using orthogonal, conditionally variance maximized way
15
Principles – Mislaid or Forgotten
• Principle 2: Common factor analysis was as much a theory of manifest variables as a theory of latent variables– Spearman – doctrine of the indifference of the
indicator, so any manifest variable was a more-or-less good indicator of g
– Thurstone – test one’s theory by developing new variables as differing mixtures of factors and then attempt to verify presumptions
– Today, focus seems largely on the latent variables
– Forgetting about manifest variables can be problematic
16
Principles – Mislaid or Forgotten
• Principle 3: Invariance of the psychological/ mathematical description of manifest variables is a fundamental issue– “It is a fundamental criterion for a valid method of
isolating primary abilities that the weights of the primary abilities for a test must remain invariant when it is moved from one test battery to another test battery”
– Much work on measurement & factorial invariance
– But, only similarities between common factors and principal components are stressed; differences are not emphasized
17
Principles – Mislaid or Forgotten
• Principle 4: Know data and model– Should know relation between data and model
– Should know all assumptions (even implicit) of model
– Frequently told:
• information in correlation matrix is difficult to discern
• so, don’t look at data
• run it through FA or PCA
• interpret the results
– This is not justifiable!
18
Common FA & Principal CA Models
• Common Factor Analysis– R = FF’ + U2 = PΦP’ + U2
– where
• R is (p x p) correlation matrix among manifest vars
• F is a (p x k) unrotated factor matrix, with loadings of p manifest variables on k factors
• U2 is a (p x p) matrix (diagonal) of unique factor variances
• P is a (p x k) rotated factor matrix, with loadings of p manifest variables on k rotated factors
• Φ is a (k x k) matrix of covariances among factors (may be I, usually diag = I)
19
Common FA & Principal CA Models
• Principal Component Analysis– R = FcFc’ = PcΦcPc’
– R = FcFc’ + GG’ = PcΦcPc’ + GG’
– R = FcFc’ + Δ = PcΦcPc’ + Δ
– where
• Fc, Pc, & Φc have same order as like-named matrices for CFA, but with c subscript to denote PCA
• G is a (p x [p-k]) matrix of loadings of p manifest variables on the (p-k) discarded components
• Δ (= GG’) is a (p x p) matrix of covariances among residuals
20
Present Day: Advice to Practicing Scientist
• Velicer & Jackson (1990): CFA vs. PCA– Four “practical” issues
• Similarity between solutions
• Issues related to # of dimensions to retain
• Improper solutions in CFA
• Differences in computational efficiency
– Three “theoretical” issues
• Factorial indeterminacy in CFA, not PCA
• CFA can be used in exploratory and confirmatory modes, PCA only exploratory
• CFA is latent procedure, PCA is manifest
21
Present Day: Advice to Practicing Scientist
• Goldberg & Digman (1994) and Goldberg & Velicer (in press): CFA vs. PCA– Results from CFA and PCA are so similar that
differences are unimportant
– If differences are large, “data are not well-structured enough for either type of analysis”
– Use “factor” to refer to factors and components
– Aim is to explain correlations among manifest vars
22
Present Day: Quantitative Approaches
• Recent paper in Psychometrika (Ogasawara, 2003)– Based work on oblique factors & components with:
• Equal number of indicators per dimension
• Independent cluster solution
• Sphericity (equal “error” variances), hence equal factor loadings
– Derived expression for SEs (standard errors) for factor and component loadings and intercorrelations
– SEs for PCA estimates were smaller than those for CFA estimates, implying greater stability of (i.e., lower variability around) population estimates
23
An Apocryphal Example
• Researcher wanted to develop a new inventory to assess three cognitive traits
• Knew to collect data in at least two initial, derivation samples
• Use exploratory procedures to verify initial, a priori hypotheses
• Then, move on to confirmatory techniques
• So, Sample 1, N = 1600, and 8 manifest variables
• 3 Components explain 51% of total variance
24
Oblique Components, Sample 1
Variable Fac 1 Fac 2 Fac 3 . h2
.
V1 .704 .002 –.005 .496
V2 .704 .002 –.005 .496
V3 .704 .002 –.005 .496
N1 .105 .715 .065 .575
N2 –.002 .725 .014 .538
N3 –.116 .670 –.089 .417
S1 –.005 .002 .735 .540
S2 –.005 .002 .735 .540
Fac 1 1.0
Fac 2 .256 1.0
Fac 3 .147 .147 1.0
25
Orthogonal Components, Sample 1
Variable Fac 1 Fac 2 Fac 3 . h2
.
V1 .698 .079 .044 .496
V2 .698 .079 .044 .496
V3 .698 .079 .044 .496
N1 .211 .717 .127 .575
N2 .104 .716 .070 .538
N3 –.025 .643 –.045 .417
S1 .050 .046 .732 .540
S2 .050 .046 .732 .540
Fac 1 1.0
Fac 2 .000 1.0
Fac 3 .000 .000 1.0
26
An Apocryphal Example
• After confirming a priori hypotheses in Sample 1, the researcher collected data from Sample 2
– Same manifest variables
– Sampled from the same general population
– Same mathematical approach – principal components followed by oblique and orthogonal rotation
– Got same results!
• Decided to “test” the theory in Sample 3 – using “replicate and extend” approach
– Major change: Switch to Confirmatory Factor Analysis
27
Confirmatory Factor Analysis, Sample 3
Variable Fac 1 Fac 2 Fac 3 . θ2
.
V1 2.50 (.18) .0 .0 18.75
V2 3.00 (.21) .0 .0 27.00
V3 3.50 (.25) .0 .0 36.75
N1 .0 2.10 (.13) .0 4.59
N2 .0 2.00 (.14) .0 12.00
N3 .0 1.50 (.16) .0 22.75
S1 .0 .0 2.40 (.44) 58.24
S2 .0 .0 2.70 (.50) 73.71
Fac 1 1.0
Fac 2 .50 (.04) 1.0
Fac 3 .50 (.10) .50 (.10) 1.0
28
Fully Standardized Solution, Sample 3
Variable Fac 1 Fac 2 Fac 3 . h2
.
V1 .50 .0 .0 .25
V2 .50 .0 .0 .25
V3 .50 .0 .0 .25
N1 .0 .70 .0 .49
N2 .0 .50 .0 .25
N3 .0 .30 .0 .09
S1 .0 .0 .30 .09
S2 .0 .0 .30 .09
Fac 1 1.0
Fac 2 .50 1.0
Fac 3 .50 .50 1.0
29
Oblique Component Solution, Sample 3
Variable Fac 1 Fac 2 Fac 3 . h2
.
V1 .704 .002 –.005 .496
V2 .704 .002 –.005 .496
V3 .704 .002 –.005 .496
N1 .105 .715 .065 .575
N2 –.002 .725 .014 .538
N3 –.116 .670 –.089 .417
S1 –.005 .002 .735 .540
S2 –.005 .002 .735 .540
Fac 1 1.0
Fac 2 .256 1.0
Fac 3 .147 .147 1.0
30
An Early Comparison
• McCloy, Metheny, & Knott (1938)– Published in Psychometrika
– Sought to compare Common FA (Thurstone’s method) vs. Principal Components Analysis (Hotelling’s)
– Stated that Principal Components can be rotated
– So, both techniques are different means to same end
– Principal difference:
• Thurstone inserts largest correlation in row in the diagonal of each residual matrix
• Hotelling begins with unities and stays with residual values in each residual matrix
31
Hypothetical Factor Matrix (McCloy et al.)
Variable Fac 1 Fac 2 Fac 3 . h2 .
1 .900 .0 .0 .810
2 .800 .0 .0 .640
3 .0 .700 .0 .490
4 .0 .800 .0 .640
5 .0 .0 .900 .810
6 .0 .0 .600 .360
7 .0 .424 .424 .360
8 .566 .566 .0 .640
9 .495 .0 .495 .490
10 .520 .520 .520 .810
32
Rotated Factor Matrix (McCloy et al.)
Variable Fac 1 Fac 2 Fac 3 . h2 .
1 .860 –.033 .035 .742
2 .819 –.025 .023 .672
3 .014 .726 .000 .527
4 .023 .766 .004 .587
5 .010 –.004 .808 .653
6 –.008 .029 .645 .417
7 –.011 .434 .466 .406
8 .587 .548 –.014 .645
9 .516 –.038 .512 .530
10 .489 .471 .537 .749
33
Rotated Component Matrix (McCloy et al.)
Variable Fac 1 Fac 2 Fac 3 . h2 .
1 .906 –.055 .063 .828
2 .874 –.053 .046 .769
3 .034 .824 –.021 .681
4 .050 .859 –.006 .740
5 –.060 .000 .885 .787
6 –.094 –.035 .773 .608
7 –.054 .519 .525 .548
8 .653 .558 .029 .739
9 .527 –.085 .605 .651
10 .527 .477 .552 .810
34
An Early Comparison
• McCloy, Metheny, & Knott (1938)– Argued that:
• both CFA and PCA were means to same end
• both led to similar pattern of loadings, but
• Thurstone’s method was more accurate (Δh2 = .056) than Hotelling’s (Δh2 = .125) – [but these were average absolute differences]
• I averaged signed differences, and Thurstone’s method was much accurate (Δh2 = -.013) than Hotelling’s (Δh2 = .120)
35
An Early Comparison
• McCloy, Metheny, & Knott (1938)– Found similar pattern of high and low loadings from
PCA and CFA
– But, they found (but did not stress) that PCA led to decidedly higher loadings
• Tukey (1969)– Amount, as well as direction, is vital
– For any science to advance, we must pay attention to quantitative variation, not just qualitative
36
Regularity Conditions or Phenomena
• Relations between population values of P and R• Features of eigenvalues• Covariances among residuals
– Need a “theory” of “errors”
– Recount my first exposure …
– Should have to acknowledge (predict? live with?) the patterns in residuals
37
Practicing Scientists vs. “Statisticians”
• Interesting dimension along which researchers fall:
Practicing “Statisticians”
scientists (Dark side)
use CFA prefer PCA
use regression warn of probs
analysis errors in vars
38
Practicing Scientists vs. “Statisticians”
• At first seems odd– Practicing scientist prefers
• CFA (which partials out errors of measurement and specific variance)
• Regression analysis – despite the implicit assumption of “perfect” measurement
– “Statistician” prefers
• To warn of ill-effects of errors in variables on results of regression analysis
• PCA (despite lack of attention to measurement error), perhaps due to elegant, reduced rank representation
39
Practicing Scientists vs. “Statisticians”
• On second thought, is rational:– Practicing scientist prefers
• Assumptions that residuals (in CFA or regression analysis) are independent, uncorrelated, normally distributed
– “Statistician” prefers
• To try to circumvent (or solve) problem of errors in variables in regression
• To relegate “errors in variables” problems in PCA to that part of solution (GG’) that is orthogonal to the retained part, thereby circumventing (or solving) this problem
40
Regularity Conditions or Phenomena
• In Common Factor Analysis,– Char. of correlations Char. of variables 1:1
– Char. of correlations Char. of variables 1:1
• In Principal Component Analysis,– Char. of correlations Char. of variables 1:1 (??)
– Char. of correlations ~[] Char. of variables many:1
41
Manifest Correlations
Var V1 V2 V3 V4 V5 V6
V1 1.00
V2 .64 1.00
V3 .64 .64 1.00
V4 .64 .64 .64 1.00
V5 .64 .64 .64 .64 1.00
V6 .64 .64 .64 .64 .64 1.00
42
Eigenvalues, Loadings, and Explained Variance
Var EV P1 P2 h2 EVc Pc1 Pc2 hc2
V1 1.92 .80 .64 2.28 .87 .76
V2 .0.80 .64 .36 .87 .76
V3 .0.80 .64 .36 .87 .76
V4
V5
V6
P1 1.0 1.0
P2
43
Residual Covariances: CFA
Var V1 V2 V3 V4 V5 V6
V1 .36 .00 .00
V2 .00 .36 .00
V3 .00 .00 .36
V4
V5
V6
Covs below diag., corrs above diag.
44
Residual Covariances: PCA
Var V1 V2 V3 V4 V5 V6
V1 .24 -.50 -.50
V2 -.12 .24 -.50
V3 -.12 -.12 .24
V4
V5
V6
Covs below diag., corrs above diag.
45
Eigenvalues, Loadings, and Explained Variance
Var EV P1 P2 h2 EVc Pc1 Pc2 hc2
V1 3.84 .80 .64 4.20 .84 .70
V2 .0.80 .64 .36 .84 .70
V3 .0.80 .64 .36 .84 .70
V4 .0 .80 .64 .36 .84 .70
V5 .0 .80 .64 .36 .84 .70
V6 .0 .80 .64 .36 .84 .70
P1 1.0 1.0
P2
46
Residual Covariances: CFA
Var V1 V2 V3 V4 V5 V6
V1 .36 .00 .00 .00 .00 .00
V2 .00 .36 .00 .00 .00 .00
V3 .00 .00 .36 .00 .00 .00
V4 .00 .00 .00 .36 .00 .00
V5 .00 .00 .00 .00 .36 .00
V6 .00 .00 .00 .00 .00 .36
Covs below diag., corrs above diag.
47
Residual Covariances: PCA
Var V1 V2 V3 V4 V5 V6
V1 .30 -.20 -.20 -.20 -.20 -.20
V2 -.06 .30 -.20 -.20 -.20 -.20
V3 -.06 -.06 .30 -.20 -.20 -.20
V4 -.06 -.06 -.06 .30 -.20 -.20
V5 -.06 -.06 -.06 -.06 .30 -.20
V6 -.06 -.06 -.06 -.06 -.06 .30
Covs below diag., corrs above diag.
48
Regularity Conditions or Phenomena
• In Common Factor Analysis,– If (a) the model fits in the population, (b) there is one
factor, and (c) communalities are estimated optimally,
– Single non-zero eigenvalue
– Factor loadings and residual variances for first three variables are unaffected by addition of 3 “identical” variables
– Residuals = specific + error variance
– Residual matrix is diagonal
49
Regularity Conditions or Phenomena
• In Principal Component Analysis,– If (a) the common factor model fits in the population,
(b) there is one factor, and (c) unities are retained on the main diagonal,
– Single large eigenvalue, plus (p – 1) identical, smaller eigenvalues
– Residual component matrix G is independent of the space defined by Fc
– But, residual covariance matrix is clearly non-diagonal
– And, (a) “population” component loadings and (b) residual variances and covariances vary as a function of number of manifest variables!
50
Manifest Correlations
Var V1 V2 V3 V4 V5 V6
V1 1.00
V2 .36 1.00
V3 .36 .36 1.00
V4 .36 .36 .36 1.00
V5 .36 .36 .36 .36 1.00
V6 .36 .36 .36 .36 .36 1.00
51
Eigenvalues, Loadings, and Explained Variance
Var EV P1 P2 h2 EVc Pc1 Pc2 hc2
V1 1.08 .60 .36 1.72 .76 .57
V2 .0.60 .36 .64 .76 .57
V3 .0.60 .36 .64 .76 .57
V4
V5
V6
P1 1.0 1.0
P2
52
Residual Covariances: CFA
Var V1 V2 V3 V4 V5 V6
V1 .64 .00 .00
V2 .00 .64 .00
V3 .00 .00 .64
V4
V5
V6
Covs below diag., corrs above diag.
53
Residual Covariances: PCA
Var V1 V2 V3 V4 V5 V6
V1 .43 -.50 -.50
V2 -.21 .43 -.50
V3 -.21 -.21 .43
V4
V5
V6
Covs below diag., corrs above diag.
54
Eigenvalues, Loadings, and Explained Variance
Var EV P1 P2 h2 EVc Pc1 Pc2 hc2
V1 2.16 .60 .36 4.80 .68 .47
V2 .0.60 .36 .64 .68 .47
V3 .0.60 .36 .64 .68 .47
V4 .0 .60 .36 .64 .68 .47
V5 .0 .60 .36 .64 .68 .47
V6 .0 .60 .36 .64 .68 .47
P1 1.0 1.0
P2
55
Residual Covariances: CFA
Var V1 V2 V3 V4 V5 V6
V1 .64 .00 .00 .00 .00 .00
V2 .00 .64 .00 .00 .00 .00
V3 .00 .00 .64 .00 .00 .00
V4 .00 .00 .00 .64 .00 .00
V5 .00 .00 .00 .00 .64 .00
V6 .00 .00 .00 .00 .00 .64
Covs below diag., corrs above diag.
56
Residual Covariances: PCA
Var V1 V2 V3 V4 V5 V6
V1 .53 -.20 -.20 -.20 -.20 -.20
V2 -.11 .53 -.20 -.20 -.20 -.20
V3 -.11 -.11 .53 -.20 -.20 -.20
V4 -.11 -.11 -.11 .53 -.20 -.20
V5 -.11 -.11 -.11 -.11 .53 -.20
V6 -.11 -.11 -.11 -.11 -.11 .53
Covs below diag., corrs above diag.
57
Regularity Conditions or Phenomena
• So, the difference between “population” parameters from CFA and PCA diverge more:
– (a) the fewer the number of indicators per dimension, and
– (b) the lower the true communality
• But, some regularities still seem to hold (although these vary with the number of indicators)
– “regular” estimates of loadings
– “regular” magnitude of residual covariance
– “regular” magnitude of residual covariance
– “regular” form of eigenvalue structure
58
Regularity Conditions or Phenomena
• But, what if we have variation in loadings?
59
Manifest Correlations
Var V1 V2 V3 V4 V5 V6
V1 1.00
V2 .64 1.00
V3 .64 .64 1.00
V4 .48 .48 .48 1.00
V5 .48 .48 .48 .36 1.00
V6 .48 .48 .48 .36 .36 1.00
60
Eigenvalues, Loadings, and Explained Variance
Var EV P1 P2 h2 EVc Pc1 Pc2 hc2
V1 3.00 .80 .64 3.47 .83 .69
V2 .0.80 .64 .64 .83 .69
V3 .0.80 .64 .64 .83 .69
V4 .0 .60 .36 .53 .68 .47
V5 .0 .60 .36 .36 .68 .47
V6 .0 .60 .36 .36 .68 .47
P1 1.0 1.0
P2
61
Residual Covariances: CFA
Var V1 V2 V3 V4 V5 V6
V1 .36 .00 .00 .00 .00 .00
V2 .00 .36 .00 .00 .00 .00
V3 .00 .00 .36 .00 .00 .00
V4 .00 .00 .00 .64 .00 .00
V5 .00 .00 .00 .00 .64 .00
V6 .00 .00 .00 .00 .00 .64
Covs below diag., corrs above diag.
62
Residual Covariances: PCA
Var V1 V2 V3 V4 V5 V6
V1 .31 -.15 -.15 -.21 -.21 -.21
V2 -.05 .31 -.15 -.21 -.21 -.21
V3 -.05 -.05 .31 -.21 -.21 -.21
V4 -.09 -.09 -.09 .53 -.20 -.20
V5 -.09 -.09 -.09 -.11 .53 -.20
V6 -.09 -.09 -.09 -.11 -.11 .53
Covs below diag., corrs above diag.
63
Regularity Conditions or Phenomena
• So, with variation in loadings
• One piece of approximate stability
– “regular” estimates of loadings
• But, sacrifice
– “regular” magnitude of residual covariance
– “regular” magnitude of residual covariance
– “regular” form of eigenvalue structure
64
Regularity Conditions or Phenomena
• But, what if we have multiple factors?• Let’s start with
– (a) equal loadings
– (b) orthogonal factors
65
Eigenvalues, Loadings, and Explained Variance
Var EV P1 P2 h2 EVc Pc1 Pc2 hc2
V1 1.08 .60 .0 .64 1.72 .76 .0 .57
V2 1.08 .60 .0 .64 1.72 .76 .0 .57
V3 .0.60 .0 .64 .64 .76 .0 .57
V4 .0 .0 .60 .64 .64 .0 .76 .57
V5 .0 .0 .60 .64 .64 .0 .76 .57
V6 .0 .0 .60 .64 .64 .0 .76 .57
P1 1.0 1.0
P2 .0 1.0 .0 1.0
66
Residual Covariances: PCA
Var V1 V2 V3 V4 V5 V6
V1 .43 -.50 -.50 .00 .00 .00
V2 -.21 .43 -.50 .00 .00 .00
V3 -.21 -.21 .43 .00 .00 .00
V4 .00 .00 .00 .43 -.50 -.50
V5 .00 .00 .00 -.21 .43 -.50
V6 .00 .00 .00 -.21 -.21 .43
Covs below diag., corrs above diag.
67
Regularity Conditions or Phenomena
• So, “strange” result:– Same factor inflation as with 1-factor, 3 indicators
– Same within-factor residual covariances as for 1-factor, 3 indicators
– But, between-factor residual covariances = 0!
• Let’s go to – (a) equal loadings, but
– (b) oblique factors
68
Eigenvalues, Loadings, and Explained Variance
Var EV P1 P2 h2 EVc Pc1 Pc2 hc2
V1 1.62 .60 .0 .64 2.26 .76 .0 .57
V2 .54 .60 .0 .64 1.18 .76 .0 .57
V3 .0.60 .0 .64 .64 .76 .0 .57
V4 .0 .0 .60 .64 .64 .0 .76 .57
V5 .0 .0 .60 .64 .64 .0 .76 .57
V6 .0 .0 .60 .64 .64 .0 .76 .57
P1 1.0 1.0
P2 .5 1.0 .31 1.0
69
Residual Covariances: PCA
Var V1 V2 V3 V4 V5 V6
V1 .43 -.50 -.50 .00 .00 .00
V2 -.21 .43 -.50 .00 .00 .00
V3 -.21 -.21 .43 .00 .00 .00
V4 .00 .00 .00 .43 -.50 -.50
V5 .00 .00 .00 -.21 .43 -.50
V6 .00 .00 .00 -.21 -.21 .43
Covs below diag., corrs above diag.
70
Regularity Conditions or Phenomena
• So, “strange” result:– Same factor inflation as with 1-factor, 3 indicators
– Reduced correlation between factors
– But, residual covariances matrix is identical!
• Let’s go to – (a) unequal loadings, and
– (b) orthogonal factors
71
Eigenvalues, Loadings, and Explained Variance
Var EV P1 P2 h2 EVc Pc1 Pc2 hc2
V1 1.16 .80 .0 .36 1.70 .83 .0 .68
V2 1.16 .60 .0 .64 1.70 .78 .0 .61
V3 .0.40 .0 .84 .79 .64 .0 .41
V4 .0 .0 .80 .36 .79 .0 .83 .68
V5 .0 .0 .60 .64 .51 .0 .78 .61
V6 .0 .0 .40 .84 .51 .0 .64 .41
P1 1.0 1.0
P2 .0 1.0 .0 1.0
72
Residual Covariances: PCA
Var V1 V2 V3 V4 V5 V6
V1 .32 -.47 -.48 .00 .00 .00
V2 -.16 .39 -.55 .00 .00 .00
V3 -.21 -.26 .59 .00 .00 .00
V4 .00 .00 .00 .32 -.47 -.48
V5 .00 .00 .00 -.16 .39 -.55
V6 .00 .00 .00 -.21 -.26 .59
Covs below diag., corrs above diag.
73
Regularity Conditions or Phenomena
• So, “strange” result:– Different factor inflation than with 1-factor, 3
indicators
– Reduced correlation between factors
– But, residual covariances matrix has unequal covariances and correlations among residuals, but between-factor covariances = 0!
• Let’s go to – (a) unequal loadings, and
– (b) oblique factors
74
Eigenvalues, Loadings, and Explained Variance
Var EV P1 P2 h2 EVc Pc1 Pc2 hc2
V1 1.74 .80 .0 .36 2.27 .77 .11 .66
V2 .58 .60 .0 .64 1.16 .77 .00 .59
V3 .0.40 .0 .84 .79 .71 -.12 .46
V4 .0 .0 .80 .36 .77 .11 .77 .66
V5 .0 .0 .60 .64 .52 .00 .77 .59
V6 .0 .0 .40 .84 .49 -.12 .71 .46
P1 1.0 1.0
P2 .5 1.0 .32 1.0
75
Residual Covariances: PCA
Var V1 V2 V3 V4 V5 V6
V1 .34 -.38 -.49 -.13 -.11 .01
V2 -.14 .41 -.59 -.11 -.04 .07
V3 -.21 -.28 .54 .01 .07 .16
V4 -.04 -.04 .00 .34 -.38 -.49
V5 -.04 -.02 .03 -.14 .41 -.59
V6 .00 .03 .08 -.21 -.28 .54
Covs below diag., corrs above diag.
76
Regularity Conditions or Phenomena
• So, “strange” result:– Extremely different factor inflation than with 1-factor,
3 indicators
– Largest loading is now UNderrepresented
– Very different population factor loadings (.8, .6, & .4) have very similar component loadings
– Now, between-factor covariances are not zero, and some are positive!
77
R from Component Parameters
• All the preceding from a CFA view:– Develop parameters from a CF model
– Analyze using CFA and PCA
– CFA procedures recover parameters
– PCA procedures exhibit failings or anomalies
– So What? What else could you expect?
• Challenge (to me): – Generate data from a PC model
– Analyze using CFA and PCA
– PCA should recover parameters, CFA should exhibit problems and/or anomalies
78
R from Component Parameters
• Difficult to do• Leads to
– Impractical, unacceptable outcomes, from the point of view of the practicing scientist
– Crucial indeterminacies with the PCA model
79
R from Component Parameters
• Impractical, unacceptable outcomes, from the point of view of the practicing scientist
80
Manifest Correlations
Var V1 V2 V3 V4 V5 V6
V1 1.00
V2 .46 1.00
V3 .46 .46 1.00
V4
V5
V6
First principal component has 3 loadings of .8
First principal factor has 3 loadings of (.46)1/2, or about .67
81
Manifest Correlations
Var V1 V2 V3 V4 V5 V6
V1 1.00
V2 .568 1.00
V3 .568 .568 1.00
V4 .568 .568 .568 1.00
V5 .568 .568 .568 .568 1.00
V6 .568 .568 .568 .568 .568 1.00
First principal component has 6 loadings of .8
First principal factor has 6 loadings of (.568)1/2, or about .75
But, one would have to alter the first 3 tests, as their “population” correlations are altered
82
R from Component Parameters
• Crucial indeterminacies with the PCA “model”– Consider case of well-identified CFA model: 6
manifest variables loading on a single factor
– One could easily construct the population matrix as FF’ + uniquenesses to ensure diag(R) = I
– With 6 manifest variables, 6(7)/2 = 21 unique elements of covariance matrix
– 12 parameter estimates
– therefore 9 df
83
R from Component Parameters
• Crucial indeterminacies with the PCA “model”– Consider now 6 manifest variables with defined
loadings on first PC
– To estimate the correlation matrix, must come up with the remaining 5 PCs
– A start: [Fc | G]’ [Fc | G] = diag, so orthogonality constraint yields 6(5)/2 = 15 equations
– Sum of squares across rows = 1, so 6 more equations
– In short, 15 equations, but 30 unknowns (loadings of 6 variables on the 5 components in G)
– Therefore, an infinite # of R matrices will lead to the stated first PC
84
R from Component Parameters
• Crucial indeterminacies with the PCA “model”– Related to the Ledermann number, but in reverse
– For example, with 10 manifest variables, one can minimally overdetermine no more than 6 factors (so use 6 or fewer factors)
– But, here, one must specify at least 6 components (to ensure more equations than unknowns) to ensure a unique R
– If fewer than 6 components are specified, an infinite number of solutions for R can be found
85
Conclusions: CFA
• CFA factor models may not hold in the population
• But, if they do (in a theoretical population):
– The notion of a population factor loading is realistic
– The population factor loading is unaffected by presence of other variables, as long as the battery contains the same factors
– In one-factor case, loadings can vary from 0 to 1 (provided reflection of variables is possible)
– This generalizes to the case of multiple factors
86
Conclusions: CFA
• CFA factor models may not hold in the population
• But, if they do:
– Residual (i.e., unique) variances are uncorrelated
– Magnitude of unique variance for a given variable is unaffected by other variables in the analysis
87
Conclusions: PCA
• PCA factor models cannot hold in the population (because all variables have measurement error)
• Moreover:
– The notion of “the population component loading” for a particular manifest variable is meaningless
– The “population” component loading is affected strongly by presence of other variables
– SEs for component loadings have no interpretation
– In the one-component case, component loadings can only vary from (1/m)1/2 to 1, where m is the number of indicators for the dimension
– Generalizes to multiple component case
88
Conclusions: PCA
• PCA factor models cannot hold in the population (because all variables have measurement error)
• Moreover:
– Residual variables are correlated, often in unpredictable and seemingly haphazard fashion
– Magnitude of unique variance and covariances for a given manifest variable are affected by other variables in the analysis
89
Conclusions: PCA
• PCA factor models cannot hold in the population (because all variables have measurement error)
• Moreover:
– Finally, generating data from a PC model leads either to
• Impractical, unacceptable outcomes
• Indeterminacies in the parameter – R relations
90
91