Comparing three effect sizes for latent class analysis./67531/metadc822835/... · Bayesian...

COMPARING THREE EFFECT SIZES FOR LATENT CLASS ANALYSIS

Elvalicia A. Granado, B.S., M.S.

Dissertation Prepared for the Degree of

DOCTOR OF PHILOSOPHY

UNIVERSITY OF NORTH TEXAS

December 2015

APPROVED:

Prathiba Natesan, Major Professor

Robin K. Henson, Committee Member

Bertina H. Combes, Committee Member

Tao Zhang, Committee Member

Abbas Tashakkori, Chair of the Department of

Educational Psychology

Mark Wardell, Dean of the Toulouse Graduate

School

Granado, Elvalicia A. Comparing three effect sizes for latent class analysis. Doctor of

Philosophy (Educational Research-Research, Measurement and Statistics), December 2015, 52

pp., 7 tables, 5 figures, references, 48 titles.

Traditional latent class analysis (LCA) considers entropy R2 as the only measure of effect

size. However, entropy may not always be reliable, a low boundary is not agreed upon, and good

separation is limited to values of greater than .80. As applications of LCA grow in popularity, it

is imperative to use additional sources to quantify LCA classification accuracy. Greater

classification accuracy helps to ensure that the profile of the latent classes reflect the profile of

the true underlying subgroups. This Monte Carlo study compared the quantification of

classification accuracy and confidence intervals of three effect sizes, entropy R2, I-index, and

Cohen’s d. Study conditions included total sample size, number of dichotomous indicators, latent

class membership probabilities (γ), conditional item-response probabilities (ρ), variance ratio,

sample size ratio, and distribution types for a 2-class model. Overall, entropy R2 and I-index

showed the best accuracy and standard error, along with the smallest confidence interval widths.

Results showed that I-index only performed well for a few cases.

ii

Copyright 2015

by

Elvalicia A. Granado

iii

TABLE OF CONTENTS

Page

LIST OF TABLES ...........................................................................................................................v

LIST OF FIGURES ....................................................................................................................... vi

COMPARING THREE EFFECT SIZES FOR LATENT CLASS ANALYSIS .............................1

Introduction ..........................................................................................................................1

Latent Class Analysis ...........................................................................................................1

Fit Indices.................................................................................................................3

Effect sizes and Their Confidence Intervals ............................................................5

Entropy .........................................................................................................5

Improvement-Over-Chance index, I ............................................................6

Cohen’s d .....................................................................................................8

Confidence Intervals ....................................................................................9

Problem Statement ...............................................................................................................9

Significance of Study .........................................................................................................10

Purpose of Study ................................................................................................................13

Methods..............................................................................................................................14

LCA Parameters and I-index Parameters ..............................................................14

Data Generation and Analysis ..............................................................................18

Results ................................................................................................................................23

Percentage of Variance Explained .........................................................................23

Bias .......................................................................................................................24

Standard Error .......................................................................................................29

iv

Confidence Intervals ..............................................................................................30

Discussion ..........................................................................................................................33

Limitation and Future Directions ...........................................................................36

APPENDIX: SAS SYNTAX FOR DATA SIMULATION ..........................................................38

REFERENCES ..............................................................................................................................48

v

LIST OF TABLES

Page

1. Large, Medium, and Small ρ Distinction Conditions with 5 items and N=100 .......................17

2. Base Empirically Derived Effect Size Values for Normal and Nonnormal Populations, γ = .9,

.1; .8,.2 .....................................................................................................................................20


.3; .6,.4 .....................................................................................................................................21


.5...............................................................................................................................................22

5. Percentage of Variance Explained (η2) of Bias, Standard Error, RMSE, CI Widths, and CI

Coverage Rates for Study Features ..........................................................................................24

6. Means for Bias (B), Precision (P), RMSE of Effect Sizes Across All Conditions for Normal

and Nonnormal Distributions ...................................................................................................26

7. Average Confidence Interval Widths for Each Factor: Nonormal and Nonnormal

Distributions ............................................................................................................................32

vi

LIST OF FIGURES

Page

1. General diagram of latent class analysis model .........................................................................2

2. Normal Distribution: Factors by Bias for All Effect Sizes ......................................................27

3. Normal Distribution: Factors by RMSE for All Effect Sizes ..................................................28

4. Normal Distribution: Factors by Standard Error for All Effect Sizes .....................................30

5. Normal Distribution: Factors by Average CI Width for All Effect Sizes................................33

1

COMPARING THREE EFFECT SIZES FOR LATENT CLASS ANALYSIS

Introduction

The application of latent class analysis (LCA) has become more widespread. The goal of

LCA is to identify heterogeneous groups, from a sample of cases, and classify them into

homogeneous subcategories, called latent classes. Two main aspects of LCA include

probabilities of latent class membership and probabilities of responses to indicators given class

membership (Cleveland, Collins, Lanza, Greenberg, & Feinberg, 2010; McCutcheon, 1987).

LCA applications span a variety of fields, ranging from sociology to business, with the result of

customizing programs and interventions targeted to specific subgroups. Some instances include

finding diagnostic classes in the clinical field (Hadzi-Pavlovic, 2009; Rindskopf & Rindskopf,

1986), subcategories of adolescent alcohol users (Lanza, Collins, Lemmon, & Schafer, 2007),

digital piracy (Higgins, Wolfe, & Ricketts, 2008), online learners (Barnard-Brak, Lan, & Paton,

2010), consumers (Green, Carmone, & Wachspress, 1976), and stocks (Angelis, 2013).

Latent class analysis

Lazarsfeld and Henry (1968) first developed LCA methodology for sociological

investigations of latent variables derived from dichotomous survey indicators. LCA helps to

identify subgroups of a categorical latent variable, in which the rules used to identify groups do

not follow a gold standard or rule of thumb. Groups identified have patterns that are unveiled in

responses to certain categorical indicators (Muthén, 2004). For example, a researcher may ask if

there are different subcategories of online learners that may be identified with responses to a ten-

2

item questionnaire. In this example, the focus of the analysis is the pattern of responses to the

questionnaire which identifies underlying subgroups.

LCA identifies latent classes from a distribution of responses to locally independent

categorical indicators. Latent classes exhibit conditional independence, where indicator variables

are locally independent within a latent class (McCutcheon, 1987). Conditional independence

assumes that correlation between indictors is due to the latent class variable, therefore there is no

residual covariance amongst indicator variables. A general diagram of the LCA model consists

of measured indicator variables (u), and a latent variable (c) (Figure 1). The latent class variable

has K underlying categories.

Figure 1. General diagram of latent class analysis model

LCA model parameter estimates include: 1. latent class prevalence (γ), in which the

probabilities are the proportion of the population in the latent classes; 2. conditional item

probabilities (ρ), which are probabilities of a response to items of the categorical manifest

indicator, given a class membership (Cleveland et al., 2010).

The LCA model assumes that underlying latent classes exhibit conditional independence,

which make indicator variables within each latent class locally independent (McCutcheon,

1987). Let C represent the latent variable consisting of c = 1,…,K; and J represents the observed

indicator variables, where j = 1,…,J. The indicator variable j has response categories rj = 1,…,Rj.

u1 u2 ur . . .

c

3

An array of possible response patterns is Y and a response pattern (vector of responses to the J

indicator) is y. An indicator function is written as I(yj = rj), which equals 1 at j=rj

and otherwise equals 0. The following mathematical latent class model aligns with detailed

specifications described by Lanza et al. (2007). The LCA model is defined as

(1)

where γc represents probability of latent class membership and the vector of all γ parameters sum

to 1. The probability of response rj to indicator variable j, conditional on latent class membership

is

. All ρ parameters form a matrix of probabilities of item-responses given class

membership. Parameter probabilities are calculated using maximum likelihood with the EM

algorithm to produce a conditional contingency table that has expected membership values.

Contingency tables contain values produced by cross-tabulating latent class variable indicators

(Collins & Lanza, 2010; Goodman, 1974). The total number of response patterns in an LCA is

2J. For example, three indicators, that is, 2

3 yields eight response patterns. This translates to

needing more individuals in the study to observe at least one individual with each response

pattern (Lanza, Dziak, Huang, Wagner, & Collins, 2014). Individuals are assigned to a latent

class using maximum-probability assignment, where basis of assignment is on the highest

posterior probability of membership. After classification into latent classes, labels are assigned to

describe the meaningful relationships within the latent class.

Fit Indices

Model selection in LCA is based on model fit. Model fit is determined by comparing the

expected frequencies the model predicts to the cross-classification frequencies. Common

4

parsimony indices applied to LCA include Akaike Information Criterion (AIC; Akaike, 1974),

Bayesian Information Criterion (BIC; Schwartz, 1978), and adjusted BIC (Sclove, 1987). Three

widely applied goodness-of-fit indices in LCA compare fit of the K-class and K-1-class models.

These information criteria (IC) are a function of the number of parameters, sample size, and log-

likelihood of the model. Because each IC weights sample size and model parameters differently,

each IC may determine a different model to retain. With all fit indices, model selection is

determined by the lowest value from competing models (Hagenaars & McCutcheon, 2002;

Magidson & Vermunt, 2004; Nylund et al., 2007a).

Magidson and Vermunt (2004) emphasized that in LCA, BIC is the most widely applied

IC for model selection. BIC is expressed as

(2)

Sclove (1987) reduced the penalty of sample size and defined the adjusted BIC as

(3)

AIC is given as

(4)

In all equations log L is the likelihood, p equals the number of model free parameters, and n is

sample size.

Nylund et al. (2007a) conducted a simulation study that showed BIC was the best choice

for identifying number of latent classes. Performance of three likelihood ratio tests, naïve chi-

square (NCS), Lo-Mendell-Rubin (LMR), and bootstrap likelihood ratio test (BLRT) were tested

against the three information criteria. The bootstrap likelihood ratio test was a consistent

indicator of the correct number of classes. Furthermore, adjusted BIC was shown to be a better

indicator than AIC to delineate latent classes in simulation studies of LCA models with

categorical outcomes (Nylund et al., 2007a; Yang, 2006).

5

Previous research exists for fit indices. However, apart from investigations of entropy

(Celeux & Soromenho, 1996; Kaplan & Keller, 2011; Magidson & Vermunt, 2005; ) less

attention has been given to researching classification accuracy for LCA, which is a measure of

effect size. Huberty’s family of effect sizes measure the difference between groups based on

classification accuracy or improvement-over-chance (Huberty, 1994; Huberty & Lowman,

2000). To the best of our knowledge, research on LCA has not investigated the application of I-

index to quantify classification accuracy of LCA. The current research studies the feasibility of

the I-index family as LCA effect sizes and compares these to entropy R2. Increased accuracy in

classification can lead to improvement of decisions about interventions and study directions that

directly impact particular subgroups.

Effect sizes and Their Confidence Intervals

Entropy

To measure the degree of “fuzziness” between latent classes, Ramaswamy, DeSarbo,

Reibstein, and Robinson (1993) developed entropy for LCA. This commonly used relative

entropy R2 measures the degree of separation between latent classes, and therefore speaks to the

precision of assignment of individual cases to the correct latent class. Entropy R2 is calculated

with estimated posterior probabilities, number of classes, and sample size (Magidson &

Vermunt, 2005). This calculation yields probability values between 0 and 1, where low entropy

values indicate less distinction between latent classes. High entropy values indicate more

precision in group membership classification, and therefore a clearer distinction between latent

classes (Celeux & Soromenho, 1996; Kaplan & Keller, 2011). For instance, an entropy value of

.85 is interpreted as 85% of the individual cases being classified accurately in a latent class.

6

As expressed by Dziak, Lanza, and Tan (2014), the Ramaswamy relative entropy applied

to a sample is

(5)

where K is number of classes; N is total sample size ; represents observed values from each

subject; is an indicator for belonging to class c; represents posterior probability of

being in class c for an individual with data using the estimated parameters. The present study

focuses on one K-class that consists of two latent classes.

Improvement-Over-Chance I-index

Effect size can also be measured as group non-overlap. Larger effect sizes translate to

smaller group overlap, which show a greater distinction between groups (Huberty, 1994).

Huberty and Lowman’s (2000) effect size is called improvement-over-chance, I-index and is

based on overlap of outcome score univariate distributions. If the overlap is small, then by

extension, group means differ by a larger degree. Group overlap is based on across-group

membership hit rate, where groups are assigned on the basis of a classification rule (Hess,

Olejnik, & Huberty, 2001). Huberty and Lowman (2000) suggested an I-index magnitude

categorization greater than .30 as high and less than .10 as low for the two-group univariate case.

These ranges were suggested for both homogeneity and heterogeneity conditions. Just as Cohen

(1988) cautioned about Cohen’s d effect size benchmarks, Huberty and Lowman (2000)

recognized that I-index may be context specific.

Indices of group overlap, declared by Huberty and Holmes (1983), are based on the

proportion of experimental units correctly classified, which is termed the hit rate. A correctly

classified unit is a unit that is assigned to the same population from which it was drawn.

7

Posterior probabilities drive the hit rate. Huberty and Lowman (2000) state that the I-index

addresses the amount of group overlap that exceeds the overlap expected by chance alone. I-

index is defined as:

(7)

where He represents the expected across-group chance hit rate and Ho is the observed across-

group hit rate. The chance hit rate, when using the proportional chance criterion, is further

calculated as:

(8)

where N represents total sample size, qg is the estimated or set prior probability for group g, and

ng represents number of units in group g. Observed across-group hit rate, Ho, can be computed

with classification rules of predictive discriminant analysis (PDA) or logistic regression analysis

(LRA). Both PDA and LRA classify units by identifying the unit’s respective group. Two types

of PDA highlighted in previous research are linear-rule PDA and quadratic-rule PDA (Huberty

& Lowman, 2000; Huberty, 1994).

Huberty and Lowman (2000) proposed I-index as another effect size measure for

univariate, multivariate, homogeneous, and heterogeneous situations. Their research compared I

indices to ANOVAs η2 and the squared point biserial correlation (r

2pb). Positive correlations were

found between η2 and r

2pb and I indices, which ranged from .90 to .97. It was suggested that an I

value of .10 or less was a low effect, while .35 or more was a high effect. To estimate I indices

under various conditions, Hess et al. (2001) manipulated variance patterns (1:1, 1:4, 1:8), total

sample sizes (40, 100, 300), population effect sizes (.2, .5, .8), equal and unequal sample sizes,

and normal (skewness=0, kurtosis=0) and non-normal (skewness=1.75, kurtosis=3.75)

distributions. Results showed that accurate I indices were estimated by PDA and LRA for

8

optimal conditions, except at small sample sizes and small separations. Large sample sizes

estimated I-index with greater precision under LRA. An advantage over Cohen’s d, is that I-

index does not assume homogeneity of variance, therefore works well under variance

homogeneity and heterogeneity. Hess et al. (2001) found that I estimates were more accurate at

larger sample sizes, variance homogeneity, larger effect sizes, and both equal and unequal

groups.

Using the same parameters, Natesan and Thompson (2007) studied the performance of I-

index for small sample sizes (10, 20, 30, 40). Results showed that LRA estimated precise I

indices down to sample sizes of 20. Henson et al. (2014) extended these studies and included five

ratios of prior probabilities (20:80, 40:60, 50:50, 60:40, 80:20). Results showed that I indices

were mainly impacted by variance pattern, prior probabilities, and estimation method. Overall,

LRA provided precise I indices. Quadratic PDA was recommended for conditions with variance

heterogeneity and unequal prior probabilities. Previous studies stressed that results were only

generalized to their conditions, which speaks to the need for more research regarding

performance of I-index in different methods and conditions. These studies showed that I-index

estimates were also robust to the assumption violations of normality and variance homogeneity

for small sample sizes.

Cohen’s d

The commonly applied effect size of Cohen’s d is driven by group mean differences. The

difference between distribution means is divided by the pooled standard deviation, which results

in a standardized mean difference (Cohen, 1988). Cohen’s d assumes homogeneity of variance,

9

therefore when the assumption is violated Cohen’s d may not provide a good measure (Wilcox,

1987; see Henson, Natesan, & Axelson, 2015). Cohen’s d is calculated using

(6)

where m1, n1, and s1 are the mean, sample size, and standard deviation of group 1 , and m2, n2,

and s2 are the mean, sample size, and standard deviation of group 2. Latent class analysis

estimates distributions of latent classes. Therefore, Cohen’s d can be used to measure differences

between resulting latent classes. The same average posterior probabilities that are used to

produce entropy (Celeux, & Soromenho, 1996) are used to compute Cohen’s d.

Confidence Intervals

Confidence intervals (CI) for I-index, entropy R2, and Cohen’s d were calculated to yield

an interval width and precision of the effect size estimate. The precision is related to variability

in the population and sampling error. The CIs are in standardized units of the observed data,

therefore are easily interpretable. Calculating CIs also supports meta-analysis, which combines

results from a variety of studies and show stability across studies. The CI for an effect size is

calculated differently than the CI for a mean. A CI for an effect size uses estimated values from

two datasets, therefore uses the noncentrality parameter of the noncentral distribution (Cumming

& Finch, 2001; Fidler & Thompson, 2001; Smithson, 2001).

Problem statement

Even though fit indices measure how well the model fits the data, fit indices do not

measure accuracy of data classification. After LCA classifies cases into latent classes, it

10

investigates the differences between the latent classes. Therefore it is important to estimate how

accurately the data was classified. Using the traditional null hypothesis significance test (NHST)

to compare latent classes, results in a dichotomous choice of reject or fail-to-reject. Therefore

NHST does not specify the magnitude of the difference between latent classes and provides no

meaningful or practical significance. Effect sizes measure the extent to which sample results

diverge from the null hypothesis (Cohen, 1988). An effect size will specify a nonzero value with

a meaningful quantification of the difference between latent classes. If no difference is detected,

the effect size is zero. In LCA, entropy is used to estimate classification accuracy, yet entropy

may not always be reliable and traditionally good separation is limited to entropy greater than .80

(Collins & Lanza, 2010; Muthén, 2004; Ramaswamy et al., 1993). For LCA, measures of effect

size can aid in quantification of the classification accuracy and magnitude of the difference

between latent classes. Entropy has been shown to not always be reliable, which prompts

investigating alternative effect sizes, such as I-index, for LCA. The present study investigates

performance of a variety of effect sizes to quantify classification accuracy of latent classes

produced in LCA.

Significance of study

The present study quantifies classification accuracy of LCA by including effect size

values, specifically the I-index effect size. In LCA research, once subgroups are determined,

typologies are developed after examining characteristics of latent classes that are unveiled.

Adolescent risk behavior (Collins & Lanza, 2010), economics (Angelis, 2013), nicotine

withdrawal (Xian, Scherrer, Madden, Lyons, Tsuang, True, & Eisen, 2005), and weight-loss

(Lanza, Savage, & Birch, 2010) studies are some examples that apply LCA. Researchers also

11

express a shifting interest towards investigating the role of latent class memberships on future

outcomes in topics such as nursing (Roberts & Ward, 2011) and peer victimization (Nylund,

Bellmore, Nishina, & Graham, 2007b). The common thread is that important decisions may be

based on which group an individual is shown to belong to (Nylund, Asparouhov, & Muthén

2007a), therefore it is imperative to determine correct group membership and number of groups.

Dziak et al. (2014) caution that not identifying an important latent class, even though the class

shows low prevalence, may violate the local independence assumption of LCA. When this

violation occurs, the latent class variable does not adequately describe population covariance

between indicators. Harnessing a good statistical estimate to quantify accuracy of classification

helps to evaluate how correctly individuals are being classified. When classification accuracy is

compromised, study results are negatively impacted. This negative impact increases as

misclassification increases. Additionally, as latent class separation becomes less clear,

interpretation and labeling of the heterogeneous groups may be more challenging.

Entropy R2 is a common measure for quantifying the distance between latent classes in

LCA. Therefore entropy is analogous to Cohen’s d, except entropy is based on probabilities

instead of differences between means. As Huberty and Holmes (1983) note, group overlap is

another way of thinking about effect size. Huberty’s I-index, is a potential choice for an effect

size of LCA, because it quantifies classification accuracy. With I-index, a larger distribution

overlap points to distributions with less distinction between groups. As a caution, Wilcox (1987)

notes that violating the assumption of homogeneity of variance produces effect sizes that do not

provide a good measure. This may lead to misinterpretation of results. Huberty and Lowman

(2000) showed that the I-index is a good measure for quantifying group distribution overlap,

even when the assumption of homogeneity is violated. I-index has been shown to perform well

12

under homogeneous and heterogeneous variances, and univariate, and multivariate distribution

conditions (Hess et al., 2001; Henson, Natesan, & Axelson, 2014; Huberty & Lowman, 2000;

Natesan & Thompson, 2007).

When using the maximum chance criterion instead of the proportional chance criterion

sometimes the final value of I is not directly impacted by the number of analysis units (Huberty

& Lowman, 2000). Interpretation of I-index is different because it focuses on the observed and

chance hit rates, while relative entropy only focuses on the observed hit rate. At high values both

indices indicate a clearer distinction between classes. By taking into account the chance hit rate,

I-index examines the degree to which group overlap exceeds overlap expected by chance. By not

considering how much group overlap exceeds overlap expected by chance, relative entropy may

be dampening differences between latent classes. For instance, a relative entropy value of .75 is

interpreted as 75% of the individual cases being classified correctly. To compare, an I-index with

an observed hit rate of .75 and chance hit rate of .70 yields an overall I-index of (.75 - .70) / 1 -

.70) = .167. This shows that about 17% more cases were correctly classified than expected by

chance. The observed hit rate of .75 alone may be interpreted as a medium to high effect. When

I-index considers the chance hit rate of .70, the overall effect is low. If relative entropy took

chance hit rate into account, the classification by chance for two classes would be 50%. In this

instance, interpreting 75% of individuals as correctly classified would be misleading. Applying

the I-index equation, would give a relative entropy value of (.75 - .50) / 1 - .50 = .50. This shows

that about 50% more cases were correctly classified than expected by chance instead of 75% of

cases classified correctly. When relative entropy considers the chance hit rate the overall hit rate

of 75% is no longer a high effect. When estimating more complex models and models with more

13

classes, Collins and Lanza (2010) caution that relative entropy may have a higher chance of

misclassifying units.

Taking these advantages into consideration, I-index may capture a more refined picture

of the magnitude of differences between latent groups. Use of I-index extends use and reporting

of another effect size measure to LCA. I-index and relative entropy are parallel concepts, which

assess the correct classification of units into their respective groups. Whereas entropy is solely

based on estimating the classification certainty rate, I-index takes the classification a step further,

and employs the expected across-group chance hit rate. Therefore, I-index may provide a more

accurate classification index for LCA, which may result in better quantification for latent class

distinction.

Purpose of the study

Using simulation, the present study compares performance of three effect size measures,

entropy R2, I-index, and Cohen’s d as classification accuracy measures for LCA. Confidence

intervals are also used to compare performance of effect sizes. The current study extends

previous research and applies Huberty’s improvement-over-chance index effect size to LCA. The

study examines a single underlying latent variable with dichotomous indicators. The current

research aims to investigate performance of Huberty’s I-index in quantifying classification

accuracy of LCA. This is accomplished by examining the following for two latent classes under

various distributions and sample sizes through simulation:

1. How does the classification accuracy of I-index compare to the classification accuracy of

entropy R2 and Cohen’s d for LCA models?

14

2. How do the confidence intervals of I-index compare to the confidence intervals of

entropy R2 and Cohen’s d for LCA models?

Methods

Study conditions included number of latent classes, total sample size, number of

indicators, class membership probabilities (γ), conditional item-response probabilities (ρ), sample

size ratios, variance ratios, and distribution shape.

LCA Parameters and I-index Parameters

The present study examined a two latent class model. Previous research recommends

sample sizes of at least 500 for optimal performance of LCA (Finch & Bronk, 2011), yet smaller

sample sizes may be more realistic. Wurpts and Geiser (2014) caution that sample sizes less than

70 may pose issues with convergence and accuracy of class assignment. No fewer than five

indicators are recommended, due to the potential of high levels of non-convergence and under-

identified solutions (Wurpts & Geiser ,2014). More indicators help to better define LCA classes

and therefore, improve LCA performance. Additionally, Morgan (2015) reviewed twenty LCA

studies from peer-reviewed education and psychology journals and summarized that sample sizes

approximately ranged from 200 to 2,500 and number of indicators ranged between 3 and 26.

Because the total possible response patterns increases exponentially as number of indicators

increases (Lanza, et al., 2014), the current study used less than 26 indicators. Therefore, five total

samples sizes (n=100, 150, 200, 500, 1000) and four different number of indicators (5, 8, 10, 15)

were used.

15

Boundaries of other LCA parameters were extended and mirrored those examined by

Nylund et al. (2007a) and Dziak et al. (2014). An equal (50%, 50%) and four unequal sets (90%,

10%; 80%, 20%; 70%, 30%, and 60%, 40%) of latent class prevalence (γ) were examined. High

(.9, .7, .6) and low (.1, .3, .4) conditional item-response probabilities (ρ) were considered. Three

levels of separation between conditional item-response probabilities on latent class membership

were considered. One set represented a large distinction between class separation with ρ = .9, .1.

A medium distinction between classes used ρ = .7 and .3. Small distinction between class

separation was indicated with ρ = .6 and .4.

Two class-specific item-response probabilities impacted the two latent classes. The

smaller group was identified with high item-response probabilities across all items. This “high”

class consisted of individuals representing the lowest percentage of the sample size. These

individuals showed the highest chance of endorsement across all items. The larger group was

identified with high item-response probabilities on a subset of items. Individuals making up the

“selective” class represented the highest percentage of the population. The selective class

showed a high chance of endorsement for most items and a low chance of endorsement for the

remaining items. For instance, a model with five indicators may result in two classes. One class

has five items with high item-response probabilities. The second class consists of a more

selective structure. Three items have high item-response probabilities and two items have low

item-response probabilities. The current LCA study yielded 15 hypothetical two-class models.

Example tables for the large, medium, and small conditional item-response probabilities

distinctions (Table 1) with two latent classes, five items, N=100, three ρ’s, and four γ’s. These

tables followed scenarios demonstrated by Dziak et al. (2014).

16

Parameters manipulated in I-index included variance ratios, sample size ratios (equal and

unequal sample sizes), and distribution shapes (normal and non-normal). Specifically, the current

design maintained variance patterns (1:1, 1:4, 1:8), normal (skewness=0, kurtosis=0), and non-

normal (skewness=1.75, kurtosis=3.75) distribution shapes from the Hess et al. (2001) and

Natesan and Thompson (2007) studies. Five total samples sizes (n=100, 150, 200, 500, 1000)

were examined under equal (50%: 50%) and four unequal sets (90%: 10%; 80%: 20%; 70%:

30%, and 60%: 40%) of sample sizes.

17

Table 1

Large, Medium, and Small ρ Distinction Conditions with 5 items and N=100

Model 1 Model 2 Model 3 Model 4 Model 5

Class

distinction

Class

label Selective High Selective High Selective High Selective High Selective High

Large γ 0.9 0.1 0.8 0.2 0.7 0.3 0.6 0.4 0.5 0.5

ρ=.9, .1

Item 1 0.9 0.9 0.9 0.9 0.9 0.9 0.9 0.9 0.9 0.9

Item 2 0.9 0.9 0.9 0.9 0.9 0.9 0.9 0.9 0.9 0.9

Item 3 0.9 0.9 0.9 0.9 0.9 0.9 0.9 0.9 0.9 0.9

Item 4 0.1 0.9 0.1 0.9 0.1 0.9 0.1 0.9 0.1 0.9

Item 5 0.1 0.9 0.1 0.9 0.1 0.9 0.1 0.9 0.1 0.9


Class

distinction

Class


Medium γ 0.9 0.1 0.8 0.2 0.7 0.3 0.6 0.4 0.5 0.5

ρ=.7, .3

Item 1 0.7 0.7 0.7 0.7 0.7 0.7 0.7 0.7 0.7 0.7

Item 2 0.7 0.7 0.7 0.7 0.7 0.7 0.7 0.7 0.7 0.7

Item 3 0.7 0.7 0.7 0.7 0.7 0.7 0.7 0.7 0.7 0.7

Item 4 0.3 0.7 0.3 0.7 0.3 0.7 0.3 0.7 0.3 0.7

Item 5 0.3 0.7 0.3 0.7 0.3 0.7 0.3 0.7 0.3 0.7


Class

distinction

Class


Small γ 0.9 0.1 0.8 0.2 0.7 0.3 0.6 0.4 0.5 0.5

ρ=.6,.4

Item 1 0.6 0.6 0.6 0.6 0.6 0.6 0.6 0.6 0.6 0.6

Item 2 0.6 0.6 0.6 0.6 0.6 0.6 0.6 0.6 0.6 0.6

Item 3 0.6 0.6 0.6 0.6 0.6 0.6 0.6 0.6 0.6 0.6

Item 4 0.4 0.6 0.4 0.6 0.4 0.6 0.4 0.6 0.4 0.6

Item 5 0.4 0.6 0.4 0.6 0.4 0.6 0.4 0.6 0.4 0.6

γ= probability of class membership, ρ = probability of response

18

Data Generation and Analysis

The LCA study included one K-class (2 latent classes) and manipulated seven

parameters: 4 sets of indicators, 5 total sample sizes, 5 latent class prevalence, 3 conditional

item-response probability distinctions, 3 variance ratio patterns, 5 sample size ratios, and 2

distribution shapes. The completely crossed design produced 9000 conditions with 500

replications simulated per condition. Monte Carlo simulations and data analyses were conducted

using SAS version 9.2 software. Generation and analyses of the LCA sample data sets were

executed with SAS code Simulate LCA Dataset and PROC LCA (Lanza, Dziak, Huang, Wagner,

& Collins, 2014). The I-index was calculated with SAS code previously provided by Hess et al.

(2001).

Bias, standard errors, root mean square errors, CI coverage rates, and CI widths were

calculated for each of the three effect sizes to examine the impact of conditions. For each

condition, population values of the three effect sizes were calculated and used as a base for

comparison. Echoing Hess et al. (2001), the current base comparison population distributions

showed a classification rate of 50% with equal sample size ratios of 50:50. Empirically derived

values for the effect sizes for normal and nonnormal population distributions were calculated

(Tables 2-4).

For each condition, the mean of each effect size was calculated across 500 samples.

Average bias (B), which is accuracy of the estimator, was calculated by subtracting Tables 2-4

base values from the mean effect size values. Overestimation of true effect resulted in a positive

bias, whereas underestimation resulted in a negative bias. Average precision (P), measured with

standard error of the estimator, was equal to the standard deviation of the 500 sample effect sizes

in each condition. Average root mean square error (RMSE) produced the difference between the

19

observed and predicted effect sizes across conditions. Factorial ANOVAs were performed to

quantify the variation in mean bias, mean standard error, mean RMSE, CI coverage rates and CI

widths of each of the three effect sizes explained by each condition.

20

Table 2

Base Empirically Derived Effect Size Values for Normal and Nonnormal Populations, γ = 9,.1; .8,.2

ρ .9, .1 (large) .7, .3 (medium) .6, .4 (small)

Ind.

Var

Ratio

I-index Entropy Cohen's d I-index Entropy Cohen's d I-index Entropy Cohen's d

γ Norm Non Norm Non Norm Non Norm Non Norm Non Norm Non Norm Non Norm Non Norm Non

.9, .1 5 1:1 .009 .015 .408 .377 .374 .252 .011 .009 .265 .310 .576 .536 .015 .010 .326 .289 .002 .111

1:4 .011 .012 .477 .565 .482 .366 .007 .012 .274 .226 .542 .187 .008 .009 .306 .249 .318 .559

1:8 .002 .005 .475 .345 .480 .432 .007 .001 .281 .212 .543 .178 .008 .006 .312 .240 .295 .540

8 1:1 .015 .015 .301 .339 .825 .301 .006 .013 .321 .263 .009 .622 .011 .010 .323 .339 .408 .332

1:4 .006 .010 .294 .307 .039 .099 .002 .010 .303 .314 .242 .460 .013 .011 .255 .310 .435 .000

1:8 .004 .008 .290 .295 .039 .101 .001 .008 .298 .320 .239 .432 .050 .013 .415 .344 .124 .214

10 1:1 .010 .010 .333 .507 .045 .549 .013 .018 .235 .196 .037 .202 .008 .011 .228 .272 .045 .099

1:4 .011 .010 .362 .584 .533 .497 .011 .014 .251 .327 .062 .183 .011 .011 .312 .307 .088 .237

1:8 .011 .011 .419 .462 .520 .539 .008 .008 .305 .360 .054 .145 .013 .011 .280 .343 .073 .051

15 1:1 .016 .013 .297 .315 .104 .045 .013 .007 .309 .401 .008 .131 .011 .017 .328 .389 .230 .238

1:4 .010 .013 .373 .326 .298 .114 .009 .006 .331 .372 .515 .197 .004 .013 .403 .377 .247 .265

1:8 .004 .011 .389 .326 .364 .106 .009 .004 .370 .380 .419 .188 .004 .010 .395 .376 .389 .252

.8, .2 5 1:1 .033 .020 .249 .188 .014 .232 .022 .020 .203 .291 .242 .610 .030 .021 .205 .223 .196 .413

1:4 .024 .018 .247 .196 .121 .243 .016 .026 .242 .248 .032 .457 .019 .025 .289 .209 .303 .197

1:8 .016 .003 .242 .190 .120 .226 .016 .011 .245 .221 .032 .441 .015 .021 .279 .198 .290 .179

8 1:1 .025 .034 .244 .224 .092 .137 .025 .025 .335 .214 .281 .028 .022 .022 .297 .270 .124 .125

1:4 .017 .021 .361 .269 .060 .250 .017 .018 .367 .245 .172 .320 .022 .021 .316 .269 .250 .083

1:8 .012 .016 .359 .264 .065 .255 .012 .023 .372 .333 .168 .490 .002 .019 .198 .295 .129 .346

10 1:1 .025 .038 .327 .276 .018 .277 .025 .034 .343 .338 .056 .522 .013 .021 .244 .347 .048 .226

1:4 .025 .038 .321 .325 .130 .231 .025 .025 .235 .395 .251 .047 .022 .022 .360 .360 .212 .018

1:8 .021 .028 .333 .410 .116 .229 .012 .020 .337 .386 .439 .282 .026 .022 .330 .353 .204 .051

15 1:1 .033 .026 .387 .338 .130 .044 .033 .027 .349 .352 .333 .279 .022 .037 .328 .232 .116 .152

1:4 .023 .019 .279 .351 .437 .105 .013 .025 .337 .357 .004 .260 .015 .021 .371 .357 .410 .050

1:8 .016 .008 .412 .418 .339 .088 .008 .016 .384 .416 .402 .309 .012 .015 .399 .344 .350 .050 Note: γ = latent class prevalence, norm = normal population (skewness=0, kurtosis=0), non = nonnormal population, (skewness=1.75, kurtosis=3.75), ρ = conditional item-response probability

distinction, Ind. = Indicators, Var Ratio = Variance Ratio

21

Table 3

Base Empirically Derived Effect Size Values for Normal and Nonnormal Populations, γ = .7,.3; .6,.4


Ind.

Var

Ratio



.7, .3 5 1:1 .056 .048 .314 .280 .198 .531 .037 .025 .225 .168 .009 .715 .042 .030 .235 .309 .240 .822

1:4 .042 .017 .268 .329 .513 .339 .029 .018 .195 .306 .325 .306 .024 .006 .273 .258 .002 .378

1:8 .039 .017 .259 .332 .509 .347 .022 .008 .192 .296 .305 .283 .024 .005 .268 .270 .001 .399

8 1:1 .031 .049 .305 .244 .281 .133 .040 .005 .151 .257 .282 .804 .037 .031 .347 .320 .148 .778

1:4 .012 .032 .268 .306 .304 .095 .006 .003 .434 .251 .279 .786 .038 .031 .354 .282 .465 .158

1:8 .012 .024 .275 .316 .311 .091 .003 .000 .419 .244 .288 .814 .016 .006 .332 .369 .201 .008

10 1:1 .024 .056 .260 .307 .198 .090 .018 .049 .312 .371 .227 .226 .031 .037 .243 .284 .073 .133

1:4 .018 .043 .291 .353 .547 .561 .023 .043 .271 .226 .142 .206 .031 .037 .280 .306 .150 .066

1:8 .016 .042 .277 .334 .533 .553 .012 .024 .327 .395 .610 .519 .038 .032 .384 .337 .372 .729

15 1:1 .081 .075 .243 .244 .110 .394 .031 .018 .288 .313 .215 .016 .031 .043 .329 .290 .149 .010

1:4 .037 .038 .412 .331 .151 .203 .025 .024 .334 .339 .473 .519 .022 .018 .385 .308 .476 .575

1:8 .024 .037 .379 .405 .590 .418 .012 .012 .446 .426 .063 .117 .012 .018 .412 .310 .368 .575

.6, .4 5 1:1 .065 .041 .215 .245 .050 .200 .041 .032 .295 .234 .673 .587 .050 .056 .199 .241 .388 .668

1:4 .030 .040 .189 .210 .612 .481 .041 .031 .204 .260 .116 .646 .024 .048 .282 .213 .251 .098

1:8 .040 .021 .263 .202 1.008 .453 .031 .014 .213 .265 .121 .616 .024 .031 .284 .225 .267 .096

8 1:1 .048 .049 .338 .317 .230 .280 .001 .049 .275 .294 .269 .066 .040 .041 .380 .305 .168 .311

1:4 .038 .032 .346 .303 .467 .029 .024 .033 .327 .339 .269 .565 .049 .048 .352 .283 .611 .216

1:8 .032 .023 .345 .291 .471 .028 .016 .030 .312 .338 .261 .577 .044 .031 .340 .330 .598 .557

10 1:1 .072 .065 .253 .265 .187 .017 .049 .057 .309 .318 .210 .316 .041 .057 .290 .349 .075 .170

1:4 .056 .056 .296 .366 .206 .050 .032 .057 .284 .270 .120 .212 .057 .050 .306 .297 .469 .220

1:8 .024 .055 .309 .349 .402 .047 .031 .032 .346 .341 .410 .691 .049 .073 .323 .353 .052 .554

15 1:1 .065 .031 .308 .268 .248 .069 .040 .046 .408 .337 .300 .079 .023 .058 .312 .328 .211 .130

1:4 .049 .008 .373 .314 .466 .013 .023 .032 .374 .321 .430 .065 .016 .032 .354 .300 .466 .305

1:8 .024 .008 .409 .438 .398 .011 .016 .008 .380 .409 .499 .424 .008 .008 .395 .356 .440 .480 Note: γ = latent class prevalence, norm = normal population (skewness=0, kurtosis=0), non = nonnormal population, (skewness=1.75, kurtosis=3.75), ρ = conditional item-response probability


22

Table 4

Base Empirically Derived Effect Size Values for Normal and Nonnormal Populations, γ = .5,.5


Ind.

Var

Ratio



.5, .5 5 1:1 .051 .050 .403 .508 .413 .899 .048 .040 .492 .431 .130 .348 .052 .040 .457 .392 .238 .271

1:4 .047 .049 .361 .501 .565 .730 .049 .030 .425 .421 .859 .873 .059 .029 .431 .547 .844 2.713

1:8 .199 .005 .362 .342 .565 .412 .040 .005 .427 .341 .898 .457 .039 .005 .427 .333 .860 .412

8 1:1 .060 .050 .251 .310 .350 .132 .048 .190 .542 .342 .165 .328 .050 .187 .586 .342 1.157 .330

1:4 .111 .049 .348 .291 1.180 .468 .070 .031 .427 .545 .869 2.653 .069 .030 .428 .547 .841 2.735

1:8 .048 .050 .356 .315 .090 .422 .050 .005 .427 .334 .903 .427 .050 .005 .428 .331 .871 .429

10 1:1 .060 .051 .198 .238 .034 .080 .030 .048 .511 .522 .148 .489 .050 .050 .260 .284 .040 .618

1:4 .050 .053 .241 .339 .003 .278 .048 .050 .429 .529 .897 .173 .051 .052 .231 .263 .042 .103

1:8 .051 .050 .398 .354 .115 .044 .053 .049 .426 .647 .848 .547 .049 .051 .351 .298 .091 .383

15 1:1 .071 .019 .308 .332 .042 .293 .052 .036 .313 .357 .226 .386 .049 .070 .336 .332 .253 .102

1:4 .020 .010 .347 .344 .292 .271 .041 .010 .361 .306 .216 .100 .040 .019 .361 .306 .376 .362

1:8 .010 .021 .416 .427 .319 .380 .021 .009 .393 .300 .414 .106 .020 .010 .380 .389 .578 .243

Note: γ = latent class prevalence, norm = normal population (skewness=0, kurtosis=0), non = nonnormal population, (skewness=1.75, kurtosis=3.75), ρ = conditional item-response probability


23

Results

Percentage of Variance Explained

The ANOVA η2 values for the planned orthogonal contrasts show variation in mean bias

and mean standard error (Table 5) of the three effect sizes. Interpretation of the η2 values is

similar to 2, which speaks to the percentage of variance accounted for by the factors. Dependent

variables of mean bias and mean standard error are associated with all the combinations of the

conditions (2 distributions shapes x 3 conditional item probabilities x 3 sample size ratios x 3

variance ratios x 4 number of indicators x 5 sample sizes x 5 latent class prevalence).

The majority of variation in bias was explained by effect size method (η2

= 22.85%).

Most of this variation in bias was derived from the orthogonal contrast of entropy to both I-index

and Cohen’s d (η2

= 21.13%). Sample size ratio explained the second largest amount of variation

in bias (η2

= 11.20%). Two-thirds of this variability (η2

= 6.81%) was due to the comparison of

the 60:40 sample size ratio to the larger sample size ratios. This shows that bias tends to break

down between the sample size ratios of 60:40 and 70:30. The results of standard error echoed

those of bias. Effect size method had the greatest percentage of variation (η2

= 52.53%). Sample

size ratio accounted for little variability in standard error (η2

= 3.88%). The rest of the factors

explained less than 5% variation in bias, standard error, CI width, and CI coverage rates.

The type of effect size explained the majority of variation in RMSE (η2 = 33.10%), CI

widths (η2

= 25.56%), and CI coverage rates (η2

= 31.34%). Most of the variation in RMSE

originated from the contrast of entropy to both I-index and Cohen’s d (η2

= 14.89%). Nearly all

of the variation in CI coverage rates was due to the comparison of entropy to I-index. Further

analysis revealed that entropy yielded the greatest amount of high CI coverage rates. Sample size

24

ratio explained the second largest amount of variation in RMSE (η2

= 9.83%) and CI widths (η2

=

14.65%). More than half of the variation in each was explained by the comparison of the 60:40

sample size ratio to the larger sample size ratios.

Table 5

Percentage of Variance Explained (η2)of Bias, Standard Error, RMSE, CI Width, and CI

Coverage Rates for Study Features

Percentages

Main and Interaction

Effects/Orthogonal Contrasts Standard Error Bias RMSE

CI

Width

CI Coverage

Rate

Effect size 52.53% 22.85% 33.10% 25.56% 31.34%

Entropy vs I-indexa 5.23% 11.05% 1.46% 0.58% 30.75%

Entropy vs I-index and Cohen's

d 2.12% 21.13% 14.89% 3.38% 26.90%

N 0.01% 0.07% 0.39% 5.12% 1.25%

Sample size ratio 3.88% 11.20% 9.83% 14.65% 0.40%

80:20 vs 90:10a

0.44% 1.32% 1.28% 2.60% 0.03%

70:30 vs 80:20 or 90:10 1.54% 3.07% 3.05% 3.57% 0.36%

60:40 vs 70:30, 80:20 or 90:10 1.90% 6.81% 5.50% 8.47% 0.00% Note: The factorial ANOVA degrees of freedom error is zero, because there is only one case per cell (i.e., mean bias or mean

standard error). η2 values (excluding orthogonal contrast subpartitions) sum to 100%, within rounding error. η2 values less than

5% are not reported. n = total sample size. a. η2 values for orthogonal contrasts that compare design effects. For a particular main effect, the contrast η2s sum to equal the

omnibus main effect η2.

Bias

In general, bias was relatively moderate for normal (M = .247, SD = .203) and nonnormal

(M = .220, SD = .181) distributions across all conditions (Table 6). A more granular examination

showed a clear ranking of effect sizes. Entropy showed the most accuracy and lowest RMSE,

followed by I-index. Bias was smallest for entropy, (Mnorm = -.007, SDnorm = .033; Mnonnorm = -

.014, SDnonnorm = .080). Both I-index (Mnorm = .294, SDnorm = .002; Mnonnorm = .297, SDnonnorm =

.006) and Cohen’s d resulted in the larger bias (Mnorm = .455, SDnorm = .026; Mnonnorm = .378,

SDnonnorm = .044).

25

Regarding sample size ratios, bias was lowest, slightly underestimated (60:40, 80:20),

and showed little variation for entropy (Figure 2). Entropy showed lowest RMSEs with little

variation across all conditions, followed by I-index (Figure 3). For I-index and Cohen’s d, bias

and RMSE increased as sample size ratio increased. Looking at conditional item-response

probability distributions, bias and RMSE showed little variation for all effect sizes. Entropy

showed the lowest RMSE, lowest and slightly underestimated bias, and little variation for latent

class prevalence and number of indicators.

26

Table 6

Means for Bias (B), Precision (P), and RMSE of Effect Sizes Across All Conditions for Normal and Nonnormal Distributions

Normal Nonnormal

I-index Entropy Cohen's d I-index Entropy Cohen's d

B P RMSE B P RMSE B P RMSE B P RMSE B P RMSE B P RMSE

Overall

.294 .036 .296 -.007 .144 .144 .455 .407 .610 .297 .033 .299 -.014 .157 .158 .378 .383 .538

Sample Size

Ratio

60:40 .100 .025 .103 -.062 .137 .150 .178 .263 .318 .093 .022 .096 -.058 .142 .153 .134 .273 .304

70:30 .233 .032 .235 .013 .107 .108 .400 .346 .529 .222 .030 .224 -.050 .143 .151 .280 .349 .447

80:20 .359 .043 .362 -.022 .149 .151 .533 .462 .705 .365 .035 .367 -.003 .160 .161 .467 .417 .626

90:10 .483 .044 .485 .045 .182 .187 .709 .555 .900 .508 .043 .510 .056 .184 .192 .629 .494 .800

ρ .6, .4 .293 .044 .296 -.009 .148 .149 .424 .375 .566 .296 .038 .298 -.008 .161 .161 .323 .365 .487

.7, .3 .297 .035 .299 -.004 .134 .135 .489 .421 .645 .299 .029 .300 -.015 .148 .149 .340 .389 .517

.9, .1 .292 .030 .293 -.007 .148 .149 .452 .424 .619 .297 .031 .298 -.019 .162 .163 .469 .396 .614

γ .5, .5 .449 .051 .452 .000 .118 .118 .460 .420 .623 .443 .045 .445 -.009 .129 .129 .316 .386 .499

.6, .4 .366 .031 .367 -.008 .148 .148 .377 .403 .552 .368 .025 .369 -.006 .164 .164 .397 .384 .552

.7, .3 .294 .034 .296 -.001 .149 .149 .452 .406 .608 .298 .033 .300 -.015 .162 .163 .280 .371 .465

.8, .2 .226 .035 .229 -.013 .151 .151 .521 .391 .652 .227 .032 .229 -.013 .164 .165 .428 .380 .572

.9, .1 .135 .030 .138 -.012 .153 .154 .464 .412 .621 .151 .027 .153 -.026 .167 .169 .466 .395 .611

Indicators 5 .304 .038 .307 -.014 .108 .109 .515 .449 .683 .266 .027 .267 -.015 .119 .120 .430 .457 .627

8 .292 .041 .295 -.007 .135 .135 .438 .415 .603 .298 .044 .301 .007 .150 .150 .328 .370 .495

10 .278 .034 .280 .018 .160 .161 .491 .392 .629 .312 .041 .315 -.017 .177 .178 .359 .358 .507

15 .301 .031 .303 -.024 .172 .174 .376 .369 .527 .313 .019 .314 -.029 .183 .185 .393 .348 .524

N 100 .281 .046 .285 .044 .162 .168 .376 .375 .531 .288 .041 .291 .034 .179 .182 .301 .346 .459

150 .285 .046 .289 .038 .160 .165 .385 .374 .537 .292 .041 .295 .028 .176 .178 .313 .351 .470

200 .297 .047 .301 .036 .162 .166 .412 .390 .567 .302 .041 .305 .027 .176 .178 .349 .358 .500

500 .299 .022 .300 -.047 .128 .136 .548 .448 .708 .301 .023 .302 -.054 .137 .147 .434 .416 .601

1000 .307 .018 .307 -.105 .108 .150 .553 .446 .711 .303 .018 .303 -.103 .118 .156 .491 .445 .663

Variance

Ratio

1:1 .315 .042 .318 .024 .117 .120 .548 .380 .667 .316 .036 .318 .003 .154 .154 .408 .379 .557

1:4 .289 .036 .291 -.015 .153 .154 .395 .408 .568 .299 .032 .300 -.015 .155 .156 .307 .373 .483

1:8 .278 .030 .280 -.030 .161 .164 .421 .432 .603 .277 .030 .279 -.029 .163 .165 .417 .398 .576

Note: γ = latent class prevalence, ρ = conditional item-response probability distinction, normal population: skewness=0, kurtosis=0, nonnormal population: skewness=1.75,

kurtosis=3.75, RMSE = root mean square error

27

Figure 2. Normal Distribution: Factors by Bias for All Effect Sizes

28

Figure 3. Normal Distribution: Factors by RMSE for All Effect Sizes

Results indicated that RMSE values for entropy and I-index were close as latent class

prevalence, number of indicators, and variance ratio increased. Two conditions, sample size ratio

of 60:40 and the latent class prevalence of .9, .1 showed slightly lower RMSE for I-index than

entropy (Figure 3). For I-index, bias and RMSE decreased as latent class prevalence increased.

Variance ratios showed little variation in bias and RMSE for all effect sizes. Entropy produced the

lowest RMSE and bias was underestimated at higher total sample sizes and larger variance ratios.

29

Standard Error

Overall, standard errors were small for normal (M = .195, SD = .164) and nonnormal (M =

.191, SD = .152) distributions across all conditions (Table 6). I-index showed the lowest standard

errors across all conditions, followed by entropy. Standard errors were small for both I-index

(Mnorm = .036, SDnorm = .008; Mnonnorm = .033, SDnonnorm = .008) and entropy (Mnorm = .144, SDnorm =

.027; Mnonnorm = .157, SDnonnorm = .016). Cohen’s d resulted in the largest standard errors (Mnorm =

.407, SDnorm = .006) and nonnormal distributions (Mnonnorm = .383, SDnonnorm = .006).

Mean standard error is displayed for each factor in Figures 4. Conditional item-response

probability distinctions, latent class prevalence, number of indicators, and variance ratio produced

little variation in standard errors for all effect sizes. For entropy, standard errors slightly decreased

as total sample size increased. Alternatively, Cohen’s d standard errors slightly increased in

tandem with sample size ratios and total sample size.

30

Figure 4. Normal Distribution: Factors by Standard Error for All Effect Sizes

Confidence Intervals

Effect size CI coverage rates were determined as the percentage of CIs in a given effect

size that captured the true value. Overall, under the normal distribution, entropy yielded the

greatest percentage of high CI coverage rates. For entropy, about 14% of cases showed a CI

coverage rate of 95% or higher. On the other hand, both I-index and Cohen’s d resulted in about

2% of cases with CI coverage rates of 95% or higher. The nonnormal distribution yielded entropy,

I-index, and Cohen’s d with 7%, 5%, and 2% of cases showing CI coverage rates of 95% or

greater, respectively.

31

Width of CIs were aggregated across conditions. Under the normal distribution, the overall

width of CI was smallest for I-index (CI.95[0.341]), followed by entropy (CI.95[0.414]), and

Cohen’s d (CI.95[0.715]) across all conditions. Generally, the nonnormal and normal distributions

showed similar widths of CIs (Table 7).

Figure 5 displays the mean CI width for each factor. All effect sizes showed little variation

in CI width for conditional item-response probability distinctions. Width of CIs decreased with

increase in latent class prevalence for I-index (γ=.9,.1:CI.95[0.523] to γ=.5,.5:CI.95[0.237]), entropy

(γ=.9,.1: CI.95[0.520] to γ=.5,.5: CI.95[0.404]), and Cohen’s d (γ=.9,.1: CI.95[0.768] to γ=.5,.5:

CI.95[0.696]). Width of CIs decreased with increase in total sample sizes for both I-index (N=100:

CI.95[0.391] to N=1000: CI.95[0.252]) and entropy (N =100: CI.95[0.532] to N=1000: CI.95[0.206]).

Width of CIs decreased with increase in number of indicators for I-index (5: CI.95[0.362] to 15:

CI.95[0.315]), entropy (5: CI.95[0.436] to 15: CI.95[0.370]) and Cohen’s d (5: CI.95[0.767] to 15:

CI.95[0.657]). For Cohen’s d, widths of CIs increased with increase in variance ratios (1:1:

CI.95[0.666] to 1:8: CI.95[0.750]). Nonnormal distribution results showed similar patterns for all

factors.

32

Table 7

Average Confidence Interval Widths for Each Factor: Normal and Nonnormal Distributions

Normal Nonnormal

I-index Entropy Cohen's d I-index Entropy Cohen's d

Overall

.341 .414 .715 .353 .427 .685

Sample Size Ratio

50:50 .190 .361 .536 .176 .369 .558

60:40 .210 .335 .534 .214 .365 .546

70:30 .374 .459 .669 .330 .388 .655

80:20 .395 .407 .827 .448 .453 .765

90:10 .539 .506 1.011 .598 .562 .904

ρ .6, .4 .364 .421 .683 .379 .433 .672

.7, .3 .360 .416 .735 .368 .425 .704

.9, .1 .301 .404 .727 .313 .424 .680

γ .5, .5 .523 .520 .768 .520 .534 .753

.6, .4 .374 .391 .724 .379 .404 .675

.7, .3 .310 .381 .707 .330 .395 .676

.8, .2 .263 .371 .680 .284 .384 .661

.9, .1 .237 .404 .696 .254 .420 .663

Indicators 5 .362 .436 .767 .340 .439 .822

8 .349 .432 .735 .363 .441 .684

10 .340 .416 .702 .391 .449 .636

15 .315 .370 .657 .319 .381 .599

N 100 .391 .532 .712 .405 .528 .669

150 .386 .519 .710 .403 .519 .673

200 .385 .511 .724 .403 .511 .679

500 .293 .299 .722 .300 .329 .690

1000 .252 .206 .709 .255 .249 .716

Variance Ratio 1:1 .355 .438 .666 .367 .420 .642

1:4 .338 .399 .730 .359 .436 .694

1:8 .332 .404 .750 0.334 0.426 0.720

Note: γ = latent class prevalence, normal population: skewness=0, kurtosis=0; nonnormal population: skewness=1.75, kurtosis=3.75, ρ = conditional item-response probability distinction

33

Figure 5. Normal Distribution: Factors by Average CI Width for All Effect Sizes

Discussion

LCA classifies cases into latent classes to better understand relationships and patterns in a

particular population. Correctly classifying individuals, objects, or cases into the subgroup of

interest has relevance in many fields, such as psychology, education, and business. Therefore it is

important to investigate different methods that measure classification accuracy. The overall goal of

this study was to explore the performance of I-index in quantifying classification accuracy of

LCA. The first objective addressed classification accuracy between three effect sizes, entropy R2,

I-index, and Cohen’s d. The second objective compared CI widths of the three effect sizes. Across

34

all conditions, entropy and I-index produced the best accuracy, standard error, and smallest CI

widths. Generally, I-index only performed well for a few cases.

Various scenarios were simulated using several factors to calculate the three effect sizes.

Entropy R2

showed the least bias and RMSE followed by I-index, while Cohen’s d performed

worst. Therefore entropy showed more accuracy than both I-index and Cohen’s d. I-index

produced the lowest standard errors and CI widths, followed by entropy R2 and Cohen’s d.

According to the ANOVA results, the type of effect size explained the greatest variation in

bias, RMSE, standard error, CI width, and CI coverage rates. Sample size ratio showed the second

highest percentage in variation in bias and tended to break down at ratio of 60:40. This indicates

that bias increased with increase in sample size ratios. It’s important to note that all effect size

estimates increased with increase in sample size ratios. For example, the sample size ratio of 90:10

showed a greater effect size than a sample size ratio of 60:40. Additionally, I-index (CI.95[.341])

showed the most narrow CI width, followed by entropy (CI.95[.414]) and Cohen’s d (CI.95[.715]).

I-index had the smallest standard errors overall, therefore showed the highest precision

with the smallest CI width. Across all conditions, entropy resulted in seven times more CI

coverage rates of 95% or greater when compared to I-index. This showed that I-index captured the

true population value less often. A few conditions showed slightly wider CIs. These included

wider CIs at lower latent class prevalence, less indicators, and smaller total sample sizes. Here I-

index has a better chance of capturing the true population value. The width of the CIs may have

been affected by sampling error and measurement error.

Current results show that I-index does not perform well for most conditions examined.

Only two extreme conditions, sample size ratio of 60:40 and latent class prevalence of .9, .1

yielded slightly lower RMSEs for I-index than entropy. In these cases, I-index may be useful as a

35

second effect size. Results also showed that RMSE values for entropy and I-index were more

similar with increase in latent class prevalence, number of indicators, and variance ratio. This may

address an issue in LCA research that has minimal attention. Collins and Lanza (2010) mentioned

that as models become more complex, there are more errors in class assignment due to chance.

Therefore, less parsimonious models may decrease the quality of the measure and produce lower

entropy when higher entropy is expected. Having a second effect size may help validate the

classification accuracy of more complex models that may be referenced in future studies.

Initial LCA research may provide a foundation for theoretical and practical implications,

and future research, therefore it is important to execute LCA as quantitatively efficiently as

possible. Incorporating multiple measures of effect sizes in LCA may help quantify classification

accuracy of latent classes. Once latent classes are identified, latent class characteristics are used to

develop typologies and describe the latent classes. Every case or individual belongs to only one

latent class. Therefore researchers perform further analysis of patterns within latent classes to gain

a better understanding about individual characteristics within a latent class. Subsequent analyses

may include latent class means, means for items within a latent class, and cross tabulations.

Greater classification accuracy in the initial analysis can lend to trusting that there is greater

accuracy in the subsequent analyses. Misclassification may skew study results, subsequent

analysis, and results in future research. Skewed results of latent class characteristics may be

compounded with subsequent research. In this case, the characteristics of the latent class may not

represent the characteristics of the true latent classes within the population.

Historically, LCA has considered entropy as the only effect size, therefore investigating I-

index as a measure of classification accuracy may help to better quantify the separation of the

resulting latent classes and help validate entropy values. Both entropy and I-index assess

36

assignment of correctly classified units with posterior probabilities. We thought I-index would

perform well, due to using the degree to which group overlap exceeds overlap expected by chance,

whereas entropy does not. Yet I-index did not perform well for most situations and only performed

well for very few cases. Future research may want to explore these areas further. In general,

entropy showed the highest accuracy followed by I-index. Standard error experienced the opposite,

as I-index showed the lowest standard error, followed by entropy. I-index showed the smallest

width in confidence intervals. These results show that I-index may be useful in a couple of

conditions, yet more research is needed to determine specifics of these conditions. Taking a

broader view, incorporating I-index in a variety of techniques may help bring to light this long lost

cousin of the effect size family.

Limitation and Future Directions

Simulation studies lend many advantages, yet results do not generalize beyond the scope of

specific conditions that are investigated in the study. For instance, our study only examined a two-

class model, with a certain number of indicators and parameters. Even though the most commonly

encountered conditional item-response probabilities and latent class prevalence were examined,

perhaps investigating smaller, larger, or values in between may be of importance. Since a two-

class model was studied with a mixture of four sample sizes, results may not generalize to models

with more classes. More classes may need larger sample sizes and perhaps more indicators for

reliable results. Numerous conditions were studied, yet additional levels of parameters exist within

the explored conditions. This study focused on dichotomous indicators. Future studies may expand

studies to include continuous and polytomous items. The impact of single or multiple covariates

can be also explored. Furthermore, average bias was calculated using actual values of bias

37

aggregated across the 500 replications. This may cause the final average bias to be less if the

biases have opposite signs across replications. The current study used the proportional chance

criterion to calculate the I-index. It may be useful to examine the I-index using the maximum

chance criterion.

38

APPENDIX

SAS SYNTAX FOR DATA SIMULATION

39

libname T 'C:\Users\Desktop\Main'; /*MAIN FOLDER WHERE DATA IS SAVED*/

%INCLUDE "C:\Users\Desktop\SimulateLcaDataset.sas"; /*CHANGE-LOCATION WHERE SIMULATION MACRO IS

SAVED*/

%LET RUN = 1; /*END OF FILE NAME FOR FINAL EXCEL FILE, CAN RUN MORE THAN ONCE*/

%LET COUNT=1000; /*CHANGE-TOTAL TIMES TO RUN PROGRAM-NUMBER OF SIMULATED LCA DATASET

ITERATIONS, RESULTS IN THE NUMBER OF END MAIN VARIABLES FOR THE DISTRIBUTION*/

%LET TOTALN= 1000; /*CHANGE-TOTAL SAMPLE*/

%LET INDNUM = 5; /*CHANGE-TOTAL NUMBER OF INDICATORS, IN DATA RHOS, DATA STARTVAL,

PROCLCA*/

%LET GAMMA1 = .8; /*CHANGE-FIRST GAMMA, PRIOR PROBABILITY OF LATENT CLASS 1*/

%LET GAMMA2 = .2; /*CHANGE-SECOND GAMMA, PRIOR PROBABILITY OF LATENT CLASS 2*/

%LET RHO1 = .6; /*CHANGE-FIRST RHO, DATA STARTVAL, FOLDERPATH1/2 */

%LET RHO2 = .4; /*CHANGE-SECOND RHO, DATA STARTVAL*/

%LET DISTCLASS = SMALL; /*CHANGE-DISTINCTION BETWEEN 2 LATENT CLASSES, BASED ON RHO*/

%LET MNUM = 12; /*CHANGE-CORRECT MODEL NUMBER*/

%LET PRINTNUM = 1; /*CHANGE-NUMBER OF UNIQUE DATASETS TO PRINT FOR EACH MAIN

OUTPUT, USE AS QUICK CHECK*/

%LET QLCA1 = .8; /*CHANGE-PRIOR PROBABILITY OF LATENT CLASS 1*/

%LET QLCA2 = .2; /*CHANGE-PRIOR PROBABILITY OF LATENT CLASS 2*/

/*DESCRIPTIVE TITLE, CHANGES AS PARAMETERS ABOVE CHANGE*/

%LET T2USE = C2, M&MNUM., N&TOTALN., I&INDNUM., gammas: &GAMMA1., &GAMMA2., rhos: &RHO1., &RHO2.,

-&DISTCLASS. distinction between classes;

/*CHANGE-LOCATION FOR LCAI AND ENTROPY EXCEL OUTPUTS*/

%LET FOLDERPATH1 = C:\Users\Main\Desktop\N&TOTALN._I&INDNUM._LCAI_ENTROPY\;

/*CHANGE-LOCATION FOR POST PROBS AND SE EXCEL OUTPUTS*/

%LET FOLDERPATH2 = C:\Users\Main\N&TOTALN._I&INDNUM._POSTPROB_SE\;

OPTIONS NODATE NONUMBER;

/*GENERIC DATA TO FEED INTO SIMULATION LCA DATASET PROGRAM, TO BUILD INITIAL DATASET THAT

FEEDS INTO PROC LCA,CAN RUN OUTSIDE OF SIMULATION, SINCE THESE ARE APPLIED TO ALL DATASETS

CREATED WITH SIMULATED LCA DATASET MACRO*/

DATA gammas; /*THESE GAMMAS - GENERIC LATENT CLASS PREFERENCES*/

INPUT gammas; /*CHANGE NUMBER OF ROWS TO ALIGN WITH CORRECT NUMBER OF LATENT CLASSES*/

DATALINES;

.5

.5

RUN;

/*THESE RHOS - GENERIC ITEM RESPONSE PROBABILITIES*/

DATA rhos; /*CHANGE NUMBER OF ROWS TO ALIGN WITH CORRECT NUMBER OF ITEMS*/

INPUT rhos1 rhos2; /*COLUMNS ARE LATENT CLASSES,ROWS ARE ITEMS*/

DATALINES;

.5 .5

.5 .5

.5 .5

.5 .5

.5 .5

run;

/*RUN_LOOPS MACRO CREATES SIMULATED LCA DATASETS MULTIPLE TIMES FOR SPECIFIED TOTAL SAMPLE

SIZE, PRODUCES UNIQUE DATASETS OF AGGREGATED RESPONSE PATTERNS THAT ARE FED INTO THE REST OF

THE PROGRAM, THE UNIQUE DATASET PRODUCED IS MANIPULATED WITH STARTVALUES AND THEN FED

INTO PROC LCA FOR ANALYSIS */

%MACRO RUN_LOOPS;

40

%DO I = 1 %TO &COUNT.;

%SimulateLcaDataset( true_gamma_dataset = gammas,

true_rho_dataset = rhos,

output_dataset_name = DATA&I.,

total_n = &TOTALN. )

%END;

%MEND RUN_LOOPS;

%RUN_LOOPS

/*----------------------------------------------------------------------------------------------------------------------------------------------------*/

/*STARTVALUES DATASET IS NOT LOOPED, CREATES ONE DATA FILE WITH DESIRED STARTING VALUES

(GAMMAS, RHOS), IT IS RUN BEFORE THE OTHER DATASETS AND PROCS, OUTSIDE OF SIMULATION. STARTVAL

IS APPLIED TO ALL, UNIQUE DATASETS PRODUCED IN ABOVE SIMULATION. THE RESULTING MANIPULATED

DATASETS ARE FED INTO PROC LCA*/

/*-----------------------------------------------------------------------------------------------------------------------------------------------------*/

/*CHANGE NUMBER OF ROWS AS NEEDED TO REFLECT NUMBER OF INDICATORS*/

DATA STARTVAL;

INPUT PARAM $ GROUP VARIABLE $ RESPCAT ESTLC1 ESTLC2;

DATALINES;

GAMMA 1 . . &GAMMA1. &GAMMA2.

RHO 1 Item001 1 &RHO1. &RHO1.










;

RUN;

/*-----------------------------------------------------------------------------------------------------------------------*/

/*RUN_LOOPS2 MACRO FEEDS EACH UNIQUE DATASET (DATA1, DATA2, DATA3...), */

/*CREATED IN RUN_LOOPS MACRO, INTO THE FINAL DATASETS AND PROCS */

/*-----------------------------------------------------------------------------------------------------------------------*/

%MACRO RUN_LOOPS2;


/*ADDS SUBJECT ID COLUMN TO SIMULATED DATASET TO PRODUCE AND USE OUTPOST FILE THAT PRODUCES

POSTERIOR PROBABILITIES, THE LCA DATA PRODUCED IS AGGREGATED, THEREFORE SUBJECT ID

REPRESENTS NUMBER OF DIFFERENT RESPONSE PATTERNS, THAT ARE PRODUCED FROM THE SIMULATED

LCA DATASET CODE ABOVE*/

DATA DATA&I.ID;

SET DATA&I.;

DO SUBID = 1,_N_;

END;

OUTPUT;

RUN;

PROC LCA DATA=DATA&I.ID OUTEST=EST&I. START=STARTVAL OUTPOST=POST&I.;

TITLE "&T2USE";

NCLASS 2; /*NUMBER OF LATENT CLASSES*/

ID SUBID; /*SHOWS AS SECOND COLUMN, NEEDED TO VIEW POSTERIOR PROBABILITEIS*/

ITEMS Item001 Item002 Item003 Item004 Item005; /*CHANGE TO REFLECT CORRECT NUMBER OF ITEMS */

FREQ Count; /*CREATES AGGREATED DATASET OF RESPONSE PATTERNS*/

CATEGORIES 2 2 2 2 2; /*CHANGE TO REFELCT CORRECT NUMBER OF ITEMS */

MAXITER 1800; /*MAXIMUM NUMBER OF ITERATIONS*/

41

RUN;

/*THIS TABULATE PRODUCES TOTAL COUNT OF 1'S (HITS) AND 2'S(MISSES) - ALSO GOOD CHECK FOR NUMBER

OF OBSERVED HITS*/

ODS LISTING CLOSE; /*PRODUCES NEEDED VARIABLES TO FEED, WITHOUT PRINTING TO OUTPUT*/

PROC TABULATE DATA=POST&I.;

CLASS BEST; /*GROUP THE RESPONSE PATTERN IS PLACED INTO, 1 OR 2 */

VAR COUNT; /*COUNT OF INDIVIDUALS FROM TOTAL SAMPLE SIZE WITH 1'S AND 2'S*/

TABLE BEST, COUNT*BEST;

RUN;

ODS LISTING;

/*ADDS TOTAL COUNT COLUMN TO POST1 DATASET, TOTAL COUNT COLUMN REPRESENTS THE NUMBER OF

EACH RESPONSE PATTERN*/

/*THAT ARE HITS (1'S)*/

DATA POST&I.TOTAL;

SET POST&I.;

TPOSTLC1 = POSTLC1*COUNT; /*TOTAL FOR POST PROBS IN LATENT CLASS 1 FOR THAT RESPONSE PATTERN*/

TPOSTLC2 = POSTLC2*COUNT; /*TOTAL FOR POST PROBS IN LATENT CLASS 2 FOR THAT RESPONSE PATTERN*/

IF BEST=1 THEN /*GROUP 1 RESPONSE PATTERNS*/

TOTALCOUNT = BEST*COUNT;/*BEGINS CUMULATIVE SUM COUNT FOR TOTAL HITS*/

BTCOUNT+TOTALCOUNT; /*RUNNING CUMULATIVE SUM FOR TOTALCOUNT FOR HITS (1'S) ONLY*/

RUN;

/*CREATES DATA FILE WITH OBSERVED HIT VALUE TO BE USED TO CALCUALTE OBSERVED HIT RATE*/

PROC SUMMARY DATA=POST&I.TOTAL;

VAR TOTALCOUNT;

OUTPUT OUT=TOTALHITS SUM=TOTALHITS SKEW=BSKEW KURT=BKURT;

RUN;


VAR TPOSTLC1;

WHERE BEST=1;

OUTPUT OUT=VAR1 VAR=V1;

RUN;


VAR TPOSTLC2;

WHERE BEST=2;

OUTPUT OUT=VAR2 VAR=V2;

RUN;

PROC SQL;

CREATE TABLE TOTALHITS2A AS

SELECT A.*, B.V1, C.V2

FROM TOTALHITS A, VAR1 B, VAR2 C;

QUIT;

DATA TOTALHITS2;

SET TOTALHITS2A;

KEEP TOTALHITS BSKEW BKURT V1 V2 RATIO_VAR;

IF V1 < V2 THEN RATIO_VAR = V1/V2;

IF V1 > V2 THEN RATIO_VAR = V2/V1;

RUN;

/********************************/

/* I-INDEX CODE STARTS */

/********************************/

/*DATA PRODCUED FROM PROC LCA TO SHOW TOTAL HITS (TOTALHITS) IS FED, INTO I-INDEX CODE TO

PRODUCE OBSERVED HIT RATE AND I-INDEX, ALSO TAGS SAMPLE RATIO AND IF DISTRIBUTION IS NORMAL

AND VARIANCE RATIO*/

DATA LCAIINDEX&I.;

42

SET TOTALHITS2;

KEEP TOTALHITS TOTALNONHITS BSKEW BS BKURT BK LCAHIT LCAHE LCAI SRATIO DIST V1

V2 RATIO_VAR RVAR;

TOTALNONHITS=&TOTALN. - TOTALHITS; /*NUMBER OF CASES WITH NO HITS*/

LCAHIT=TOTALHITS/&TOTALN.; /*PERCENT OF OBSERVED HITS (H0) [THE TOTAL CASES WITH 1'S]*/

/*LABEL NORMAL AND NON-NORMAL DISTRIBUTIONS*/

IF BSKEW = 0 OR BSKEW =. THEN BS='S';

IF BKURT = 0 OR BKURT =. THEN BK='K';

IF BS='S' AND BK='K' THEN DIST='NORM';

IF BSKEW = 1.75 AND BKURT = 3.75 THEN DIST='OTHE';

/*LABEL SAMPLE SIZE RATIOS,CHANGES WTIH TOTAL SAMPLE SIZE,BASED ON TOTAL HITS WHICH IS FIRST

LATENT CLASS*/

IF TOTALN = 100 THEN DO;

/*100 SAMPLE**/

IF TOTALHITS = 10 THEN SRATIO = '10:90';









END;

ELSE IF TOTALN = 150 THEN DO;

/*150 SAMPLE*/










END;


/*200 SAMPLE*/










END;


/*500 SAMPLE*/







43




END;


/*1000 SAMPLE*/










END;

/*LABEL VARIANCE RATIO*/ /*CHANGE TO EXACT 1 DIVIDED BY 8 AND SUCH*/

IF RATIO_VAR = .125 THEN RVAR = '1:8';



IF RATIO_VAR = 1 THEN RVAR = '1:1';

/*LCA I-INDEX CALCULATED*/

LCAHE = ((&QLCA1.*TOTALHITS)+(&QLCA2.*TOTALNONHITS))/&TOTALN.;/*EXPECTED HIT

RATE (HE), PROPORTIONAL CHANCE CRITERION*/

LCAI = (LCAHIT-LCAHE)/(1-LCAHE); /*CALCULATION OF THE I-INDEX */

IF LCAI < 0 THEN LCAI=0;

OUTPUT;

RUN;

/********************************/

/* I-INDEX CODE ENDS */

/********************************/

/********************************/

/* COHENS D CODE STARTS */

/********************************/

/*THE DATA PRODCUED FROM POSTTOTAL GET POSTERIOR PROBABILITIES IS FED*/

/*INTO THE COHENS D CODE TO PRODUCE EFFECT SIZE OF COHENS D*/

PROC SQL;

CREATE TABLE CD2PREP&I. AS

SELECT subid, BEST,

SUM (CASE WHEN BEST=1 THEN TPOSTLC1 ELSE 0 END) AS POSTLC1_SUM,

SUM (CASE WHEN BEST=1 THEN COUNT ELSE 0 END) AS BC1,

CALCULATED POSTLC1_SUM/ CALCULATED BC1 AS MEAN1,

((POSTLC1 - CALCULATED MEAN1)**2)*COUNT AS PRE_SS1

FROM POST&I.TOTAL

WHERE BEST=1;

QUIT;

PROC SQL;

CREATE TABLE CD3PREP&I. AS

SELECT subid, BEST,

SUM (CASE WHEN BEST=2 THEN TPOSTLC2 ELSE 0 END) AS POSTLC2_SUM,

SUM (CASE WHEN BEST=2 THEN COUNT ELSE 0 END) AS BC2,

CALCULATED POSTLC2_SUM/ CALCULATED BC2 AS MEAN2,

((POSTLC2 - CALCULATED MEAN2)**2)*COUNT AS PRE_SS2

FROM POST&I.TOTAL

WHERE BEST=2;

44

QUIT;

PROC SQL;

CREATE TABLE COHENSD&I. AS

SELECT DISTINCT

A.MEAN1,

B.MEAN2,

B.MEAN2 - A.MEAN1 AS MEANDIFF,

C.SS1,

D.SS2,

C.VARLC1,

D.VARLC2,

C.SD1,

D.SD2,

&TOTALN.-2 AS DF,

SQRT((C.SS1 + D.SS2)/(&TOTALN.)) AS POOLEDSD,

(B.MEAN2 - A.MEAN1)/( SQRT((C.SS1 + D.SS2)/(&TOTALN.)) ) AS COHENSD

FROM CD2PREP&I. A, CD3PREP&I. B,

(SELECT subid, SUM(PRE_SS1) AS SS1,

(CALCULATED SS1)/(BC1-1) AS VARLC1,

SQRT(CALCULATED VARLC1) AS SD1

FROM CD2PREP&I.) C,

(SELECT subid, SUM(PRE_SS2) AS SS2,

(CALCULATED SS2)/(BC2-1) AS VARLC2,

SQRT(CALCULATED VARLC2) AS SD2

FROM CD3PREP&I.) D

where a.subid = c.subid and b.subid = d.subid;

QUIT;

/********************************/

/* COHENS D CODE ENDS */

/********************************/

/*DATASET CONTAINS FIT INDICES AND ENTROPY*/

DATA EST&I.ALL;

SET EST&I.;

KEEP AIC BIC ABIC ENTROPY;

RUN;

%END;

%MEND RUN_LOOPS2;

%RUN_LOOPS2

/*COMBO_LCA, COMBO_COHENSD, COMBO_EST MACROS, APPEND A DECLARED NUMBER OF */

/*DATASETS USED TO EXPORT TO EXCEL, CAN PROC PRINT LATER */

%MACRO COMBO_LCA;

DATA COMBO_LCAIINDEX;

SET


LCAIINDEX&I.

%END;

;

DO CASEID = 1,_N_;

END;

OUTPUT;

RUN;

%MEND COMBO_LCA;

45

%COMBO_LCA

%MACRO COMBO_COHEN;

DATA COMBO_COHENSD;

SET


COHENSD&I.

%END;

;

DO CASEID = 1,_N_;

END;

OUTPUT;

RUN;

%MEND COMBO_COHEN;

%COMBO_COHEN

%MACRO COMBO_EST;

DATA COMBO_ESTALL;

SET


EST&I.ALL

%END;

;

DO CASEID = 1,_N_;

END;

OUTPUT;

RUN;

%MEND COMBO_EST;

%COMBO_EST

/* MERGE LCAIINDEX, ENTROPY, COHEN'S D VALUES FOR FOR UNIVARIATE ANALYSIS */

PROC SQL;

CREATE TABLE MAIN_COMBO AS

SELECT A.CASEID,

A.TOTALHITS,

A.TOTALNONHITS,

C.DF,

A.SRATIO,

A.BSKEW,

A.BS,

A.BK,

A.BKURT,

A.RATIO_VAR,

A.RVAR,

A.DIST,

A.LCAI,

B.ENTROPY,

B.AIC,

B.BIC,

B.ABIC,

C.COHENSD

FROM COMBO_LCAIINDEX A

LEFT JOIN COMBO_ESTALL B ON A.CASEID=B.CASEID

LEFT JOIN COMBO_COHENSD C ON A.CASEID=C.CASEID;

QUIT;

PROC SORT DATA=MAIN_COMBO;

BY TOTALHITS;

RUN;

/*CONFIDENCE INTERVALS*/

DATA MAINCI;

46

SET MAIN_COMBO ;

DROP TOTALHITS TOTALNONHITS BSKEW BKURT BS BK RATIO_VAR;

CLASSTYPE = "&DISTCLASS.";

TOTALNSIZE = "&TOTALN.";

MODEL = "&MNUM.";

ITEMN = "&INDNUM.";

LCAPP = "&GAMMA1., &GAMMA2.";

GAMMAS = "&GAMMA1., &GAMMA2.";

RHOS = "&RHO1., &RHO2.";

N1 = TOTALHITS;

N2 = TOTALNONHITS;

CONF = .95;

PRLOW = 1 - (1-CONF)/2;

PRUPP = (1-CONF)/2;

/*CI FOR COHENS D*/

TC = COHENSD *(SQRT((N1*N2)/(N1+N2)));

CNCPLOW = TNONCT (TC, DF, PRLOW);

CNCPUPP = TNONCT (TC, DF, PRUPP);

CDLOW = CNCPLOW / (SQRT ((N1*N2)/(N1+N2)));

CDUPP = CNCPUPP / (SQRT ((N1*N2)/(N1+N2)));

CDCI_RANGE = CDUPP - CDLOW;

/*CI FOR LCAI*/

TI = LCAI *(SQRT((N1*N2)/(N1+N2)));

LNCPLOW = TNONCT (TI, DF, PRLOW);

LNCPUPP = TNONCT (TI, DF, PRUPP);

LDLOW = LNCPLOW / (SQRT ((N1*N2)/(N1+N2)));

LDUPP = LNCPUPP / (SQRT ((N1*N2)/(N1+N2)));

LCAI_CI_RANGE = LDUPP - LDLOW;

/*CI FOR ENTROPY*/

TE = ENTROPY *(SQRT((N1*N2)/(N1+N2)));

ENCPLOW = TNONCT (TE, DF, PRLOW);

ENCPUPP = TNONCT (TE, DF, PRUPP);

EDLOW = ENCPLOW / (SQRT ((N1*N2)/(N1+N2)));

EDUPP = ENCPUPP / (SQRT ((N1*N2)/(N1+N2)));

ENTROPY_CI_RANGE = EDUPP - EDLOW;

RUN;

DATA MAINDISS&RUN.;

SET MAINCI;

KEEP CLASSTYPE TOTALNSIZE ITEMN MODEL LCAPP GAMMAS RHOS LDLOW LCAI LDUPP

LCAI_CI_RANGE CDLOW COHENSD CDUPP CDCI_RANGE EDLOW

ENTROPY EDUPP ENTROPY_CI_RANGE SRATIO RVAR DIST AIC BIC ABIC CASEID;

RUN;

PROC SORT DATA = MAINDISS&RUN.;

BY DIST RVAR;

RUN;

/******************************************************************/

47

/*EXPORT ALL DATA INTO SEPARATE SHEETS IN THE SAME EXCEL DOCUMENT */

/******************************************************************/

/*EXPORT LCAIINDEX TO EXCEL*/

PROC EXPORT DATA=COMBO_LCAIINDEX

OUTFILE = "&FOLDERPATH1.N&TOTALN._I&INDNUM._M&MNUM._ALL1_r&RUN..xls"

DBMS=EXCEL97 REPLACE;

SHEET = "LCA_I_INDEX";

RUN;

/*EXPORT FIT INDICES AND ENTROPY TO EXCEL*/

PROC EXPORT DATA=COMBO_ESTALL



SHEET = "ENT_FITS";

RUN;

/*EXPORT COHENS D TO EXCEL*/

PROC EXPORT DATA=COMBO_COHENSD



SHEET = "COHENSD";

RUN;

/*EXPORT LCAIINDEX, ENTROPY, COHEN'S D*/

PROC EXPORT DATA=MAIN_COMBO



SHEET = "MAIN_COMBO";

RUN;

PROC EXPORT DATA=MAINCI



SHEET = "MAINCI";

RUN;

PROC EXPORT DATA=MAINDISS&RUN.



SHEET = "MAINDISS";

RUN;

%MACRO EXPORT_LOOP;

%DO I = 1 %TO 10;

/*EXPORT POST1TOTAL TO EXCEL*/

PROC EXPORT DATA=POST&I.TOTAL



SHEET = "POSTERIOR_PROBS_&I.";

RUN;

%END;

%MEND EXPORT_LOOP;

%EXPORT_LOOP

48

REFERENCES

Akaike, H. (1974). A new look at the statistical model identification. IEEE Transactions on

Automatic Control, 19, 716–723.

Angelis, L. D. (2013). Latent class models for financial data analysis: some statistical

developments. Statistical Methods and Applications, 22, 227-242.

Barnard-Brak, L., Lan, W. Y., & Paton, V. O. (2010). Profiles in Self-regulated Learning in

Online Learning Environment. The International Review of Research in Open and

Distance Learning, 11, 61-80.

Celeux, G., & Soromenho, G. (1996). An entropy criterion for assessing the number of clusters

in a mixture model. Journal of Classification, 13, 195-212.

Cleveland, M.J., Collins, L. M., Lanza, S. T., Greenberg, M. T., & Feinberg, M. E. (2010). Does

individual risk moderate the effect of contextual-level protective factors? A latent

class analysis of substance use. Journal of Prevention & Intervention in the

Community, 38, 213-228.

Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences. (2nd ed.). Hillsdale.

NJ: Erlbaum.

Collins, L. M., & Lanza, S. T. (2010). Latent Class and Latent Transition Analysis: With

Applications in the Social, Behavioral, and Health Sciences. Hoboken, NJ: John Wiley &

Sons, Inc.

Dziak, J. J., Lanza, S. T., & Tan, X. (2014). Effect Size, Statistical Power and Sample Size

Requirements for the Bootstrap Likelihood Ratio Test in Latent Class Analysis.

Structural Equation Modeling: A Multidisciplinary Journal, 21, 534-552.

Fidler, F., & Thompson, B. (2001). Computing correct confidence intervals for ANOVA fixed

49

and random-effects effect sizes. Educational and Psychological Measurement, 61, 575-

604.

Finch, W. H., & Bronk, K. C. (2011). Conducting confirmatory latent class analysis using

Mplus. Structural Equation Modeling: A Multidisciplinary Journal, 18, 132–151.

doi:10.1080/10705511.2011.532732

Grant, M. B. (2015). Mixed Mode Latent Class Analysis: An Examination of Fit Index

Performance for Classification. Structural Equation Modeling: A Multidisciplinary

Journal, 22, 76-86. doi: 10.1080/10705511.2014.935751

Green, P.E., Carmone, F. J., & Wachspress, D. P. (1976). Consumer Segmentation via Latent

Class Analysis. Journal of Consumer Research, 3, 170-174.

Goodman, L. A. (1974). Exploratory latent structure analysis using both identifiable and

unidentifiable models. Biometrika, 61, 215–231.

Hagenaars, J., & McCutcheon, A. (Eds.). (2002). Applied Latent Class Analysis Models. New

York: Cambridge University Press.

Hadzi-Pavlovic, D. (2009). Finding patterns and groupings: I. Introduction to latent class

analysis. Acta Neuropsychiatrica, 21, 312–313.

Henson, R. K., Natesan, P., & Axelson, E. D. (2014). Comparisons of improvement-over-

chance effect sizes for two groups under variance heterogeneity and prior

probabilities. The Journal of Experimental Education, 82, 205-228.

Hess, B., Olejnik, S., & Huberty, C.J. (2001). The efficacy of two improvement-

over-chance effect sizes for two-group univariate comparisons under variance

heterogeneity and nonnormality. Educational and Psychological Measurement, 61, 909-93.

Higgins, G. E., Wolfe, S. E., & Ricketts, M. L. (2009). Digital Piracy: A Latent Class Analysis.

50

Social Science Computer Review, 27, 24-40.

Huberty, C.J. (1994). Applied Discriminant Analysis. New York: John Wiley.

Huberty, C. J, & Holmes, S. E. (1983). Two-group comparisons and univariate classification.

Educational and Psychological Measurement, 43, 15-26.

Huberty, C. J, & Lowman, L. L. (2000). Group overlap as a basis for effect size. Educational and

Psychological Measurement, 60, 543-563.

Kaplan, D., & Keller, B. (2011). A note on cluster effect in latent class analysis. Structural

Equation Modeling, 18, 525-536.

Lanza, S. T., Collins, L. M., Lemmon, D. R., & Schafer, J. L. (2007). PROC LCA: A SAS

procedure for latent class analysis. Structural Equation Modeling, 14, 671–694.

Lanza, S. T., Dziak, J. J., Huang, L., Wagner, A. T., & Collins, L. M. (2014). PROC LCA &

PROC LTA User's Guide (Version 1.3.1). University Park: The Methodology Center,

Penn State. Available from methodology.psu.edu.

Lanza S. T., Savage, J. S., & Birch, L. L. (2010). Identification and prediction of latent

classes of weight-loss strategies among women. Obesity, 18, 833-40.

Lazarsfeld, P. F., Henry, N. W. (1968). Latent Structure Analysis. Boston: Houghton Mill.

Magidson, J., & Vermunt, J. (2004). Latent class models. In D. Kaplan (Ed.), Handbook of

Quantitative Methodology for the Social Sciences (pp. 175–198). Newbury Park, CA:

Sage.

Magidson, J., & Vermunt, J.K. (2005) Technical Guide for Latent GOLD 4.0: Basic and

Advanced. Statistical Innovations Inc.: Belmont, Massachussetts.

McCutcheon, A.C. (1987). Latent Class Analysis. Beverly Hills: Sage Publications.

Morgan, G. (2015). Mixed Mode Latent Class Analysis: An Examination of Fit Index

51

Performance for Classification. Structural Equation Modeling: A Multidisciplinary

Journal, 22, 76-86.

Muthén, B. (2004). Latent variable analysis: Growth mixture modeling and related techniques

for longitudinal data. In D. Kaplan (Ed.), Handbook of Quantitative Methodology for the

Social Sciences (pp. 345–368). Newbury Park, CA: Sage.

Natesan, P., & Thompson, B. (2007). Extending improvement-over-chance I-Index effect size

simulation studies to cover some small-sample cases. Educational and Psychological

Measurement, 67, 59-72.

Nylund, K. L., Asparouhov, T., & Muthén, B. (2007a). Deciding on the number of classes in

latent class analysis and growth mixture modeling: A Monte Carlo simulation study.

Structural Equation Modeling: A Multidisciplinary Journal, 14, 535–569.

Nylund, K., Bellmore, A., Nishina, A., & Graham, S. (2007b). Subtypes, severity, and

structural stability of peer victimization: What does latent class analysis say? Child

Development, 78, 1706-1722.

Ramaswamy, V., DeSarbo, W. S., Reibstein, D. J., & Robinson, W. T. (1993). An empirical

pooling approach for estimating marketing mix elasticities with PIMS data. Marketing

Science, 12, 103−124.

Rindskopf, D. (2003). Mixture or homogeneous? Comment on Bauer and Curran (2003).

Psychological Methods, 8, 364–368.

Rindskopf, R., & Rindskopf, W. (1986). The value of latent class analysis in medical diagnosis.

Statistics in Medicine, 5, 21‐27.

Roberts, T. J., & Ward, S. E. (2011). Using latent transition analysis in nursing research

to explore change over time. Nursing Research, 60, 73-79.

Schwartz, G. (1978). Estimating the dimension of a model. The Annals of Statistics, 6, 461–464.

52

Sclove, L. (1987). Application of model-selection criteria to some problems in multivariate

analysis. Psychometrika, 52, 333–343.

SimulateLcaDataset SAS Macro (Version 1.1.0) [Software]. (2011). University Park: The

Methodology Center, Penn State. Retrieved from http://methodology.psu.edu

Smithson, M. (2001). Correct confidence intervals for various regression effect sizes and

parameters: The importance of noncentral distributions in computing intervals.

Educational and Psychological Measurement, 61, 605-632.

Thompson, B. (2002). What future quantitative social science research could look like:

Confidence intervals for effect sizes. Educational Researcher, 31, 24-31.

Thompson, B. (2007). Effect sizes, confidence intervals, and confidence intervals for effect sizes.

Psychology in the Schools, 44, 423-432.

Wilcox, R. R. (1987). New designs in analysis of variance. Annual Review of Psychology, 54,

594-604.

Wurpts, I. C., & Geiser, C. (2014). Is adding more indicators to a latent class analysis beneficial

or detrimental? Results of a Monte-Carlos study. Frontiers in Psychology, 5, 1-15.

doi:10.3389/fpsyg.2014.00920

Xian, H., Scherrer, J. F., Madden P. A., Lyons M. J., Tsuang M., True W. R. & Eisen S.A.

(2005). Latent class typology of nicotine withdrawal: genetic contributions and

association with failed smoking cessation and psychiatric disorders. Psychological

Medicine, 35, 409-419.

Yang, C. (2006). Evaluating latent class analyses in qualitative phenotype identification.

Computational Statistics & Data Analysis, 50, 1090–1104.

http://methodology.psu.edu/

Comparing three effect sizes for latent class analysis./67531/metadc822835/... · Bayesian...

Documents

Transcript of Comparing three effect sizes for latent class analysis./67531/metadc822835/... · Bayesian...