2011 Kayan, Assessing Power of Structural Equation Modeling WORD

8/18/2019 2011 Kayan, Assessing Power of Structural Equation Modeling WORD

1/10

Education Research Journal Vol. 1(3), pp. 37-42, August 2011Available online at http://www.resjournals.com/ARJISSN: 2026 – 6332 ©2011 International Research Journals

Full l Length Research paper

Assessing Power of Structural Equation Modeling

Studies: A Meta-Analysis

Fatma KAYAN FADLELMULA

Middle East Technical University, Faculty of Education Department of Elementary Education

06531, Ankara, TURKEY

Emil:[email protected]

Abstract

A meta-analysis was performed to determine the statistical power of research studies utilizing

Structural Equation Modeling as the primary analysis technique. An extensive literature review was

conducted in the Turkish Social Sciences Database, for journals published in the field of education. The

database covered 34 journals in this field. After a full text review, 12 studies, published from 2007 to

2010, were found suitable for analysis. Power was calculated using SAS; consideration was given to,

indicated degrees of freedom, significance level, sample size, and root-mean-square error ofapproximation. Results revealed that one fourth of the studies achieved power less than 0.50.

Considering that rejecting a false null hypothesis with a power of 0.50 is the same as guessing the

outcome by chance, the reported findings of these studies would possess relatively little practical

significance. Recommendations are made for future research and practice.

Keywords:Structural equation modeling, power, meta-analysis

INTRODUCTION

Structural equation modeling (SEM) is a statistical

technique that takes a hypothesis testing approach tothe analysis of a structural theory (Raykov andMarcoulides, 2006). It is used for both constructvalidation and theory development (Pedhazur andPedhazur, 1991). Procedurally, structural equationmodeling works with a correlation or a covariance datamatrix, derived from a set of observed or latentvariables (Kunnan, 1998), and it attempts to explain thepatterns of covariance among the variables included in

the structural model (Kelloway, 1998).

Nowadays, SEM has become increasingly popularamong researchers from many different disciplines.Interest in SEM is evident by the growing number of

software programs developed to apply SEM (such asAMOS, EQS, LISREL, and Mplus), numerous graduatelevel courses and continuing education workshopsprepared for explaining SEM, and many published


2/10

empirical works where the researchers describe theirSEM results (Kline, 2011). This is mostly due to the factthat most of the data analytical techniques, such asexploratory factor analysis, confirmatory factor analysis,multiple regression, path analysis, and several analysisof variance as well as multivariate analysis of variance,are special cases of structural equation modeling(Sapp, 2006). Basically, SEM combines those statistical

techniques into one comprehensive analysis (Kunnan,1998). However, it should be noted that although SEMis a powerful multivariate technique, the conclusionsreached from SEM may not be “as strong as thoseobtained from true experimental research designs”,specially regarding the threats to internal validity (Sapp,2006, p.47).

Most applications of structural equation modeling

include five stages. These are model specification,model identification, model estimation, model testing,and model


3/10

38 Educ. Res. J.

Table 1.Model Fit Indices and Their Criterion

Fit Indices Criterion

Chi-Squared (χ2)

Goodness of Fit Index (GFI)

Adjusted Goodness of Fit Index (AGFI)

Root Mean Square Error of Approximation (RMSEA)

Root Mean Square Residual (RMR)

Standardized Root Mean Square Residual

(S-RMR)

Normed Fit Index (NFI)

Non-Normed Fit Index (NNFI)

Comparative Fit Index (CFI)

Incremental Fit Index (IFI)

Relative Fit Index (RFI)

Non-significant

GFI>0.90

AGFI>0.90

0.050.90

modification, or pruning (Bollen and Long,1993). Among these stages, model testingrefers to the evaluation of whether the

theoretical model is supported by the sampledata and is crucial in the analysis. In theliterature, there are a number of fit indices thatprovide a statistical assessment of what itmeans to say the model fits the data (Jöreskogand Sörbom, 1993). The most commonly used

ones are the chi square (χ2), the goodness of

fit index (GFI), the adjusted goodness of fitindex (AGFI), and the root mean squared errorof approximation (RMSEA) (Schumacker andLomax, 2004).

Table 1 summarizes a number of model fitindices and their criterion used in SEM

analyses. In particular, chi square (χ2) is a

measure for overall fit of the model to the data

(Jöreskog and Sörbom, 1993). A significantχ2

value indicates that the observed andestimated variance-covariance matrices differ,

whereas a nonsignificantχ2 value indicates

that there is no significant difference. Hence, a

nonsignificant χ2 value with associated

degrees of freedom implies that the model fitsthe data well (Kelloway, 1998). GFI measureshow much better the model fits as compared tono model at all (Jöreskog and Sörbom, 1993).The measure of GFI ranges from 0 (no fit) to 1(perfect fit), with values exceeding 0.90,indicating a good fit to the data (Kelloway,1998). In a similar vein, AGFI adjusts GFI fordegrees of freedom; values exceeding 0.90indicate a good fit to data (Schumacker andLomax, 2004).In addition, RMR is the square root of themean of the squared discrepancies betweenthe implied and observed covariance matrices(Schumacker and Lomax, 2004). It is used tocompare the fit of two different models with the


4/10

same data. The lowest bound of RMR is 0 andlow values are taken to indicate good fit(Kelloway, 1998). However, this index issensitive to the scale of measurement of themodel variables. Therefore, it is difficult todetermine what a low value actually is. So,standardized root mean square residual (S-

RMR) is used as an alternative index, which isa standardized summary of the averagecovariance residuals. Covariance residuals arethe differences between the observed andestimated covariance. Values of the S-RMRrange from 0 (perfect fit) to 1 (poor fit). As thedifference between the observed and predictedcovariances increases, the value of the S-RMRincreases as well. The values of S-RMR lessthan 0.05 indicate a good fit to the data;however, values less than 0.10 are also

accepted (Kline, 1998).

Unlike the other fit indices, RMSEA is based

on the analysis of residuals, where smaller

values indicate a better fit to the data. It has

the advantage of going beyond point estimates

to the provision of 90 % confidence intervals

(Kelloway, 1998). In particular, RMSEA is a

parsimony-adjusted index. Its value decreases

as there are more degrees of freedom (greater

parsimony) or a larger sample size, keepingthe others constant (Kline, 2011). Values less

than 0.10 indicate a good fit to the data and

values less than 0.05 indicate a very good fit

(Schumacker and Lomax, 2004).

In SEM, hypothesis testing consists ofconfirming that a theoretical specified modelfits sample variance-covariance data, bytesting the significance of structuralcoefficients or testing the equality ofcoefficients between groups (Schumacker and

Lomax, 2004). The initial model represents thenull hypothesis (Ho) and the final model


5/10


6/10

conducted in Turkish Social SciencesDatabase, available at the Turkish AcademicNetwork and Information Centre (ULAKBIM).The journals included in the database provideaccess to the national literature in the field ofsocial sciences, and they are evaluated byULAKBIM database committee according tothe ‘Journal Evaluation Criteria’ conforming tothe international standards.

There were 176 journals included in thisdatabase, which covered journals from 18different subject matters. For the purpose ofthis study, only journals related with educationfield were considered. There were 34 journalsin this field. These journals were searched forterms such as ‘structural’, ‘structural model’,

‘model’, and ‘structural equation modeling’.Through this process, more than 80 studieswere identified for possible consideration.However, after a full text review, only 12research studies, published from 2007 to2010, were found suitable for the purpose ofthis study, as they were the only studiesconducted with structural equation modeling

technique.

Characteristics of Selected Studies

The descriptive analyses of the selectedstudies were summarized in Table 2.

Among these studies, two studies werepublished in 2007, three were published in2008, four were published in 2009, and threewere published in 2010. Moreover, 11


7/10

40 Educ. Res. J.

Table 3.Research Variables used for Calculating Power of SEM Studies

Study df N α ε 0 ε a Power

Study A 1 695 0.05 0.00 0.000 0.05

Study B 18 646 0.05 0.00 0.056 0.98

Study C 415 218 0.05 0.00 0.030 0.83

Study D 107 642 0.05 0.00 0.100 1.00

Study E 19 7841 0.05 0.00 0.082 1.00

Study F 18 755 0.05 0.00 0.029 0.49

Study G 2 7841 0.05 0.00 0.050 1.00

Study H 68 17647 0.05 0.00 0.057 0.92

Study I 844 730 0.05 0.00 0.044 1.00

Study J 4 310 0.05 0.00 0.090 0.72

Study K 284 7841 0.05 0.00 0.052 1.00Study L 27 357 0.05 0.00 0.027 0.23

studies were published in Social Science CitationIndex (SSCI), and two were published only innational index. Regarding the samplecharacteristic, 6 studies were conducted withundergraduate students, 1 was conducted withhigh school students, 4 were conducted with

elementary school students, and 1 wasconducted with teachers. The sample size rangedbetween 218 and 17,647. In addition, thesestudies were conducted on many different subjectareas, including educational sciences,educational psychology, elementary education,secondary education, and technology education.

Meta-Analytic Procedure

MacCallum, Brown and Sugawara (1996)suggested method to calculate power in order tomeasure the fit of structural equation modelsbased upon different fit indices, such as RMSEAfit index. According to this method, in order toperform a statistical power analysis, five factorswere needed to be taken into consideration.These factors were degrees of freedom (df),significance level (α), sample size (N), the nullvalue of RMSEA (ε 0), and the alternative valueof RMSEA (ε a). Statistically in power analysis,the difference betweenε 0 andε a refers

to the effect size, which is conceptualized as thedegree to which Ho is incorrect.For this study, the null RMSEA value of eachstudy was taken asε 0 = 0.00 for the test of exactfit, as suggested by MacCallum et al. The otherresearch variables were obtained from the resultssections of each study or from the researcherspersonally if the values were not


8/10

indicated in the original papers. Then, statisticalprocedures were administered by StatisticalAnalysis System (SAS) program and R routines,which are free software environments forstatistical computing and graphics. In particular,df,α, N,ε 0, andε a values were entered in theSAS program. So, the program provided a codefor each set of variables. Then, this code was

inserted in R routines, which gave the exact valueof achieved power for each study.

RESULTS

Table 3 shows the research variables used forpower analysis of each study. The result of poweranalysis revealed that 25% of the selectedstudies achieved power less than 0.50.

Particularly, one study had a power of 0.05 (df =1,α = 0.05, N=695, andε a= 0.00), one studyhad a power of 0.23 (df = 27,α = 0.05, N=357,andε a = 0.027), and one study had a power of 0.49(df = 18,

α = 0.05, N=755, andε a = 0.029). Among thesethree studies, one had a very small degrees offreedom (df = 1), one had a small sample size(just around 300), and one had a very small

RMSEA value (ε a = 0.029).Further, among the other studies, one had apower of 0.72 (df = 4,α = 0.05, N=310, andε a =0.090), which was less than Cohen’s (1992)conventions for an ideal power of 0.80.Nevertheless, the other studies attained powerlarger than 0.80, indicating good approximate fits.Specially, five studies had full power of 1.00.Among these five studies, three had largeamount of degrees of freedom (Study D, I, andK), and two had large sample

size (Study E and G). During this meta-analysis,none of the examined studies were found tomention about the power issue of their structuralequation models.

FADLELMULA41

indices, they achieved highly different levels ofpower. In brief, power was low with small degreesof freedom, sample size and RMSEA indices.

DISCUSSION AND CONCLUSION

This study employed a meta-analytic procedureto examine the statistical power of SEM studies

published in the Turkish Social SciencesDatabase. After reviewing 34 journals in theeducation field, 12 studies, published from 2007to 2010, were found suitable for researchconsideration. Power was calculated by SASprogram and R routines, using the indicateddegrees of freedom, significance level, samplesize, and root-mean-square error ofapproximation indices. The results revealed thatone third of the selected studies achieved powerless than 0.80. Specially, three of them achievedpower less than 0.50.

Statistically in SEM studies, achieving a lowpower implies that if the researcher’s model isfalse, then the probability of detecting thisspecification error is low (Kline, 2011). Thismeans that there is a high possibility that theresearcher accepts a false model as significant,and s/he is unlikely to detect that that this modelis actually false. On the other hand, achieving a

high power implies a higher probability ofdetecting a reasonably correct model (Kline,2011). Therefore, achieving low power can beregarded as a highly serious problem as itcompromises the scientific value of the research

study. Especially, considering the fact thatrejecting a false null hypothesis with a power of0.50 is the same as guessing the outcome of acoin toss, the results of these studies can beregarded as having very small practicalsignificance. In these cases, tossing a coin wouldbe more beneficial as it saves time and money(Schmidt and Hunter, 1997).

Furthermore, the results of this study alsoindicated how power of SEM studies varied as afunction of degrees of freedom, sample size, and

RMSEA indices. For instance, power was lowwhen there were small degrees of freedom, evenfor a reasonably large sample size or RMSEAvalue. Hence, for models with only one or twodegrees of freedom, sample sizes in thousandswere required for power to be greater than 0.80.Like Study G, which had very small degrees offreedom, sample size over 5000 was required forachieving a high level of power. On the otherhand, for Study C, Study D, and Study I, which


9/10

had large degrees of freedom, powers werereasonably high, even for sample sizes around500.

Similarly, power was low when the differencebetween the null and alternative value of RMSEA

was small, evenfor large sample sizes or degreesof freedom. For instance, Study B and Study Fhad similar amount of degrees of freedom and

sample size. However, due to the differencesbetween the null and alternative RMSEA

IMPLICATIONS

Power is a critical issue in designing and planningresearch studies. Especially for SEM studies,greater power implies a higher probability ofdetecting a reasonably correct model (Kline,2011). This is why; getting a significant result

from model testing is not enough to indicate thatthe model fit is adequate. Indeed, beforetentatively concluding that the model is fit,researchers need to pay special attention topower level of their studies. In this study, powerwas not issued in any of analyzed studies.

In the literature, there are a number of methodsfor estimating power of structural equationmodels, including those of Saris and Satorra’method (1993), MacCallum, Browne, andSugawara (1996), and Kim (2005). No matter

which method is implemented, researchers needto estimate the power value of their study and toensure that the predetermined power is at least0.80 (Cohen, 1992). For studies with power wellbelow this desired level, researchers need toeither change the conditions under which theresearch would be conducted or postpone thestudy until a reasonably high sample size isreached.

REFERENCES

Akın A (2008). Self-compassion and achievementgoals: A structural equation modeling approach.Eğitim Ara tırmaları, 31, 1-15.

Akın A (2009). Self-compassion and submissivebehavior. Eğitim ve Bilim, 34(152), 138-147.

Arıcak OT (2009). Psychiatric symptomatology as apredictor of cyberbullying among university students.Eğitim Ara tırmaları, 34, 167-184.

Aslan S, Güven M (2010). Bağlanma ve ki isel uyumarasındaki ili kide ayrı ma bireyle menin aracılığı.

Eğitim ve Bilim, 35(157), 181-191.Bollen KA, Long J S (Eds.). (1993). Testing structuralequation models. Newbury Park, CA: Sage.

Ceylan E, Berberoğlu G (2007). Öğrencilerin fen baarısını açıklayan etmenler: Bir modelleme çalı ması.Eğitim ve Bilim, 32(144): 36-48.

Cohen, J. (1992). Quantitative methods in psychology:A power primer. Psychological Bulletin, 112(1): 155-159.

Çetin B, Gündüz HB, Akın A (2008). An investigation of

the relationships between self-compassion,


10/10

motivation, and burnout with structural equation

modeling. Abantİzzet Baysal Üniversitesi Eğitim

Fakültesi Dergisi, 8(2):39-45.

Demir K (2008). Transformational leadership and

collective efficacy: The moderating roles of

collaborative culture

42 Educ. Res. J.

and teachers’ self-efficacy. Eğitim Ara tırmaları, 33:

93-112.

Ha laman T, A kar P.(2007). Programlama dersi ile ilgili

özdüzenleyici öğrenme stratejileri ve ba arı

arasındaki ili kinin incelenmesi. Hacettepe

Üniversitesi Eğitim Fakültesi Dergisi, 32: 110-122.

Jöreskog KG, Sörbom D (1993). LISREL 8: Structuralequation modeling with the SIMPLIS commandlanguage. Chicago: Scientific Software International.

Karadağ E (2009). Ruhsal liderlik ve örgüt kültürü : Biryapısal e itlik modelleme çalı ması. Kuram veUygulamada Eğitim Bilimleri, 9(3): 1357-1405.

Kelloway EK (1998). Using LISREL for structural

equation modeling: A researcher’s guide. ThousandOaks: Sage Publications.

Kim KH (2005). The relation among fit indexes, power,and sample size in structural equation modeling.Structural Equation Modeling, 12: 368–390.

Kline RB (2011). Principles and practice of structural

equation modeling (3rd ed.). New York: The Guilford

Press.

Kunnan AJ (1998). An introduction to structuralequation modeling for language assessmentresearch. Language Testing, 15(3): 295-332.

MacCallum RC, Browne MW, Sugawara HM (1996).Power analysis and determination of sample size forcovariance structure modeling. PsychologicalMethods, 1(2):130-149.

Pedhazur EJ, Pedhazur-Schmelkin LP (1991).Measurement, design, and analysis: An integratedapproach. Hillsdale, NJ: Lawrence EarlbawmAssociate.

Raykov T, Marcoulides GA (2006). A first course instructural equation modeling (2nd Edition). LawrenceErlbaum Associates, Inc. Publishers.

Sap M (2006). Basic psychological measurement,

research designs, and statistics without math.Springfield, IL: Charles C. Thomas.

Saris WE, Satorra A (1993). Power evaluations in

structural equation models. In Bollen KA, LongJS

(Eds.), Testing structural equation models (pp. 181–

204). Newbury Park, CA: Sage.

Schumacker RE, Lomax RG (2004). A beginner’sguide to structural equation modeling. Mahwah NJ:Lawrence Erlbaum.

Schmidt FL, Hunter JE (1997). Eight common but falseobjections to the discontinuation of significancetesting in the analysis of research data. In HarlowLL, Mulaik SA, JH Steiger (Eds.), What if there wereno significance tests? Mahwah, NJ: Erlbaum. pp.37–64.

ẞim ek GölbaZ G, Noyan F (2009). The effect ofperceived instructional effectiveness on studentloyalty: A multilevel structural equation model.Hacettepe Üniversitesi Eğitim Fakültesi Dergisi,36;109-118.

Uzun B, Öğretmen T (2010). Fen ba arısı ile ilgili bazı

deği kenlerin TIMSS-R Türkiye örneklemindecinsiyete göre ölçme deği mezliğinindeğerlendirilmesi. Eğitim ve Bilim, 35(155), 26-35.

Uzun NB, Gelbal S, Öğretmen T (2010). TIMSS-R fenba arısı ve duyu sal özellikler arasındaki ili kininmodellenmesi ve modelin cinsiyetler bakımından karıla tırılması. Kastamonu Üniversitesi KastamonuEğitim Fakültesi Dergisi, 18(2): 531-544.

Yavuz M (2009). Ortaöğretim kurumları öğrenci seçme

ve yerle tirme sınavında öğrencilerin Matematik-Fen (

MF ) puanlarını etkilediği dü ünülen bazı faktörlerin

yapısal e itlik modeli ile incelenmesi. Kuram ve

Uygulamada Eğitim Bilimleri, 9(3): 1543-1572.

2011 Kayan, Assessing Power of Structural Equation Modeling WORD

Documents

Transcript of 2011 Kayan, Assessing Power of Structural Equation Modeling WORD