Empirically Based Characteristics of Effect Sizes used in ANOVA

J. Jackson Barnette, PhD

Community and Behavioral Health

College of Public Health

University of Iowa

Examine characteristics of four commonly used effect sizes

• Standardized Effect Size

• Measures of Association:

– Eta-Squared– Omega-Squared– Intraclass Correlation Coefficient

Standardized Effect Size

It represents mean differences in units of common population standard deviation.

Population Form Statistic Form

1 – µ2 X1 – X2

= d= s

In practice, the Std. Deviation is typically replaced with the Root Mean Square Error

Standardized Effect Size in ANOVA

Mean Range

Std. Effect Size =

MSE

Cohen’s Standards

Cohen needed to base his research on power

on some effect sizes so he pretty much

arbitrarily chose three values that had been

used extensively as standards for effect sizes:

.2 is a “small effect”

.5 is a “moderate effect”

.8 is a “large effect”

Mean Standardized Effect Size by Sample Size for K=2, 4, and 10

0

0.2

0.4

0.6

0.8

1

1.2

1.4

0 25 50 75 100 125 150 175 200 225 250 275 300 325 350 375 400 425 450 475 500

Sample Size

SE

S

K=2

K=4

K=10

Observed Effect Sizes when K= 2

n= 5, mean= .55, sd= .47, p>.2= .76, p>.5= .44, p>.8= .24

n= 30, mean= .21, sd= .16, p>.2= .44, p>.5= .06, p>.8= .00

n= 60, mean= .15, sd= .11, p>.2= .27, p>.5= .01, p>.8= .00

n=100, mean= .11, sd= .09, p>.2= .16, p>.5= .00, p>.8= .00

Observed Effect Sizes when K= 4

n= 5, mean= .97, sd= .46, p>.2= .99, p>.5= .85, p>.8= .59

n= 30, mean= .38, sd= .16, p>.2= .87, p>.5= .22, p>.8= .01

n= 60, mean= .29, sd= .11, p>.2= .70, p>.5= .03, p>.8= .00

n=100, mean= .21, sd= .09, p>.2= .49, p>.5= .00, p>.8= .00

Observed Effect Sizes when K= 10

n= 5, mean= 1.40, sd= .40, p>.2= 1.00, p>.5= 1.00, p>.8= .96

n= 30, mean= .56, sd= .15, p>.2= 1.00, p>.5= .64, p>.8= .06

n= 60, mean= .40, sd= .10, p>.2= .99, p>.5= .15, p>.8= .00

n=100, mean= .31, sd= .08, p>.2= .92, p>.5= .00, p>.8= .00

Eta-Squared (Pearson and Fisher)

SStreatment

2=

Sstotal

A 2 of .25 would indicate that 25% of the

total variation is accounted for by the

treatment variation.

Eta-Squared

Positives: easy to compute and easy to

interpret.

Negatives: it is more of a descriptive than

inferential statistic, it has a tendency to be

positively biased and chance values are a

function of number and size of samples.

Mean Eta Squared by Sample Size for K= 2, 4, and 10

0

0.02

0.04

0.06

0.08

0.1

0.12

0 25 50 75 100 125 150 175 200 225 250 275 300 325 350 375 400 425 450 475 500

Sample Size

Eta

Sq

. K=2

K=4

K=10

The Bias in Eta-Squared

Mean sampled 2

Sample Size

K 5 30 60 100

2 .110 .017 .008 .005

4 .159 .025 .013 .008

6 .173 .028 .014 .008

8 .180 .029 .015 .009

10 .183 .030 .015 .009

Omega-Squared (Hays, 1963)

When a fixed effect model of ANOVA is used, Haysproposed more of an inferential strength of association measure, referred to as Omega-Squared (2) to specifically reduce the recognized bias in 2.

It provides an estimate of the proportion of variancethat may be attributed to the treatment in a fixeddesign. 2 = .32 means 32% of variance attributed tothe treatment.

Omega-Squared

2 is computed using terms from the ANOVA

SStreatment – ( K – 1) MSerror

2 =

SStotal – MSerror

Mean Omega Squared by Sample Size for K=2, 4, and 10

-0.0005

0

0.0005

0.001

0.0015

0.002

0 50 100 150 200 250 300 350 400 450 500

Sample Size

Om

eg

a S

q.

K=2

K=4

K=10

Omega-Squared

Positives and Negatives (Pun intended) of 2

Positives: it is an inferential statistic that can be usedfor predicting population values, easily computed, itdoes remove much of the bias found in 2.

Negatives: it can have negative values, not just rounding error type, but relatively different than 0.If you get one that is negative, call it zero.

Intraclass Correlation

Omega-squared is used when the independent

variable is fixed. Occasionally, the

independent variable may be “random” in

which case the intraclass correlation is used to

assess strength of association.

Intraclass CorrelationValues to determine the ICC come from the ANOVA.

MStreatment – MSerror

I=

MStreatment + ( n – 1) Mserror

The ICC is a variance-accounted-for statistic,

interpreted in the same way as is Omega-Squared. It

also has the same strengths and weaknesses.

Mean Intraclass Correlation by Sample Size for K=2, 4, and 10

-0.035

-0.03

-0.025

-0.02

-0.015

-0.01

-0.005

0

0.005

0 50 100 150 200 250 300 350 400 450 500

Sample Size

ICC

K=2

K=4

K=10