chapter

7

Statistical Issues in Research Planning and Evaluation

Chapter Outline

• Probability• Meaningfulness• Power• Using information in context of the study• Reporting statistical data

Probability

• What are the odds that a certain thing(s) will happen?– Equally likely events– Relative frequency

• Alpha (α): – Level of chance occurrence set by the researcher

prior to the study– Typical: p < .05 or p < .01

• Odds that findings are due to chance are either 5 or 1 in 100

– Controls for Type I error

Truth Table forthe Null Hypothesis

Ho true Ho false

Accept Correct decision Type II error (beta)

Reject Type I error (alpha)

Correct decision

Probability

• Beta (β):– Magnitude of a Type II error– Acceptance of null hypothesis when it is FALSE

• As alpha is set increasingly smaller, beta becomes larger

Sampling for Null Hypothesis

From Experimental procedures for behavioral science, 3rd ed., by R.E. Kirk © 1995. Reprinted with permission of Brooks/Cole, an imprint of the Wadsworth Group, a division of Thomson Learning. Fax 800-730-2215.

Meaningfulness

• The practical significance of an effect or relationship

• Effect Size (ES)– Standardized value that is the difference between

the means divided by the standard deviation

• Formula:

– ES = (M1 – M2)/s

Estimating Effect Size

• ES allows comparison between studies using different dependent variables because it puts data in standard deviation units.

• Scale:– ES of 0 is no difference,

– ES of 0.2 or less is small,

– ES of ~0.5 is medium, and

– ES of 0.8 or more is large

Power

• The probability of rejecting the null hypothesis when it is false (detecting a real difference), or

• The probability of making a correct decision• This concepts answers the following

questions:– How large a difference is important in theory or

practice?– How many participants are needed to declare an

important difference as significant?

Effect Size Curve to EstimateSample Size When p = .05

Effect Size Curve to EstimateSample Size When p = .01

Using Information in the Context of the Study

• How do findings from the study fit within the context of– Theory– Practice

• Context is what matters with regard to “meaningfulness”– Estimates of significance are driven by sample size– Estimates of meaningfulness are driven by the size

of the difference– Context is driven by how the findings will be used

Summary

• Information needed when planning research1. Alpha

• Establishes the acceptable magnitude of Type I error• Usually .05 or .01

2. Effect size• The outcome of a study expressed in standard deviation

units

3. Power• The chance of rejecting a false null hypothesis• 1 - beta

4. Sample size• Number of participants in the study

Using the Power Calculator WhenReading a Research Study

When reading research, often sample size, means, and standard deviations are supplied. You can calculate the effect size by the formula in chapter 7. Using this data and the Power Calculator at the Web site below, you can estimate the power to detect a difference or relationship.

http://calculators.stat.ucla.edu/powercalc/

Using the Power Calculatorto Plan Research

If you are planning your own research, you can often estimate the effect size from other studies. By setting your alpha (say .05) and power (say .8), you can use the Power Calculator at the website below to estimate the sample size you need.

http://calculators.stat.ucla.edu/powercalc/