Psbe2 08 research methods 2011-2012 - week 2
-
Upload
vlady-fckfb -
Category
Technology
-
view
157 -
download
1
Transcript of Psbe2 08 research methods 2011-2012 - week 2
PSBE2-08
Research Methods
Week 2
1Tassos Sarampalis
Null Hypothesis Significance
Testingand
Power
2
Null Hypothesis Significance
Testing
• Goal– determine whether mean differences among groups
in an experiment are greater than differences expected simply because of chance (error variation)
• First step– assume that the groups do not differ (H0)
• = null hypothesis
• assume the independent variable did not have an effect
3
Null Hypothesis Significance
Testing
• Next steps
– Probability theory: estimate likelihood of observed outcome, while assuming null hypothesis is true.
– “statistically significant”
• outcome has small likelihood of occurring under H0
• reject H0
• conclude IV had an effect on DV– difference between means is larger than what would be expected
if error variation alone caused the outcome
4
5
6
pro
ba
bili
ty
0 2 4 6 8 10 12 14 16 18
“heads” count
7
pro
ba
bili
ty
0 2 4 6 8 10 12 14 16 18
“heads” count
Null Hypothesis Significance
Testing
• How small does the likelihood have to be to decide outcome isn’t due to chance?
• scientific consensus: p < .05
• = alpha (α) or level of significance
• What does a statistically significant outcome tell us?– outcome at p ≈ .05 has about a 50/50 chance of being repeated
(at p < .05) in an exact replication
– as probability of outcome decreases (e.g., p = .025, p = .01), likelihood of observing a statistically significant outcome (p < .05) in an exact replication increases
– APA recommends reporting exact probability of outcome
8
9
Null Hypothesis Significance
Testing
• What do we conclude when a finding is notstatistically significant?
– do not reject the null hypothesis of no difference
– don’t accept the null hypothesis
• don’t conclude that the IV didn’t produce an effect
– cannot make a conclusion about the effect of an IV
• some factor in experiment may have prevented us from observing an effect of the IV
• most common factor: too few participants
10
NHST Criticisms
• A difference between populations can almost always be found, given a large enough sample
• A statistically significant finding may not be relevant in practice, whilst a true effect of practical significance may not appear statistically significant if the test lacks the power
• Fairness of exclusion
• Publication bias and the file-drawer problem
11
Experimental Sensitivity and
Power
• Sensitivity
– likelihood an experiment will detect the effect of an IV when in fact, the IV has an effect
• affected by experiment methods and procedures
• sensitivity increases with good research design and methods
– high degree of experimental control
– little opportunity for biases
12
Experimental Sensitivity and
Power
• Power
– likelihood that a statistical test will allow researchers to reject correctly H0
• low statistical power increases Type II errors
• Power = 1 - β
• three factors affect power of statistical tests– level of significance (alpha)
– size of the effect of the IV
– sample size (N)
13
Experimental Sensitivity and
Power
• Prospective Power Analysis• step 1: estimate effect size of IV
– examine previous research involving the IV
• step 2: refer to “Power Tables”– identify sample size needed to observe effect of IV
• step 3: use adequate sample size– most studies in psychology are “underpowered” because of
low sample size
• Retrospective Power Analysis• Determine the power of a study based on the effect
size, sample size, and significance level
14