Advanced Methods and Analysis for the Learning and Social Sciences
description
Transcript of Advanced Methods and Analysis for the Learning and Social Sciences
![Page 1: Advanced Methods and Analysis for the Learning and Social Sciences](https://reader035.fdocuments.us/reader035/viewer/2022070421/56816258550346895dd2abb4/html5/thumbnails/1.jpg)
Advanced Methods and Analysis for the Learning and Social Sciences
PSY505Spring term, 2012
April 18, 2012
![Page 2: Advanced Methods and Analysis for the Learning and Social Sciences](https://reader035.fdocuments.us/reader035/viewer/2022070421/56816258550346895dd2abb4/html5/thumbnails/2.jpg)
Today’s Class
• Post-hoc Adjustments
![Page 3: Advanced Methods and Analysis for the Learning and Social Sciences](https://reader035.fdocuments.us/reader035/viewer/2022070421/56816258550346895dd2abb4/html5/thumbnails/3.jpg)
The Problem
• If you run 20 statistical tests, you get a statistically significant effect in one of them
• If you report that effect in isolation, as if it were significant, you add junk to the open literature
![Page 4: Advanced Methods and Analysis for the Learning and Social Sciences](https://reader035.fdocuments.us/reader035/viewer/2022070421/56816258550346895dd2abb4/html5/thumbnails/4.jpg)
The Problem
• To illustrate this, let’s run a simulation a few times, and do a probability estimation
• spurious-effect-v1.xlsx
![Page 5: Advanced Methods and Analysis for the Learning and Social Sciences](https://reader035.fdocuments.us/reader035/viewer/2022070421/56816258550346895dd2abb4/html5/thumbnails/5.jpg)
The Problem
• Comes from the paradigm of conducting a single statistical significance test
• How many papers have just one statistical significance test?
• How big is the risk if you run two tests, or eight tests?– Back to the simulation!
![Page 6: Advanced Methods and Analysis for the Learning and Social Sciences](https://reader035.fdocuments.us/reader035/viewer/2022070421/56816258550346895dd2abb4/html5/thumbnails/6.jpg)
The Solution
• Adjust for the probability that your results are due to chance, using a post-hoc control
![Page 7: Advanced Methods and Analysis for the Learning and Social Sciences](https://reader035.fdocuments.us/reader035/viewer/2022070421/56816258550346895dd2abb4/html5/thumbnails/7.jpg)
Two paradigms
• FWER – Familywise Error Rate– Control for the probability that any of your tests
are falsely claimed to be significant (Type I Error)
• FDR – False Discovery Rate– Control for the overall rate of false discoveries
![Page 8: Advanced Methods and Analysis for the Learning and Social Sciences](https://reader035.fdocuments.us/reader035/viewer/2022070421/56816258550346895dd2abb4/html5/thumbnails/8.jpg)
Bonferroni Correction
![Page 9: Advanced Methods and Analysis for the Learning and Social Sciences](https://reader035.fdocuments.us/reader035/viewer/2022070421/56816258550346895dd2abb4/html5/thumbnails/9.jpg)
Bonferroni Correction
• Ironically, derived by Miller rather than Bonferroni
![Page 10: Advanced Methods and Analysis for the Learning and Social Sciences](https://reader035.fdocuments.us/reader035/viewer/2022070421/56816258550346895dd2abb4/html5/thumbnails/10.jpg)
Bonferroni Correction
• Ironically, derived by Miller rather than Bonferroni
• Also ironically, there appear to be no pictures of Miller on the internet
![Page 11: Advanced Methods and Analysis for the Learning and Social Sciences](https://reader035.fdocuments.us/reader035/viewer/2022070421/56816258550346895dd2abb4/html5/thumbnails/11.jpg)
Bonferroni Correction
• A classic example of Stigler’s Law of Eponomy– “No scientific discovery is named after its original
discoverer”
![Page 12: Advanced Methods and Analysis for the Learning and Social Sciences](https://reader035.fdocuments.us/reader035/viewer/2022070421/56816258550346895dd2abb4/html5/thumbnails/12.jpg)
Bonferroni Correction
• A classic example of Stigler’s Law of Eponomy– “No scientific discovery is named after its original
discoverer”– Stigler’s Law of Eponomy proposed by
Robert Merton
![Page 13: Advanced Methods and Analysis for the Learning and Social Sciences](https://reader035.fdocuments.us/reader035/viewer/2022070421/56816258550346895dd2abb4/html5/thumbnails/13.jpg)
Bonferroni Correction
• If you are conducting n different statistical tests on the same data set
• Adjust your significance criterion a to be– a / n
• E.g. For 4 statistical tests, use statistical significance criterion of 0.0125 rather than 0.05
![Page 14: Advanced Methods and Analysis for the Learning and Social Sciences](https://reader035.fdocuments.us/reader035/viewer/2022070421/56816258550346895dd2abb4/html5/thumbnails/14.jpg)
Bonferroni Correction
• Sometimes instead expressed by multiplying p * n, and keeping statistical significance criterion a = 0.05
• Mathematically equivalent…• As long as you don’t try to treat p like a probability
afterwards… or meta-analyze it… etc., etc.• For one thing, can produce p values over 1, which doesn’t
really make sense
![Page 15: Advanced Methods and Analysis for the Learning and Social Sciences](https://reader035.fdocuments.us/reader035/viewer/2022070421/56816258550346895dd2abb4/html5/thumbnails/15.jpg)
Bonferroni Correction: Example
• Five tests– p=0.04, p=0.12, p=0.18, p=0.33, p=0.55
• Five corrections– All p compared to a= 0.01– None significant anymore– p=0.04 seen as being due to chance
![Page 16: Advanced Methods and Analysis for the Learning and Social Sciences](https://reader035.fdocuments.us/reader035/viewer/2022070421/56816258550346895dd2abb4/html5/thumbnails/16.jpg)
Bonferroni Correction: Example
• Five tests– p=0.04, p=0.12, p=0.18, p=0.33, p=0.55
• Five corrections– All p compared to a= 0.01– None significant anymore– p=0.04 seen as being due to chance– Does this seem right?
![Page 17: Advanced Methods and Analysis for the Learning and Social Sciences](https://reader035.fdocuments.us/reader035/viewer/2022070421/56816258550346895dd2abb4/html5/thumbnails/17.jpg)
Bonferroni Correction: Example
• Five tests– p=0.001, p=0.011, p=0.02, p=0.03, p=0.04
• Five corrections– All p compared to a= 0.01– Only p=0.001 still significant
![Page 18: Advanced Methods and Analysis for the Learning and Social Sciences](https://reader035.fdocuments.us/reader035/viewer/2022070421/56816258550346895dd2abb4/html5/thumbnails/18.jpg)
Bonferroni Correction: Example
• Five tests– p=0.001, p=0.011, p=0.02, p=0.03, p=0.04
• Five corrections– All p compared to a= 0.01– Only p=0.001 still significant– Does this seem right?
![Page 19: Advanced Methods and Analysis for the Learning and Social Sciences](https://reader035.fdocuments.us/reader035/viewer/2022070421/56816258550346895dd2abb4/html5/thumbnails/19.jpg)
Bonferroni Correction
• Advantages
• Disadvantages
![Page 20: Advanced Methods and Analysis for the Learning and Social Sciences](https://reader035.fdocuments.us/reader035/viewer/2022070421/56816258550346895dd2abb4/html5/thumbnails/20.jpg)
Bonferroni Correction
• Advantages• You can be “certain” that an effect is real if it makes it
through this correction• Does not assume tests are independent (in the same data
set, they probably aren’t!)
• Disadvantages• Massively over-conservative• Essentially throws out every effect if you run a lot of tests
![Page 21: Advanced Methods and Analysis for the Learning and Social Sciences](https://reader035.fdocuments.us/reader035/viewer/2022070421/56816258550346895dd2abb4/html5/thumbnails/21.jpg)
Often attacked these days
• Arguments for rejecting the sequential Bonferroni in ecological studies. MD Moran - Oikos, 2003 - JSTOR
• Beyond Bonferroni: less conservative analyses for conservation genetics. SR Narum - Conservation Genetics, 2006 – Springer
• What's wrong with Bonferroni adjustments. TV Perneger - Bmj, 1998 - bmj.com
• p Value fetishism and use of the Bonferroni adjustment. JF Morgan - Evidence Based Mental Health, 2007
![Page 22: Advanced Methods and Analysis for the Learning and Social Sciences](https://reader035.fdocuments.us/reader035/viewer/2022070421/56816258550346895dd2abb4/html5/thumbnails/22.jpg)
Holm Correction
• Also called Holm-Bonferroni Correction• And the Simple Sequentially Rejective Multiple
Test Procedure• And Holm’s Step-Down• And the Sequential Bonferroni Procedure
![Page 23: Advanced Methods and Analysis for the Learning and Social Sciences](https://reader035.fdocuments.us/reader035/viewer/2022070421/56816258550346895dd2abb4/html5/thumbnails/23.jpg)
Holm Correction
• Order your n tests from most significant (lowest p) to least significant (highest p)
• Test your first test according to significance criterion a / n
• Test your second test according to significance criterion a / (n-1)
• Test your third test according to significance criterion a / (n-2)
• Quit as soon as a test is not significant
![Page 24: Advanced Methods and Analysis for the Learning and Social Sciences](https://reader035.fdocuments.us/reader035/viewer/2022070421/56816258550346895dd2abb4/html5/thumbnails/24.jpg)
Holm Correction: Example
• Five tests– p=0.001, p=0.01, p=0.02, p=0.03, p=0.04
![Page 25: Advanced Methods and Analysis for the Learning and Social Sciences](https://reader035.fdocuments.us/reader035/viewer/2022070421/56816258550346895dd2abb4/html5/thumbnails/25.jpg)
Holm Correction: Example
• Five tests– p=0.001, p=0.011, p=0.02, p=0.03, p=0.04
• First correction– p = 0.001 compared to a= 0.01– Still significant!
![Page 26: Advanced Methods and Analysis for the Learning and Social Sciences](https://reader035.fdocuments.us/reader035/viewer/2022070421/56816258550346895dd2abb4/html5/thumbnails/26.jpg)
Holm Correction: Example
• Five tests– p=0.001, p=0.011, p=0.02, p=0.03, p=0.04
• Second correction– p = 0.011 compared to a= 0.0125– Still significant!
![Page 27: Advanced Methods and Analysis for the Learning and Social Sciences](https://reader035.fdocuments.us/reader035/viewer/2022070421/56816258550346895dd2abb4/html5/thumbnails/27.jpg)
Holm Correction: Example
• Five tests– p=0.001, p=0.011, p=0.02, p=0.03, p=0.04
• Third correction– p = 0.02 compared to a= 0.0166– Not significant
![Page 28: Advanced Methods and Analysis for the Learning and Social Sciences](https://reader035.fdocuments.us/reader035/viewer/2022070421/56816258550346895dd2abb4/html5/thumbnails/28.jpg)
Holm Correction: Example
• Five tests– p=0.001, p=0.011, p=0.02, p=0.03, p=0.04
• Third correction– p = 0.02 compared to a= 0.0166– Not significant– p=0.03 and p=0.04 not tested
![Page 29: Advanced Methods and Analysis for the Learning and Social Sciences](https://reader035.fdocuments.us/reader035/viewer/2022070421/56816258550346895dd2abb4/html5/thumbnails/29.jpg)
Less Conservative
• p=0.011 no longer seen as not statistically significant
• But p=0.02, p=0.03, p=0.04 still discarded
• Does this seem right?
![Page 30: Advanced Methods and Analysis for the Learning and Social Sciences](https://reader035.fdocuments.us/reader035/viewer/2022070421/56816258550346895dd2abb4/html5/thumbnails/30.jpg)
Tukey’s Honestly Significant Difference (HSD)
![Page 31: Advanced Methods and Analysis for the Learning and Social Sciences](https://reader035.fdocuments.us/reader035/viewer/2022070421/56816258550346895dd2abb4/html5/thumbnails/31.jpg)
Tukey’s HSD
• Method for conducting post-hoc correction on ANOVA
• Typically used to assess significance of pair-wise comparisons, after conducting omnibus test– E.g. We know there is an overall effect in our
scaffolding * agent 2x2 comparison– Now we can ask is Scaffolding+Agent better than
Scaffolding + ~Agent, etc. etc.
![Page 32: Advanced Methods and Analysis for the Learning and Social Sciences](https://reader035.fdocuments.us/reader035/viewer/2022070421/56816258550346895dd2abb4/html5/thumbnails/32.jpg)
Tukey’s HSD
• The t distribution is adjusted such that the number of means tested on is taken into account
• Effectively, the critical value for t goes up with the square root of the number of means tested on
• E.g. for 2x2 = 4 means, critical t needed is double• E.g. for 3x3 = 9 means, critical t needed is triple
![Page 33: Advanced Methods and Analysis for the Learning and Social Sciences](https://reader035.fdocuments.us/reader035/viewer/2022070421/56816258550346895dd2abb4/html5/thumbnails/33.jpg)
Tukey’s HSD
• Not quite as over-conservative as Bonferroni, but errs in the same fashion
![Page 34: Advanced Methods and Analysis for the Learning and Social Sciences](https://reader035.fdocuments.us/reader035/viewer/2022070421/56816258550346895dd2abb4/html5/thumbnails/34.jpg)
Other FWER Corrections
• Sidak Correction– Less conservative than Bonferroni– Assumes independence between tests– Often an undesirable assumption
• Hochberg’s Procedure/Simes Procedure– Corrects for number of expected true hypotheses
rather than total number of tests– Led in the direction of FDR
![Page 35: Advanced Methods and Analysis for the Learning and Social Sciences](https://reader035.fdocuments.us/reader035/viewer/2022070421/56816258550346895dd2abb4/html5/thumbnails/35.jpg)
Questions? Comments?
![Page 36: Advanced Methods and Analysis for the Learning and Social Sciences](https://reader035.fdocuments.us/reader035/viewer/2022070421/56816258550346895dd2abb4/html5/thumbnails/36.jpg)
FDR Correction
![Page 37: Advanced Methods and Analysis for the Learning and Social Sciences](https://reader035.fdocuments.us/reader035/viewer/2022070421/56816258550346895dd2abb4/html5/thumbnails/37.jpg)
FDR Correction
• Different paradigm, probably a better match to the original conception of statistical significance
![Page 38: Advanced Methods and Analysis for the Learning and Social Sciences](https://reader035.fdocuments.us/reader035/viewer/2022070421/56816258550346895dd2abb4/html5/thumbnails/38.jpg)
Comparison of Paradigms
![Page 39: Advanced Methods and Analysis for the Learning and Social Sciences](https://reader035.fdocuments.us/reader035/viewer/2022070421/56816258550346895dd2abb4/html5/thumbnails/39.jpg)
Statistical significance
• p<0.05
• A test is treated as rejecting the null hypothesis if there is a probability of under 5% that the results could have occurred if there were only random events going on
• This paradigm accepts from the beginning that we will accept junk (e.g. Type I error) 5% of the time
![Page 40: Advanced Methods and Analysis for the Learning and Social Sciences](https://reader035.fdocuments.us/reader035/viewer/2022070421/56816258550346895dd2abb4/html5/thumbnails/40.jpg)
FWER Correction
• p<0.05
• Each test is treated as rejecting the null hypothesis if there is a probability of under 5% divided by N that the results could have occurred if there were only random events going on
• This paradigm accepts junk far less than 5% of the time
![Page 41: Advanced Methods and Analysis for the Learning and Social Sciences](https://reader035.fdocuments.us/reader035/viewer/2022070421/56816258550346895dd2abb4/html5/thumbnails/41.jpg)
FDR Correction
• p<0.05
• Across tests, we will attempt to accept junk exactly 5% of the time– Same degree of conservatism as the original
conception of statistical significance
![Page 42: Advanced Methods and Analysis for the Learning and Social Sciences](https://reader035.fdocuments.us/reader035/viewer/2022070421/56816258550346895dd2abb4/html5/thumbnails/42.jpg)
Example
• Twenty tests, all p=0.05
• Bonferroni rejects all of them as non-significant
• FDR notes that we should have had 1 fake significant, and 20 significant results is a lot more than 1
![Page 43: Advanced Methods and Analysis for the Learning and Social Sciences](https://reader035.fdocuments.us/reader035/viewer/2022070421/56816258550346895dd2abb4/html5/thumbnails/43.jpg)
FDR Procedure
• Order your n tests from most significant (lowest p) to least significant (highest p)
• Test your first test according to significance criterion a*1 / n
• Test your second test according to significance criterion a*2 / n
• Test your third test according to significance criterion a*3 / n
• Quit as soon as a test is not significant
![Page 44: Advanced Methods and Analysis for the Learning and Social Sciences](https://reader035.fdocuments.us/reader035/viewer/2022070421/56816258550346895dd2abb4/html5/thumbnails/44.jpg)
FDR Correction: Example
• Five tests– p=0.001, p=0.011, p=0.02, p=0.03, p=0.04
• First correction– p = 0.001 compared to a= 0.01– Still significant!
![Page 45: Advanced Methods and Analysis for the Learning and Social Sciences](https://reader035.fdocuments.us/reader035/viewer/2022070421/56816258550346895dd2abb4/html5/thumbnails/45.jpg)
FDR Correction: Example
• Five tests– p=0.001, p=0.011, p=0.02, p=0.03, p=0.04
• Second correction– p = 0.011 compared to a= 0.02– Still significant!
![Page 46: Advanced Methods and Analysis for the Learning and Social Sciences](https://reader035.fdocuments.us/reader035/viewer/2022070421/56816258550346895dd2abb4/html5/thumbnails/46.jpg)
FDR Correction: Example
• Five tests– p=0.001, p=0.011, p=0.02, p=0.03, p=0.04
• Third correction– p = 0.02 compared to a= 0.03– Still significant!
![Page 47: Advanced Methods and Analysis for the Learning and Social Sciences](https://reader035.fdocuments.us/reader035/viewer/2022070421/56816258550346895dd2abb4/html5/thumbnails/47.jpg)
FDR Correction: Example
• Five tests– p=0.001, p=0.011, p=0.02, p=0.03, p=0.04
• Fourth correction– p = 0.03 compared to a= 0.04– Still significant!
![Page 48: Advanced Methods and Analysis for the Learning and Social Sciences](https://reader035.fdocuments.us/reader035/viewer/2022070421/56816258550346895dd2abb4/html5/thumbnails/48.jpg)
FDR Correction: Example
• Five tests– p=0.001, p=0.011, p=0.02, p=0.03, p=0.04
• Fifth correction– p = 0.04 compared to a= 0.05– Still significant!
![Page 49: Advanced Methods and Analysis for the Learning and Social Sciences](https://reader035.fdocuments.us/reader035/viewer/2022070421/56816258550346895dd2abb4/html5/thumbnails/49.jpg)
FDR Correction: Example
• Five tests– p=0.04, p=0.12, p=0.18, p=0.33, p=0.55
• First correction– p = 0.04 compared to a= 0.01– Not significant; stop
![Page 50: Advanced Methods and Analysis for the Learning and Social Sciences](https://reader035.fdocuments.us/reader035/viewer/2022070421/56816258550346895dd2abb4/html5/thumbnails/50.jpg)
How do these results compare
• To Bonferroni Correction
• To Holm Correction
• To just accepting p<0.05, no matter how many tests are run
![Page 51: Advanced Methods and Analysis for the Learning and Social Sciences](https://reader035.fdocuments.us/reader035/viewer/2022070421/56816258550346895dd2abb4/html5/thumbnails/51.jpg)
q value extension in FDR(Storey, 2002)
• Remember how Bonferroni Adjustments could either be made by adjusting a or adjusting p?
• There is a similar approach in FDR, where p values are transformed to q values
• Unlike in Bonferroni, q values can be interpreted the same way as p values
![Page 52: Advanced Methods and Analysis for the Learning and Social Sciences](https://reader035.fdocuments.us/reader035/viewer/2022070421/56816258550346895dd2abb4/html5/thumbnails/52.jpg)
q value extension in FDR(Storey, 2002)
• p = probability that the results could have occurred if there were only random events going on
• q = probability that the current test is a false discovery, given the post-hoc adjustment
![Page 53: Advanced Methods and Analysis for the Learning and Social Sciences](https://reader035.fdocuments.us/reader035/viewer/2022070421/56816258550346895dd2abb4/html5/thumbnails/53.jpg)
q value extension in FDR(Storey, 2002)
• q can actually be lower than p
• In the relatively unusual case where there are many statistically significant results
![Page 54: Advanced Methods and Analysis for the Learning and Social Sciences](https://reader035.fdocuments.us/reader035/viewer/2022070421/56816258550346895dd2abb4/html5/thumbnails/54.jpg)
Questions? Comments?
![Page 55: Advanced Methods and Analysis for the Learning and Social Sciences](https://reader035.fdocuments.us/reader035/viewer/2022070421/56816258550346895dd2abb4/html5/thumbnails/55.jpg)
Asgn. 10
• Kim, can you please go over your solution?
![Page 56: Advanced Methods and Analysis for the Learning and Social Sciences](https://reader035.fdocuments.us/reader035/viewer/2022070421/56816258550346895dd2abb4/html5/thumbnails/56.jpg)
Asgn. 11
• No assignment 11!
![Page 57: Advanced Methods and Analysis for the Learning and Social Sciences](https://reader035.fdocuments.us/reader035/viewer/2022070421/56816258550346895dd2abb4/html5/thumbnails/57.jpg)
Next Class
• I need to move class next Monday to either Thursday or Friday
• What do folks prefer?
![Page 58: Advanced Methods and Analysis for the Learning and Social Sciences](https://reader035.fdocuments.us/reader035/viewer/2022070421/56816258550346895dd2abb4/html5/thumbnails/58.jpg)
Next Class• Wednesday, April 25• 3pm-5pm• AK232
• Social Network Analysis
• Readings• Haythornthwaite, C. (2001) Exploring Multiplexity: Social Network
Structures in a Computer-Supported Distance Learning Class. The Information Society: An International Journal, 17 (3), 211-226.
• Dawson, S. (2008) A study of the relationship between student social networks and sense of community. Educational Technology & Society, 11(3),224-238.
• Assignments Due: None
![Page 59: Advanced Methods and Analysis for the Learning and Social Sciences](https://reader035.fdocuments.us/reader035/viewer/2022070421/56816258550346895dd2abb4/html5/thumbnails/59.jpg)
The End