How to Lie With Statistics

I'm so confident!• The universe is not one of clockwork• Everything we measure has

uncertainty• How can we confidently draw

conclusions and make decisions without sacrificing our intelligence?

Quantify Your Uncertainty!• Family of intervals to handle this• Confidence• Prediction

• Comprised of• Point estimator • Degree of Confidence• Uncertainty

• Rely on• Underlying distribution (assumption)• Data collection

Formula

𝑃𝑜𝑖𝑛𝑡 𝐸𝑠𝑡𝑖𝑚𝑎𝑡𝑒±𝐶𝑜𝑛𝑓𝑖𝑑𝑒𝑛𝑐𝑒𝐿𝑒𝑣𝑒𝑙∗𝑈𝑛𝑐𝑒𝑟𝑡𝑎𝑖𝑛𝑡𝑦

𝑥 , �̂� , �̂� , �̂� Z*,t*

Margin of Error

𝑠√𝑛,√ 𝑝(1−𝑝)

𝑛 , 𝜎√𝑛

What could go wrong?• Poor data collection• "I talked to my 10 closest friends – Romney has got the election in

the BAG!"• Wrong assumptions• Excessive uncertainty• Unrealistic expectations• Degree of confidence is too high/low

• Wrong type of interval• Prediction versus Confidence

Degree of Confidence• What are we confident in?• The Process

• If we collect data from the same population, over and over and over again, we can expect our interval to contain the true (correct) value x% of thetime where x is ourdegree of confidence(confidence level)

• "The feeling or belief thatone can rely on someoneor something"-- What I got Googling"What is Confidence"

Degree of Confidence• What is confidence NOT?• A probability of any single interval containing the true value• The % of all population values contained in the interval• A product of your self-esteem

Lancet Survey• October 29, 20041 The Lancet published a study on the effects

of the US invasion of Iraq, specifically on the number of additional civilian deaths due to the invasion• The data estimated 98,000 excess deaths• They were 95% confident that the true number of excess deaths

was between 8,000 and 194,000

1 Les, R., Lafta, R., Garfield, R., Khudhairi, J., & Burnham, G. (2004). "Mortality before and after the 2003 invasion of Iraq: Cluster sample survey. The Lancet, 1857-1864.

Lancet Survey• October 11, 20062 The Lancet published a follow-up study• The data now estimated 654,965 excess deaths• They were 95% confident that the true number of excess deaths

was between 392,979 and 942,636• Claimed that they asked for death certificates 87% of the time

and that 92% of the time they asked they were shown a certificate (80% of reported deaths had a certificate to verify the death)• Iraqi Ministry of Health only issued ~50,000 death certificates for

post-war violent deaths

2 Burnham, G., Lafta, R., Doocy, S., & Roberts, L. (2006). Mortality after the 2003 invasion of Iraq: a cross-sectional cluster sample survey. The Lancet, Retrieved from http://www.brussellstribunal.org/pdf/lancet111006.pdf

Statistical Ties• Des Moines Register Iowa Poll Results Published 9/29/2012• Obama 49%• Romney 45%• Other 4%• Undecided 2%• Margin of Error ±3.8%

• Who is winning?• Register claims Obama has a 4% lead• Statistically inconclusive

• The poll results give an ~84% chance that Obama is really ahead

Multiplicity of Error

Multiplicity of Error• Dr. Elisabeth Targ made headlines when she identified an

effect of remote faith healing• The study results were published in the peer-reviewed journal

The Western Journal of Medicine under the title "A randomized double-blind study of the effect of distant healing in a population with advanced AIDS. Report of a small scale study"

• Results from the double blind study showed that members of the control group spent 600% more days in the hospital and contracted 300% as many AIDS-related illnesses

• Considered the most rigorous, scientifically valid experiment to demonstrate the effect of remote faith healing

Multiplicity of Error• What was learned until after the fact:• Study was designed to look for differences in mortality rates

• Triple-drug Anti-Retroviral therapy became common practice one month into the six month study

• Only one person died – insufficient data to make conclusions• Adjusted the scope to look at HIV physical symptoms and Quality of Life

• Also inconclusive• Adjusted scope again to look at Mood State scores

• Control group performed better than treatment group• Adjusted scope again to look at Hospital Stays and Doctor Visits

• Treatment group performed statistically significantly better• Adjusted scope to include at 23 illnesses associated with AIDS

• Data hadn't been collected as part of the study design, so blindness was broken and data was aggregated from hospital charts

• Sharpshooter fallacy• Spray bullets down range, then draw your target around a cluster

Multiplicity of Error• Keeping ourselves honest• Bonferroni correction

• Don't use α, use where m is the number of comparisons• Conservative

• Holm-Bonferroni correction• Compare the smallest p-value to • If it is significant, continue testing• Otherwise, stop

• Compare the next smallest p-value to • Continue comparing the p-values until one is not significant, then stop• Less conservative

• Others that are implemented in statistics packages (Minitab, JMP, etc…)• Tukey's HSD• Dunnet's• And many more (you can't get a Ph.D. in statistics without inventing a

correction to name after yourself)

Biggest Lie Ever Told• Vitamin C prevents or helps cure the Common Cold• Linus Pauling (an incredibly bright, talented scientist)

championed the need for megadoses of Vitamin C to prevent or cure the Common Cold (and cancer) with the publishing of his book Vitamin C and the Common Cold in 1970• Had a whole institute setup to further study the benefits of

megadosing Vitamin C• At least 30 experiments have been performed to test the

ability of Vitamin C to prevent or cure the Common Cold in large groups of people (dates of experiments go from 1967-2001)• Not a single one has found a link between Vitamin C and the

Common Cold