Hypothesis Testing I: Introduction and Comparing means · Example scenario to use hypothesis...
70
Hypothesis Testing I: Introduction and Comparing means BIO5312 FALL2017 STEPHANIE J. SPIELMAN, PHD
Transcript of Hypothesis Testing I: Introduction and Comparing means · Example scenario to use hypothesis...
HypothesisTestingI:IntroductionandComparingmeansBIO5312FALL2017
STEPHANIE J. SPIELMAN,PHD
StatisticalinferencePopulation Sample
Random sampling
Statistical inferencePopulation parameters !, "
Sample estimates x, s
RecallfromlastweekFor𝑋~𝑁(0,1),whatistheprobabilityP(X≤0.47)?
0.0
0.1
0.2
0.3
0.4
−5 −4 −3 −2 −1 0 1 2 3 4 5Z
Probability
0.47
# CDF: P(X <= 0.47)> pnorm(0.47)
[1] 0.6808225
ComputingprobabilitiesforapopulationClownfishlengthsarenormallydistributedasN(11,1.9).
Whatistheprobabilitythataclownfishislessthan13cm?
𝑍 = ,-./= 01-00
0.03� = 1.83> pnorm(1.83) [1] 0.966375
ClownfishlengthsarenormallydistributedasN(11,1.9).
Whatistheprobabilitythatasampleof100clownfishhasameanlessthan11.5?
𝑍 = 𝑥 − 𝜇𝑆𝐸
Computingprobabilitiesforasample
𝑍 = 𝑥 − 𝜇𝜎
𝑆𝐸 = /;�= 0.3
0<<� = 0.19
= 00.=-00<.03
= 2.63 > pnorm(2.63) [1] 0.9957308
AreturntosamplingdistributionsThesamplingdistribution isthedistributionofallpossiblevaluesanestimateforaparametercantakebasedonrandomsampling◦ Samplingdistributionofthemeans=allthemeansonecouldmeasureacrosssamples
Forclownfish,samplingdistributionofthemean,forN=100,hasastandarddeviationof1.1◦ SE=standarddeviationofthesamplingdistributionofthemean=1.1
Andifwedidn’tknowthepopulation𝜎 tocomputeSE?◦ WecouldapproximateasSE= >
;�,wheres isstandarddeviationofasample
HypothesistestingComparedata(randomsample)totheexpectationofaspecific nullhypothesis
Hypothesistestingusesprobabilitytoanswerwhetheranobservedeffectoccurredbychance
ExamplescenariotousehypothesistestingThepoliovaccinewasfirsttestedin1954.~400,000studentsweredividedintotworandomgroups:◦ Halfreceivedthevaccine,halfreceivedplacebo◦ Invaccinegroup,0.016%developedpolio.◦ Inplacebogroup,0.057%developedpolio.
Wecanusehypothesistestingtoaskifthevaccinelikelyworked,orwhetherrandomchancelikelycausedresults.
HypothesistestinghasnullandalternativehypothesesThenullhypothesisH0 makesaclaimabouttheunderlyingpopulationparameter◦ "Nothinginterestingisgoingon"◦ Specifictestshavespecificnullhypotheses
ThealternativehypothesisHA iswhatwewouldliketo"knowifit'strue"◦ "Somethingwecareaboutisgoingon"
Parametricvs.nonparametrichypothesistestsParametrictestsassumethatthedatafollowaparticularknowndistribution◦ Distributionssuchasnormal,binomial,chi-squared,etc.
Nonparametric testsmakenoassumptionaboutthedataandare"distribution-free"
ThenulldistributionrepresentsH0Thenulldistribution isthesamplingdistributionofoutcomesforateststatisticassumingthenullistrue
Hypothesistestingissometimesreferredtoasnullhypothesissignificancetesting
Hypothesistestsask:Towhatextentaremydataexpectedunderthenullhypothesis?
0.00
0.05
0.10
0.15
−4 −3 −2 −1 0 1 2 3 4test statistic
Prob
abilit
y de
nsity
Samplingdistributionofthenullhypothesis
Xvaluesinthetails areunlikely
Xvaluesinmiddleofthedistributionarefairlylikely
0.00
0.05
0.10
0.15
−4 −3 −2 −1 0 1 2 3 4test statistic
Prob
abilit
y de
nsity
Samplingdistributionofthenullhypothesis
ShadedareasaretheCriticalregions
criticalvalue criticalvalue
Thetotalareaofthecriticalregionsisdefinedas𝛂
CriticalvalueswhereN(0,1)isthenull
TheP-valueistheareaunderthecurveforyourteststatisticResultofhypothesistestissignificant ifteststatisticfallsinthecriticalregion
0.00
0.05
0.10
0.15
−4 −3 −2 −1 0 1 2 3 4test statistic
Prob
abilit
y de
nsity
Forteststatistic=2.25,thesumofareaislessthan𝛂
0.00
0.05
0.10
0.15
−4 −3 −2 −1 0 1 2 3 4test statistic
Prob
abilit
y de
nsity
0.00
0.05
0.10
0.15
−4−3−2−101234test statistic
Probability density
-2.25 2.25
NOTE:Itwillnotalwaysbesymmetric
FormingconclusionsBasedonyourpre-chosenα:
1. P-value<=α◦ Significantresultsallowustorejectthenullhypothesisandconclude
evidenceinfavorofthealternativehypothesis
2. P-value> α◦ Wedonothavesignificantresults.Wefailtorejectthenullhypothesis,
andwehavenoevidenceinfavorofthealternativehypothesis.
Thechoiceofαistotallyarbitrary,butusuallyyouwillsee0.05or0.01
Errorratesα setstheoverallfalsepositiverate forourtestprocedure◦ Ifthenullistrue,wefalselyrejectthenull5%ofthetimeforα=0.05
Truthaboutpopulation(generallyunknown)
Conclusion
Nullistrue Alternativeistrue
Rejectnull(P<= α) TypeIerror(Falsepositive)
Truepositive
Failtorejectnull(P> α) True negative TypeIIError(Falsenegative)
Whattypeoferrorisit(orisit?)Anewarthritisdrugdoesnothaveaneffectinclinicaltrials,eventhoughitactuallydoesreducearthritispain.ApersonwithHIVreceivesapositivetestresultforHIV.Apersonusingillegalperformingenhancingdrugspassesatestclearingthemofdruguse.Astudyfoundasignificantrelationshipbetweenneckstrainandjogging,whenrealitythereisnorelationship.Anhealthyindividualgetsapositivecancerbiopsyresult.
FN(typeII)
Noerror
FN(typeII)
FP(typeI)
FP(typeI)
One-sidedvs.Two-sided(or–tailed)One-sidedtestsaredirectional◦ Aremydatalarger/smallerthannull?
Two-sidedtestsarenon-directional◦ Domydatadifferfromnull?
Total area=𝛂
Doesdatadifferfromnull?
Isdata<null?
Isdata>null?
One-sidedvs.Two-sidedtestsOne-sidedtestshavemorepowerthantwo-sidedtests◦ Power=theabilitytodetectatrueeffect◦ Alsoknownastruepositiverate
One-sidedtestsaremorelimitedinscopeandcangetyouintroubleifyouchoosethewrongdirection
Youmustchooseonlyonetestbeforeyoulookatyourdata
Approachtohypothesistesting1. Decidewhatquestionyouareinterestedinanswering2. Determinetheappropriatehypothesistesttouse3. Checkthatyourdatameettheassumptions*ofthetest4. Computetheteststatisticforyourhypothesistestand
thecorrespondingP-value5. Drawconclusionsusinganapriorispecified𝛂 (P-value
threshold)*Parametriconly
HypothesisteststocomparemeansTestthenullhypothesis:◦ Onesampletest:Themeanofmysampleequalsanullvalue◦ Twosampletest:Themeanofmytwosamplesareequal(differenceinmeansis0)
Twooptions:◦ Z-test,wherethenulldistributionisthestandardnormalN(0,1)◦ TeststatisticisZ
◦ t-test,wherethenulldistributionistheStudent'stdistribution◦ Teststatisticist
Z-testsvst-testsNormalpopulationshavethesamplingdistribution𝑋~𝑁(𝜇, /
@
;�)
◦ Itsteststatisticis𝑍 = ,-.AB whereSE=𝜎/ 𝑛�
◦ Usedforz-tests
Because𝜎 israrelyknown,wecanapproximateSE = >;�
𝑡 = ,̅-.ABHI
,where𝑆𝐸,̅ = >;�
StandardnormalandStudent'st
0.0
0.1
0.2
0.3
0.4
−4 −2 0 2 4Z or t
Prob
abilit
y de
nsity
0.0
0.1
0.2
0.3
0.4
−4 −2 0 2 4Z or t
Prob
abilit
y de
nsity
0.0
0.1
0.2
0.3
0.4
−4 −2 0 2 4Z or t
Prob
abilit
y de
nsity
0.0
0.1
0.2
0.3
0.4
−4 −2 0 2 4Z or t
Prob
abilit
y de
nsity
StandardnormalN(0,1)Addingt distributionswithincreasingdegreesoffreedom
df =2 df =5 df =15
Student'stwheredf =100
0.0
0.1
0.2
0.3
0.4
−4 −2 0 2 4Z or t
Prob
abilit
y de
nsity
Propertiesofthet distributionAsdf approaches∞,thet distributionapproachesN(0,1)◦ Usually,df =n– 1,wherenissamplesize
Likethenormaldistribution…◦ t distributionissymmetric◦ Mean=median=mode
Unlikethenormaldistribution…◦ t has*muchfattertails*
0.0
0.1
0.2
0.3
0.4
−4 −2 0 2 4Z or t
Prob
abilit
y de
nsity
TestAssumptionsSampleisarandomsamplefromitspopulation
Sampledataisnormallydistributed
Diverightin:Onesamplet-testNullhypothesisH0 :𝒙I = 𝝁
One-sidedtestHA :𝒙I > 𝝁HA :𝒙I < 𝝁
Directional
Two-sidedtestHA :𝒙I ≠ 𝝁
Non-directional
Performingaone-samplet-testIwanttoknowifthedisease,BadDisease,influenceshumanbodytemperature.Weknowthatstandardhumanbodytemperatureis98.6degreesF.Imeasuredthetemperaturesforarandomsampleof15individualswithBadDisease.Onaverage,theyhaveatempof99.59 degreesF.
DoesBadDiseaseraisebodytemperature?
H0 :BadDiseasedoesnotraisebodytemperature.𝒙I = 𝟗𝟖. 𝟔HA :BadDiseaseraisesbodytemperature.𝒙I > 𝟗𝟖. 𝟔
> head(bad.disease.temp)temp
1 98.174202 97.621373 99.609204 100.441585 99.754836 100.28846
> mean(bad.disease.temp$temp)[1] 99.594
CheckingassumptionsofthetestThet-testassumesthatoursampledataisnormallydistributed
Hardtotellifnormalfromhistogram!
0
1
2
3
98 99 100 101 102 103Bad disease temperature for n=15
coun
t
AssessingnormalitywithaQ-QplotQuantile-Quantileplotsgraphicallyshowiftwodatasetscomefromthesamedistribution◦ Ifthepointsfollowthe"expectation"line,datasetsaresimilarlydistributed
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●
●
●●
●
●
−2 −1 0 1 2
01
23
45
Normal Q−Q Plot
Theoretical Quantiles
Sam
ple
Qua
ntile
s●
●
●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
−2 −1 0 1 2
4.4
4.6
4.8
5.0
5.2
5.4
5.6
Normal Q−Q Plot
Theoretical Quantiles
Sam
ple
Qua
ntile
s
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
−2 −1 0 1 2
020
4060
80
Normal Q−Q Plot
Theoretical Quantiles
Sam
ple
Qua
ntile
s
Idealscenario.Dataisnormal Dataisnotnormal Closeenough,let'ssaynormal
Granulardataisalsonormal!
! !
!"#$%&#"'()*+#%,-.*/)*"0%$.'$1*.0-,*$0(*'$.'+#(-*.-2'#('0$*3"04*$0"4#&'()
$0"4#&
MakingaNormalQ-Qplot> head(bad.disease.temp)
temp1 98.174202 97.621373 99.609204 100.441585 99.754836 100.28846
## Uses base R, not ggplot ##> qqnorm(bad.disease.temp$temp, pch=20)> qqline(bad.disease.temp$temp, col="red")
DataapproximatelyfollowtheQQline.Therefore,assumptionshavebeenmetandwecanrunthetest.
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
−1 0 1
9899
100
102
Normal Q−Q Plot
Theoretical Quantiles
Sam
ple
Qua
ntile
s
Performingthet-test1. Calculatetheteststatistic
2. Seewhereteststatisticfallsonitsdistribution
3. ComputeP-valueasareaunderthecurve pastthisstatistic
o TheP-valueistheprobabilityofobtainingateststatisticaslargeorlargerthanthatrecovered
Computetheteststatistic,tH0 :BadDiseasedoesnotraisebodytemperature.𝒙I = 𝟗𝟖. 𝟔HA :BadDiseaseraisesbodytemperature.𝒙I > 𝟗𝟖. 𝟔
𝑡 = ,̅-.RS�= 33.=3-3T.UV.WW
VX�= 2.66
> mean(bad.disease.temp$temp)[1] 99.594
> sd(bad.disease.temp$temp) [1] 1.438273
> nrow(bad.disease.temp)[1] 15
thisisourteststatistic,t2.66
Moreprecisely,t2.66,df=14
FindwherethestatisticfallsindistributionOurnulldistributionisat distributionwithdf =14(15-1)
0.0
0.1
0.2
0.3
0.4
−4 −2 0 2 4t
Prob
abilit
y D
ensi
ty
2.66
ThisareaisourP-valuebecausewetestif 𝒙I > 𝟗𝟖. 𝟔
## P(X >= 2.66) ##> 1 – pt(2.66, 14)
0.009
Formingconclusions
OurP=0.009,whichislessthanα=0.05.ThereforewerejectthenullhypothesisandwehaveevidencethatBadDiseaseraisesbodytemperature.
H0 :BadDiseasedoesnotraisebodytemperature.𝒙I = 𝟗𝟖. 𝟔HA :BadDiseaseraisesbodytemperature.𝒙I > 𝟗𝟖. 𝟔
Approachtohypothesistesting1. Decidewhatquestionyouareinterestedinanswering
Isthemeanofmydataequalto98.6?
2. DeterminetheappropriatehypothesistesttouseUseaone-samplet-test
3. CheckthatyourdatameettheassumptionsofthetestWeconfirmedthedataisnormallydistributed
4. ComputetheteststatisticforyourhypothesistestandthecorrespondingP-value
Wefoundt=2.66andP=0.009
5. DrawconclusionsusingaspecifiedP-valuethresholdAt𝛂 =0.05,werejectthenullhypothesisandfindevidencethatBadDiseaseraisesbody
temp.
EffectsizeTheeffectweobservedhereis99.59– 98.6=0.99
Statisticalsignificanceisnotthesameasbiologicalsignificance
0
1
2
3
4
1.0 1.5 2.0 2.5 3.0x
y
Comparisonbetweenredandbluesampleswillshowasignificantdifferencebetweenmeans.Butdoesitmatter?
ConfidenceintervalsRangeofvaluessurroundingthesampleestimatethatislikelytocontainthepopulationparameter
Generallywecalculatethe95%confidenceinterval(goeswith𝛂 =0.05)
In95%ofrandomsamples,thiswillbetrue:
𝒙I − 𝒕𝟎.𝟎𝟐𝟓 ∗ 𝑺𝑬𝒙I < 𝝁 < 𝒙I + (𝒕𝟎.𝟎𝟐𝟓 ∗ 𝑺𝑬𝒙I)
95%ConfidenceInterval
ConceptualizingtheCI
True population mean
10 random samples, in no particular order
sample effect
95% CI
9/10(~95%)oftheserandomsampleshasa95%confidenceintervalthatoverlapsthetruemean
Doesthisfigurerepresent95%CIs?
True population mean
10 random samples, in no particular order
NO.Ifanything,theseare40%CIs
CalculatingtheCIConstructupperandlowerlimitsof95%CI◦ Lower:�̅� − (𝑡<.<c= ∗ 𝑆𝐸,̅)◦ Upper:�̅� + (𝑡<.<c= ∗ 𝑆𝐸,̅)
### Calculate t_0.025> qt(0.025, 14)[1] -2.144787
### Calculate t_0.025*SE> t <- abs(qt(0.025, 14)) > se <- sd(bad.disease.temp$temp)/sqrt(15)> t * se
[1] 0.3713605
95%CI=𝒙I ± 𝒕𝟎.𝟎𝟐𝟓 ∗ 𝑺𝑬𝒙I
= 𝟗𝟗. 𝟓𝟗 ± 0.371
Thetruepopulationmeanis95%likelytobeintherange99.22– 99.96
BringitalltogetherWehaveasamplemeanof99.59withastandarderrorof0.19.
Ourteststatistictdf=14 =2.66,givingaP-value=0.009.Werejectthenullhypothesisat𝛂 =0.05 andhaveevidencethatBadDiseaseraisestemperature.
Oureffectsizeis0.99.
Wefounda95%CIof99.59±0.371,givingthelikelyrangeforthetruepopulationparameter.Notethatthenullof98.6isnotintheCI.
Reportingnon-significantresultsLet'ssaywefoundtdf=14 =1.05à P=0.15
At𝛂 =0.05,wefailtorejectthenullhypothesis.WehavenoevidencethatBadDiseaseraisesbodytemperature.◦ DoesnotmeanthatBadDiseasedoesn'traisethetemperature– oursamplejusthadnoevidenceforthiseffect.
### Calculate P-value> 1 - pt(1.05, 14)[1] 0.1557531
Whatis aP-value?TheP-valueisanareaunderthecurveofthenulldistribution◦ Itisthereforetheprobabilityofobservingthiseffectorlargerassumingthenullhypothesisistrue
◦ P-value=P(effectormoreobserved|H0 istrue)◦ P-value=0.009:IfH0 istrue,Iwouldobtainthiseffectorlarger(t>=2.66)in0.9%ofsuchstudiesduetorandomsamplingerror
Whatis aP-value?AlowP-valuemeansthatthedataareunlikelyunderthenull◦ Wethereforemakeaneducatedguessthatthereisprobablysomethingelsegoingon,suchasthealternativehypothesis
◦ Wecannever ruleoutthepossibilitythatresultswerefullyconsistentwithnull,justunlikely
P-valuesarenotmagicP-valuescannot evaluatewhetherH0 orHA istrue◦ LargeP-valuesdonot provethenullistrue◦ SmallP-valuesdonot provethealternativeistrue.Theymerelysuggestthenulllikelyisn't.
P-valuesdonot givetheprobabilitythatyoumadetherightconclusion
TwostudieswiththesameP-valuedonotprovidethesameweightofevidence
DistributionofP-valuesIperform1000t-testsonrandomsamplesfromN(3.5,1)andcomparetonullmean=3.5.Therefore,nullistrue.
Histogram of ps
Pvalues
Count
0.0 0.2 0.4 0.6 0.8 1.0
020
4060
80
P<=0.05à Falsepositives
DistributionofP-valuesIperform1000t-testsonrandomsamplesfromN(3.5,1)andcomparetonullmean=2.Therefore,nullisfalse.
P−values
Count
0.0 0.2 0.4 0.6 0.8 1.0
0200
400
600
P<=0.05à TruepositivesOnly~67%ofdistribution!
P-valuesandeffectsize
cally important. This is often untrue. First, the difference maybe too small to be clinically important. The P value carries noinformation about the magnitude of an effect, which is cap-tured by the effect estimate and confidence interval. Second,the end point may itself not be clinically important, as canoccur with some surrogate outcomes: response rates versussurvival, CD4 counts versus clinical disease, change in a mea-surement scale versus improved functioning, and so on.25-27
Misconception #4: Studies with P values on opposite sides of.05 are conflicting. Studies can have differing degrees of sig-nificance even when the estimates of treatment benefit areidentical, by changing only the precision of the estimate, typ-ically through the sample size (Figure 2A). Studies statisti-cally conflict only when the difference between their results isunlikely to have occurred by chance, corresponding to whentheir confidence intervals show little or no overlap, formallyassessed with a test of heterogeneity.
Misconception #5: Studies with the same P value provide thesame evidence against the null hypothesis. Dramatically differentobserved effects can have the same P value. Figure 2B showsthe results of two trials, one with a treatment effect of 3%(confidence interval [CI], 0% to 6%), and the other with aneffect of 19% (CI, 0% to 38%). These both have a P value of.05, but the fact that these mean different things is easilydemonstrated. If we felt that a 10% benefit was necessary tooffset the adverse effects of this therapy, we might well adopta therapy on the basis of the study showing the large effectand strongly reject that therapy based on the study showingthe small effect, which rules out a 10% benefit. It is of coursealso possible to have the same P value even if the lower CI isnot close to zero.
This seeming incongruity occurs because the P value de-fines “evidence” relative to only one hypothesis—the null.There is no notion of positive evidence—if data with a P !.05 are evidence against the null, what are they evidence for?In this example, the strongest evidence for a benefit is for 3%in one study and 19% in the other. If we quantified evidencein a relative way, and asked which experiment provided
greater evidence for a 10% or higher effect (versus the null),we would find that the evidence was far greater in the trialshowing a 19% benefit.13,18,28
Misconception #6: P ! .05 means that we have observeddata that would occur only 5% of the time under the null hypoth-esis. That this is not the case is seen immediately from the Pvalue’s definition, the probability of the observed data, plusmore extreme data, under the null hypothesis. The result withthe P value of exactly .05 (or any other value) is the mostprobable of all the other possible results included in the “tailarea” that defines the P value. The probability of any individ-ual result is actually quite small, and Fisher said he threw inthe rest of the tail area “as an approximation.” As we will seelater in this chapter, the inclusion of these rarer outcomesposes serious logical and quantitative problems for the Pvalue, and using comparative rather than single probabilitiesto measure evidence eliminates the need to include outcomesother than what was observed.
This is the error made in the published survey of medicalresidents cited in the Introduction,3 where the following fouranswers were offered as possible interpretations of P ".05:
a. The chances are greater than 1 in 20 that a differencewould be found again if the study were repeated.
b. The probability is less than 1 in 20 that a difference thislarge could occur by chance alone.
c. The probability is greater than 1 in 20 that a differencethis large could occur by chance alone.
d. The chance is 95% that the study is correct.
The correct answer was identified as “c”, whereas the ac-tual correct answer should have read, “The probability isgreater than 1 in 20 that a difference this large or larger couldoccur by chance alone.”
These “more extreme” values included in the P-value def-inition actually introduce an operational difficulty in calcu-lating P values, as more extreme data are by definition unob-served data. What “could” have been observed depends onwhat experiment we imagine repeating. This means that twoexperiments with identical data on identical patients couldgenerate different P values if the imagined “long run” weredifferent. This can occur when one study uses a stoppingrule, and the other does not, or if one employs multiplecomparisons and the other does not.29,30
Misconception #7: P ! .05 and P !.05 mean the samething. This misconception shows how diabolically difficult itis to either explain or understand P values. There is a bigdifference between these results in terms of weight of evi-dence, but because the same number (5%) is associated witheach, that difference is literally impossible to communicate. Itcan be calculated and seen clearly only using a Bayesian evi-dence metric.16
Misconception #8: P values are properly written as inequal-ities (eg, “P !.02” when P ! .015). Expressing all P values asinequalities is a confusion that comes from the combinationof hypothesis tests and P values. In a hypothesis test, a pre-set“rejection” threshold is established. It is typically set at P !.05, corresponding to a type I error rate (or “alpha”) of 5%. Insuch a test, the only relevant information is whether the
-10
-5
0
5
10
15
20
25
30
35
40
Perc
ent b
enef
it
P=0.30 P=0.002
P=0.05 P=0.05
Same effect, different P-value
Different effect, same P-value
A BFigure 2 Figure showing how the P values of very different signifi-cance can arise from trials showing the identical effect with differentprecision (A, Misconception #4), or how same P value can be de-rived from profoundly different results (B, Misconception #5).
Twelve P-value misconceptions 137
P-valuesarestronglyinfluencedbysamplesize
𝑡 = ,̅-.RS�= 33.=3-3T.UV.WW
𝟏𝟓�= 2.66à P=0.009
𝑡 = ,̅-.RS�= 33.=3-3T.UV.WW
𝟏𝟎𝟎�= 6.88à P=2.81e-10
Increasingsamplingsizeincreasespower
Poweristheprobabilityyoudetectatrueeffect,i.e.truepositiverate
P-valuesarekindofanaccidentPersonally,thewriterpreferstosetalowstandardofsignificanceatthe5percentpoint....Ascientificfactshouldberegardedasexperimentallyestablishedonlyifaproperlydesignedexperimentrarelyfailstogivethislevelofsignificance.
- R.A.Fisher
TorecapWehaveperformedaone-sidedt-testtotestthealternativehypothesisthat𝒙I > 𝟗𝟖. 𝟔
Wecanalsoperformatwo-sidedt-testtotestthenon-directional alternativehypothesisthat 𝒙I ≠ 𝟗𝟖. 𝟔
Computetheteststatistic,tH0 :BadDiseasedoesnotaffectbodytemperature.𝒙I = 𝟗𝟖. 𝟔HA :BadDiseaseaffects bodytemperature.𝒙I ≠ 𝟗𝟖. 𝟔
𝒙I =99.59𝜇 =98.6n=15
𝑡 = ,̅-.RS�= 33.=3-3T.UV.WW
VX�= 2.66
> head(bad.disease.temp)temp
1 98.174202 97.621373 99.609204 100.441585 99.754836 100.28846
> mean(bad.disease.temp$temp)[1] 99.594
> sd(bad.disease.temp$temp) [1] 1.438273
thisisourteststatistic,t2.66
Moreprecisely,t2.66,df=14
ComputingtheP-valueForatwo-sidedtest,weconsiderbothextremes
## P(X >= 2.66) or P(X<=-2.66)> 2 * (1 - pt(2.66, 14))
0.01806
0.0
0.1
0.2
0.3
0.4
−4 −2 0 2 4t
Prob
abilit
y D
ensi
ty
2.66
ThiscombinedareaisourP-value=0.018
-2.66
Whenrunningaone-sidedt-testgoeswrongForsampleofn=20,Iwanttotest�̅� < 𝜇 where𝜇 =5.8> head(example.sample)
values1 5.5113072 5.0126123 6.1784214 7.5875685 6.8921656 5.197049
> mean(example.sample$values)[1] 6.325366
> sd(example.sample$values)[1] 0.8295474
𝑡 = �̅� − 𝜇𝑠𝑛�
= 6.32 − 5.80.8320�
= 2.801
Theareabelow thestatisticisourP-valuefor�̅� < 𝜇
0.0
0.1
0.2
0.3
0.4
−4 −2 0 2 4t
Prob
abilit
y D
ensi
ty2.801
> pt(2.801, 19)[1] 0.9943006
Two-samplet-testsComparemeansoftwosamples(�̅�0and�̅�c )toeachother.Aretheunderlyingpopulationmeans𝜇0and𝜇c thesame?
NullhypothesisH0 :𝝁𝟏 =𝝁𝟐
H0 :𝝁𝟏 - 𝝁𝟐 = 𝟎
One-sidedtestHA :𝝁𝟏 > 𝝁𝟐HA :𝝁𝟏 < 𝝁𝟐
HA :𝝁𝟏 −𝝁𝟐 > 𝟎HA :𝝁𝟏 −𝝁𝟐 < 𝟎
Two-sidedtestHA :𝝁𝟏 ≠ 𝝁𝟐
HA :𝝁𝟏 − 𝝁𝟐 ≠ 𝟎
Formulasfortwo-samplet-test
𝒕 = 𝒙I𝟏 − 𝒙I𝟐𝑺𝑬𝒙I𝟏-𝒙I𝟐
𝑆𝐸,̅V-,̅@ = 𝑠jc 1𝑛0+
1𝑛c
�
𝑠jc =𝑑𝑓0𝑠0c + 𝑑𝑓c𝑠cc
𝑑𝑓0 + 𝑑𝑓c
𝑑𝑓 = 𝑑𝑓0 + 𝑑𝑓c = 𝑛0 + 𝑛c − 2
Standarderrorfortwosamples
Pooledsamplevariance
Degreesoffreedomfort distribution
Two-samplet-testassumptionsBothsamplesmustbedistributednormally
Samplesshouldhaveequalvariances◦ F-test cancomparevariancesofsamplestocheckassumption,butitishighlysensitiveandwill"toooften"rejectthenull.
◦ Levene's test willtestforhomogeneityofvariancesaswell◦ CanuseWelch'st-test whenvarianceassumptionisnotmet
Forthisclass,wefocusonnormalassumption
Two-samplevs.pairedt-testPairedt-testisaspecialcasewherethetwosamplesbeingcomparedhaveanaturalpairing◦ Effectivelyaone-samplet-test wherewetesttheifdifference betweentwosamples=0
Pairedt-testmustcheckassumptionthatdifferencebetweenmeansisnormalYoucanalwaysperformatwo-sampleinsteadofpaired,butpairedwillhavemorepower
PairedscenariosMakingtwomeasurementsoneachsubject
Makingrepeatmeasurementsonthesamesubjectattwotimepoints◦ Beforeandaftertreatment
Matchingsubjectswithsimilarage,sex,etc.
Placingsubjectandcontrolincloseproximity
Isitpairedorindependent?Triglyceridelevelsofagroupofsubjectsiscomparedbeforeandaftertakingavitaminsupplement.Foraclinicaltrial,onegroupisgivenavitaminandtheotheraplacebo.Theirtriglyceridelevelsarecompared.Foraclinicaltrial,twogroupswithindividualsmatchedforage,sex,andhealthhistoryaregivenvitaminandplacebo,respectively.Triglyceridelevelsarecompared.Imeasurefreeenergies,betweeninactiveandactiveconformations,for30enzymes.Aclinicaltrialtestsanewdrugon20setsoftwins,giving,foreachtwinpair,oneaplaceboandonethedrug.Comparisonsbetweenplaceboanddruggroupsaremade.Aclinicaltrialtestsanewdrugon20setsoftwins,randomizingallindividualsintoaplacebogroupandoradruggroup.Comparisonsbetweengroupsaremade.
Paired
Independent
PairedPaired
Paired
Independent
Isitpairedorindependent?Paired:CanIdrawalinebetweenindividualsingroups?◦ MustbesameNpergroup,bydefinition
Independent:Nonaturalwaytodrawthelines.
5
10
15
before afterPaired groups
Mea
sure
d va
lue
Troubleshooting:Failuretomeetnormalassumption1.Ifsamplesizeislargeenough(>~30),CentralLimitTheoremkicksinandassumptionsareeffectivelymet
2.Ifsampleissizeissmall(<~30)wecaneither:◦ Transform thedatatobenormal◦ Useanonparametrictest
Non-normaldataright skew
coun
t
0 5 10 15 20 25 30
010
25
left skew
coun
t
−20 −10 0 5 10
020
40
long tails
coun
t
−20 0 10 20 30
010
30
●
●●
●
●●
●
●
● ●
●
●●
●●
●●●●●
●●●
●●
●●
●
●
●
●
●●
●●● ●
●●●
●●
●
●●●●●
●●●
●
●
●●
●●
●
●
●● ●●●
●●
●
●
●
●
● ●● ●●
●●●●
●●●●●
●●
●
●
●●
●●
●●●
●
●●
●●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●
●
●●
●
●
●
●
−2 −1 0 1 2
010
2030
Normal Q−Q Plot
Theoretical Quantiles
Sam
ple
Qua
ntile
s
●● ●●
●● ●
● ●●
● ●●
●●●●
●
●● ●
●●
●
●
●●
●●
●
●●
●
●
●●●●
●●●
●
●
●●
●
●●●
●●
●
●
●
●
●
●●
●
●● ●
● ●
●
●
●
●
●
●●
●●●
●
●●
●
●
●●
●
●●
●●●●
●●
●
● ●●
●●
●●
●●
●
●
●
●
●
●●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
−2 −1 0 1 2
−20
−55
Normal Q−Q Plot
Theoretical Quantiles
Sam
ple
Qua
ntile
s
●
●
●●● ●●
●●
●●
●
●●●
● ● ● ●● ●
●●●●●●
●● ●●●
●● ●●●●
●● ●● ●
●
●●
●●●●●
●
●●●● ●
●● ●
●
●
●
●●●
●●
●
●
●●●●
●●
● ●
●●
●
●●
●
●●
● ●●
●●●
●●●●●
● ●●
●
●●
●
●
●
●
●●
●
●
●
●●
●●●
●●
●●●
●
●●
●●●
●
●●
●
●●
●
●
●
●
●●
●
●●
●●
●●
●
●
●
−2 −1 0 1 2
−20
020
Normal Q−Q Plot
Theoretical Quantiles
Sam
ple
Qua
ntile
s
DatatransformationsLogofdata:xà log(x)
Squarerootofdata:xà sqrt(x)
Inverseofdata:xà 1/x
Datatransforms:CautionYourtestwillnowrunontransformeddata◦ Assumelogtransformperformedandresulthaseffectsize1.5◦ Actualeffectsizeisexp(1.5)=4.48
Becarefulof0'sindata◦ 1/0andlog(0)areundefined◦ Hack:Replaceall0'swithtinynumberlike1e-8
InstructorEditorializing:
MuchlikeZ-tables,datatransformsaremostlyhistorical
