Understanding P- values and CI 20Nov08 (1)
-
Upload
santoshchitra -
Category
Documents
-
view
214 -
download
0
Transcript of Understanding P- values and CI 20Nov08 (1)
-
8/8/2019 Understanding P- values and CI 20Nov08 (1)
1/37
Understanding P-values andConfidence Intervals
Thomas B. Newman, MD, MPH
20 Nov 08
-
8/8/2019 Understanding P- values and CI 20Nov08 (1)
2/37
Announcements
Optional reading about P-values andConfidence Intervals on the website
Exam questions due Monday 11/24/08 5:00
PM Next week (11/27) is Thanksgiving
Following week Physicians and Probability(Chapter 12) and Course Review
Final exam to be distributed in SECTION 12/4and posted on web
Exam due 12/11 8:45 AM
Key will be posted shortly thereafter
-
8/8/2019 Understanding P- values and CI 20Nov08 (1)
3/37
Overview Introduction and justification
What P-values and Confidence Intervals dontmean
Whatthey do mean: analogy betweendiagnostic tests and clinical researc
Useful confidence interval tips
CI for negative studies; absolute vs.
relative risk Confidence intervals for small numerators
-
8/8/2019 Understanding P- values and CI 20Nov08 (1)
4/37
Why cover this material here?
P-values and confidence intervals are
ubiquitous in clinical research
Widely misunderstood and mistaught
Pedagogical argument:
Is itimportant?
Can you handle it?
-
8/8/2019 Understanding P- values and CI 20Nov08 (1)
5/37
Example: Douglas Altman Definition of
95% Confidence Intervals* "A strictly correct definition of a 95% CI is,
somewhat opaquely, that 95% of such
intervals will contain the true population
value.
Little is lost by the less pure interpretation of
the CI as the range of values within which we
can be 95% sure thatthe population valuelies.
*Quoted in: Guyatt, G., D. Rennie, et al. (2002). Users' guides to the medical
literature : essentials of evidence-based clinical practice. Chicago, IL,AMAPress.
-
8/8/2019 Understanding P- values and CI 20Nov08 (1)
6/37
Understanding P-values and
confidence intervals is important
because
It explains things which otherwise do
not make sense, e.g. the need to state
hypotheses in advance and correction
for multiple hypothesis testing
You will be using them all the time
You are future leaders in clinicalresearch
-
8/8/2019 Understanding P- values and CI 20Nov08 (1)
7/37
You can handle it because
We have already covered the important
concepts at length earlierin this course
Priorprobability
Posteriorprobability
What you thought before + new
information = what you think now We will support you through the process
-
8/8/2019 Understanding P- values and CI 20Nov08 (1)
8/37
Review of traditional statistical
significance testing
State null (Ho) and alternative (Ha)
hypotheses
Choose
Calculate value oftest statistic from
your data
Calculate P- value from test statistic
If P-value < , reject Ho
-
8/8/2019 Understanding P- values and CI 20Nov08 (1)
9/37
Problem:
Traditional statistical significance testing
has led to widespread misinterpretation
of P-values
-
8/8/2019 Understanding P- values and CI 20Nov08 (1)
10/37
What P-values dontmean
Ifthe P-value is 0.05, there is a 95%
probability that
The results did not occur by chance
The null hypothesis is false
There really is a difference between the
groups
-
8/8/2019 Understanding P- values and CI 20Nov08 (1)
11/37
So if P = 0.05, what IS there a 95%
probability of?
-
8/8/2019 Understanding P- values and CI 20Nov08 (1)
12/37
White board:
2x2 tables and false positive confusion
Analogy with diagnostic tests
(This is covered step-by-stepin thecourse book.)
-
8/8/2019 Understanding P- values and CI 20Nov08 (1)
13/37
Analogy between diagnostic tests
and research studies
Diagnostic Test Research Study
Absence ofDiseasePresence of disease
Severit of disease in t e
diseased group
Cutoff for distinguishingpositive and negative
results
Test result
-
8/8/2019 Understanding P- values and CI 20Nov08 (1)
14/37
Analogy between diagnostic tests
and research studies
Diagnostic Test Research Study
Negative result (test
withinnormallimits)
Positive resultSensitivity
False positive rate (1-
specificity)
Prior probability ofdisease (ofa given
severity)
Posterior probability of
disease, given test result
-
8/8/2019 Understanding P- values and CI 20Nov08 (1)
15/37
Extending the Analogy
Intentionally ordered tests and
hypotheses stated in advance
Multiple tests and multiple hypotheses
Laboratory error and bias
Alternative diagnoses and confounding
-
8/8/2019 Understanding P- values and CI 20Nov08 (1)
16/37
Bonferroni
Inequality: If we do k differenttests,
each with significance level , the
probability that one or more will be
significantis less than or equal to k v
Correction: If we test k different
hypotheses and want ourtotal Type 1
error rate to be no more than alpha,then we should reject H0 only if P < /k
-
8/8/2019 Understanding P- values and CI 20Nov08 (1)
17/37
Derivation
Let A & B = probability of a Type 1 error forhypotheses A and B
P(A or B) = P(A) + P(B) P(A & B)
Under Ho, P(A) = P(B) =
So P(A or B) = + - P(A & B) = 2 - P(A & B). Of course, itis possible to falsely reject 2 different null
hypotheses, so P(A & B) > 0. Therefore, the
probability of falsely rejecting either ofthe null
hypotheses must be less than 2.
Note that often A & B are notindependent, in which
case Bonferroni will be even more excessively
conservative
-
8/8/2019 Understanding P- values and CI 20Nov08 (1)
18/37
Problems withBonferroni correction
Overly conservative (especially whenhypotheses are notindependent)
Maintains s
pec
ificity a
tthe ex
pense ofsensitivity
Does nottake priorprobability intoaccount
Not clear when to use it BUT can be useful if results still
significant
-
8/8/2019 Understanding P- values and CI 20Nov08 (1)
19/37
CONFIDENCE INTERVALS
-
8/8/2019 Understanding P- values and CI 20Nov08 (1)
20/37
What Confidence Intervals dont
mean There is a 95% chance thatthe true
value is within the interval
If you conclude thatthe true value iswithin the interval you have a 95%chance of being right
The range of values within which wecan be 95% sure thatthe populationvalue lies
-
8/8/2019 Understanding P- values and CI 20Nov08 (1)
21/37
One source of confusion: Statistical
confidence
(Some) statisticians say: You can be 95%
confident thatthe population value is in the
interval. This is NOT the same as There is a 95%
probability thatthe population value is in the
interval.
Confidence is tautologously defined by
statisticians as what you get from a
confidence interval
-
8/8/2019 Understanding P- values and CI 20Nov08 (1)
22/37
Illustration
If a 95% CI has a 95% chance of containing
the true value, then a 90% CI should have a
90% chance and a 40% CI should have a
40% chance.
Study: 4 deaths in 10 subjects in each group
RR= 1.0 (95% CI: 0.34 to 2.9)
40% CI: 0.75 to 1.33
Conclude from this study thatthere is 60%
chance thatthe true RR is 1.33?
-
8/8/2019 Understanding P- values and CI 20Nov08 (1)
23/37
Confidence Intervals apply to a
Process Consider a bag with 19 white and 1 pink
grapefruit
The process of selecting a grapefruit atrandom has a 95% probability of yielding awhite one
But once Ive selected one, does it still have a95% chance of being white?
You may have prior knowledge that changesthe probability (e.g., pink grapefruit havethinnerpeel are denser, etc.)
-
8/8/2019 Understanding P- values and CI 20Nov08 (1)
24/37
Confidence Intervals for negative
studies: 5 levels of sophistication
Example 1: Oral amoxicillin to treatpossible occult bacteremia in febrilechildren*
Randomized, double-blind trial
3-36 month old children with T 39 C (N=955)
Treatment: Amox 125 mg/tid ( 10 kg) or250 mg tid (> 10 kg)
Outcome: majorinfectious morbidity
*Jaffe et al., New Engl J Med 1987;317:1175-80
-
8/8/2019 Understanding P- values and CI 20Nov08 (1)
25/37
Amoxicillin for possible occult
bacteremia 2: Results Bacteremia in 19/507 (3.7%) with amox,
vs 8/448 (1.8%) with placebo (P=0.07)
Major Infectious Morbidity 2/19(10.5%) with amox vs 1/8 (12.5%) withplacebo (P = 0.9)
Conclusion: Data do not supportroutine use of standard doses ofamoxicillin
-
8/8/2019 Understanding P- values and CI 20Nov08 (1)
26/37
5 levels of sophistication
Level 1: P > 0.05 = treatment does notwork
Level 2: Look atpower for study.
(Authors reported power = 0.24 forOR=4. Therefore, study underpowered
and negative study uninformative.)
-
8/8/2019 Understanding P- values and CI 20Nov08 (1)
27/37
5 levels of sophistication, contd
Level 3: Look at 95% CI!Authors calculated OR= 1.2 (95% CI:
0.02 to 30.4)
This is based on 1/8 (12.5%) with placebovs 2/19 (10.5%) with amox
(They putplacebo on top)
(Silly to use OR)
With amox on top, RR = 0.84 (95% CI:0.09 to 8.0)
This was level of TBN in letterto theeditor (1987)
-
8/8/2019 Understanding P- values and CI 20Nov08 (1)
28/37
5 levels of sophistication, contd
Level 4: Make sure you do an intentionto treat analysis!
Itis notOK to restrict attention to
bacteremic patients So it should be 2/507 (0.39%) with amox
vs 1/448 (0.22%) with placebo
RR= 1.8 (95% CI: 0.05 to 6.2)
-
8/8/2019 Understanding P- values and CI 20Nov08 (1)
29/37
Level 5: the clinically relevant quantity
is the Absolute Risk Reduction (ARR)!
2/507 (0.39%) with amox vs 1/448 (0.22%)with placebo
ARR = 0.17% {amoxicillin worse} 95% CI (0.9% {harm} to +0.5% {benefit})
Therefore, LOWER limit of 95% CI for benefit(I.e., best case) is NNT= 1/0.5% = 200
So this study suggests need to treat 200children to prevent Major InfectiousMorbidity in one
-
8/8/2019 Understanding P- values and CI 20Nov08 (1)
30/37
Stata output. csi 2 1 505 447
| Exposed Unexposed | Total
-----------------+------------------------+----------
Cases | 2 1 | 3
Noncases | 505 447 | 952
-----------------+------------------------+----------Total | 507 448 | 955
| |
Risk | .0039448 .0022321 | .0031414
| |
| Point estimate | [95% Conf. Interval]
|------------------------+----------------------
Risk difference | .0017126 | -.005278 .0087032
Risk ratio | 1.767258 | .1607894 19.42418
Attr. frac. ex. | .4341518 | -5.219315 .9485178
Attr. frac. pop | .2894345 |
+-----------------------------------------------
chi2(1) = 0.22 Pr>chi2 = 0.6369
-
8/8/2019 Understanding P- values and CI 20Nov08 (1)
31/37
Example 2: Pyelonephritis and new renal
scarring in the International Reflux
Study in Children*
RCT of ureteral reimplantation vs prophylactic
antibiotics for children with vesicoureteral
reflux Overall result: surgery group fewer episodes
ofpyelonephritis (8% vs 22%; NNT = 7; P chi2 = 0.0437
-
8/8/2019 Understanding P- values and CI 20Nov08 (1)
34/37
Conclusions
No evidence that new pyelonephritis causesscarring
Some evidence thatit does not
P-values and confidence intervals are
approximate, especially for small sample
sizes
There is nothing magical about 0.05
Key concept: calculate 95% CI for negative
studies
ARR for clinical questions (less generalizable)
RR for etiologic questions
-
8/8/2019 Understanding P- values and CI 20Nov08 (1)
35/37
Confidence intervals for small
numeratorsObserved
numer t r
Appr x mate
Numerat rfor
UpperLimit of 95%CI
0 3
1 5
2 7
3 9
4 10
-
8/8/2019 Understanding P- values and CI 20Nov08 (1)
36/37
When P-values and Confidence Intervals
Disagree
Usually P < 0.05 means 95% CI excludes null value.
But both 95% CI and P-values are based on
approximations, so this may not be the case Illustrated by IRSC slide above
If you want 95% CI and P- values to agree, use test-
based confidence intervals see next slide
-
8/8/2019 Understanding P- values and CI 20Nov08 (1)
37/37
Alternative Stata output: Test-
based CI
.
. csi 2 18 28 58,tb
| Exposed Unexposed | Total
-----------------+-----------------------+------------
Cases | 2 18 | 20
Noncases | 28 58 | 86
-----------------+-----------------------+------------
Total | 30 76 | 106
| |
Risk | .0666667 .2368421 | .1886792
| |
| Point estimate | [95% Conf. Interval]
|-----------------------+------------------------
Risk difference | -.1701754 | -.3363063 -.0040446 (tb)
Risk ratio | .2814815 | .0816554 .9703199 (tb)
Prev. frac. ex. | .7185185 | .0296801 .9183446 (tb)
Prev. frac. pop | .2033543 |
+------------------------------------------------- chi2(1) = 4.07 Pr>chi2 = 0.0437