Academic VivaPOWER and ERROR

T R Wilson

Impact Factor

• Measure reflecting the average number of citations to recent articles published in that journal.

• For given year, the average number of citations received per paper published from that journal during the two preceding years.

• E.g. The 2008 impact factor of a journal isNumber of times all items published in that journal in 2006 and 2007 were cited by indexed publications during 2008The total number of "citable items" (not letters or editorials) published by that journal in 2006 and 2007

Concept of sampling

• We are interested in attribute of a population• Can’t measure attribute in everyone• Therefore we need a representative sample– Random, Unbiased

• If representative, attribute in sample → estimation of attribute in population

• Size of sample is important– ↑ sample size →↑accuracy of estimate

Statistically speaking

• Measure attribute in sample– → Sample Mean

• Estimate of population mean– 95% confidence interval (CI) of sample mean– = 95% chance that the population mean lies within

the CI• Confidence interval depends on size sample– Larger the sample → the smaller the CI– Smaller the CI → more accurate the estimation

Concept statistical testing

• Interested in difference between an attribute in two populations

• Take two samples (one from each population)– Measure attribute in each sample– Calculate the difference between the attributes in

the two samples– Use this to estimate the difference between the

attribute in the two populations– Similarly can produce a statistical estimation of true

difference with confidence intervals

Concept Null hypothesis

• From statistical stand point– Assumption no difference between the groups– (Null hypothesis = H0)

• In the two samples you will almost always find a difference– Is that difference TRUE? (reject Null hypothesis)– Is there really no difference ? (Don’t Reject H0)

• Need statistical test → P value

P Value - definition

• If you assume that there is no difference in the populations, P value = Probability of finding a difference that large in the samples

• Small P value– Very unlikely that the observed difference in the

samples occurred by chance– Therefore likely to be a difference in populations

• If P < 0.05 (5%) → difference in sample is unlikely to be coincidental = (statistically) significant

P value and Confidence Interval

Significant result (p value is <0.05)≡

95% CI estimation of population difference will not include 0

Error

• Statistical testing is not infallible• Risk of getting a significant result when there is

no difference in the populations– False positive (OPTIMISTIC)– Probability of this = TYPE I Error

• Risk of not getting a significant result when there really is a difference in the populations– False negative (PESSIMISTIC)– Probability of this = TYPE II Error

Type I ERROR

• Even if the P value is small (e.g. p=0.001), there is still a chance that you could have seen the same difference in your samples when there really isn’t any difference in the population (e.g. 1 in 1000)

• If you took 1000 samples of the two populations with the same mean for testing, one of these 1000 tests would be significant

• TYPE I ERROR = Level of significance

Type II Error

• TYPE II Error• If there is a difference between two

populations then 95% CI should exclude 0• If the Sample size is small → the 95%CI for the

population difference will be wide and more likely to include 0 giving a false negative result

• The Type II error can be reduced by increasing the sample size

POWER

• POWER is opposite of TYPE II ERROR• TYPE II ERROR = Probability of not getting a

significant result when there really is a difference in the population means

• POWER = Probability of getting a significant result when there is a difference in the population means

• POWER = 100 – TYPE II ERROR

Calculating Power

• Complicated calculation• Requires– Sample size (both samples)– Difference in samples– The spread (standard deviation) of (both) samples– Statistical level of significance of test (usually 5%)

• ↑Power with – ↑Sample size– ↑Difference

– ↑ Level Significance– ↓ Standard deviation

Calculating Sample Size

• Same as equation as for power calculation• Requires– Size of difference you want to detect– Estimation of standard deviation in samples– Statistical level of significance of test (usually 5%)– Power that you want test to have (usually 80%)

• ↑sample size needed for – Small difference– ↑ Standard deviation

– ↓ Level Significance (1%)– ↑ Power

Beware Errors

• TYPE I: If a paper does 20 tests at 5% level on two similar populations one of these is likely to be significant

• TYPE 2: If the study is underpowered and concludes that there is no difference in the main outcome when it looks like there might be (unlikely)

• TYPE 2: If paper measures a secondary outcome that it is underpowered to detect and concludes that there is no significant difference when it looks like there might be

Guide for assessing RCTs