THE RISK OF RANDOM ERROR (PLAY OF CHANCE)

Goran Poropat

Introduction

No physical quantity can be measured with perfect certainty

All measurements are prone to errors

Experimental errors

Do not refer to mistakes, blunders, or miscalculations

(eg. measuring a width when the lenght should have been measured)

Inherent in the measurement process

Experimental errors

Are measured by a) Accuracy – how close a measured value is to

the true value or accepted value

b) Precision – how closely two or more measurements agree with each other (repeatability, reproducibility)

Experimental errors

Three dimensions particularly influence the reliability of our observations in clinical research:

1. RANDOM ERRORS (PLAY OF CHANCE)

2. SYSTEMATIC ERRORS (BIAS)

3. DESIGN ERRORS

Bias

A systematic error – deviation from the truth, in results or inferences

Overestimation or underestimation of the true intervention effect

Bias

Affect accuracy of a measurement

Should not be confused with imprecision

Multilpe replications of the same study – wrong answer on average

Random error

Imprecision – refers to random error

The unpredictable variation between observed values and some “true” value

Possible reason of misleading results in RCTs and meta-analyses

Random error

Affects precision of measurement

Multilpe replications of the same study

Different effect estimates

SAMPLING VARIABILITY

Sampling variability

The actual study result will vary depending on who is actually in the study sample

A sample – a subset of a population of manageable size

Epidemiological studies

Impossible to evaluate every member of the entire population

The relationship between exposure and health-related event is judged from observations on a SAMPLE of the population

STATISTICAL INFERENCES(extrapolations)

Sampling variability50 x x 50

N = 4

2 x

2 x

x 4

Sampling variability

Different inferences – various possible samples

Hypothesis Information size

Probability of drawing a bad sample

Random errors tend to decrease as information size increases

Sampling variability

q

RR

Number of patients randomised

1Required

sample size

Clinically important overestimate!

Epidemiological studies

The measure of association we observe (inference) in our data may differ from the “true” measure of association

- by chance alone

Probability that the observed difference is due to play of chance

Hypothesis testing

Quantify the degree to which sampling variability (chance) can explain the observed association

Assume H0 is true, and not Ha

The probability of obtaining an observed effect (or larger) under a null-hypothesis

Assessing H0 P-value

P-value

The likelihood of observing certain data given that the null-hypothesis is true

P-value threshold = 0.05 – arbitrary

Data yielding a P-value = 0.05 – a 5% chance obtaining the observed result, if no real effect exists

P-value

A P-value is the probability of an observed (or more extreme) result arising by chance

Misinterpretations

“the intervention has no effect”

“not strong evidence that the intervention has an effect”

“ an intervention has a strong benefit”

The P value addresses the question of whether the intervention effect is precisely nil

P>0.05

P<0.05

Confidence intervals

Another approach to quantify sampling variability

Range within which the true magnitude of effect lies with a stated probability, or a certain degree of assurance (usually 95%)

Confidence intervals

Point estimate – the actual measure of association given by the data (OR, RR, RD)

The best guess of the magnitude and direction of the experimental intervention’s effect compared with the control intervention

Confidence intervals

Wider intervals – greater imprecision

CI width• Sample size• Precision of individual study estimates• Number of studies combined

Confidence intervals

P-value – to which extent the null-hypothesis is compatible with the data

CI – the range of hypothesis compatible with the data

Sum up

• Random error (due to ’play of chance’) is the unpredictable variation between observed values and some ’true’ value

• Everything we attempt to estimate may be subject to some degree of random error

Sum up

• Random error affects• Statistical significance • Estimated treatment effects• Heterogeneity estimates

• Only a sufficient number of trials and patients will ensure an acceptable risk of random error

Thank you for your attention!