12630 Statistics for Authors Canadian Journal of Anesthesia Journal canadien d’anesthésie...

12630

Statistics for AuthorsCanadian Journal of Anesthesia Journal canadien d’anesthésie

Winterlude Anesthesia SymposiumFebruary 1, 2015Ottawa, Ontario

Canadian Journal of Anesthesia Journal canadien d‘anesthésie

springer.com/12630

Short version of this session• Hire an experienced PhD biostatistician• … before you submit to ethics• … understand your data, but don’t touch it• … statistician writes statistical methods (results).• Thank you, let’s go watch the SuperBowl.


springer.com/12630

Disclosure• No financial COI• Not, definitely not, a statistician• Not particularly good at math• MSc Epidemiology• Deputy Editor-in-Chief, Canadian Journal of Anesthesia• I do a lot of peer review


springer.com/12630

Objectives• I will not tell you how to do statistics• I will not show you formulae• Discuss common statistical problems identified at

peer review• Design• Analysis• Reporting

• Links and take home points on Twitter• @glbryson• #wntrane15


springer.com/12630

• Studies are small• Effect sizes are small• Greater number of less predetermined statistical tests• Flexibility in design, outcomes, and analysis• Conflict of interest• Multiple teams competing


springer.com/12630


springer.com/12630

What is the research question?

P = PopulationI = InterventionC = ComparatorO = Outcome


springer.com/12630

Q1. What percent of trials change their primary outcome between registration and publication?

A. NoneB. 3%C. 10%D. 30%


springer.com/12630

Trials N (%)Identified 323Registered 147 (46)Different primary outcome 46 (31)Change statistically significant 19 (41)

Things change …

JAMA. 2009;302(9):977-984


springer.com/12630

Q1. What percent of trials change their primary outcome between registration and publication?

A. NoneB. 3%C. 10%D. 30%


springer.com/12630

Q2. What is a meaningful change in pain score?

A. 1 pointB. 10%C. 2 pointsD. 30%


springer.com/12630

Does it feel better?

• 2,700 patients in 10 pregabalin-neuropathy trials• 5 to 12 weeks of therapy• NRS pain cw 7 point “improved” scale

• -1.74 points or -27.9% decrease • 134 patients, fentanyl cancer-pain breakthrough

• Measurements every 15 minutes for 1 hour• Success = not requiring additional medication at 30 minutes

• - 2 points or <33% decrease

Farrar JT Pain 2001;94;149–158

Farrar JT. J Pain Sympt Manage 2003;25:406-11


springer.com/12630

Q2. What is a meaningful change in pain score?

A. 1 pointB. 10%C. 2 pointsD. 30%


springer.com/12630

Q3. What’s wrong with this sample size estimate?

In this three-group trial the authors state“… therapy would reduce [pain] by 30%; power analysis with α = 0.05 and β = 0.80 revealed that we would need to enrol 24 patients in each group”


springer.com/12630

Required Elements for Sample Size Estimate

Mean ProportionAlpha (type 1) error Alpha (type 1) errorBeta (1-power) error Beta (1-power) errorMean intervention Rate interventionMean control Rate controlStandard deviation


springer.com/12630

Q3. Identify errors in this sample size estimate

In this three-group trial the authors state“… therapy would reduce [pain] by 30%; power analysis with α = 0.05 and β = 0.80 revealed that we would need to enrol 24 patients in each group”

Category ErrorCentral tendency Mean pain in target populationDispersion SD pain in target population


springer.com/12630

Q4. How many comparisons in 3 group trial?

A

CB

0.05

0.01670.0167

0.0167


springer.com/12630

Q5. Are we any good at sample size?• Two years worth of top medicine journals

• NEJM, JAMA, PLoS Med, Lancet, BMJ, Ann Int Med

• 215 citations with median sample size of 425• 113 (53%) reported all elements required• 146 (68%) assumptions were <30% off observed result• 73 (34%) estimates were complete and accurate• 96 (45%) were registered with a sample size estimate• 46 (21%) registration matched report

BMJ 2009;338:b1732

NOT SO MUCH


springer.com/12630

TreatmentN=30

ControlN=30

P Value

48 hr morphine 27 (15) 32 (16) 0.12

NRS pain 2 (3) 3(3) 0.23

BPI function 4 (5) 4 (6) 0.78

SF-MPQ -2 3.2 (3.6) 3.6 (3.1) 0.78

Continuous3.2 (2.5) 4.0 (2.0) 0.04*

Intermittent3.3 (2.6) 2.8 (3.0) 0.23

Neuropathic2.2 (2.9) 2.1 (2.4) 0.87

Affective 2.6 (2.8) 2.4 (2.4) 0.75

Q6. What’s wrong with this table?

Data presented as mean (SD). P for unpaired T-test indicated


springer.com/12630

TreatmentN=30

ControlN=30

P Value

PACU 7 (8) 8 (10) 0.45

6 hr 6 (6) 8 (10) 0.23

12 hr 2 (5) 6 (8) 0.04*

24 hr 6 (8) 6 (9) 0.78

48 hr 6 (10) 4 (10) 0.32

Total 27 (15) 32 (16) 0.12

Q7. Or this one?

Data presented as mean (SD). P for unpaired T-test indicated


springer.com/12630

Answer Qs 6 and 7 = Multiplicity• Alpha errors accumulate• FWER = 1 – (1-alpha)number of tests

• FWER = 1 – (1-0.05)8 = 1-0.663 = 33.7%• Multiple primary outcomes• Multiple related outcomes• Repeated measures• Multiple treatment groups• Interim analyses

• The more cuts at the data the more like you are to make a type 1 error.


springer.com/12630

P-hacking• Coined by Joseph Simmons, Leif Nelson, and Uri Simonsohn• Generally refers to repeated analysis of data until P < 0.05

appears• Complicated debate, Bayes’ theorem, false discovery rates• Bottom line.

• Analyze only what you need• In the way your protocol described it


springer.com/12630

Q8. Do we favor positive results?

Kühberger, Fritz, and Schendl. PLoSOne 2014;9(9):e105825.


springer.com/12630

Q8. Do we favor positive results?

RT @mc_hankins

You bet we do


springer.com/12630

P values by XKCD

http://xkcd.com/1478/

Or if all else fails, use “significant at P>0.05” and hope no one notices.


springer.com/12630

Q9. Which of the following is “better?”

A. Morphine consumption reduced cw placebo (P 0.017)B. 33% reduction in morphine consumption cw placebo (P 0.017)C. Morphine consumption 18mg (7) v 12mg (6) (P 0.017)D. Mean difference morphine consumption 6mg (95% CI 2-9)

• P values conflate statistical and clinical significance• Must report measure of effect• Absolute measure preferred to relative.• Mean difference with 95% confidence limits• Absolute risk reduction or NNT


springer.com/12630

• Registration• Replication• Open data• Standardized outcomes• More stringent statistical approach

12630 Statistics for Authors Canadian Journal of Anesthesia Journal canadien d’anesthésie...

Documents

Transcript of 12630 Statistics for Authors Canadian Journal of Anesthesia Journal canadien d’anesthésie...