Biomedizinische Forschung: Mehr Wert, weniger Müll

Freiburg, 22.2.2017ulrich.dirnagl@charite.dehttp://dirnagl.com

http://bit.ly/freiburgdirnagl

Katharina FritschKatharina Fritsch

PLoS Biol. 2010;8(3):e1000344.

Metaanalysis of 20.000 animals in neuroprotection trials – all therapies are highly effective _____________________________

I.v. thrombolysis is the only clinically proven pharmacological therapy of acute ischemic stroke. Benefit only to a small percentage of stroke victims.(ARR 2%)

There is no therapeutic ‚neuroprotection or 'neuroregeneration' in human stroke.

Only thrombolysis clinically effective! ______

In vitro and in vivo - 1026

Tested in vivo - 603

Effective in vivo - 374

Tested in clinical trial - 97

Effective in clinical trial - 1

O’Collins et al, 2006

1026 interventions in experimental stroke_____

>100.000.000.000 € per year spent on preclinical research

> 4.000.000 researchers and clinicians work globally in academic and pharmaceutical biomedicine

How effective is research and development in biomedicine__________________________________

“85% of health research is wasted.”

Public concern _______________________________

Alarm in academia ____________________________

Help is at hand (on paper) __________________

John Ioannidis________________________________

3500 cit.

Ten years later ______________________________

Spotlight on preclinical _____________________

• Personal scientific midlife crisis

• Overwhelming evidence through Meta Research

• Preclinical research fundament of clinical trials - Translational roadblock

• selection bias (creating groups with different confounders; solved by randomization)

• performance bias and detection bias (investigators respectively treating or assessing more positively those subjects on the treatment arm; controlled by blinding interventions and outcome assessments)

• attrition bias (dropouts of subjects with a negative outcome not included in the final result)

Internal validity ____________________________

Macleod MR, et al. (2015) Risk of Bias in Reports of In Vivo Research:

A Focus for Improvement. PLoS Biol 13: e1002273.

Low prevalence of methods to prevent bias ____

PLoS Biol. 2016;14:e1002331

Effects of attrition in experimental biomedical research __________________________

PLoS Biol. 2016;14:e1002331

Effects of attrition in experimental biomedical research __________________________

Mean group size n ≈ 8

Mean statistical power ≈ 45 %

False positive rate (p ≤ 0.05): ≈ 50 %

Overestimation of true effects: ≈ 50 %

Low n's = low power, many false positives,inflated effect sizes ________________________

Overall median power of 730 primary neuroscience studies: 21 %

False positives and inflation of effect sizes _

• Low base rates (low prior probability)

• Low power (small n's, high variance, low

effect size)

• Winner's curse

• Regression to the mean

Even without bias and p-hacking, many statistically significant results are false positives, and effect sizes are inflated ______

Bias against the NULL hypothesis ______________

p > 0.05Repeat experiment, add animals or repeat statistics with different test (e.g. contrast) (i.e. p-hack), remove outliers (to nudge effect size in proper direction), try different strategy (antibody, assay, claim that the previous one 'did not work'), etc.

Once mission accomplished (p<0.05): don't talk about how you got there.

p < 0.05Move on to next experiment, write paper

Non-publication of results: Publication bias __

PLoS Biol. 2010; 8 e1000344

What can we do about it ?_____________________

Open access Education/Training (Statistics, study design etc.)Enforce compliance with existing guidelinesElectronic labbooks (preclinical)Authentication of reagents and biologicals (incl.

animals)Open data / Repositories / Publication of negative

resultsReplication (culture)Structured quality management (preclinical)Better study designs and analysisEnforced registration (studies, protocols, etc.)(Peer-) Auditing (preclinical)Large-scale cooperation / Data sharingEnforce publication of results (evidence)

Novel indicators and incentives

Impact

Practicability

How to make more published research findings true? ________________________________________

Reduce Bias!Use blinding, randomization,in/exclusion criteria.Publish Null results.Report results according to guidelines (e.g. ARRIVE)

Increase Power!Check your power. Achieve at least 80%.Do apriori sample size calculations.Probably you need to increase n‘s.Replicate.

Question ‚Statistical Significance‘!P-values do not provide evidence regarding a model or hypothesis.Think biological significance, think effect size.Use confidence intervals, not SEMs.Replicate.

Preclinical multicenter trials _______________

2015;7:299ra121

Grant Agreement FP7 2007-13 No. 278709

Grant Agreement n°HEALTH-F2-2013-603043

NIH/NINDS RFI NOT-NS-14-006

MULTIPART

Systematic replication (with higher n’s)______

Publication of NULL results - preregistration _

Prevents:Publication bias

Prevents:Outcome switching,Cherry picking of results

OPEN SCIENCE POLICY: Find, Access, Interoperate, Reuse Data (FAIR)_______________

• CAMARADES

• Collaborative Approach to Meta-Analysis and Review of Animal Data from Experimental Studies

• Look systematically across the modelling of a range of conditions

• Data Repository

– 19 Diseases

– 7,000 studies

– from over 200,000 animals

Data aggregation, Metaanalysis _______________

Training in critical appraisal of publications in biomedicine (IICARUS)

https://ecrf1.clinicaltrials.ed.ac.uk/iicarus/Training

http://syrf.org.uk/

https://www.nc3rs.org.uk/experimental-design-assistant-eda

Tools ________________________________________

http://www.nc3rs.org.uk/

PLoS Biol 2014;12: e1001756PLoS Biol 2014;12: e1001756PLoS Biol 2014;12: e1001756

Guidelines ___________________________________

Standardization and authentication ___________

Rewards and incentives ________________________

Source: Slate

Scientists need to publish new, positive

and spectacular results for professional

advancement

Journals need to publish new,

positive and spectacular results

to promote their IF

Institutions and Funders support

researchers who publish new,

positive and spectacular results

in high IF journals

Non-reproducible research findings

Failure to translate bench findings into

effective therapies

The vicious cycle of academic biomedical research _____________________________________

Reduce Bias• Use blinding, randomization,in/exclusion criteria.• Publish Null results.• Report results according to guidelines (e.g.

ARRIVE)

Increase Power• Check your power. Achieve at least 80%.• Do apriori sample size calculations.• Probably you need to increase n‘s.• Replicate.

Question ‚Statistical Significance‘• P-values do not provide evidence regarding a model

or hypothesis.• Think biological significance, think effect size.• Replicate.

What scientists can (must) do _________________

What institutions can (must) do _______________

Incentives and rewards• Professorships, tenure• Performance oriented funding• Teaching and training• …

Infrastructure• Office for Good Scientific Practice• Open data officers and policy• Electronic labbooks• …

Safeguard • adherence to guidelines• (Pre)Registration• Publication • …

What funders can (must) do ____________________

Project funding• Request measures to prevent bias and improve quality• Implement funding criteria regarding quality • Request open science policy • Request data management plan• Request preregistration • Request publication of NULL and negative results• Audit/monitor

Institutional funding• Request quality improving measures and structures• Request teaching and training concepts

Fund specific programs• Meta-research incl. systematic reviews• Replication funds (for critical findings)• Preclinical multicenter trials• Innovative training / education• Development of quality standards and management systems

for academic preclinical biomedicine

Academic researchers will not be able to substantia lly

improve the quality of current biomedical research

without changes in the academic system.

Most of the required measures involve the institutions

and funders , are straightforward, and could be

implemented swiftly.

Key to this: Development and implementation of nove l

metrics for appraising and rewarding biomedical

research (careers, funding).

Everything else will follow.

Conclusions ___________________________________ http://bit.ly/freiburgdirnagl