Lies, Damned Lies, & Search Marketing Statistics By Adria Kyne
STATISTICAL LEARNING FROM DATA “LIES, DAMNED LIES, AND STATISTICS’’, Mark Twain. Senate...
-
Upload
curtis-armstrong -
Category
Documents
-
view
216 -
download
3
Transcript of STATISTICAL LEARNING FROM DATA “LIES, DAMNED LIES, AND STATISTICS’’, Mark Twain. Senate...
![Page 1: STATISTICAL LEARNING FROM DATA “LIES, DAMNED LIES, AND STATISTICS’’, Mark Twain. Senate Approves Tighter Policing of Drug Makers, May 8, 2007 Mark van.](https://reader031.fdocuments.us/reader031/viewer/2022032722/56649cf15503460f949c0aee/html5/thumbnails/1.jpg)
STATISTICAL LEARNING FROM DATA
“LIES, DAMNED LIES, AND STATISTICS’’, Mark Twain.
Senate Approves Tighter Policing of Drug Makers, May 8, 2007
Mark van der Laan, www.stat.berkeley.edu/~laan
![Page 2: STATISTICAL LEARNING FROM DATA “LIES, DAMNED LIES, AND STATISTICS’’, Mark Twain. Senate Approves Tighter Policing of Drug Makers, May 8, 2007 Mark van.](https://reader031.fdocuments.us/reader031/viewer/2022032722/56649cf15503460f949c0aee/html5/thumbnails/2.jpg)
OVERVIEW How good is the human statistical intuition? Statistics are a tool of modern life. The loose statistical world of data analyses and scientific
publishing: “Why more than half [data based] published research findings are false?” (John P.A. Ioannidis)
The rigid statistical world of the FDA and Drug Manufacturers: Role of statistics in clinical trials.
Advances in statistics can help but are heavily un-used: The challenge.
Can we reduce the cost of prescription drugs through improvements in statistical practice and design?
Can statistics improve safety reviews of drugs in use? Concluding remarks.
![Page 3: STATISTICAL LEARNING FROM DATA “LIES, DAMNED LIES, AND STATISTICS’’, Mark Twain. Senate Approves Tighter Policing of Drug Makers, May 8, 2007 Mark van.](https://reader031.fdocuments.us/reader031/viewer/2022032722/56649cf15503460f949c0aee/html5/thumbnails/3.jpg)
THE QUIZZ-MASTER PROBLEM
Behind one of these doors is a car.Behind the other two is a goat.Click on the door that you think the car is behind.
![Page 4: STATISTICAL LEARNING FROM DATA “LIES, DAMNED LIES, AND STATISTICS’’, Mark Twain. Senate Approves Tighter Policing of Drug Makers, May 8, 2007 Mark van.](https://reader031.fdocuments.us/reader031/viewer/2022032722/56649cf15503460f949c0aee/html5/thumbnails/4.jpg)
YOU SELECT DOOR 1
To keep it exciting, I open one of the other two doors without a car behind it.Obviously the car is not behind door 3.But before I open door 1, the door you selected,I'm going to let you switch to door 2 if you like.Again, click on the door in which you think the car is behind.
![Page 5: STATISTICAL LEARNING FROM DATA “LIES, DAMNED LIES, AND STATISTICS’’, Mark Twain. Senate Approves Tighter Policing of Drug Makers, May 8, 2007 Mark van.](https://reader031.fdocuments.us/reader031/viewer/2022032722/56649cf15503460f949c0aee/html5/thumbnails/5.jpg)
Congratulations! You're a winner!
Recap: You originally picked door 1 and then switched to door 2.
Here is a summary of how previous contestants have fared.
# of Players Winners Percent Winners
Switched 131 88 67.2
Didn't Switch 116 42 36.2
![Page 6: STATISTICAL LEARNING FROM DATA “LIES, DAMNED LIES, AND STATISTICS’’, Mark Twain. Senate Approves Tighter Policing of Drug Makers, May 8, 2007 Mark van.](https://reader031.fdocuments.us/reader031/viewer/2022032722/56649cf15503460f949c0aee/html5/thumbnails/6.jpg)
This problem was given the name The Monty Hall Paradox in honor of the long time host of the television game show "Let's Make a Deal." Articles about the controversy appeared in the New York Times and other papers around the country.
Marilyn's answer to the reader was that the contestant should switch doors and she received nearly 10,000 responses from readers, most of them disagreeing with her. Hundreds of them were Professors in mathematics and scientists whose responses ranged from hostility to disappointment at the nation's lack of mathematical skills.
Quizz-Master Problem was raised by reader in Marilyn Vos Savant's Sunday Parade Column
![Page 7: STATISTICAL LEARNING FROM DATA “LIES, DAMNED LIES, AND STATISTICS’’, Mark Twain. Senate Approves Tighter Policing of Drug Makers, May 8, 2007 Mark van.](https://reader031.fdocuments.us/reader031/viewer/2022032722/56649cf15503460f949c0aee/html5/thumbnails/7.jpg)
Statistics are a tool of modern life Identifying associations,
correlations and patterns Establishing causation
based on randomized trials and observational studies.
Making predictions Shaping strategies and
future behavior Of people and societies Of machines and computing Of complex systems
![Page 8: STATISTICAL LEARNING FROM DATA “LIES, DAMNED LIES, AND STATISTICS’’, Mark Twain. Senate Approves Tighter Policing of Drug Makers, May 8, 2007 Mark van.](https://reader031.fdocuments.us/reader031/viewer/2022032722/56649cf15503460f949c0aee/html5/thumbnails/8.jpg)
Statistics are often cited to denotea certainty that does not, in fact, exist “There is increasing concern
that in modern research, false findings may be the majority, or even the vast majority, of published research claims.” “Simulations show that for most
study designs and settings it is more likely for a research claim to be false than true.”
“For many current scientific fields, claimed research findings often may be simply accurate measures of the prevailing bias.”
- J.P.A. Ioannidis, “Why Most Published Research Findings are False”, Chance, Vol. 18, No. 4, 2005
![Page 9: STATISTICAL LEARNING FROM DATA “LIES, DAMNED LIES, AND STATISTICS’’, Mark Twain. Senate Approves Tighter Policing of Drug Makers, May 8, 2007 Mark van.](https://reader031.fdocuments.us/reader031/viewer/2022032722/56649cf15503460f949c0aee/html5/thumbnails/9.jpg)
![Page 10: STATISTICAL LEARNING FROM DATA “LIES, DAMNED LIES, AND STATISTICS’’, Mark Twain. Senate Approves Tighter Policing of Drug Makers, May 8, 2007 Mark van.](https://reader031.fdocuments.us/reader031/viewer/2022032722/56649cf15503460f949c0aee/html5/thumbnails/10.jpg)
![Page 11: STATISTICAL LEARNING FROM DATA “LIES, DAMNED LIES, AND STATISTICS’’, Mark Twain. Senate Approves Tighter Policing of Drug Makers, May 8, 2007 Mark van.](https://reader031.fdocuments.us/reader031/viewer/2022032722/56649cf15503460f949c0aee/html5/thumbnails/11.jpg)
Causal Inference and Curse of Dimensionality
Causal Model and Data
Text from Taubes NYTimes Article
![Page 12: STATISTICAL LEARNING FROM DATA “LIES, DAMNED LIES, AND STATISTICS’’, Mark Twain. Senate Approves Tighter Policing of Drug Makers, May 8, 2007 Mark van.](https://reader031.fdocuments.us/reader031/viewer/2022032722/56649cf15503460f949c0aee/html5/thumbnails/12.jpg)
Text from Taubes NYTimes Article
![Page 13: STATISTICAL LEARNING FROM DATA “LIES, DAMNED LIES, AND STATISTICS’’, Mark Twain. Senate Approves Tighter Policing of Drug Makers, May 8, 2007 Mark van.](https://reader031.fdocuments.us/reader031/viewer/2022032722/56649cf15503460f949c0aee/html5/thumbnails/13.jpg)
![Page 14: STATISTICAL LEARNING FROM DATA “LIES, DAMNED LIES, AND STATISTICS’’, Mark Twain. Senate Approves Tighter Policing of Drug Makers, May 8, 2007 Mark van.](https://reader031.fdocuments.us/reader031/viewer/2022032722/56649cf15503460f949c0aee/html5/thumbnails/14.jpg)
![Page 15: STATISTICAL LEARNING FROM DATA “LIES, DAMNED LIES, AND STATISTICS’’, Mark Twain. Senate Approves Tighter Policing of Drug Makers, May 8, 2007 Mark van.](https://reader031.fdocuments.us/reader031/viewer/2022032722/56649cf15503460f949c0aee/html5/thumbnails/15.jpg)
![Page 16: STATISTICAL LEARNING FROM DATA “LIES, DAMNED LIES, AND STATISTICS’’, Mark Twain. Senate Approves Tighter Policing of Drug Makers, May 8, 2007 Mark van.](https://reader031.fdocuments.us/reader031/viewer/2022032722/56649cf15503460f949c0aee/html5/thumbnails/16.jpg)
![Page 17: STATISTICAL LEARNING FROM DATA “LIES, DAMNED LIES, AND STATISTICS’’, Mark Twain. Senate Approves Tighter Policing of Drug Makers, May 8, 2007 Mark van.](https://reader031.fdocuments.us/reader031/viewer/2022032722/56649cf15503460f949c0aee/html5/thumbnails/17.jpg)
False conclusions are expensive In medicine
False positives lead to expensive additional tests and anxiety False negatives lead to delayed treatment with escalated costs
and illness In drug discovery
False positives lead to failed trials The average cost of a phase III clinical trial is $4m-$20m,
some cost more than $100m False negatives lead to failed trials
Missed contraindications, negative interactions and imprecise dosages
In genomics, proteomics and chemoinformatics False positives are abundant and lead to wasted time, effort and
experimentation False negatives lead to missed business opportunities
In public policy False positives and false negatives lead to action based on false
premises and, frequently, public cynicism
![Page 18: STATISTICAL LEARNING FROM DATA “LIES, DAMNED LIES, AND STATISTICS’’, Mark Twain. Senate Approves Tighter Policing of Drug Makers, May 8, 2007 Mark van.](https://reader031.fdocuments.us/reader031/viewer/2022032722/56649cf15503460f949c0aee/html5/thumbnails/18.jpg)
Bias is a hazard of statistics Statistical data samples can be
biased The sample selected does not
represent the population Example: There are five red
heads in a town of 100 people. Our sample of 20 people happens to include all five.
Statistical methods for learning from data can be biased The statistical model selected is
not the one that best fits the data…
… for the question being asked! Statistical interpretations of
findings can be biased.
![Page 19: STATISTICAL LEARNING FROM DATA “LIES, DAMNED LIES, AND STATISTICS’’, Mark Twain. Senate Approves Tighter Policing of Drug Makers, May 8, 2007 Mark van.](https://reader031.fdocuments.us/reader031/viewer/2022032722/56649cf15503460f949c0aee/html5/thumbnails/19.jpg)
Variable Importance of HIV resistance mutations Goal: Rank a set of genetic mutations based on their
importance for determining an outcome Mutations (A) in the HIV protease enzyme
Measured by sequencing Outcome (Y) = change in viral load 12 weeks after starting
new regimen containing saquinavir How important is each mutation for viral resistance to this
specific protease inhibitor drug? Inform genotypic scoring systems
![Page 20: STATISTICAL LEARNING FROM DATA “LIES, DAMNED LIES, AND STATISTICS’’, Mark Twain. Senate Approves Tighter Policing of Drug Makers, May 8, 2007 Mark van.](https://reader031.fdocuments.us/reader031/viewer/2022032722/56649cf15503460f949c0aee/html5/thumbnails/20.jpg)
Stanford Drug Resistance Database
All Treatment Change Episodes (TCEs) in the Stanford Drug Resistance Database Patients drawn from 16 clinics in Northern CA
Baseline Viral Load
Viral GenotypeTCE (Change >= 1 Drug)
Final Viral Load
Change in Regimen
<24 weeks 12 weeks
333 patients on saquinavir regimen
![Page 21: STATISTICAL LEARNING FROM DATA “LIES, DAMNED LIES, AND STATISTICS’’, Mark Twain. Senate Approves Tighter Policing of Drug Makers, May 8, 2007 Mark van.](https://reader031.fdocuments.us/reader031/viewer/2022032722/56649cf15503460f949c0aee/html5/thumbnails/21.jpg)
Parameter of Interest Need to control for a range of other covariates W
Include: past treatment history, baseline clinical characteristics, non-protease mutations, other drugs in regimen
Parameter of Interest: Variable Importanceψ = E[E(Y|Aj=1,W)-E(Y|Aj=0,W)] For each protease mutation (indexed by j)
![Page 22: STATISTICAL LEARNING FROM DATA “LIES, DAMNED LIES, AND STATISTICS’’, Mark Twain. Senate Approves Tighter Policing of Drug Makers, May 8, 2007 Mark van.](https://reader031.fdocuments.us/reader031/viewer/2022032722/56649cf15503460f949c0aee/html5/thumbnails/22.jpg)
Analytic approach Standard approach:
Fit a single multivariable regression E(Y|A,W) i.e. Regress clinical response on mutations,
covariates Is this the best approach for answering the
scientific question of interest? What is the scientific question?
Construct best predictor vs. Estimate importance of each mutation
![Page 23: STATISTICAL LEARNING FROM DATA “LIES, DAMNED LIES, AND STATISTICS’’, Mark Twain. Senate Approves Tighter Policing of Drug Makers, May 8, 2007 Mark van.](https://reader031.fdocuments.us/reader031/viewer/2022032722/56649cf15503460f949c0aee/html5/thumbnails/23.jpg)
Prediction vs. Importance
Prediction – create a model that the clinician will use to help predict risk of a disease for the patient.
Explanation – trying to investigate the causal association of a treatment or risk factor and a disease outcome.
![Page 24: STATISTICAL LEARNING FROM DATA “LIES, DAMNED LIES, AND STATISTICS’’, Mark Twain. Senate Approves Tighter Policing of Drug Makers, May 8, 2007 Mark van.](https://reader031.fdocuments.us/reader031/viewer/2022032722/56649cf15503460f949c0aee/html5/thumbnails/24.jpg)
Targeted Maximum Likelihood MLE- aims to do good job of estimating whole
density Targeted MLE- aims to do good job at parameter
of interest General decrease in bias for parameter of Interest Protection under the null hypothesis Honest p-values, inference, multiple testing
![Page 25: STATISTICAL LEARNING FROM DATA “LIES, DAMNED LIES, AND STATISTICS’’, Mark Twain. Senate Approves Tighter Policing of Drug Makers, May 8, 2007 Mark van.](https://reader031.fdocuments.us/reader031/viewer/2022032722/56649cf15503460f949c0aee/html5/thumbnails/25.jpg)
Targeted Maximum Likelihood
In regression case, implementation just involves adding a covariate h(A,W) to the regression model
Requires estimating g(A|W) E.g. distribution of each mutation given covariates
Robust: Estimate of ψ is consistent if either g(A|W) is estimated consistently E(Y|A,W) is estimated consistently
More on this later . . .
)|Pr()|( where,)|0(
)0(
)|1(
)1(),( WaAWag
Wg
AI
Wg
AIWAh
![Page 26: STATISTICAL LEARNING FROM DATA “LIES, DAMNED LIES, AND STATISTICS’’, Mark Twain. Senate Approves Tighter Policing of Drug Makers, May 8, 2007 Mark van.](https://reader031.fdocuments.us/reader031/viewer/2022032722/56649cf15503460f949c0aee/html5/thumbnails/26.jpg)
Mutation Rankings Based on Variable Importance
Current Score Mutation VIM VIM p-value Crude Crude p-value
35 90M 0.70 0.00 0.76 0.00
40 48VM 0.79 0.00 1.07 0.00
0 30N -0.78 0.00 -1.06 0.00
10 82AFST 0.46 0.01 0.35 0.03
10 54VA 0.46 0.01 0.31 0.11
10 73CSTA 0.67 0.03 0.80 0.00
2 20IMRTVL 0.32 0.07 0.26 0.18
1 36ILVTA 0.28 0.10 0.27 0.12
2 10FIRVY 0.27 0.13 0.48 0.00
5 88DTG -0.23 0.24 -0.50 0.33
2 71TVI 0.18 0.29 0.14 0.37
5 32I -0.18 0.58 -0.20 0.55
2 63P 0.06 0.77 0.11 0.56
5 46ILV 0.13 0.98 0.27 0.10
![Page 27: STATISTICAL LEARNING FROM DATA “LIES, DAMNED LIES, AND STATISTICS’’, Mark Twain. Senate Approves Tighter Policing of Drug Makers, May 8, 2007 Mark van.](https://reader031.fdocuments.us/reader031/viewer/2022032722/56649cf15503460f949c0aee/html5/thumbnails/27.jpg)
“Better Evaluation Tools – Biomarkers and Disease” #1 highly-targeted research project in FDA “Critical Path Initiative”
Requests “clarity on the conceptual framework and evidentiary standards for qualifying a biomarker for various purposes”
“Accepted standards for demonstrating comparability of results, … or for biological interpretation of significant gene expression changes or mutations”
Proper identification of biomarkers can . . .
Identify patient risk or disease susceptibility Determine appropriate treatment regime Detect disease progression and clinical outcomes Access therapy effectiveness Determine level of disease activity etc . . .
![Page 28: STATISTICAL LEARNING FROM DATA “LIES, DAMNED LIES, AND STATISTICS’’, Mark Twain. Senate Approves Tighter Policing of Drug Makers, May 8, 2007 Mark van.](https://reader031.fdocuments.us/reader031/viewer/2022032722/56649cf15503460f949c0aee/html5/thumbnails/28.jpg)
Evaluation of Biomarker Discovery Methods
> Univariate Linear Regression Importance measure: Coefficient value
with associated p-valueMeasures marginal association
> RandomForest (Breiman 2001)Importance measures (no p-values)
RF1: variable’s influence on error rate
RF2: mean improvement in node splits due to
variable
> Variable Importance with LARS
• Importance measure: causal effect
Formal inference, p-values providedLARS used to fit initial E[Y|A,W]
estimate W={marginally significant covariates}
All p-values are FDR adjusted
> Test methods ability to determine “true” variables under increasing correlation conditions
• Ranking by measure and p-value• Minimal list necessary to get all “true”?
> Variables Block Diagonal correlation structure: 10 independent sets of 10
Multivariate normal distributionConstant ρ, variance=1ρ={0,0.1,0.2,0.3,…,0.9}
> OutcomeMain effect linear model10 “true” biomarkers, one variable from each set of 10
Equal coefficients Noise term with mean=0 sigma=10
“realistic noise”
),0|(),|( WAYEWaAYE pp
Methods Simulation Study
![Page 29: STATISTICAL LEARNING FROM DATA “LIES, DAMNED LIES, AND STATISTICS’’, Mark Twain. Senate Approves Tighter Policing of Drug Makers, May 8, 2007 Mark van.](https://reader031.fdocuments.us/reader031/viewer/2022032722/56649cf15503460f949c0aee/html5/thumbnails/29.jpg)
Simulation Results
No appreciable difference in ranking by importance measure or p-value plot above is with respect to ranked importance measures
List Length for linear regression and randomForest increase with increasing correlation, Variable Importance w/LARS stays near minimum (10) through ρ=0.6, with only small decreases in power
Linear regression list length is 2X Variable Importance list length at ρ=0.4 and 4X at ρ=0.6
RandomForest (RF2) list length is consistently short than linear regression but still is 50% than Variable Importance list length at ρ=0.4, and twice as long at ρ=0.6
Variable importance coupled with LARS estimates true causal effect and outperforms both linear regression and randomForest
Minimal List length to obtain all 10 “true” variables
0
20
40
60
80
100
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
Correlation
Lis
t L
en
gth
Linear Reg
VImp w/LARS
RF1
RF2
![Page 30: STATISTICAL LEARNING FROM DATA “LIES, DAMNED LIES, AND STATISTICS’’, Mark Twain. Senate Approves Tighter Policing of Drug Makers, May 8, 2007 Mark van.](https://reader031.fdocuments.us/reader031/viewer/2022032722/56649cf15503460f949c0aee/html5/thumbnails/30.jpg)
THE ‘’RIGID’’ STATISTICAL WORLD OF THE FDA Clinical Trials Rigid statistical methodology and designs
required for FDA approval.
![Page 31: STATISTICAL LEARNING FROM DATA “LIES, DAMNED LIES, AND STATISTICS’’, Mark Twain. Senate Approves Tighter Policing of Drug Makers, May 8, 2007 Mark van.](https://reader031.fdocuments.us/reader031/viewer/2022032722/56649cf15503460f949c0aee/html5/thumbnails/31.jpg)
Clinical trials are expensive Time to market is critical
Half of the time-to-approval (currently 15.3 yrs) is spent in clinical trials Each day of delay is expensive
Moderately successful drug: $1m per day in lost sales Blockbuster drug: $3m per day
A lot of money is involved Spending on US-sponsored clinical trials is $25.6b in 2006
Biotech + Pharma = $22.6b, NIH = $3 9,937 trials this year
Pharma: 71% of R&D goes to drug development, 45% of this goes to clinical trials
Recruiting patients is expensive Direct costs of patient recruitment are high: $440m per year Indirect costs due to delays
#1 contributor to drug application delays 94% of trials in US miss their enrollment deadlines (Europe: 82%) 80% are delayed at least one month
Drop outs are a major problem 1 of 4 volunteers drops out of a study after it begins
![Page 32: STATISTICAL LEARNING FROM DATA “LIES, DAMNED LIES, AND STATISTICS’’, Mark Twain. Senate Approves Tighter Policing of Drug Makers, May 8, 2007 Mark van.](https://reader031.fdocuments.us/reader031/viewer/2022032722/56649cf15503460f949c0aee/html5/thumbnails/32.jpg)
MOVING TOWARDS ADAPTIVE CLINICAL TRIALS‘’A widely noted survey by Accenture provided some alarming
figures a few years ago: Eighty-nine percent of all drug candidates from the initiation of Phase I through FDA approval fail and many of them in the clinic.
Clearly, any techniques that could give an earlier read on these issues would be valuable. In too many cases , the chief result of a trial is to show that the trial itself was set up wrong, in ways that only became clear after the data were un-blinded. Did the numbers show that your dosage was suboptimal partway into a two year trial? Too bad- you probably weren'tallowed to know that. Were several arms of your study obviously pointless from the start? Even if you know, what could you do about it without harming the validity of the whole effort? Over the last years, such concerns have stimulated an unprecedented amount of work on new approaches. Ideas have come from industry, academia, and regulatory agencies such as the FDA's critical path initiative. A common theme in these efforts has been to move toward adaptive clinical trials.’’
![Page 33: STATISTICAL LEARNING FROM DATA “LIES, DAMNED LIES, AND STATISTICS’’, Mark Twain. Senate Approves Tighter Policing of Drug Makers, May 8, 2007 Mark van.](https://reader031.fdocuments.us/reader031/viewer/2022032722/56649cf15503460f949c0aee/html5/thumbnails/33.jpg)
Approval of Drugs, and Post-Market Safety Reviews of Drugs
FDA approvals are based on inefficient and often biased statistical methods (e.g., in how
they deal with informative drop out.) FDA does not have expertise to do post market
safety reviews, since this requires expertise in the challenging field of causal inference.
FDA needs to be modernized and need to make strong alliances with academic and industrial centers of excellence.
![Page 34: STATISTICAL LEARNING FROM DATA “LIES, DAMNED LIES, AND STATISTICS’’, Mark Twain. Senate Approves Tighter Policing of Drug Makers, May 8, 2007 Mark van.](https://reader031.fdocuments.us/reader031/viewer/2022032722/56649cf15503460f949c0aee/html5/thumbnails/34.jpg)
Senate Approves Tighter Policing of Drug Makers, May 8, 2007
![Page 35: STATISTICAL LEARNING FROM DATA “LIES, DAMNED LIES, AND STATISTICS’’, Mark Twain. Senate Approves Tighter Policing of Drug Makers, May 8, 2007 Mark van.](https://reader031.fdocuments.us/reader031/viewer/2022032722/56649cf15503460f949c0aee/html5/thumbnails/35.jpg)
Statistical Innovations are Available Statistical Inference for Adaptive designs. Targeted (Maximum Likelihood) Learning in
Biomarker Discovery Targeted (Maximum Likelihood) Learning of
Causal Effects of drugs and other interventions Targeted Learning of Treatment effect
modification due to genetic and genomic factors (Multiple Testing).
Learning of Individualized Treatment Rules (e.g., individualized medicine).
Super Learning in Prediction
![Page 36: STATISTICAL LEARNING FROM DATA “LIES, DAMNED LIES, AND STATISTICS’’, Mark Twain. Senate Approves Tighter Policing of Drug Makers, May 8, 2007 Mark van.](https://reader031.fdocuments.us/reader031/viewer/2022032722/56649cf15503460f949c0aee/html5/thumbnails/36.jpg)
CONCLUDING REMARKS: ‘’Statistics’’ can do a lot of harm in the hands of
people. Any published statistical analysis should be based
on publicly available data. Any statistical analysis should be based on a
priori specified analysis plan (MACHINE LEARNING) and any HUMAN INDUCED deviation from it should be documented (just like the FDA requires!).
Statistical tools used in practice need to improved: FDA needs to be modernized with well founded statistical innovations.