Quality assurance in clinical trials Marc Buyse IDDI, Louvain-la-Neuve, Belgium Sensible Guidelines...
-
Upload
nicolas-tillerson -
Category
Documents
-
view
214 -
download
1
Transcript of Quality assurance in clinical trials Marc Buyse IDDI, Louvain-la-Neuve, Belgium Sensible Guidelines...
![Page 1: Quality assurance in clinical trials Marc Buyse IDDI, Louvain-la-Neuve, Belgium Sensible Guidelines for the Conduct of Clinical Trials Washington, January.](https://reader036.fdocuments.us/reader036/viewer/2022081602/5518b31b550346a61f8b4fd0/html5/thumbnails/1.jpg)
Quality assurance in clinical trials
Marc Buyse
IDDI, Louvain-la-Neuve, Belgium
Sensible Guidelines for the Conduct of Clinical Trials
Washington, January 25-26, 2007
![Page 2: Quality assurance in clinical trials Marc Buyse IDDI, Louvain-la-Neuve, Belgium Sensible Guidelines for the Conduct of Clinical Trials Washington, January.](https://reader036.fdocuments.us/reader036/viewer/2022081602/5518b31b550346a61f8b4fd0/html5/thumbnails/2.jpg)
Outline
1. Data errorsDo they matter?
2. On-site monitoring Is it useful?
3. Statistical data checking Is it possible?
![Page 3: Quality assurance in clinical trials Marc Buyse IDDI, Louvain-la-Neuve, Belgium Sensible Guidelines for the Conduct of Clinical Trials Washington, January.](https://reader036.fdocuments.us/reader036/viewer/2022081602/5518b31b550346a61f8b4fd0/html5/thumbnails/3.jpg)
Quality Assurance – why ?
The purpose of quality assurance is not to ensure that the data are 100% error-free.
Its purpose is to ensure that the clinical trial results are reliable, i.e.
• observed treatment effects are real• their estimated magnitude is unbiased
![Page 4: Quality assurance in clinical trials Marc Buyse IDDI, Louvain-la-Neuve, Belgium Sensible Guidelines for the Conduct of Clinical Trials Washington, January.](https://reader036.fdocuments.us/reader036/viewer/2022081602/5518b31b550346a61f8b4fd0/html5/thumbnails/4.jpg)
A taxonomy of errors
Random errors*– Measurement errors (eg due to assay precision or
frequency of visits)– Errors due to slopiness (eg transcription errors)– Many types of fraud (most cases of data fabrication)
Systematic errors– Design flaws (eg exclusion of patients with
incomplete treatment or unequal schedule of visits)– Some types of fraud (most cases of data falsification)
* Random with respect to treatment assignment
![Page 5: Quality assurance in clinical trials Marc Buyse IDDI, Louvain-la-Neuve, Belgium Sensible Guidelines for the Conduct of Clinical Trials Washington, January.](https://reader036.fdocuments.us/reader036/viewer/2022081602/5518b31b550346a61f8b4fd0/html5/thumbnails/5.jpg)
– Random errors do not matter much– Systematic errors do matter but are largely
preventable through proper trial design
Do data errors matter?
![Page 6: Quality assurance in clinical trials Marc Buyse IDDI, Louvain-la-Neuve, Belgium Sensible Guidelines for the Conduct of Clinical Trials Washington, January.](https://reader036.fdocuments.us/reader036/viewer/2022081602/5518b31b550346a61f8b4fd0/html5/thumbnails/6.jpg)
A randomized trial of anti-VEGF therapy for age-related macular
degeneration
Sham
3 mg
Patientswith
exudativeAMD
Stratify by - Center- Lesion subtype- Prior therapy
Intra-ocular injections
0.3 mg
1 mg
Ref: Gragoudas et al. N Engl J Med 2004;351:2805
![Page 7: Quality assurance in clinical trials Marc Buyse IDDI, Louvain-la-Neuve, Belgium Sensible Guidelines for the Conduct of Clinical Trials Washington, January.](https://reader036.fdocuments.us/reader036/viewer/2022081602/5518b31b550346a61f8b4fd0/html5/thumbnails/7.jpg)
Trial endpoint: visual acuity over time
(assessed through vision chart)
Visual acuity = numbers of letters read correctly
![Page 8: Quality assurance in clinical trials Marc Buyse IDDI, Louvain-la-Neuve, Belgium Sensible Guidelines for the Conduct of Clinical Trials Washington, January.](https://reader036.fdocuments.us/reader036/viewer/2022081602/5518b31b550346a61f8b4fd0/html5/thumbnails/8.jpg)
Changes in visual acuity from baseline to 1 year by treatment
arm
Visit time
VA
S c
hange
Week 0 Week 6 Week 12 Week 18 Week 24 Week 30 Week 36 Week 42 Week 48 Week 54
-17
-16
-15
-14
-13
-12
-11
-10
-9
-8
-7
-6
-5
-4
-3
-2
-1
0 0.3 mg1 mg3 mgSham
0.3 mg vs sham P = 0.00011 mg vs sham P = 0.00033 mg vs sham P = 0.03
![Page 9: Quality assurance in clinical trials Marc Buyse IDDI, Louvain-la-Neuve, Belgium Sensible Guidelines for the Conduct of Clinical Trials Washington, January.](https://reader036.fdocuments.us/reader036/viewer/2022081602/5518b31b550346a61f8b4fd0/html5/thumbnails/9.jpg)
Impact of adding random errors to visual acuity
Let ² be the within-patient variance of visual acuity over time
Add random error N(0, ²) to given proportion of patients selected at random
Simulate 1,000 trials with added random errors
Calculate 1,000 t-test P-values Report the median (and quartiles) of the
distribution of these P-values
![Page 10: Quality assurance in clinical trials Marc Buyse IDDI, Louvain-la-Neuve, Belgium Sensible Guidelines for the Conduct of Clinical Trials Washington, January.](https://reader036.fdocuments.us/reader036/viewer/2022081602/5518b31b550346a61f8b4fd0/html5/thumbnails/10.jpg)
Median simulated t-test P-values (and interquartile range)
0.00001
0.0001
0.001
0.01
0.1
1
0 0.1 0.2 0.3 0.4 0.5
Proportion of patients with random error
P-V
alu
e (
Lo
g s
ca
le)
3mg 1mg 0.3mg
![Page 11: Quality assurance in clinical trials Marc Buyse IDDI, Louvain-la-Neuve, Belgium Sensible Guidelines for the Conduct of Clinical Trials Washington, January.](https://reader036.fdocuments.us/reader036/viewer/2022081602/5518b31b550346a61f8b4fd0/html5/thumbnails/11.jpg)
Systematic errors
Errors avoidable by design (and analysis), e.g. No post-randomization exclusions No “per-treatment received” analyses Identical follow-up schedules Blinding to avoid endpoint assessment bias Etc.
Fraud (“intention-to-cheat”)
![Page 12: Quality assurance in clinical trials Marc Buyse IDDI, Louvain-la-Neuve, Belgium Sensible Guidelines for the Conduct of Clinical Trials Washington, January.](https://reader036.fdocuments.us/reader036/viewer/2022081602/5518b31b550346a61f8b4fd0/html5/thumbnails/12.jpg)
Prevalence of fraud?
Industry (Hoechst, 1990-1994)1 case of fraud in 243 (0.43%) randomly selected centers
FDA (1985-1988)1% of 570 routine audits led to a for-cause investigation
CALGB (1982-1992)2 cases of fraud in 691 (0.29%) on-site audits
SWOG (1983-1990)no case (0%) of fraud in 1,751 patients
fraud is probably rare (but possible underestimation ?)
Ref: Buyse et al, Statist in Med 1999;18:3435
![Page 13: Quality assurance in clinical trials Marc Buyse IDDI, Louvain-la-Neuve, Belgium Sensible Guidelines for the Conduct of Clinical Trials Washington, January.](https://reader036.fdocuments.us/reader036/viewer/2022081602/5518b31b550346a61f8b4fd0/html5/thumbnails/13.jpg)
Impact of fraud
Most frauds have little impact on the trial results because: they introduce random but not systematic errors (i.e. noise but
no bias) in the analyses they affect secondary variables (e.g. eligibility criteria) their magnitude is too small to have an influence (one site
and/or few patients)
Refs: Altman, Practical Statistics for Medical Research 1991
Peto et al, Controlled Clin Trials 1997;18:1
![Page 14: Quality assurance in clinical trials Marc Buyse IDDI, Louvain-la-Neuve, Belgium Sensible Guidelines for the Conduct of Clinical Trials Washington, January.](https://reader036.fdocuments.us/reader036/viewer/2022081602/5518b31b550346a61f8b4fd0/html5/thumbnails/14.jpg)
On-site monitoring
”(...) the trial management procedures ensuring validity and reliability of the results are vastly more important than absence of clerical errors. Yet, it is clerical inconsistencies referred to as ’errors’ that are chased by the growing GCP-departments.”
Refs: Lörstad, ISCB-27, Geneva, August 28-31, 2006
Grimes et al, Lancet 2005;366:172
![Page 15: Quality assurance in clinical trials Marc Buyse IDDI, Louvain-la-Neuve, Belgium Sensible Guidelines for the Conduct of Clinical Trials Washington, January.](https://reader036.fdocuments.us/reader036/viewer/2022081602/5518b31b550346a61f8b4fd0/html5/thumbnails/15.jpg)
A phase IV randomized trial of adjuvant treatment for breast
cancer
6 x FEC *
4 x FEC 4 x T **
Patientswith node-
positiveresectedbreast cancer
Stratify by - Center- Patient’s age- Number of positive
nodes
* 5FU, Epirubicin, Cyclophosphamide** Taxol
![Page 16: Quality assurance in clinical trials Marc Buyse IDDI, Louvain-la-Neuve, Belgium Sensible Guidelines for the Conduct of Clinical Trials Washington, January.](https://reader036.fdocuments.us/reader036/viewer/2022081602/5518b31b550346a61f8b4fd0/html5/thumbnails/16.jpg)
A randomized study of the impact
of on-site monitoring
Group A (site visits)
Group B (no visits)
Centersaccruing patientsin trial AERO B2000
Stratify by - Type (Academic vs
Private)- Location (Paris vs
Province)
Ref: Liénard et al, Clinical Trials 2006;3:1-7
![Page 17: Quality assurance in clinical trials Marc Buyse IDDI, Louvain-la-Neuve, Belgium Sensible Guidelines for the Conduct of Clinical Trials Washington, January.](https://reader036.fdocuments.us/reader036/viewer/2022081602/5518b31b550346a61f8b4fd0/html5/thumbnails/17.jpg)
No difference
Impact of initiation visits on patient accrual
Nr patients accrued by
opened center
A (site visits)68 centers
B (no visits)67 centers
0 33 (48%) 33 (49%)
1-2 8 (12%) 7 (11%)
3-5 12 (18%) 11 (16%)
6 + 15 (22%) 16 (24%)
![Page 18: Quality assurance in clinical trials Marc Buyse IDDI, Louvain-la-Neuve, Belgium Sensible Guidelines for the Conduct of Clinical Trials Washington, January.](https://reader036.fdocuments.us/reader036/viewer/2022081602/5518b31b550346a61f8b4fd0/html5/thumbnails/18.jpg)
No difference
Impact of initiation visits on volume of data submitted
Nr CRF pages submitted by
patient
A (site visits)302 patients
B (no visits)271 patients
0 162 (54%) 114 (42%)
1-2 51 (17%) 44 (16%)
3-5 77 (25%) 96 (36%)
6 + 12 (4%) 17 (6%)
![Page 19: Quality assurance in clinical trials Marc Buyse IDDI, Louvain-la-Neuve, Belgium Sensible Guidelines for the Conduct of Clinical Trials Washington, January.](https://reader036.fdocuments.us/reader036/viewer/2022081602/5518b31b550346a61f8b4fd0/html5/thumbnails/19.jpg)
No difference
Impact of initiation visits on quality of data submitted
Nr queries generated by
CRF page
A (site visits)444 pages
B (no visits)571 pages
0 102 (23%) 91 (16%)
1-2 195 (44%) 314 (55%)
3-5 120 (27%) 132 (23%)
6 + 27 (6%) 34 (6%)
![Page 20: Quality assurance in clinical trials Marc Buyse IDDI, Louvain-la-Neuve, Belgium Sensible Guidelines for the Conduct of Clinical Trials Washington, January.](https://reader036.fdocuments.us/reader036/viewer/2022081602/5518b31b550346a61f8b4fd0/html5/thumbnails/20.jpg)
Statistical approaches to data checking
• Humans are poor random number generators test randomness (e.g. Benford’s law)
• Plausible multivariate data are hard to fabricate test correlation structure
• Clinical trial data are highly structured compare expected vs observed
• Clinical trial data are rich in meaning test plausibility (e.g. dates)
• Fraud or gross errors usually occur at one center compare centers
![Page 21: Quality assurance in clinical trials Marc Buyse IDDI, Louvain-la-Neuve, Belgium Sensible Guidelines for the Conduct of Clinical Trials Washington, January.](https://reader036.fdocuments.us/reader036/viewer/2022081602/5518b31b550346a61f8b4fd0/html5/thumbnails/21.jpg)
Non-parametric approach
• In multicentric trials, the distribution of all variables can be compared between each center and all others, through ² statistics for discrete variables– t-test to compare means of continuous variables– F-test to compare variances– mutivariate test statistics for more than one variable– etc.
![Page 22: Quality assurance in clinical trials Marc Buyse IDDI, Louvain-la-Neuve, Belgium Sensible Guidelines for the Conduct of Clinical Trials Washington, January.](https://reader036.fdocuments.us/reader036/viewer/2022081602/5518b31b550346a61f8b4fd0/html5/thumbnails/22.jpg)
Brute force approach
• These tests can be applied automatically, without regard to meaning or plausibility
• They yield very large number of center-specific statistics
• Meta-statistics can be applied to these statistics to identify outlying centers
• These ideas are currently implemented in the project « SAFE » (Statistical Alerts For Errors)
![Page 23: Quality assurance in clinical trials Marc Buyse IDDI, Louvain-la-Neuve, Belgium Sensible Guidelines for the Conduct of Clinical Trials Washington, January.](https://reader036.fdocuments.us/reader036/viewer/2022081602/5518b31b550346a61f8b4fd0/html5/thumbnails/23.jpg)
SAFE – prototype overview
Working datasets
Data checking modules
Reports
Results datasets
Meta-statistics modules
Data preparation modules
Trial original datasets
![Page 24: Quality assurance in clinical trials Marc Buyse IDDI, Louvain-la-Neuve, Belgium Sensible Guidelines for the Conduct of Clinical Trials Washington, January.](https://reader036.fdocuments.us/reader036/viewer/2022081602/5518b31b550346a61f8b4fd0/html5/thumbnails/24.jpg)
Quality Assurance – how ?
“Total Quality Management” has been used successfully in other industries (nuclear power, aviation and space industry). It requires
- A working definition of quality (“fitness for use”)- Error prevention (rather than cure)- Performance data - Statistical process control- Continuous improvement
Ref: Lörstad, ISCB-27, Geneva, August 28-31, 2006
![Page 25: Quality assurance in clinical trials Marc Buyse IDDI, Louvain-la-Neuve, Belgium Sensible Guidelines for the Conduct of Clinical Trials Washington, January.](https://reader036.fdocuments.us/reader036/viewer/2022081602/5518b31b550346a61f8b4fd0/html5/thumbnails/25.jpg)
Conclusions
We lack evidence on the (cost-)effectiveness of current trial procedures, such as intensive monitoring and 100% source data verification
A statistical approach to quality assurance could yield huge cost savings without compromising the reliability of the trial results
Quality assurance in clinical trials is in great need of re-engineering. More regulations such as GCP or ICH, useful as they are, will not achieve this goal.