Evolution of Translational “Omics” & Lessons Learned from the Duke Saga

Thomas R Fleming, PhDUniversity of Washington

Topics Covered

Considerable interest exists in using genomic data to develop ‘predictors’ or ‘effect modifiers’ for treatment response

A rigorous process needs to be followed to validate genomic biomarkers before being used to guide clinical treatment

Despite the large infrastructure in place at Duke, inadequaciesin the scientific process yielded very misleading results.

IOM received a request to provide guidance

Outline for today’s presentation:

Lessons learned from the Duke Saga

Summary of IOM Recommendation

Institute of MedicineThe National Academies Press, 2012

IOM Multidisciplinary Committee* GILBERT S. OMENN (Chair), University of Michigan Medical School

CATHERINE D. DEANGELIS, Johns Hopkins School of MedicineDAVID L. DEMETS, University of Wisconsin-Madison THOMAS R. FLEMING, University of Washington GAIL GELLER, Johns Hopkins University JOE GRAY, Oregon Health & Science University Knight Cancer InstituteDANIEL F. HAYES, University of Michigan Comprehensive Cancer Center I. CRAIG HENDERSON, University of California San FranciscoLARRY KESSLER, University of Washington School of Public Health STANLEY LAPIDUS, SynapDx CorporationDEBRA LEONARD, Weill Medical College of Cornell UniversityHAROLD L. MOSES, Vanderbilt-Ingram Cancer Center WILLIAM PAO, Vanderbilt University School of Medicine REBECCA D. PENTZ, Emory School of MedicineNATHAN D. PRICE, Institute for Systems BiologyJOHN QUACKENBUSH, Dana-Farber Cancer Institute ELDA RAILEY, Research Advocacy Network DAVID RANSOHOFF, University of North Carolina School of Medicine  E. ALBERT REECE, University of Maryland School of Medicine DANIELA M. WITTEN, University of Washington

2003 The ‘Institute for Genome Sciences & Policy’ created at Duke

2004 Anil Potti joins laboratory of Joseph Nevins

‘06-’07 ▪ Nature Medicine: ( Potti et al; 2006a) genomic signatures to guide use of chemotherapeutics▪ NEJM: (Potti et al; 2006b) genomic biomarker for lung

cancer▪ JCO: (Dressman et al) genomic biomarker for ovarian ca.▪ JCO: (Hsu et al) genomic biomarker for cisplatin-resistant

pts

‘06-’07 Keith Baggerly et al (MD Anderson statisticians) correspond with Duke authors, unable to reproduce their results

‘07-’08 ‘Clinical Genomics Studies Unit’ established at DukeClinical trials initiated using marker driven treatment

strategies ▪ BOP0801 in early stage Breast Cancer▪ TOP0602 in stage IIIB/IV Non Small Cell Lung Cancer▪ TOP0703 in early stage Non Small Cell Lung Cancer

Precipitating Events at Duke: Summary(IOM Report Table B-2)

Precipitating Events at Duke: Summary Omics tests developed at Duke to predict sensitivity to chemoRx

• Papers suggested major advance in directing therapy• Concerns about accuracy and validity raised immediately• 2009 Annals of Applied Statistics by Baggerly and Coombes:

Numerous errors in test development Data used in articles inconsistent with primary dataFailure to reproduce Duke investigators’ results

Trials scrutinized by NCI statistician Lisa McShane & colleagues

Criticisms rebuffed for 4 years while:• Hundreds of publications cited papers in question• Clinical trials initiated in 2007, using tests to direct patient care

IOM asked by NCI & Duke to review situation & provide guidance ‘10-’12 IOM Omics Committee documents wide misuse of biomarkers and formulates recommendations for scientific processes

By 2012, > two-thirds of 40 publications partially or fully retracted

Topics Covered

Outline for today’s presentation:

Lessons learned from the Duke Saga

Summary of IOM Recommendations

…How to prevent it from Happening Again…

Nevins presented at 3/30-31/2011 IOM Omics Committee meeting:

Key confirmation study, (Bonnefoi et al DATE), revealed the signature score was strongly correlated with response, for the 99 responders and 44 non-responders. A miscoding of 12 responders and 12 non-responders was discovered later. When corrections were made, the signature score was no longer correlated with the response status.

Omics Characteristics

• Complex, high dimensional data• Many more variables than samples• High risk that computational models will overfit data

Omics-based Test

• Composed or derived from multiple molecular measurements and interpreted by a fully specified computational model to produce a clinically actionable result

Errors may be Attributable to:

• Inherent complexity• Inadequate attention to detail• Intentional misrepresentation

ma

Some Specific Issues at Duke • Genomic trials conducted outside the Duke Cancer Center

• Trials conducted within the Institute for Genomics, which is outside the medical center

• Lacked clinical trials infrastructure and oversight

• Genomic predictors were published in leading journals, even though these biomarkers were not appropriately validated• Test data not locked down• Algorithm not locked down• Not validated on blinded data set

• Clinical trials launched at Duke used these invalid predictors that• Aided in the selection of patients for their risk• Aided in making an ‘optimal’ choice among available drugs

Some Specific Issues at Duke

• Duke IRB interpreted FDA lack of response as an approval for not needing an Investigational Device Exemption (IDE)• Inadequate communication with regulatory authorities• Did not recognize an algorithm is a device

• Statistical input largely was from a junior faculty member whose funding heavily dependent on the Nevins-Potti lab; • offered ownership in the spin-off companies• did not have independence necessary to raise issues

• Institution did not recognize the seriousness of the issues• Did not want to challenge a proven senior investigator• Believed the issues to be statistical subtleties• Did not fully understand issues until a meeting with NCI

late in the process

Some Specific Issues at Duke

• NCI understood there were problems but thought they could not engage until they realized that they were funding some aspects of one of the trials (TOP0602: Advanced NSCLC Lung Cancer)

• COI issues at many levels

• Potti & Nevins were inventors, investors and investigators

• Colleagues did not know about each other’s COI

• A DMC member was a colleague in related trials

• Duke had conflicting interests financially in the spin off companies started by Potti and Nevins

• Duke administrators had professional COI in protecting the reputations of the institution and their colleagues

• The investigation led by Duke IRB that allowed Nevins to influence external reviewers’ access to relevant information

Some Specific Issues at Duke• Conducting a serious external review likely will require months

of effort – as by an NCI statistician and by the IOM• Can’t make this commitment easily

• Journals accepted one round of letters; then rebuffed the questions and challenges by MD Anderson statisticians (Baggerly & Coombes )- did not facilitate scientific challenges to their publications

• 40 papers, 160 authors, but no one inquired as to manuscript validity even after research integrity issues became known

• Despite Duke having substantial infrastructure in place, all oversight systems simultaneously failed

• Another cancer center used genomic predictors, perhaps based on work at Duke, to assign treatment

Topics Covered

Outline for today’s presentation:

Lessons learned from the Duke Saga

Summary of IOM Recommendations

…How to prevent it from Happening Again…

Goals of Committee’s Recommendations

GOAL I: Define best practices for discovery and translation of an omics-based test into a clinical trial.

[Recommendations 1-3]

GOAL II: Recommend actions to ensure adoption of and adherence to the development and evaluation process.

[Recommendations 4-7]

Three Stages of Omics Test Development

1. Discovery

2. Test Validation

3. Evaluation for Clinical Utility and Use

Omics-Based Test Development Framework

Discovery Phase

RECOMMENDATION 1:

If candidate omics-based tests are intended for clinical development:

a. The tests should be confirmed using an independent set of samples*.

b. Data, code, and metadata should be made available.

c. Candidate test should be defined precisely:

• Molecular measurements

• Computational procedures

• Intended clinical use

*In some cases (e.g. tests developed using preclinical models or data generated in early clinical trials, independent data sets may not exist)

Recommendation 2: Test Validation

Test should be discussed with FDA prior to validation

studies.

Test development and validation should be performed in a CLIA-certified clinical laboratory.

CLIA lab should design, optimize, validate, and implement the test under current clinical laboratory standards.

Analytical validation and CLIA requirements should be met by each laboratory in which test will be performed.

Omics-Based Test Development Framework

Evaluation for Clinical Utility and Use

Three pathways:

Prospective–retrospective studies using archived specimens from previously conducted clinical trials.

Prospective clinical trials that directly address the utility of the omics-based test, where The test does not direct patient management Shadows clinical practice

Prospective clinical trials that directly address the utility of the omics-based test, where The test does direct patient management by assessing risk or

treatment response

Omics-Based Test Development Framework

Sargent, D. J., et al.; J Clin Oncol; 2005. Freidlin, B., et al.; J Natl Cancer Inst; 2010

All patients

All patients Omics-based test

R

R

R

New therapeutic strategy

Standard of Care

Test +

Test -

New therapeutic strategy

Standard of Care

(R= randomization)

Standard of Care

All patients

New therapeutic strategy

Completely Randomized Design

SOURCE: Adapted from Sargent, D. J., et al.; J Clin Oncol; 2005. Freidlin, B., et al.; J Natl Cancer Inst; 2010; and McShane, 2011

Test-Stratified Design

Omics-Based Test Development Framework

Sargent, D. J., et al.; J Clin Oncol; 2005. Freidlin, B., et al.; J Natl Cancer Inst; 2010

Omics-based testdetermines treatment

R

Test +

Test -

New therapeutic strategy

Standard of Care

Standard of Care

(R= randomization)

All patientsAll patients

Test-Guided Strategy Versus Standard of Care

SOURCE: Adapted from Sargent, D. J., et al.; J Clin Oncol; 2005. Freidlin, B., et al.; J Natl Cancer Inst; 2010; and McShane, 2011

Evaluation for Clinical Utility and Use

RECOMMENDATION 3:

a. Investigators should communicate early with the FDA

regarding Investigational Device Exemption (IDE) process.

b. Omics-based tests should not be changed during the clinical trial without a protocol amendment and discussion with the FDA. A substantive change to the test may require restarting study.

Goals of Committee’s Recommendations

GOAL II: Recommendations to ensure adoption of andadherence to the development & evaluation process

Investigators and institutions are responsible for the scientific culture in which omics-based tests are developed.

Recommendations also for funders, FDA & journals

Recommendations are not intended to create new barriers, but rather to maintain the integrity of the scientific process

Recommendation 4: Institutions

Recommendation 4a:

Institutions are responsible for establishing, supporting, and overseeing the infrastructure and research processes for omics-based test development and evaluation.

Recommendation 4b:

Institutional leaders should provide oversight and promote culture of integrity and transparency by designating

officials responsible for:

i. IDE and IND requirements

ii. management of financial and non-financial conflicts of interest (individual and institutional)

iii. a system for preventing, reporting, adjudicating lapses in integrity

iv. establishing clear procedures for response to inquiries

Recommendation 4: Institutions

Recommendation 4c:

Institutions should ensure that individuals who collaborate on omics research and test development, including biostatisticians and bioinformaticians, are:

i. Treated as equal co-investigators and co-owners of responsibility

ii. Represented on relevant review and oversight bodies

iii. Intellectually independent

Recommendation 4: Institutions

Recommendation 5: Funders

5a: All funders of omics-based translational research should:

i. Require investigators to make data, prespecified analysis plans, code, and computational models publicly available

ii. Provide continuing support for independent repositories to guarantee ongoing access to omics and clinical data

iii. Support test validation in a CLIA-certified laboratory and the independent confirmation of a candidate omics-based

iv. Alert the institutional leadership about serious questions

v. Establish lines of communication with other funders to be used when serious problems arise

5b: Federal funders of omics-based research should have authority to investigate research being conducted by a funding

recipient.

Recommendation 6: FDA

6a: FDA should develop and finalize a risk-based guidance or a regulation on:

i. Bringing omics-based tests to the FDA for review ii. Oversight of LDTs

6b: FDA should communicate IDE requirements for use of omics-based tests in clinical trials to OHRP, IRBs, and others.

Recommendation 7: Journals

Journal editors should:

7a: Require authors describing clinical evaluations of omics- based tests to:

i. Register all clinical trials

ii. Make data, metadata, analysis plans, code, and fully specified computational models publicly available

iii. Provide relevant sections of the research protocol

iv. State roles and attest to study integrity

v. Use appropriate reporting guidelines

7b: Develop mechanisms to resolve possible serious errors

7c: Alert the institutional leadership and all authors when a serious question of accuracy or integrity has been raised

Summary: Lessons Learned

• Importance of:

Data provenance and data management

Locking down the computational model required

Making data, code, and other information publicly available

Independent confirmation of the test & Test validation

Effective multidisciplinary collaboration

Institutional oversight, including fresh review of the science when serious criticisms are raised or clinical trials or spinoff companies are proposed

Consultation with FDA and submission of IDE

To download or read the report online:

www.nap.edu