REGULATORY CONSIDERATIONS AND …...2020/01/29 · REGULATORY CONSIDERATIONS AND ASSESSMENT OF...
Transcript of REGULATORY CONSIDERATIONS AND …...2020/01/29 · REGULATORY CONSIDERATIONS AND ASSESSMENT OF...
![Page 1: REGULATORY CONSIDERATIONS AND …...2020/01/29 · REGULATORY CONSIDERATIONS AND ASSESSMENT OF AI/ML DEVICES Nicholas Petrick, Ph.D. Division of Imaging, Diagnostics and Software](https://reader030.fdocuments.us/reader030/viewer/2022041011/5ebce52bedf7c949547d4a13/html5/thumbnails/1.jpg)
REGULATORY CONSIDERATIONS ANDASSESSMENT OF AI/ML DEVICES
Nicholas Petrick, Ph.D.Division of Imaging, Diagnostics and Software Reliability (DIDSR)
Office of Science and Engineering Labs (OSEL)Center for Devices and Radiological Health (CDRH)
U.S. Food and Drug Administration (FDA)[email protected]
![Page 2: REGULATORY CONSIDERATIONS AND …...2020/01/29 · REGULATORY CONSIDERATIONS AND ASSESSMENT OF AI/ML DEVICES Nicholas Petrick, Ph.D. Division of Imaging, Diagnostics and Software](https://reader030.fdocuments.us/reader030/viewer/2022041011/5ebce52bedf7c949547d4a13/html5/thumbnails/2.jpg)
2
• None
1/29/2020
DISCLOSURES
![Page 3: REGULATORY CONSIDERATIONS AND …...2020/01/29 · REGULATORY CONSIDERATIONS AND ASSESSMENT OF AI/ML DEVICES Nicholas Petrick, Ph.D. Division of Imaging, Diagnostics and Software](https://reader030.fdocuments.us/reader030/viewer/2022041011/5ebce52bedf7c949547d4a13/html5/thumbnails/3.jpg)
3
• What we do in CDRH/OSEL/DIDSR• Device Regulations• Software as a Medical Device (SaMD)
– AI/ML development and assessment
• DIDSR AI/ML research areas
1/29/2020
OUTLINE
![Page 4: REGULATORY CONSIDERATIONS AND …...2020/01/29 · REGULATORY CONSIDERATIONS AND ASSESSMENT OF AI/ML DEVICES Nicholas Petrick, Ph.D. Division of Imaging, Diagnostics and Software](https://reader030.fdocuments.us/reader030/viewer/2022041011/5ebce52bedf7c949547d4a13/html5/thumbnails/4.jpg)
4
• Center for Device and Radiological Health– Office of Science And Engineering Labs
• Division of Imaging, Diagnostics and Software Reliability
1/29/2020
INTRODUCTION
![Page 5: REGULATORY CONSIDERATIONS AND …...2020/01/29 · REGULATORY CONSIDERATIONS AND ASSESSMENT OF AI/ML DEVICES Nicholas Petrick, Ph.D. Division of Imaging, Diagnostics and Software](https://reader030.fdocuments.us/reader030/viewer/2022041011/5ebce52bedf7c949547d4a13/html5/thumbnails/5.jpg)
51/29/2020
FDA CAMPUSSilver Spring, MD
CDRH
FDA
![Page 6: REGULATORY CONSIDERATIONS AND …...2020/01/29 · REGULATORY CONSIDERATIONS AND ASSESSMENT OF AI/ML DEVICES Nicholas Petrick, Ph.D. Division of Imaging, Diagnostics and Software](https://reader030.fdocuments.us/reader030/viewer/2022041011/5ebce52bedf7c949547d4a13/html5/thumbnails/6.jpg)
6
CDRH IN PERSPECTIVE
EMPLOYEES
22k/yearPremarketSubmissionsincludes supplements and amendments
18kMedical Device Manufacturers
570kProprietary Brands
25kMedical Device Facilities Worldwide
1900 183k Medical DevicesOn the U.S. Market
Reports on medical device adverse events and malfunctions
1.4 MILLION/year
1/29/2020
![Page 7: REGULATORY CONSIDERATIONS AND …...2020/01/29 · REGULATORY CONSIDERATIONS AND ASSESSMENT OF AI/ML DEVICES Nicholas Petrick, Ph.D. Division of Imaging, Diagnostics and Software](https://reader030.fdocuments.us/reader030/viewer/2022041011/5ebce52bedf7c949547d4a13/html5/thumbnails/7.jpg)
7
CDRH MISSION
.. protect and promote the health of the public by ensuring the safety and effectiveness of medical devices and the safety of radiation-emitting electronic products…
We facilitate medical device innovation by advancing regulatory science, providing industry with predictable, consistent, transparent, and efficient regulatory pathways, and assuring consumer confidence in devices marketed in the U.S.
1/29/2020
![Page 8: REGULATORY CONSIDERATIONS AND …...2020/01/29 · REGULATORY CONSIDERATIONS AND ASSESSMENT OF AI/ML DEVICES Nicholas Petrick, Ph.D. Division of Imaging, Diagnostics and Software](https://reader030.fdocuments.us/reader030/viewer/2022041011/5ebce52bedf7c949547d4a13/html5/thumbnails/8.jpg)
81/29/2020
OSELLabs and Offices
![Page 9: REGULATORY CONSIDERATIONS AND …...2020/01/29 · REGULATORY CONSIDERATIONS AND ASSESSMENT OF AI/ML DEVICES Nicholas Petrick, Ph.D. Division of Imaging, Diagnostics and Software](https://reader030.fdocuments.us/reader030/viewer/2022041011/5ebce52bedf7c949547d4a13/html5/thumbnails/9.jpg)
9
OSEL IN PERSPECTIVE
FEDERAL EMPLOYEESUp to 180 visiting scientists
2,500k/year
PremarketRegulatory consults
140 ProjectsIn 27 Laboratories and Program Areas
75Standards and conformity assessment committees
70%Staff with post graduate degree
183 400/yearPeer reviewed presentations, articles, and other public disclosures
55,000 ft2Lab facilities
1/29/2020
![Page 10: REGULATORY CONSIDERATIONS AND …...2020/01/29 · REGULATORY CONSIDERATIONS AND ASSESSMENT OF AI/ML DEVICES Nicholas Petrick, Ph.D. Division of Imaging, Diagnostics and Software](https://reader030.fdocuments.us/reader030/viewer/2022041011/5ebce52bedf7c949547d4a13/html5/thumbnails/10.jpg)
10
• Conduct laboratory-based regulatory research to facilitate development and innovation of safe and effective medical devices and radiation emitting products
• Provide scientific and engineering expertise, data, and analyses to support regulatory processes
• Collaborate with colleagues in academia, industry, government, and standards development organizations to develop, translate, and disseminate science and engineering-based information regarding regulated products
1/29/2020
WHAT OSEL DOES
![Page 11: REGULATORY CONSIDERATIONS AND …...2020/01/29 · REGULATORY CONSIDERATIONS AND ASSESSMENT OF AI/ML DEVICES Nicholas Petrick, Ph.D. Division of Imaging, Diagnostics and Software](https://reader030.fdocuments.us/reader030/viewer/2022041011/5ebce52bedf7c949547d4a13/html5/thumbnails/11.jpg)
11
WHY OSEL IS SO IMPORTANT
Design Freeze
1/29/2020
![Page 12: REGULATORY CONSIDERATIONS AND …...2020/01/29 · REGULATORY CONSIDERATIONS AND ASSESSMENT OF AI/ML DEVICES Nicholas Petrick, Ph.D. Division of Imaging, Diagnostics and Software](https://reader030.fdocuments.us/reader030/viewer/2022041011/5ebce52bedf7c949547d4a13/html5/thumbnails/12.jpg)
12
DIDSR IN PERSPECTIVE
FEDERAL EMPLOYEES14 Fellows/Students3 Open Staff Positions
550/year
PremarketRegulatory consults
5 Program Areas• AI/ML• Credibility Assessment• Digital Pathology
Medical Imaging & Diagnostics
• Mixed Reality
35 145/yearPeer reviewed articles, code and presentations
~15,000 ft2DIDSR Lab and facilities
1/29/2020
![Page 13: REGULATORY CONSIDERATIONS AND …...2020/01/29 · REGULATORY CONSIDERATIONS AND ASSESSMENT OF AI/ML DEVICES Nicholas Petrick, Ph.D. Division of Imaging, Diagnostics and Software](https://reader030.fdocuments.us/reader030/viewer/2022041011/5ebce52bedf7c949547d4a13/html5/thumbnails/13.jpg)
13
• Conducts laboratory research across a wide variety of imaging & diagnostic devices
DIDSR
LDH --------- 167 AST (SGOT) ----- 17ALK PHOS ----- 44 TOT. BILI ------ 0.9ALBUMIN ----- 4.4 GLOBULIN -------- 2.5 GLUCOSE ------ 66 BUN -------------- 16CREATININE -- 1.0 PLATELETS ------- 185
1/29/2020
![Page 14: REGULATORY CONSIDERATIONS AND …...2020/01/29 · REGULATORY CONSIDERATIONS AND ASSESSMENT OF AI/ML DEVICES Nicholas Petrick, Ph.D. Division of Imaging, Diagnostics and Software](https://reader030.fdocuments.us/reader030/viewer/2022041011/5ebce52bedf7c949547d4a13/html5/thumbnails/14.jpg)
14
• Develop least burdensome approaches to clinical trials for imaging and big-data devices– Methods that account for reader variability, imperfect or
missing reference standards, limited data for training and testing of machine classifiers, etc.
• Develop measures of technical effectiveness of imaging and big-data technologies as surrogates for clinical trials– Phantoms, lab measures, and computational models
1/29/2020
DIDSR’S OVERARCHING RESEARCH GOALS
![Page 15: REGULATORY CONSIDERATIONS AND …...2020/01/29 · REGULATORY CONSIDERATIONS AND ASSESSMENT OF AI/ML DEVICES Nicholas Petrick, Ph.D. Division of Imaging, Diagnostics and Software](https://reader030.fdocuments.us/reader030/viewer/2022041011/5ebce52bedf7c949547d4a13/html5/thumbnails/15.jpg)
15
DEVICE REGULATIONS OVERVIEW
1/29/2020
![Page 16: REGULATORY CONSIDERATIONS AND …...2020/01/29 · REGULATORY CONSIDERATIONS AND ASSESSMENT OF AI/ML DEVICES Nicholas Petrick, Ph.D. Division of Imaging, Diagnostics and Software](https://reader030.fdocuments.us/reader030/viewer/2022041011/5ebce52bedf7c949547d4a13/html5/thumbnails/16.jpg)
16
CDRH MISSION• Protect and promote the health of the public by ensuring the
safety and effectiveness of medical devices and the safety of radiation-emitting electronic products
RisksBenefits
![Page 17: REGULATORY CONSIDERATIONS AND …...2020/01/29 · REGULATORY CONSIDERATIONS AND ASSESSMENT OF AI/ML DEVICES Nicholas Petrick, Ph.D. Division of Imaging, Diagnostics and Software](https://reader030.fdocuments.us/reader030/viewer/2022041011/5ebce52bedf7c949547d4a13/html5/thumbnails/17.jpg)
17
DEVICE CLASS & PRE-MARKET REQUIREMENTS
Device Class ControlsPremarket Review
ProcessClass I(lowest risk)
General Controls Most are exempt
Class II General ControlsSpecial Controls
Premarket Notification [510(k)]
De Novo
Class III(highest risk)
General ControlsPremarket Approval
Premarket Approval [PMA]
17
![Page 18: REGULATORY CONSIDERATIONS AND …...2020/01/29 · REGULATORY CONSIDERATIONS AND ASSESSMENT OF AI/ML DEVICES Nicholas Petrick, Ph.D. Division of Imaging, Diagnostics and Software](https://reader030.fdocuments.us/reader030/viewer/2022041011/5ebce52bedf7c949547d4a13/html5/thumbnails/18.jpg)
18
DEVICE CLASS & PRE-MARKET REQUIREMENTS
Device Class ControlsPremarket Review
ProcessClass I(lowest risk)
General Controls Most are exempt
Class II General ControlsSpecial Controls
Premarket Notification [510(k)]
De Novo
Class III(highest risk)
General ControlsPremarket Approval
Premarket Approval [PMA]
Demonstrate substantial equivalence to predicate device
18
![Page 19: REGULATORY CONSIDERATIONS AND …...2020/01/29 · REGULATORY CONSIDERATIONS AND ASSESSMENT OF AI/ML DEVICES Nicholas Petrick, Ph.D. Division of Imaging, Diagnostics and Software](https://reader030.fdocuments.us/reader030/viewer/2022041011/5ebce52bedf7c949547d4a13/html5/thumbnails/19.jpg)
19
DEVICE CLASS & PRE-MARKET REQUIREMENTS
19
Device Class ControlsPremarket Review
ProcessClass I(lowest risk)
General Controls Most are exempt
Class II General ControlsSpecial Controls
Premarket Notification [510(k)]
De Novo
Class III(highest risk)
General ControlsPremarket Approval
Premarket Approval [PMA]
Means for new device, without a valid predicate, to be classified into Class I or II
![Page 20: REGULATORY CONSIDERATIONS AND …...2020/01/29 · REGULATORY CONSIDERATIONS AND ASSESSMENT OF AI/ML DEVICES Nicholas Petrick, Ph.D. Division of Imaging, Diagnostics and Software](https://reader030.fdocuments.us/reader030/viewer/2022041011/5ebce52bedf7c949547d4a13/html5/thumbnails/20.jpg)
20
DEVICE CLASS & PRE-MARKET REQUIREMENTS
Device Class ControlsPremarket Review
ProcessClass I(lowest risk)
General Controls Most are exempt
Class II General ControlsSpecial Controls
Premarket Notification [510(k)]
De Novo
Class III(highest risk)
General ControlsPremarket Approval
Premarket Approval [PMA]
Demonstrate reasonable assurance of safety and effectiveness
20
![Page 21: REGULATORY CONSIDERATIONS AND …...2020/01/29 · REGULATORY CONSIDERATIONS AND ASSESSMENT OF AI/ML DEVICES Nicholas Petrick, Ph.D. Division of Imaging, Diagnostics and Software](https://reader030.fdocuments.us/reader030/viewer/2022041011/5ebce52bedf7c949547d4a13/html5/thumbnails/21.jpg)
21
MEDICAL DEVICES BY CLASS
1/29/2020
Class I Class IICT, MR, US imaging systemsAny imaging AI/MLSome IVD tests
Class IIINovel Imaging systems (DBT)Leadless PacemakersSome AI/MLSome IVD Tests
![Page 22: REGULATORY CONSIDERATIONS AND …...2020/01/29 · REGULATORY CONSIDERATIONS AND ASSESSMENT OF AI/ML DEVICES Nicholas Petrick, Ph.D. Division of Imaging, Diagnostics and Software](https://reader030.fdocuments.us/reader030/viewer/2022041011/5ebce52bedf7c949547d4a13/html5/thumbnails/22.jpg)
22
SOFTWARE AS A MEDICAL DEVICE (SAMD)
1/29/2020
![Page 23: REGULATORY CONSIDERATIONS AND …...2020/01/29 · REGULATORY CONSIDERATIONS AND ASSESSMENT OF AI/ML DEVICES Nicholas Petrick, Ph.D. Division of Imaging, Diagnostics and Software](https://reader030.fdocuments.us/reader030/viewer/2022041011/5ebce52bedf7c949547d4a13/html5/thumbnails/23.jpg)
231/29/2020
• IMDRF conceived 2011 as a forum to discuss future directions in medical device regulatory harmonization
• Voluntary group of medical device regulators from around the world … to accelerate international medical device regulatory harmonization and convergence
![Page 24: REGULATORY CONSIDERATIONS AND …...2020/01/29 · REGULATORY CONSIDERATIONS AND ASSESSMENT OF AI/ML DEVICES Nicholas Petrick, Ph.D. Division of Imaging, Diagnostics and Software](https://reader030.fdocuments.us/reader030/viewer/2022041011/5ebce52bedf7c949547d4a13/html5/thumbnails/24.jpg)
24
SOFTWARE AS A MEDICAL DEVICE (SAMD)
1/29/20205http://www.imdrf.org/workitems/wi-samd.asp
![Page 25: REGULATORY CONSIDERATIONS AND …...2020/01/29 · REGULATORY CONSIDERATIONS AND ASSESSMENT OF AI/ML DEVICES Nicholas Petrick, Ph.D. Division of Imaging, Diagnostics and Software](https://reader030.fdocuments.us/reader030/viewer/2022041011/5ebce52bedf7c949547d4a13/html5/thumbnails/25.jpg)
25
• Treatment planning software for radiation therapy
• Image display and processing software tools (PACS software)
• Software that detect lesions/conditions from diagnostic medical images
• Software that diagnoses lesions/conditions from diagnostic medical images
• Software making treatment recommendations from diagnostic images
1/29/2020
SAMD EXAMPLES
![Page 26: REGULATORY CONSIDERATIONS AND …...2020/01/29 · REGULATORY CONSIDERATIONS AND ASSESSMENT OF AI/ML DEVICES Nicholas Petrick, Ph.D. Division of Imaging, Diagnostics and Software](https://reader030.fdocuments.us/reader030/viewer/2022041011/5ebce52bedf7c949547d4a13/html5/thumbnails/26.jpg)
26
IMDRF WORKING TOWARD GLOBAL CONVERGENCEIN CHARACTERIZING SAMD
2014 –Risk framework based on impact to patients
2013 Foundational vocabulary
2015 –QMS control Translating Software development practices to regulatory QMS
2017 – SaMD Clinical Evaluation Generating evidence for clinically meaningful SaMD
1/29/2020
![Page 27: REGULATORY CONSIDERATIONS AND …...2020/01/29 · REGULATORY CONSIDERATIONS AND ASSESSMENT OF AI/ML DEVICES Nicholas Petrick, Ph.D. Division of Imaging, Diagnostics and Software](https://reader030.fdocuments.us/reader030/viewer/2022041011/5ebce52bedf7c949547d4a13/html5/thumbnails/27.jpg)
27
IMDRF SAMD: CLINICAL EVALUATION• Adopted as FDA guidance in 2017
– FDA adopted the converged principles of IMDRF in our evolving approach to SaMD review
https://www.fda.gov/downloads/medicaldevices/deviceregulationandguidance/guidancedocuments/ucm524904.pdf
1/29/2020
![Page 28: REGULATORY CONSIDERATIONS AND …...2020/01/29 · REGULATORY CONSIDERATIONS AND ASSESSMENT OF AI/ML DEVICES Nicholas Petrick, Ph.D. Division of Imaging, Diagnostics and Software](https://reader030.fdocuments.us/reader030/viewer/2022041011/5ebce52bedf7c949547d4a13/html5/thumbnails/28.jpg)
28
SAMD RISK CATEGORIES
1/29/2020
State of Healthcare Situation or Condition
Significance of Information Provided by SaMD to Healthcare Decision
Treat or Diagnose
Drive Clinical
Management
Inform Clinical
Management
Critical IV III IISerious III II INon-Serious II I I
Increasing SignificanceIn
crea
sing
crit
ical
ity
![Page 29: REGULATORY CONSIDERATIONS AND …...2020/01/29 · REGULATORY CONSIDERATIONS AND ASSESSMENT OF AI/ML DEVICES Nicholas Petrick, Ph.D. Division of Imaging, Diagnostics and Software](https://reader030.fdocuments.us/reader030/viewer/2022041011/5ebce52bedf7c949547d4a13/html5/thumbnails/29.jpg)
29
• Clinical association is indicator of level of clinical acceptance– How much meaning and confidence can be assigned to the clinical
significance of the SaMD’s output
SAMD: CLINICAL EVALUATION
1/29/2020
![Page 30: REGULATORY CONSIDERATIONS AND …...2020/01/29 · REGULATORY CONSIDERATIONS AND ASSESSMENT OF AI/ML DEVICES Nicholas Petrick, Ph.D. Division of Imaging, Diagnostics and Software](https://reader030.fdocuments.us/reader030/viewer/2022041011/5ebce52bedf7c949547d4a13/html5/thumbnails/30.jpg)
30
• Evidence generation– Literature– Professional guidelines– Secondary data analysis– Clinical trials/studies
SAMD: CLINICAL EVALUATION
1/29/2020
![Page 31: REGULATORY CONSIDERATIONS AND …...2020/01/29 · REGULATORY CONSIDERATIONS AND ASSESSMENT OF AI/ML DEVICES Nicholas Petrick, Ph.D. Division of Imaging, Diagnostics and Software](https://reader030.fdocuments.us/reader030/viewer/2022041011/5ebce52bedf7c949547d4a13/html5/thumbnails/31.jpg)
31
• Confirms and provides objective evidence SaMD meet technical requirements
– Provide evidence that software correctly constructed– Demonstrate it meets specifications and conforms to user needs
SAMD: CLINICAL EVALUATION
1/29/2020
![Page 32: REGULATORY CONSIDERATIONS AND …...2020/01/29 · REGULATORY CONSIDERATIONS AND ASSESSMENT OF AI/ML DEVICES Nicholas Petrick, Ph.D. Division of Imaging, Diagnostics and Software](https://reader030.fdocuments.us/reader030/viewer/2022041011/5ebce52bedf7c949547d4a13/html5/thumbnails/32.jpg)
32
• Evidence generation– Verification and Validation (V & V) testing
• Verification: objective evidence that specified requirements fulfilled• Validations: objective evidence that requirements for specific intended use or application fulfilled
SAMD: CLINICAL EVALUATION
1/29/2020
![Page 33: REGULATORY CONSIDERATIONS AND …...2020/01/29 · REGULATORY CONSIDERATIONS AND ASSESSMENT OF AI/ML DEVICES Nicholas Petrick, Ph.D. Division of Imaging, Diagnostics and Software](https://reader030.fdocuments.us/reader030/viewer/2022041011/5ebce52bedf7c949547d4a13/html5/thumbnails/33.jpg)
34
• Clinical validation measures ability SaMD to yield clinically meaningful output associated
• Clinically meaningful means positive impact on the health of an individual or population
SAMD: CLINICAL EVALUATION
1/29/2020
![Page 34: REGULATORY CONSIDERATIONS AND …...2020/01/29 · REGULATORY CONSIDERATIONS AND ASSESSMENT OF AI/ML DEVICES Nicholas Petrick, Ph.D. Division of Imaging, Diagnostics and Software](https://reader030.fdocuments.us/reader030/viewer/2022041011/5ebce52bedf7c949547d4a13/html5/thumbnails/34.jpg)
35
• Evidence generation– Existing data from studies for same intended use– Existing data from studies for a different intended use
• Where extrapolation of such data can be justified– New clinical data for a specific intended use
SAMD: CLINICAL EVALUATION
1/29/2020
![Page 35: REGULATORY CONSIDERATIONS AND …...2020/01/29 · REGULATORY CONSIDERATIONS AND ASSESSMENT OF AI/ML DEVICES Nicholas Petrick, Ph.D. Division of Imaging, Diagnostics and Software](https://reader030.fdocuments.us/reader030/viewer/2022041011/5ebce52bedf7c949547d4a13/html5/thumbnails/35.jpg)
37
AI/ML SAMD
Assessment Framework
1/29/2020
![Page 36: REGULATORY CONSIDERATIONS AND …...2020/01/29 · REGULATORY CONSIDERATIONS AND ASSESSMENT OF AI/ML DEVICES Nicholas Petrick, Ph.D. Division of Imaging, Diagnostics and Software](https://reader030.fdocuments.us/reader030/viewer/2022041011/5ebce52bedf7c949547d4a13/html5/thumbnails/36.jpg)
38
INTRODUCTION• AI
– Algorithm that can sense, reason, act or adapt
• ML– Algorithms who’s
performance is based training with data
• Deep learning– ML based on multilayer
neural network
1/29/2020
*https://towardsdatascience.com/cousins-of-artificial-intelligence-dda4edc27b55
![Page 37: REGULATORY CONSIDERATIONS AND …...2020/01/29 · REGULATORY CONSIDERATIONS AND ASSESSMENT OF AI/ML DEVICES Nicholas Petrick, Ph.D. Division of Imaging, Diagnostics and Software](https://reader030.fdocuments.us/reader030/viewer/2022041011/5ebce52bedf7c949547d4a13/html5/thumbnails/37.jpg)
39
MANY DEVICES INCORPORATING AI/ML IN MEDICAL IMAGING
Computer-Aided Classification of Breast
Lesions on Magnetic Resonance Imaging:
Computer-Aided Diagnosis (CADx)
Computer-Aided Detection and
Diagnosis of Fractures on Radiographs:
CADe + CADx
Notification of Specialists for Suspicion of Stroke
on Computerized Tomography Images:
Radiological Computer Aided Triage
Detection of Diabetic Retinopathy on Retina
Fundus Images:Retinal Diagnostic
Software
1/29/2020
![Page 38: REGULATORY CONSIDERATIONS AND …...2020/01/29 · REGULATORY CONSIDERATIONS AND ASSESSMENT OF AI/ML DEVICES Nicholas Petrick, Ph.D. Division of Imaging, Diagnostics and Software](https://reader030.fdocuments.us/reader030/viewer/2022041011/5ebce52bedf7c949547d4a13/html5/thumbnails/38.jpg)
40
AI/ML INTERACTING WITH HCPS• Echocardiography
– Cardiac sonography acquisition requires training and skill
– AI can providing naïve users real-time guidance
1/29/2020
http://www.thepocusatlas.com/echocardiography/sgz53ib6vai913wzgyn9qkhqo6ioe7
![Page 39: REGULATORY CONSIDERATIONS AND …...2020/01/29 · REGULATORY CONSIDERATIONS AND ASSESSMENT OF AI/ML DEVICES Nicholas Petrick, Ph.D. Division of Imaging, Diagnostics and Software](https://reader030.fdocuments.us/reader030/viewer/2022041011/5ebce52bedf7c949547d4a13/html5/thumbnails/39.jpg)
41
ASSESSMENT OF IMAGING-BASED AI/ML
• Computer-aided Detection (CADe)– ML-based disease
prompting devices
• FDA CADe Guidance documents
1/29/2020 *Yamada et al, Radiol 287(1):333, 2018.
Reference locationCADe Prompt
![Page 40: REGULATORY CONSIDERATIONS AND …...2020/01/29 · REGULATORY CONSIDERATIONS AND ASSESSMENT OF AI/ML DEVICES Nicholas Petrick, Ph.D. Division of Imaging, Diagnostics and Software](https://reader030.fdocuments.us/reader030/viewer/2022041011/5ebce52bedf7c949547d4a13/html5/thumbnails/40.jpg)
42
FUNDAMENTALS OF IMAGE-BASED ML SAMD ASSESSMENT
• Device description• Data• Performance assessment
– Standalone performance– Reader performance (when appropriate)– …
• Human factors or other information/testing as appropriate• …
1/29/2020
![Page 41: REGULATORY CONSIDERATIONS AND …...2020/01/29 · REGULATORY CONSIDERATIONS AND ASSESSMENT OF AI/ML DEVICES Nicholas Petrick, Ph.D. Division of Imaging, Diagnostics and Software](https://reader030.fdocuments.us/reader030/viewer/2022041011/5ebce52bedf7c949547d4a13/html5/thumbnails/41.jpg)
43
DEVICE DESCRIPTION• Device & algorithm descriptions
– Device usage (mode of operation, patient population, …)
– Algorithm design and function• Including structure of traditional and deep learning networks• Inputs
– Type and range of signals/data• Outputs
– Training process
– Training/test database– Reference standard– …
1/29/2020
![Page 42: REGULATORY CONSIDERATIONS AND …...2020/01/29 · REGULATORY CONSIDERATIONS AND ASSESSMENT OF AI/ML DEVICES Nicholas Petrick, Ph.D. Division of Imaging, Diagnostics and Software](https://reader030.fdocuments.us/reader030/viewer/2022041011/5ebce52bedf7c949547d4a13/html5/thumbnails/42.jpg)
44
• ML algorithms are data-driven– Versus, for example, physics or biology based
• ML algorithm development now facilitated by standardized ML platforms– Brings ML to a wider array of users
– The good• Access to high-quality data streamlines design of novel ML applications
– The bad• Garbage in - garbage out
DATA
ML Features ML Training1/29/2020
![Page 43: REGULATORY CONSIDERATIONS AND …...2020/01/29 · REGULATORY CONSIDERATIONS AND ASSESSMENT OF AI/ML DEVICES Nicholas Petrick, Ph.D. Division of Imaging, Diagnostics and Software](https://reader030.fdocuments.us/reader030/viewer/2022041011/5ebce52bedf7c949547d4a13/html5/thumbnails/43.jpg)
45
• Performance of ML algorithm on an independent data– Ideally, identifies problems with training process
1/29/2020
PERFORMANCE TESTING
Learning Process
Training Data
Training Set
Learned Models Selected Model
Tuning Set
Independent Test Data
Learned Model (Fixed)
TestPerformance
![Page 44: REGULATORY CONSIDERATIONS AND …...2020/01/29 · REGULATORY CONSIDERATIONS AND ASSESSMENT OF AI/ML DEVICES Nicholas Petrick, Ph.D. Division of Imaging, Diagnostics and Software](https://reader030.fdocuments.us/reader030/viewer/2022041011/5ebce52bedf7c949547d4a13/html5/thumbnails/44.jpg)
46
PERFORMANCE TESTING• Standalone performance
– Performance of algorithm alone– Assesses robustness and
generalizability of algorithm
• Clinical reader performance– Assessment of clinical aids– Clinicians’ performance utilizing
device• Multi-reader multi-case designs• Compare clinician’s performance
with the ML SaMD aid to without the aid
1/29/2020
Apply ML SaMDTest Dataset
Statistical/Regression
Analysis
StandalonePerformance
Apply ML SaMD
Clinicalaided read
Clinicalw/o aid read
Test Dataset
ReaderPerformance(Difference)
Statistical Analysis
![Page 45: REGULATORY CONSIDERATIONS AND …...2020/01/29 · REGULATORY CONSIDERATIONS AND ASSESSMENT OF AI/ML DEVICES Nicholas Petrick, Ph.D. Division of Imaging, Diagnostics and Software](https://reader030.fdocuments.us/reader030/viewer/2022041011/5ebce52bedf7c949547d4a13/html5/thumbnails/45.jpg)
47
• Clinical Association– Likely from literature, professional guidelines– Device description used to support clinical association
CADE SAMD EVALUATION
1/29/2020
![Page 46: REGULATORY CONSIDERATIONS AND …...2020/01/29 · REGULATORY CONSIDERATIONS AND ASSESSMENT OF AI/ML DEVICES Nicholas Petrick, Ph.D. Division of Imaging, Diagnostics and Software](https://reader030.fdocuments.us/reader030/viewer/2022041011/5ebce52bedf7c949547d4a13/html5/thumbnails/46.jpg)
48
• Analytical validation– Verification and validation testing of software– Standalone testing may support analytical validation, in part
CADE SAMD EVALUATION
1/29/2020
![Page 47: REGULATORY CONSIDERATIONS AND …...2020/01/29 · REGULATORY CONSIDERATIONS AND ASSESSMENT OF AI/ML DEVICES Nicholas Petrick, Ph.D. Division of Imaging, Diagnostics and Software](https://reader030.fdocuments.us/reader030/viewer/2022041011/5ebce52bedf7c949547d4a13/html5/thumbnails/47.jpg)
49
• Clinical validation– Standalone performance
• Establish performance of ML SaMD alone (compared to clinicians)– Clinical reader performance
• Assess clinical aids provided by ML SaMD
CADE SAMD EVALUATION
1/29/2020
![Page 48: REGULATORY CONSIDERATIONS AND …...2020/01/29 · REGULATORY CONSIDERATIONS AND ASSESSMENT OF AI/ML DEVICES Nicholas Petrick, Ph.D. Division of Imaging, Diagnostics and Software](https://reader030.fdocuments.us/reader030/viewer/2022041011/5ebce52bedf7c949547d4a13/html5/thumbnails/48.jpg)
50
• Who we are in CDRH• Introduced to device regulation
– Software as a Medical Device (SaMD)– Assessment framework imaging-based ML
1/29/2020
SUMMARY
![Page 49: REGULATORY CONSIDERATIONS AND …...2020/01/29 · REGULATORY CONSIDERATIONS AND ASSESSMENT OF AI/ML DEVICES Nicholas Petrick, Ph.D. Division of Imaging, Diagnostics and Software](https://reader030.fdocuments.us/reader030/viewer/2022041011/5ebce52bedf7c949547d4a13/html5/thumbnails/49.jpg)
51
• Postdoctoral fellowships and summer internships at all levels in OSEL/DIDSR– AI/ML– Color Quality– Computational modeling– Mixed Reality– Imaging system development– …
1/29/2020
DIDSR INTERNSHIP AVAILABLE
![Page 50: REGULATORY CONSIDERATIONS AND …...2020/01/29 · REGULATORY CONSIDERATIONS AND ASSESSMENT OF AI/ML DEVICES Nicholas Petrick, Ph.D. Division of Imaging, Diagnostics and Software](https://reader030.fdocuments.us/reader030/viewer/2022041011/5ebce52bedf7c949547d4a13/html5/thumbnails/50.jpg)
52
• I’ d like to acknowledge Berkman Sahiner for his help in developing this presentation
1/29/2020
ACKNOWLEDGMENTS
![Page 51: REGULATORY CONSIDERATIONS AND …...2020/01/29 · REGULATORY CONSIDERATIONS AND ASSESSMENT OF AI/ML DEVICES Nicholas Petrick, Ph.D. Division of Imaging, Diagnostics and Software](https://reader030.fdocuments.us/reader030/viewer/2022041011/5ebce52bedf7c949547d4a13/html5/thumbnails/51.jpg)