LEARNING ANALYTICS - New York University€¦ · LEARNING ANALYTICS “Learning analytics refers to...

4/25/2019

1

Martin Pusic MD PhD, Marc Triola MD on behalf of…Institute for Innovations in Medical Education

NYU School of Medicine

Predictive Analytics: An NYU Case Study

Disclosures

• Grant funding from:

– American Medical Association

– Aquifer

– RCPS(C)

– Department of Defense

• Am here representing a wonderful team

LEARNING

ANALYTICS“Learning analytics refers to the interpretation of

a wide range of data produced by and gathered on

behalf of students in order to assess academic

progress, predict future performance, and spot

potential issues”

- DOE 2012

Learning Analytics

Learning Analytics

Computer Science Statistics

Learning Science

Data Science

How is Big Data Different?Traditional Big Data

Data Collection Purposeful Incidental / Opportunistic

Intrusiveness High Low

Data Acquisition Cost High Low

Frequency e.g. Quarterly Continuously

Temporality Static Reports Dynamic Dashboards

Hypotheses Causal Associations

Sample Size Small biopsies Large swaths

4/25/2019

2

Outline

NYU Approach to Learning Analytics:

“The Football Field” – Listeners everywhere

“The Blade of Grass”

– Insights at a Fine Level

“The New CME” – Action Oriented

“ The Football Field”

NYU Education Data Warehouse

Education Data

Warehouse

LMS

Curricular

Content

Curricular

Content

AdmissionsExams

ePortfolioPatient

Log

Learning

ModulesEvaluations

SIS

Reporting and Analytics

Data Marts for

Education Research

Simulation

Clinical Data

Warehouse

NYU Practice

Network

NYU Practice

NetworkEpic EMREpic EMR

Activity Logging

iBeacons EPIC

4/25/2019

3

iBeacons

Radiology Pilot Project

New “Listeners”

• iPads

• Point-of-care digital forms

• Immediate Lecture evaluations

• Learning interactions with EHR

• Anything a SmartPhone can do


Education Data

Warehouse

LMS

Curricular

Content

Curricular

Content

AdmissionsExams

ePortfolioPatient

Log

Learning

ModulesEvaluations

SIS


Data Marts for

Education Research

Simulation

Clinical Data

Warehouse

NYU Practice

Network

NYU Practice



Education Data

Warehouse

LMS

Curricular

Content

Curricular

Content

AdmissionsExams

ePortfolioPatient

Log

Learning

ModulesEvaluations

SIS


Data Marts for

Education Research

Simulation

Clinical Data

Warehouse

NYU Practice

Network

NYU Practice


Gartner Model

© Gartner Group

LEARNING ANALYTICSDescriptive Analytics: Y: Counts, averages, %, min/max

Understand what happened in past

Diagnostic Analytics: y=mx +b

Understand what influences what

Why did this happen?

Focus is on group level coefficients

Predictive Analytics: y=mx + b

Knowing something sooner

Early warning system

Focus is on individual level prediction

Prescriptive Analytics: IF-Y-THEN-WHAT?

Adjustment on the fly (uses the prediction)

Supports Individualization

4/25/2019

4














NYU Medical Knowledge Report

NYU Analytics CenterClinical Curriculum

High level

view for

chairs

Curriculum Stage














Path Diagram for the Class of 2014

MCATMCAT

College GPA

College GPA

USMLE1USMLE1

0.26

0.37 0.62USMLE2USMLE2SHELFSHELF

0.36

0.42

0.29

0.57Med_GPAMed_GPA

4/25/2019

5

Path Diagram for the Class of 2014

MCATMCAT

College GPA

College GPA

USMLE1USMLE1

0.26

0.37 0.62USMLE2USMLE2SHELFSHELF

0.36

0.42

0.29

0.57Med_GPAMed_GPA

Not significant














Gartner Model

© Gartner Group

ADMISSIONS MODEL

NEW TARGETS for the PREDICTIONS

Created a data mart with several combined sources, all linked

via the NPI

– AMA Masterfile data for NYU UME/GME graduates

– CMS Physician Compare: Directory and Quality

– Medicare Part D Prescribing Data

– CMS Utilization and Payment Data

– NPI Database

– New York State SPARCS

4/25/2019

6

Create a database of your graduates with their NPI numbers

Our Approach at NYU Langone

Education Data

Warehouse

Curricular

Content

Admissions ePortfolio

Case and

Procedure Logs

Care Guideline and

Decision Support

Evaluations /

Assessments

Reporting and

Analytics

Data Marts for

Education Research

Simulation

Clinical Data

WarehouseNYU Practice

Network

Epic EMR

AMA Masterfile data for NYU

CMS Physician Compare

CMS Part D Prescribing

CMS Utilization and Payment

NPI Database

New York State SPARCS

EPA /

Milestones

NYU GME Grads

Measure National Rate NYU Med School Non-NYU Med School

Breast Cancer Screening 59.78%

n=13004

78.14%

n=7

48.11%

n=37

Care Plan 62.05%

n=10921

68.69%

n=13

51.04%

n=48

Colorectal Cancer Screening 58.51%

n=15215

53.6%

n=5

37.93%

n=45

Documentation of Current Medications in the Medical Record 93.69%

n=64816

97.73%

n=37

92.66%

n=163

Pneumonia Vaccination Status for Older Adults 62.81%

n=19722

66.13%

n=8

53.64%

n=59

Body Mass Index (BMI) Screening and Follow-Up Plan 70.94%

n=38491

57.41%

n=17

71.97%

n=91

Screening for High Blood Pressure and Follow-Up Documented 72.43%

n=15454

70.25%

n=12

69.04%

n=50

Use of High-Risk Medications in the Elderly* 11.62%

n=10996

2.6%

n=5

5.49%

n=47

2015 Physician Quality Reporting System and non-PQRS Qualified Clinical Data Registry measures

Physician Quality Reporting System (PQRS) Qualified Clinical Data Registry (QCDR) measure performance rates

Dreyfus Model of Expertise Development

Compound InterestModel of Expertise Development

4/25/2019

7

LEARNING ANALYTICSDescriptive Analytics: Counts, averages, %, min/max













Gartner Model

© Gartner Group

Individualized Report

Example Text Color

Upper 50th “A predictive model based on your quantitative data available to the 18th month point of medical school suggests you are on-track to pass the USMLE”

25-50th percentile “A predictive model based on your quantitative data available to the 18th month point of medical school suggests …”

<25th percentile “Students with your profile of quantitative data up to the 18th month point of medical school have gone on to score between xx and yy on the USMLE Step 1 in 90% of cases.

Individualized ReportExample Text Color

<25th

percentile“Students with your profileof quantitative data up to the 18th month point of medical school have gone on to score between xx and yy on the USMLE Step 1 in 90% of cases.

A 10% improvement in your score on each of these 3 medical knowledge report categories would have resulted in the maximum USMLE Step 1 Score improvement, based on our statistical models. Your mileage may vary.

• Neuroanatomy• Physiology• Histology

Individualized ReportExample Text Color

<25th

percentile“Students with your profileof quantitative data up to the 18th month point of medical school have gone on to score between xx and yy on the USMLE Step 1 in 90% of cases.

A 10% improvement in your score on each of these 3 medical knowledge report categories would have resulted in the maximum USMLE Step 1 Score improvement, based on our statistical models. Your mileage may vary.

• Neuroanatomy• Physiology• Histology

Gartner Model

© Gartner Group

4/25/2019

8

“ The Blade of Grass”

The Atomic Unit of Emergency

Medicine

DECISION

Item Bank

Case 1

Item Bank

Case 1 Case2

4/25/2019

9

Item Bank

. . . Case 1 Case2 Case3 Case4 Case 5 Case xx

0.50

0.55

0.60

0.65

0.70

0.75

0.80

0.85

0 50 100 150 200 250

Number of Cases Completed

Se

nsitiv

ity

0.50

0.55

0.60

0.65

0.70

0.75

0.80

0.85

0 50 100 150 200 250


Se

nsitiv

ity

Time Based

4/25/2019

10

0.50

0.55

0.60

0.65

0.70

0.75

0.80

0.85

0 50 100 150 200 250


Se

nsitiv

ity

Co

mp

ete

ncy B

ased

Most -- Rank by Number of Cases to Achieve Competency -- Fewest

1

Nu

mb

er

of

Ca

se

s

• Confidence

• Fluidity

• Retention

• Sequence Effects

• Case Mix

• Challenge Level

“ The New CME”

LEARNING





potential issues”

- DOE 2012


Education Data

Warehouse

LMS

Curricular

Content

Curricular

Content

AdmissionsExams

ePortfolioPatient

Log

Learning

ModulesEvaluations

SIS


Data Marts for

Education Research

Simulation

Clinical Data

Warehouse

NYU Practice

Network

NYU Practice


4/25/2019

11

Door to Needle Time

iBeacons

Radiology Pilot Project

UK National Audit:

Satisfaction with Surgeon

Adjusted mean scores for “Satisfaction with Surgeon” for hospital

organisations based on women’s responses on the 3 month post

surgery questionnaire

01

02

03

04

05

06

07

08

09

01

00

Me

an

Sco

re

0 50 100 150 200 250

Provider volume

Overall average 95% limits

Provider rate 99.8% limits

UK National Audit:

Satisfaction with Surgeon

Adjusted mean scores for “Satisfaction with Surgeon” for hospital

organisations based on women’s responses on the 3 month post

surgery questionnaire

01

02

03

04

05

06

07

08

09

01

00

Me

an

Sco

re

0 50 100 150 200 250

Provider volume

Overall average 95% limits

Provider rate 99.8% limits

The New CME “Listeners”

• Un-Announced Standardized Patients

• In-Situ Simulations

• Patient Reported Outcome Measures

• Process Metrics

– E.g. Door to Needle Time in Stroke Activations

– E.g. Surgical Video

LEARNING





potential issues”

- DOE 2012

4/25/2019

12

LEARNING

ANALYTICS“Quality improvement refers to the interpretation

of a wide range of data produced by and gathered

on behalf of clinicians in order to assess progress,

predict future performance, and spot potential

issues”

- DOE 2012

Quality

Improvement

















FeedbackThe New CME

• More, better data about individuals

• Learning data and Quality data will

overlap

• Shift from time-based to performance-

based metrics

• Shift from outcome to process metrics

• Shift from measures of knowing to

measures of doing

Limitations

• Big Brother aspect of this needs to be

worked out

• Physician “safety” will be a necessary

component of any system

• Implies resource re-alignments

The New CME

• More, better data about individuals

• Learning data and Quality data will

overlap

• Shift from time-based to performance-

based metrics

• Shift from outcome to process metrics

• Shift from measures of knowing to

measures of doing

4/25/2019

13

“ The End”

Discussion

Education Data For Innovations Committee

Data Access Policies Data Best Practices Innovations

LEARNING ANALYTICS - New York University€¦ · LEARNING ANALYTICS “Learning analytics refers to...

Documents

Transcript of LEARNING ANALYTICS - New York University€¦ · LEARNING ANALYTICS “Learning analytics refers to...