Nan Rothrock, Ph.D.

Northwestern University

May 22, 2012

Problems in patient-reported outcome measures

PROMIS approach to PRO instrument development

Available PROMIS instruments Reliability, validity

PROMIS and the FDA

Many measures of same health concept

Widely varying quality

Difficult to compare and combine data

. . . across studies

. . . across conditions

Complex

Long

01

23

- 1

- 2

- 3

Questionnairewith a widerange -but low precision

12 Questionnaire

with a highprecision -but small range

National Institutes of Health, 2003

• Attack the Patient-Reported Outcome (PRO) “Tower of Babel”

• Harness modern psychometric methods

• Improve quality and interpretability of PROs

Bruegel, 1563

Nine-year commitment of NIH $80+ million investment 15 funded research sites

Methodology

Measures (Instruments)

Software

Item = question or statement a patient answers

Instrument = collection of items Legacy = existing instrument that is

“gold standard” or a commonly used and widely accepted instrument

Domain focused, not disease focused Domain = feeling, function or perception you

want to measure (e.g., anxiety, physical function, general health perceptions)

Item Banks A large collection of items measuring one

domain Any and all items can be used to provide a

score Can be administered as Computerized Adaptive

Tests (CATs) or fixed-length short forms

Literature review

Focus groups

Archival data

analysis Expert review/

consensus

Binning and

winnowing

Literacy level

analysis

Expert item revision

Cognitive interviews

Translation review

Large-scale testing

Validation studies

Calibration decisions

Intellectual property

Short formCAT

Statistical analysis

Domain Framework

Self-Reported

Health

Social Social HealthHealth

Mental Mental HealthHealth

Physical Physical HealthHealth

SymptomsSymptoms

FunctionFunction

AffectAffect

BehaviorBehavior

CognitionCognition

RelationshipsRelationships

FunctionFunction

Pain Behavior

Pain Interference

Sleep-related Impairment

Sleep Disturbance

Fatigue

Physical Function

Pain Interference

Fatigue

Upper Extremity

Mobility

Asthma Impact

Adult Pediatric/Parent Proxy

Sexual Function

Physical Health

Pain Intensity

Mental Health

Anxiety

Depression

Psychosocial Illness Impact

Anger

Applied Cognition Concerns

Anxiety

Depression

Anger

Adult

Applied Cognition Abilities

Alcohol Use

Alcohol Consequences

Alcohol Expectancies

Pediatric/Parent Proxy

Social Health

Ability to Participate in

Roles & Activities

Emotional Support

Companionship

Informational Support

Peer Relationships

Adult

Instrumental Support

Satisfaction with Roles &

Activities

Social Isolation

Pediatric/Parent Proxy

In the past 7 days … Never Rarely Some-times

Often Always

FATEXP 20

How often did you feel tired? 1

2

3

4

5

FATEXP 5

How often did you experience extreme exhaustion? 1

2

3

4

5

FATEXP 18

How often did you run out of energy? 1

2

3

4

5

FATIMP 33

How often did your fatigue limit you at work (include work at home)?

1

2

3

4

5

FATIMP 30

How often were you too tired to think clearly? 1

2

3

4

5

FATIMP 21

How often were you too tired to take a bath or shower? 1

2

3

4

5

FATIMP 40

How often did you have enough energy to exercise strenuously?

1

2

3

4

5

Reprinted with permission of the PROMIS Health Organization and the PROMIS Cooperative Group © 2007.

In the past 7 days … Never Rarely Some-times

Often Always

FATEXP 20

How often did you feel tired? 1

2

3

4

5

FATEXP 5

How often did you experience extreme exhaustion? 1

2

3

4

5

FATEXP 18

How often did you run out of energy? 1

2

3

4

5

FATIMP 33

How often did your fatigue limit you at work (include work at home)?

1

2

3

4

5

FATIMP 30

How often were you too tired to think clearly? 1

2

3

4

5

FATIMP 21

How often were you too tired to take a bath or shower? 1

2

3

4

5

FATIMP 40

How often did you have enough energy to exercise strenuously?

1

2

3

4

5

Available Universal Spanish

In Process German Portuguese Mandarin Chinese French Italian Norwegian Others – see

nihpromis.org/measures/translations

T Score Mean = 50 Standard Deviation = 10

Referenced to the US General Population

Adult GI Symptoms Self-efficacy for management of chronic disease

Pediatric Pain Behavior, Quality, Intensity Physical Activity Experience of Stress Subjective Well-being Impact of Child Illness on Family Family Belongingness

SE = 3.3rel = 0.90

SE = 2.3rel = 0.95

0

0.1

0.2

0.3

0.4

0.5

-4 -3 -2 -1 0 1 2 10 20 30 40 50 60 70

SF-36 10 items

HAQ 20 items

CAT 10 items

PROMISShort Form20 items

PROMISShort Form10 items

rheumatoid arthritis patients US general population

Err

or

Depression

CE

SD

-2 -1 0 1 2 3 4

010

2030

4050

60

r =0.84

-2 -1 0 1 2 3 4

080

Depression

Depression

CE

SD

-2 -1 0 1 2 3 4

010

2030

4050

60

010

2030

4050

60

0 100 250

CE

SD

PROMIS Depression

Importance of PRO development to include patient voices

Importance of sound measurement Confusion in selecting an instrument

because of huge array of choices

Ongoing discussions via Interagency Clinical Outcomes Assessment Working Group to qualify PROMIS Fatigue measures, attendance and presentations at PRO Consortium

FDA Approach evaluate content validity in each clinical population in which the measure may be used

PROMIS Approach there is commonality in patients’ experiences of symptoms/outcomes and their impact on QOL Need to re-validate a well-developed & valid

instrument in a target population is questionable

Magasi, S. et al (2011) Content validity of patient-reported outcome measures: Perspectives from a PROMIS meeting. Quality of Life Research

  

N

PROMIS FatigueSF v1.0 PROMIS FatigueMS

Mean SD Mean SD Expanded Disability Status Scale (EDSS)            Mild (0-4) 83 52.2 8.2 52.5 9.2

  Moderate (4.5-6.5)104 60.5 6.4 60.7 5.6

  Severe (7.0-9.5) 43 60.7 8.3 60.5 8.7Fatigue Severity (0-10 NRS)            None/Mild (0-1) 18 43.0 4.5 42.5 5.4  Moderate (2-4) 58 51.0 6.0 51.3 6.6

  Severe (5-10)154 61.7 5.8 61.9 5.5

Vitality (item from the MOS)            None/A little 52 63.8 5.3 64.2 5.4  Some 88 59.9 6.3 60.1 5.5  Quite a lot 44 55.7 6.6 56.0 6.8  Very Much 45 47.5 7.3 47.0 7.9

PROMIS FatigueSFv1.0 and PROMIS FatigueMS Scores

by Disability Status, Fatigue Severity, and Vitality Scores

Cook et al, QOLR, 2011

Supplement with targeted measures Item banking allows flexible item choice

without loss of a standard scoring base Alternative is a messy array of

contenders that fail to communicate across themselves regarding severity or result interpretation

Comparability Provide the ability to compare or combine results

from multiple studies.

Reliability and Validity Reduce response burden. Improve measurement precision.

Simplify administration via computer-based administration, scoring, and reporting

IRT-based MIDs on a T-Score scale

Multiple cross-sectional and longitudinal anchors (18)

Summarized with nonparametric statistics (median, interquartile range)

Instrument T-Score MID

Points

T-Score MID

Effect Sizes*

Raw Score MID

Points

Raw Score MID

Effect Sizes‡

Fatigue 3 -5 0.4 - 0.7 2-3 0.4 - 0.6

Pain Interference 4 -6 0.5 - 0.7 4-7 0.3 - 0.6

Physical Function 4 -6 0.5 - 0.7 3-6 0.3 - 0.6

Anxiety 3-5 0.3 - 0.6 3-4 0.5 - 0.6

Depression 3-5 0.3 - 0.5 3-4 0.4 - 0.6

*Calculated as the T-Score MID divided by the Assessment 1 T-Score standard deviation‡Calculated as the Raw Score MID divided by the Assessment 1 Raw Score standard deviation

Recommended IRT-based T-Score MIDs and Raw Score MIDs for PROMIS-Cancer Short Forms in Advanced Cancer Patients

CATT-Score MID

PointsT-Score MID Effect Sizes

Fatigue 4.5-5.0 0.57-0.63

Pain Interference 3.0-6.0 0.34-0.67

Physical Function 4.5-7.5 0.21-0.80

Anxiety 3.0-4.5 0.38-0.58

Depression 2.5-4.5 0.32-0.58

Recommended IRT-based T-Score MIDs for PROMIS-Cancer CATs