www.kmin-vumc.nl Interpretation of PROMs:

linking measurement error to minimal important change

Caroline Terwee

Knowledgecenter Measurement InstrumentsDepartment of Epidemiology and Biostatistics

VU University Medical Center

www.kmin-vumc.nl Proposition

detectable change is conceptually different from

important change

www.kmin-vumc.nl

Content

• Measurement error (detectable change) –

difference from reliability

• Important change –

anchor-based MIC distribution

• Linking measurement error to minimal important change in individual patients

• Intermezzo –

why distribution-based methods should not be used to define MIC

• Alternative ways of interpreting change scores•

consider type I error

•

estimate the probability of belonging to the importantly improved group

• Linking measurement error to minimal important change on group level

www.kmin-vumc.nl

Measurement error

Terminology

Minimal Detectable Change

Smallest Detectable Change

Real change

Smallest Real Change

Significant change

www.kmin-vumc.nl

Measurement error

Terminology

Minimal Detectable Change

Smallest Detectable Change SDCReal change

Smallest Real Change

Significant change

www.kmin-vumc.nl

Measurement error

Reliability = The proportion of the total variance in the measurements which is due to ‘true’

differences

between patients

Measurement error = The systematic and random error of a patient’s score that is not attributed to true changes in the construct to be measured

www.cosmin.nl

Reliability

InternalConsistency Reliability

(test-retest,Inter-rater,Intra-rater)

Measurement error

(test-retest,Inter-rater,Intra-rater)

www.kmin-vumc.nl

Measurement error – consistency

errorpatientspatientsICCyReliabilit yconsistenc 22

2

σσσ

+==

www.kmin-vumc.nl

Measurement error – consistency

errorpatientspatientsICCyReliabilit yconsistenc 22

2

σσσ

+==

error)(SEM nt Measuremeof Error Standard yconsistenc2σ=

www.kmin-vumc.nl

Measurement error – consistency

errorpatientspatientsICCyReliabilit yconsistenc 22

2

σσσ

+==

ICCSDSEM yconsistencyconsistenc −= 1*

error)(SEM nt Measuremeof Error Standard yconsistenc2σ=

www.kmin-vumc.nl

Measurement error – consistency

errorpatientspatientsICCyReliabilit yconsistenc 22

2

σσσ

+==

yconsistencyconsistenc SEM2*1.96 (SDC Change Detectable Smallest *) =

ICCSDSEM yconsistencyconsistenc −= 1*

error)(SEM nt Measuremeof Error Standard yconsistenc2σ=

www.kmin-vumc.nl

Measurement error – consistency

errorpatientspatientsICCyReliabilit yconsistenc 22

2

σσσ

+==

error)(SEM nt Measuremeof Error Standard yconsistenc2σ=

agreement of limits SD*1.96 SDC changeyconsistenc ==

yconsistencyconsistenc SEM2*1.96 (SDC Change Detectable Smallest *) =

ICCSDSEM yconsistencyconsistenc −= 1*

www.kmin-vumc.nl

Measurement error – agreement

errortsmeasuremenpatientspatientsICC y Reliabilit agreement 222

2

σσσσ

++==

www.kmin-vumc.nl

Measurement error – agreement

errortsmeasuremen)(SEM nt Measuremeof Error Standard agreement22 σσ +=

errortsmeasuremenpatientspatientsICC y Reliabilit agreement 222

2

σσσσ

++==

www.kmin-vumc.nl

Measurement error – agreement

errortsmeasuremen)(SEM nt Measuremeof Error Standard agreement22 σσ +=

errortsmeasuremenpatientspatientsICC y Reliabilit agreement 222

2

σσσσ

++==

ICCSDSEM agreementagreement −= 1*

www.kmin-vumc.nl

Measurement error – agreement

agreementagreement SEM2*1.96 SDC *=

errortsmeasuremen)(SEM nt Measuremeof Error Standard agreement22 σσ +=

errortsmeasuremenpatientspatientsICC y Reliabilit agreement 222

2

σσσσ

++==

ICCSDSEM agreementagreement −= 1*

www.kmin-vumc.nl

Measurement error – agreement

agreementagreement SEM2*1.96 SDC *=

errortsmeasuremen)(SEM nt Measuremeof Error Standard agreement22 σσ +=

errortsmeasuremenpatientspatientsICC y Reliabilit agreement 222

2

σσσσ

++==

ICCSDSEM agreementagreement −= 1*

agreement of limits SDCagreement ≠

www.kmin-vumc.nl

Measurement error

agreementagreement SEM2*1.96 SDC *=

SDC is the smallest change in score that you CAN detect with the instrument, above measurement error

in individual patients

www.kmin-vumc.nl

Measurement error - example

www.kmin-vumc.nl

Measurement error - example

Scores 0-100

Measurement error of OES Function = 19 points on a scale from 0-100

www.kmin-vumc.nl

Mimimal Important Change (MIC)

MIC is the smallest change in score that you WANT to detect with the instrument

MIC is determined by an anchor-based method

www.kmin-vumc.nl

Mimimal Important Change (MIC)

Example anchor: Global rating of change

1.

Completely recovered

2.

Much improved

3.

Slightly improved

4.

No change

5.

Slightly worse

6.

Much worse

Importantly improved

Not importantly changed

www.kmin-vumc.nl

Achor-based MIC distribution

+

_

Importantly improved

ANCHORANCHOR

Not importantly changed

O

Cha

nge

on in

stru

men

t

www.kmin-vumc.nl

Anchor-based MIC distribution

+

_

Importantly improved

ANCHORANCHOR

Not importantly changed

O

Cha

nge

on in

stru

men

t

ROC cut-off point

MIC = ROC cut-off point

www.kmin-vumc.nl

Anchor-based MIC distribution - example

www.kmin-vumc.nl

Mimimal Important Change (MIC) - example

Improved = ‘slightly better’, ‘much better’, ‘no problems now’Not improved = ‘no change’, ‘worse’

MIC for OES Function is 5 points

www.kmin-vumc.nl Proposition

Smallest Detectable Change (SDC)is conceptually different from

Minimal Important Change (MIC)

SDC is the smallest change in score that you CAN detect with the instrument, above measurement error

MIC is the smallest change in score that you WANT to detect with the instrument

www.kmin-vumc.nl

Linking SDC to MIC

In individual patients SDC should be smaller than MIC to distinghuish important change from measurement error

For may PROMs this is not the case

www.kmin-vumc.nl

Linking SDC to MIC - example

For OES Function SDC (19) is larger than MIC (5)For OES Pain SDC (8) is smaller than MIC (12.5)

Thus with the OES pain we can distinguish important change from measurement errobut with the OES Function this is not possible

www.kmin-vumc.nl

Linking SDC to MIC

SDC and MIC are two reference points in the scale that can help interpret change scores

Example 1 -

SDC is smaller than MIC

Change NOT statistically significant and NOT important

Change statistically significant AND important

Change statistically significant, but not important

SDC

MICno change maximum change

OES Pain -

SDC (8) is smaller than MIC (12.5)

www.kmin-vumc.nl

Change NOT statistically significant and NOT important Change statistically significant AND important

Change important, but can NOT be distinguished from measurement error

SDCMICno change maximum change

Linking SDC to MIC

Example 2 -

SDC is larger than MIC

OES Function -

SDC (19) is larger than MIC (5)

www.kmin-vumc.nl

Intermezzo

Why distribution-based methods should not be used to define MIC

Distribution-based methods

e.g. MIC = 1*SEM or MIC = 0.5*SD

www.kmin-vumc.nl

Intermezzo

Why distribution-based methods should not be used to define MIC

Distribution-based methods

e.g. MIC = 1*SEM or MIC = 0.5*SD

If MIC is defined as 1*SEM the SDC will always (by definition) be larger than the MIC, because SDC=1.96*√2*SEM.

This would mean that one can never distinguish important change from measurement error in individual patients

www.kmin-vumc.nl

Taking type I error into account

www.kmin-vumc.nl

Taking type I error into account

agreementagreement SEM2*1.96 SDC *=

If we say that a patient who changed as much as the SDC has been

‘really’ changed (statistically significant change), there is a 5% probability (type I

error) that in fact this patient has not changed.

Patients with change scores smaller than the SDC have a higher probability that they are in fact not changed (larger type 1 error).

Thus if the SDC is larger than the MIC, there is a higher type 1

error if we call patients who changed as much as the MIC importantly improved.

www.kmin-vumc.nl

Change NOT statistically significant and NOT important Type 1 error <5%

Change important, but type 1 error may be substantial

SDCMICno change maximum change

Taking type I error into account

Type I error

www.kmin-vumc.nl

Estimate the probability of belonging to the ‘importantly improved’ group

www.kmin-vumc.nl

Estimate the probability of belonging to the ‘importantly improved’ group

+

_

Importantly improved

ANCHORANCHOR

Not importantly changed

O

Cha

nge

on in

stru

men

t

MIC

www.kmin-vumc.nl

Estimate the probability of belonging to the ‘importantly improved’ group

Logistic regression analysis

Dependent variable = group (0=not importantly changed,1= importantly improved)

Independent variable = change score on the instrument

)(11

Xeyprobabilit predicted βα +−+=

www.kmin-vumc.nl

Example

Hypothetical health questionnaire, score 0-100 (higer score = better health)

Sample n=20.000

Anchor:50% = importantly improved50% = not importantly changed

Prior probability of belonging to the ‘importantly improved’

group = 50%

www.kmin-vumc.nl

B S.E. Wald df Sig. Exp(B)95,0% C.I.for

EXP(B)

Lower Upper

Change score ,050 ,001 4005,047 1 ,000 1,052 1,050 1,053

Constant -,966 ,022 1940,184 1 ,000 ,381

Example

Logistic regression analysis

α=-0.966 and β=0.050

)050.0966.0(11

Xey probabilit predicted +−−+=

www.kmin-vumc.nl

Example

0.28 e

p =+

= +−− )0*050.0966.0(11

Patient 1 –

change score = 0

Patient 3 –

change score = 50 0.82 e

p =+

= +−− )50*050.0966.0(11

Patient 2 –

change score = 25 0.43 e

p =+

= +−− )25*050.0966.0(11

www.kmin-vumc.nl

Example

Predicted probabilities of belonging to the ‘importantly improved’ group for each possible change score on the questionnaire

Change score on instrument

Pred

icted

prob

abilit

y

www.kmin-vumc.nl

Finallylinking SDC to MIC – on group level

In research we use mean change scores of groups

Measurement error is reduced because of repeated measurements

nSDC SDC individual

group =

MIC on group level = smaller ???

www.kmin-vumc.nl

Summary

Smallest Detectable Change (SDC) is conceptually different from Minimal Important Change (MIC)

SDC is the smallest change in score that you CAN detect with the instrument, above measurement error

preferably based on SEMagreement

MIC is the smallest change in score that you WANT to detect with the instrument

Should be based on anchor-based methods

The SDC and the MIC are two important reference points on a scale that can help interpret change scores in individual patients

As an alternative, estimating the probability of belonging to the importantly improved group may be useful

www.kmin-vumc.nlThank you

cb.terwee@vumc.nl

Terwee CB, Roorda LD, Knol DL, de Boer MR, de Vet HCW. Linking measurement error to minimal important change of patient-reported outcomes. J Clin Epidemiol 2009;62:1062-1067.

de Vet HCW, Ostelo RWJG, Terwee CB, van der Roer N, Knol DL, Beckerman H, Boers M, Bouter LM. Minimally import ant change determined by a visual method integrating an anchor-

based and a distribution-based approach. Qual Life Res 2007;16:131-142.

de Vet HCW, Terwee CB, Knol DL, Bouter LM. When to use agreement versus reliability measures. J Clin Epidemiol 2006;59:1033-1039.