Ensuring Consistency in Assessment of Continuing Care Needs: An Application of Differential Item...
-
Upload
albert-stewart -
Category
Documents
-
view
217 -
download
0
description
Transcript of Ensuring Consistency in Assessment of Continuing Care Needs: An Application of Differential Item...
![Page 1: Ensuring Consistency in Assessment of Continuing Care Needs: An Application of Differential Item Functioning Analysis R. Prosser, M. Gelin, D. Papineau,](https://reader034.fdocuments.us/reader034/viewer/2022052607/5a4d1b667f8b9ab0599b0621/html5/thumbnails/1.jpg)
Ensuring Consistency in Assessment of Continuing Care Needs:
An Application of Differential Item Functioning Analysis
R. Prosser, M. Gelin, D. Papineau, B. Zumbo,
Presentation at CES ConferenceJune 3, 2003
![Page 2: Ensuring Consistency in Assessment of Continuing Care Needs: An Application of Differential Item Functioning Analysis R. Prosser, M. Gelin, D. Papineau,](https://reader034.fdocuments.us/reader034/viewer/2022052607/5a4d1b667f8b9ab0599b0621/html5/thumbnails/2.jpg)
Outline
• Background– The MDS-HC Assessment Tool & Its Uses– The Challenge
• Methodology– The IADL Scale– Differential Item Functioning Analysis
• Results of Our Look at 5 Raters• Implications for Practice
![Page 3: Ensuring Consistency in Assessment of Continuing Care Needs: An Application of Differential Item Functioning Analysis R. Prosser, M. Gelin, D. Papineau,](https://reader034.fdocuments.us/reader034/viewer/2022052607/5a4d1b667f8b9ab0599b0621/html5/thumbnails/3.jpg)
What is MDS?• Standardized assessment of functional,
psychological, social and environmental needs, administered by clinician raters
• Target groups include older adults and people with disabilities or chronic diseases
• Minimum Data Set: a tool for collecting the minimum amount of data needed for producing a comprehensive and accurate profile
• Our focus is on MDS for Home Care (HC) clients• Mandated for use in B.C.
![Page 4: Ensuring Consistency in Assessment of Continuing Care Needs: An Application of Differential Item Functioning Analysis R. Prosser, M. Gelin, D. Papineau,](https://reader034.fdocuments.us/reader034/viewer/2022052607/5a4d1b667f8b9ab0599b0621/html5/thumbnails/4.jpg)
A Word About InterRAI• International group of 40+ researchers and
clinicians– Registered as not-for-profit corporation and owns
international copyright on RAI instruments– Conducts multinational collaborative research to
develop, implement and evaluate the instruments and their related applications
– Tools available in 12 languages
– Lead Canadian researcher: Dr. John Hirdes (University of Waterloo)
![Page 5: Ensuring Consistency in Assessment of Continuing Care Needs: An Application of Differential Item Functioning Analysis R. Prosser, M. Gelin, D. Papineau,](https://reader034.fdocuments.us/reader034/viewer/2022052607/5a4d1b667f8b9ab0599b0621/html5/thumbnails/5.jpg)
Uses of MDS
Assessment
Care Planning
Outcome Measures Quality Indicators
Resource Allocation
![Page 6: Ensuring Consistency in Assessment of Continuing Care Needs: An Application of Differential Item Functioning Analysis R. Prosser, M. Gelin, D. Papineau,](https://reader034.fdocuments.us/reader034/viewer/2022052607/5a4d1b667f8b9ab0599b0621/html5/thumbnails/6.jpg)
Components of the MDS System
– Minimum Data Set assessment form– Assessment protocols (trigger issues to
review in developing client’s care plan)– Quality indicators (standards and targets)– Health outcome scales (effectiveness)– Resource utilization groups: case mix
system of groups with homogeneous resource requirements
![Page 7: Ensuring Consistency in Assessment of Continuing Care Needs: An Application of Differential Item Functioning Analysis R. Prosser, M. Gelin, D. Papineau,](https://reader034.fdocuments.us/reader034/viewer/2022052607/5a4d1b667f8b9ab0599b0621/html5/thumbnails/7.jpg)
Key Outcome Scales
• Cognitive Performance Scale (CPS)• Depression Rating Scale (DRS)• Activities of Daily Living (ADL) Performance
Scales• Independent Activities of Daily Living (IADL)
Scales
![Page 8: Ensuring Consistency in Assessment of Continuing Care Needs: An Application of Differential Item Functioning Analysis R. Prosser, M. Gelin, D. Papineau,](https://reader034.fdocuments.us/reader034/viewer/2022052607/5a4d1b667f8b9ab0599b0621/html5/thumbnails/8.jpg)
Uses of Outcome Scales
• Provide additional information to determine service needs and to plan care
• Evaluate the effects of treatments• Compare results in different settings for
client groups with similar health issues and needs
![Page 9: Ensuring Consistency in Assessment of Continuing Care Needs: An Application of Differential Item Functioning Analysis R. Prosser, M. Gelin, D. Papineau,](https://reader034.fdocuments.us/reader034/viewer/2022052607/5a4d1b667f8b9ab0599b0621/html5/thumbnails/9.jpg)
Uses of MDS in Planning & Implementing a New Service Model
• Establish / refine service inclusion and exclusion criteria using RAI-HC
• Permits more reliable and equitable service allocation
• Determine number of clients on current caseloads who could benefit from this service
• Estimate future demand for this service
![Page 10: Ensuring Consistency in Assessment of Continuing Care Needs: An Application of Differential Item Functioning Analysis R. Prosser, M. Gelin, D. Papineau,](https://reader034.fdocuments.us/reader034/viewer/2022052607/5a4d1b667f8b9ab0599b0621/html5/thumbnails/10.jpg)
The Challenge• Need to have an objective basis for evaluating
and demonstrating assessor comparability and improving the MDS rating process – To ensure that service allocation decisions based on
assessments are made fairly—and transparently so (i.e., can be explained and justified)
– To provide reliable information for care planning and program evaluation
• Time pressure (long instrument)• Small samples
![Page 11: Ensuring Consistency in Assessment of Continuing Care Needs: An Application of Differential Item Functioning Analysis R. Prosser, M. Gelin, D. Papineau,](https://reader034.fdocuments.us/reader034/viewer/2022052607/5a4d1b667f8b9ab0599b0621/html5/thumbnails/11.jpg)
The IADL Scale• 7 items
– Meal Preparation– Ordinary Housework– Managing Finance– Managing Medications– Phone Use– Shopping– Transportation
• Four point scale for each item– Independent, Some Help, Full Help, By Others
![Page 12: Ensuring Consistency in Assessment of Continuing Care Needs: An Application of Differential Item Functioning Analysis R. Prosser, M. Gelin, D. Papineau,](https://reader034.fdocuments.us/reader034/viewer/2022052607/5a4d1b667f8b9ab0599b0621/html5/thumbnails/12.jpg)
Differential Item Functioning (DIF)• Analytic Foundation: Item Response Theory of
measurement• Answers the question “Do item response curves
differ appreciably for subgroups of a population, e.g., males & females or blacks & whites?”
• Often used to provide evidence in relation to item bias in achievement testing
• In our case, subgroups are seniors rated by DIFFERENT RATERS
• Our Question: Do different raters use an item the same way?
![Page 13: Ensuring Consistency in Assessment of Continuing Care Needs: An Application of Differential Item Functioning Analysis R. Prosser, M. Gelin, D. Papineau,](https://reader034.fdocuments.us/reader034/viewer/2022052607/5a4d1b667f8b9ab0599b0621/html5/thumbnails/13.jpg)
Item Characteristic Curves & DIF
Ability
43210-1-2-3-4
Pro
babi
lity
of a
"Ye
s"1.0
.9
.8
.7
.6
.5
.4
.3
.2
.10.0
![Page 14: Ensuring Consistency in Assessment of Continuing Care Needs: An Application of Differential Item Functioning Analysis R. Prosser, M. Gelin, D. Papineau,](https://reader034.fdocuments.us/reader034/viewer/2022052607/5a4d1b667f8b9ab0599b0621/html5/thumbnails/14.jpg)
Our Method
• Began by looking at DIF via logistic regression
• Decided that graphical approach is much more informative J. Ramsay’s Testgraf software
![Page 15: Ensuring Consistency in Assessment of Continuing Care Needs: An Application of Differential Item Functioning Analysis R. Prosser, M. Gelin, D. Papineau,](https://reader034.fdocuments.us/reader034/viewer/2022052607/5a4d1b667f8b9ab0599b0621/html5/thumbnails/15.jpg)
Our Sample
• Seniors from one of the four subregions within North & West Vancouver
• Selected the five raters for that area who had the largest caseloads
• Included only cases with complete data (i.e., no ratings omitted due to the target activity not being performed within the time frame) N = 335
![Page 16: Ensuring Consistency in Assessment of Continuing Care Needs: An Application of Differential Item Functioning Analysis R. Prosser, M. Gelin, D. Papineau,](https://reader034.fdocuments.us/reader034/viewer/2022052607/5a4d1b667f8b9ab0599b0621/html5/thumbnails/16.jpg)
Raters’ Subgroups
5489847038N =
rater id
54321
clie
nt a
ge a
t ass
essm
ent
100
90
80
70
60
50
40
30
20
10
0
283
157155314329
330334289
282
125114123
3204
310
97
134177
129
34321
256
281
71
57
73
67
79
0
10
20
30
40
50
60
70
80
90
1 2 3 4 5
Rater ID
Perc
ent F
emal
e
![Page 17: Ensuring Consistency in Assessment of Continuing Care Needs: An Application of Differential Item Functioning Analysis R. Prosser, M. Gelin, D. Papineau,](https://reader034.fdocuments.us/reader034/viewer/2022052607/5a4d1b667f8b9ab0599b0621/html5/thumbnails/17.jpg)
The IADL Scale
0
0.5
1
1.5
2
2.5
IADL Item Number
mean 1.54 2.07 1.27 0.82 0.4 1.79 1.57
SD 1.16 1.06 1.26 1.12 0.84 1.15 1.24
1 2 3 4 5 6 7
Scree Plot
Factor Number
7654321
Eig
enva
lue
5
4
3
2
1
0
![Page 18: Ensuring Consistency in Assessment of Continuing Care Needs: An Application of Differential Item Functioning Analysis R. Prosser, M. Gelin, D. Papineau,](https://reader034.fdocuments.us/reader034/viewer/2022052607/5a4d1b667f8b9ab0599b0621/html5/thumbnails/18.jpg)
Curve for Meal Preparation Item
![Page 19: Ensuring Consistency in Assessment of Continuing Care Needs: An Application of Differential Item Functioning Analysis R. Prosser, M. Gelin, D. Papineau,](https://reader034.fdocuments.us/reader034/viewer/2022052607/5a4d1b667f8b9ab0599b0621/html5/thumbnails/19.jpg)
Curve for Meal Preparation Item
![Page 20: Ensuring Consistency in Assessment of Continuing Care Needs: An Application of Differential Item Functioning Analysis R. Prosser, M. Gelin, D. Papineau,](https://reader034.fdocuments.us/reader034/viewer/2022052607/5a4d1b667f8b9ab0599b0621/html5/thumbnails/20.jpg)
Curves for Housework Item (5 raters)
![Page 21: Ensuring Consistency in Assessment of Continuing Care Needs: An Application of Differential Item Functioning Analysis R. Prosser, M. Gelin, D. Papineau,](https://reader034.fdocuments.us/reader034/viewer/2022052607/5a4d1b667f8b9ab0599b0621/html5/thumbnails/21.jpg)
Housework item (Option curves)Option 1 Option 2
Option 3 Option 4
![Page 22: Ensuring Consistency in Assessment of Continuing Care Needs: An Application of Differential Item Functioning Analysis R. Prosser, M. Gelin, D. Papineau,](https://reader034.fdocuments.us/reader034/viewer/2022052607/5a4d1b667f8b9ab0599b0621/html5/thumbnails/22.jpg)
Curves for Transportation Item(Item curve)
![Page 23: Ensuring Consistency in Assessment of Continuing Care Needs: An Application of Differential Item Functioning Analysis R. Prosser, M. Gelin, D. Papineau,](https://reader034.fdocuments.us/reader034/viewer/2022052607/5a4d1b667f8b9ab0599b0621/html5/thumbnails/23.jpg)
Transportation Item (Option curves)Option 1 Option 2
Option 3 Option 4
![Page 24: Ensuring Consistency in Assessment of Continuing Care Needs: An Application of Differential Item Functioning Analysis R. Prosser, M. Gelin, D. Papineau,](https://reader034.fdocuments.us/reader034/viewer/2022052607/5a4d1b667f8b9ab0599b0621/html5/thumbnails/24.jpg)
Summary of Key Findings
• The five raters behaved similarly for most items
• Rater 5 appears to be using items differently than the other raters, particularly item 7
![Page 25: Ensuring Consistency in Assessment of Continuing Care Needs: An Application of Differential Item Functioning Analysis R. Prosser, M. Gelin, D. Papineau,](https://reader034.fdocuments.us/reader034/viewer/2022052607/5a4d1b667f8b9ab0599b0621/html5/thumbnails/25.jpg)
Implications for Practice• Consider elaboration / clarification of scoring
instructions provided to raters• Conduct periodic reviews / monitoring with
raters to ensure that consistency is maintained• Provide additional training to particular raters
on particular items as needed
• May want to have raters rate random samples of clients or stratify the analyses
![Page 26: Ensuring Consistency in Assessment of Continuing Care Needs: An Application of Differential Item Functioning Analysis R. Prosser, M. Gelin, D. Papineau,](https://reader034.fdocuments.us/reader034/viewer/2022052607/5a4d1b667f8b9ab0599b0621/html5/thumbnails/26.jpg)
Benefits of this Approach to DIF Analysis
• Can be used with relatively small samples of the population being rated
• Raters can rate different seniors provided the samples are equivalent random assignment
• Short scales (small numbers of items) are OK• Graphical approach is easy to interpret yet
reveals details needed for making changes to instructions / training of raters
![Page 27: Ensuring Consistency in Assessment of Continuing Care Needs: An Application of Differential Item Functioning Analysis R. Prosser, M. Gelin, D. Papineau,](https://reader034.fdocuments.us/reader034/viewer/2022052607/5a4d1b667f8b9ab0599b0621/html5/thumbnails/27.jpg)
Areas for Future Work
• Testing with raters– How well does the feedback work in practice?– Is transparency increased?
• Determining rules of thumb for use– Minimum numbers of cases to rate– Scale features, e.g., minimum length needed
![Page 28: Ensuring Consistency in Assessment of Continuing Care Needs: An Application of Differential Item Functioning Analysis R. Prosser, M. Gelin, D. Papineau,](https://reader034.fdocuments.us/reader034/viewer/2022052607/5a4d1b667f8b9ab0599b0621/html5/thumbnails/28.jpg)
For Further Information
• InterRAI Web Site: http://nt8380.hrca.harvard.edu/
• Differential Item Functioning: http://www.educ.ubc.ca/faculty/zumbo/DIF/
• TestGraf Web Site: ftp://ego.psych.mcgill.ca/pub/ramsay/testgraf/
• Author e-mail: [email protected]