Toolkit for Evaluation, Assessment, and Monitoring for ...
Transcript of Toolkit for Evaluation, Assessment, and Monitoring for ...
Toolkit for Evaluation, Assessment, and Monitoring for Adult Community Services and Supports Programs
Sacramento Regional MeetingMarch 22, 2017
Agenda Overview
Time Activity
11:30 – 12:00 Welcome and Introduction (Conference call begins at 11:30)Overview of Toolkit
12:00 – 1:00 Lunch (Presentations will resume at 12:30)Background and rationale of Toolkit
1:00 – 2:30Toolkit Components (Conference call ends at 2:30)
Preview of Toolkit componentsQuestions/Discussion
2:30 – 2:45 Break
2:45 – 3:45 Breakout groupsRound table discussions about specific components of the Toolkit
3:45 – 4:15 Wrap up
4:15 – 4:30 Final Comments
Project Team
Health Services Research Center (HSRC) Andrew Sarkin, PhD
Project Director
Frances Reyes, MAProject Manager
Gina Beal, MPA, MAProgram Evaluation Associate
Kyle Choi, MPHInformatics Project Manager
Edith Wilson, PhDResearch Associate
Lucyna Klinicka, BAProgram Evaluation Specialist
Jessica Rampton, MPHResearch Evaluation Specialist
Justine Zimiles, BAProgram Evaluation Specialist
Rami Hamid, MBBS, MPSResearch Assistant
Merissa Kado, BAResearch Assistant
Nicholas Panagon, MPHResearch Assistant
Project Team
Mental Health Services Oversight and Accountability Commission (MHSOAC)
Fred Molitor, PhDDirector of Research and Evaluation
Marijoyce Naguit, MPHSenior Researcher
Kayla Landry, BAResearch Associate
What is the Toolkit for Evaluation, Assessment, and Monitoring(TEAM)?
A set of resources and information provided by MHSOAC to support and empower California counties and programs to: Determine readiness for adopting a tracking, monitoring,
and evaluation systems for collecting and reporting evaluation data
Implement the measures and data collection processes successfully, and provide training
Create and use evaluation reports on various data collected from their systems
Guide the program/county on which parts of the Toolkit should be used based on their specific needs
Toolkit Components
1. Introductory materials2. Feasibility checklist3. Decision support tools4. Guidance on integration into electronic data
collection systems5. Data-entry and reporting tools6. Training materials7. Supplemental documents
Why was the Toolkit for developed?
MHSOAC wants to empower all counties to evaluate their behavioral health services in meaningful ways, without adding additional state requirements.
Counties and programs need better resources to help them collect, process, and analyze data that will inform service improvement and resource planning.
Clinicians want data that is clinically meaningful and will contribute to caring for the people they serve.
Clients and their families want to have a strong voice in their own care.
The toolkit is designed to address these needs and more!
How can the Toolkit be used?
Implement all elements from the Toolkit to develop a plan, provide assessment measures, collect data, and report evaluation outcomes.
Use certain components of the Toolkit to align with local needs and resources.
Select features from the Toolkit that will supplement clinical work and evaluation measures already used.
Who should use the Toolkit?
Clinical or Medical Directors Program Directors and
Managers Mental Health Researchers Program Evaluation and
Quality Improvement Analysts
County or Program Informatics Coordinators
Behavioral Healthcare Providers
Continuous Program Improvement Identifying training and technical assistance needs
Test program changes or new programs
Identifying client groups that need attention
Accountability to Stakeholders Staff, Funders, Community, Clients and Families
Securing Future Funding
Importance of MeasurementWhy invest time in program evaluation activities?
Tracking Individual Progress and Treatment Plans Promoting an Integrated Recovery Orientation Facilitating Recovery and Communication
Between people receiving and providing services
Between service providers in an integrated system
Importance of MeasurementHow will this help our individual clients?
To better understand the people we are helping To comprehensively assess a person’s needs To aid in treatment planning and goal setting To assess recovery progress and outcomes Identify critical indicators Identify strengths that can aid in recovery Enhancing the therapeutic dialogue to
Increase client involvement in treatment planning
Set shared goals and monitor recovery together
Increase sharing of information between team members
Clinical Utility of Evaluation Data
Phase I: CSS Tracking, Monitoring, and Evaluation Project
TimeframeMay 2014 – June 2016 (completed successfully)
Goals Develop and pilot a standardized data collection system for adults
receiving non-FSP CSS services
Allows for evaluation of clients and services
Can be integrated into other Electronic Health Records or Data Collection and Reporting Systems (such as eBHS and Avatar).
Creation of policy and practice recommendations for how to improve upon current CSS services evaluation and quality improvement systems.
Phase I: CSS Tracking, Monitoring, and Evaluation Project
Primary Evaluation Questions Answered What statewide methods should be employed to ensure proper
tracking, monitoring, and evaluation of adults receiving CSS services?
What policies, practices, systems, and infrastructure should be created and/or modified to better track, monitor, and evaluate adults receiving CSS services?
What policies, practices, systems, and infrastructure should be created and/or modified to better use information to serve adults within the CSS component?
Domains and Data Elements
Review of relevant documents and guidelines Mental Health Services Act requirements and goals Bronzan-McCorquodale and other laws MHSOAC and DHCS goals National Behavioral Healthcare Quality Framework Relevant published work on measuring outcomes Results of other similar projects
MOQA, MHDATA DCR Tools, County-level efforts, SAMHSA, etc.
Stakeholders and the Evaluation Advisory Group provided input on what measures to implement and what measurement procedures to use
Review of current systems such as CSI, DCR, and Electronic Health Records and other Data Collection Systems utilized
Evaluation and comparison of validated health measures
Validated Measures Review
Contract requires the use of validated measures.
Data elements in the DCR have mostly not been validated, and some indicators have been shown to have significant validity problems.
No validated measure would cover everything, so we had to supplement with DCR-type items.
Validated measures allow for standardized comparisons to other programs, and data-based performance criteria.
Assessment Instrument Quality Checklist (AIQC)
Measures Viewer Survey with stakeholders
Maximize Usefulness to Staff
Usefulness to People Getting Services
Validity for Measuring Goals and Outcomes
Cultural Competence and Sensitivity
Minimize Burden to Staff
Burden to People Getting Services
Costs to the Programs
Selection of MeasuresQualities
Selection of MeasuresProcess
Review Available Measures
Relevance to Goals
Clinical Utility
Psychometric Validity
Cultural Competence
Cost, Copyright, and Translation Issues
Development of Some Questions
Identify needs
Preference to validated instruments
Meetings with Stakeholders
Program Directors
CORs and Administration
Local Experts and Academics
Staff delivering services
People getting services
Contributing Stakeholders
MHSOAC California Department of Health Care Services (DHCS) Racial and Ethnic Mental Health Disparities Coalition (REMHDCO) California Mental Health Planning Council County Administrators and Contract Supervisors Clinicians and Staff Subject Matter Experts Policy Makers (Bruce Bronzan, Rusty Selix, more) People with lived experience who use services Family members of people with lived experience Evaluation Advisory Group
Assembled information from stakeholders, current legislation, mental health agencies, existing data collection systems, outcome measures, and reviews of validated instruments
Developed a system to include outcome measures, assessment completion protocols, and data collection and reporting, that can work across multiple systems
Implemented a pilot system in fall 2015 to spring 2016
Gathered feedback from end user surveys, focus groups, and the Mental Health Services Evaluation Advisory Group (MHSEAG)
Developed or adapted measures to fill gaps expressed by stakeholders, such as a measure for family/friends and a strength-based self-report
Sample of Products from Phase I
Clinical Measure Recommendations Use of validated, recovery-oriented outcome measures A measure completed by family member or other close supporter Client-completed measures adapted for use with various populations
Data Collection System Recommendations Optional data elements that programs may choose to complete Data collection software that is integrated with other electronic health
record systems
Reporting Recommendations Customizable reporting formats designed to ensure the usability of
data collected for various audiences Training for clinicians in using the data to provide better services
Sample of Select Recommendations
Purpose of the Project
Development of a toolkit that will:
Provide guidance on how to prepare for a tracking, monitoring, and evaluation system
Help counties and programs determine what measures to implement, and how to implement them
Aid MHSOAC in providing assistance to counties planning to adopt a tracking, monitoring, and evaluation system
Improve mechanisms for the negotiation and sharing standardized client outcomes data
Enhance the clinical utility of new or current tracking, monitoring, and evaluation systems and improve how data is reported
Toolkit Development
Reviewed results and recommendations from Phase I
Developed the project plan and timeline in collaboration with the MHSOAC
Selected subject matter experts and key stakeholders
Formed a Toolkit Evaluation Advisory Group (TEAG) and conducted expert interviews
Developed a draft toolkit
Now holding these regional meetings for your input
Create final toolkit for distribution
Project Timeline October 2016 – July 2017
October
November
December
January
February
March
April
May
June
July
Regional Meetings
March 15‐31
Prepare Final Toolkit and Recommendations for
ImplementationDevelop Draft Toolkit
Deliverable 1: Work Plan
October 2016
Deliverable 2: Draft ToolkitFebruary 2017
Deliverable 3: Regional Meetings
ReportMay 2017
Deliverable 4: Final ToolkitJuly 2017
Toolkit Evaluation Advisory Group (TEAG)
Review toolkit materials
Ensure needs of staff and clients are addressed to increase clinical utility
Discuss strategies to maximize the utility of the toolkit for all stakeholders
Help to plan dissemination of the toolkit
TEAG Members
Dave Pilon Dawn Kaiser Joshua Morgan Lezlie Murch Liz Miles Marshall Lewis
Patricia Wentzel Rebecca Ballinger Rick Heller Saumitra SenGupta Sunjung Cho Tony Hobson
Toolkit Features
Materials to help counties and programs determine readiness for a data collection and
reporting system
Assessment measures for tracking, monitoring,
and evaluating client outcomes
Guidance on integrating a data
collection and reporting system into existing client health
record systems
Data entry templates and automated
dashboards depicting client progress
Materials that can be customized to meet
local needs
Recommendations for data analysis and
reporting
Toolkit Sections
Assessment Measures Integration into
Existing Data Collection Systems
Data Entry and Reporting
Preparation Promotional Materials
Toolkit Sections
Assessment Measures Integration into
Existing Data Collection Systems
Data Entry and Reporting
Preparation Promotional Materials
Assessment Measures
The Toolkit includes a set of questionnaires and supplemental materials for tracking, monitoring, and evaluating client outcomes, including: Clinician and client-completed assessments Decision tools to aid in the selection of
measures and how to implement them Training materials with guidance on the
implementation of new measures
Minimize Burden to Staff Burden to People Getting Services Costs to the Programs
Maximize Usefulness to Staff Usefulness to People Getting Services Validity for Measuring Goals and Outcomes Cultural Competence and Sensitivity
Selection of Measures – Qualities
Selection of Measures – Process
Review Available Measures
Relevance to Goals
Clinical Utility
Psychometric Validity
Cultural Competence
Cost, Copyright, and Translation Issues
Development of Some Questions
Identify specific needs not covered by the validated measures
Preference to validated instruments when possible
Meetings with Stakeholders
Program Directors
County Contract Administrators
Local Experts and Academics
Staff delivering services
People getting services
Measures
Assessment Description
IntakeIntake InformationClient Information (Demographics)
Clinician
Key Indicators: Housing, Employment, Critical Events, and Health
Discharge Information
IMR, Goals, and MORS
CHAMPSSS(Integrated Self-Assessment)
Client-Rated OutcomesSatisfaction (brief)
IMR-FF Family/Friend-Rated Outcomes
Consumer Perception Survey Satisfaction (extended)
Clinician Assessment
Illness Management and Recovery Scales (IMR)
Milestones of Recovery Scale (MORS)
Progress towards goals
Housing level and stability
Employment
Frequency of critical events such as hospitalization
Demographics May already be collected in some systems and imported
Categories adhere to MHSA Innovations and PEI regulations
Illness Management and Recovery (IMR)
Captures clinician perspective of mental health recovery and wellness activities
15 Items, each with 5 anchored responses Measures:
Management of illness and symptom reduction Recovery-oriented behaviors and self-management Substance abuse
Administration Intake Follow-up Discharge
Sample IMR Items
Knowledge: How much do you feel s/he knows about symptoms, treatment, coping strategies (coping methods), and medication?
Not very much A little Some Quite a bit A great deal
Time in structured roles: How much time does s/he spend working, volunteering, being a student, being a parent, taking care of someone else or someone else’s house or apartment? That is, how much time does s/he spend in doing activities for or with another person that are expected of him/her? (This would not include self‐care or personal home maintenance.)
2 hours or less/week 3‐5 hours/week 6‐15 hours/week 16‐30 hours/week More than 30 hours/wk
Impairment of functioning: How much do symptoms get in the way of his/her doing things that s/he would like to do or need to do?
Symptoms really get in his/her way a lot
Symptoms get in his/her way quite a bit
Symptoms get in his/her way somewhat
Symptoms get in his/her way very little
Symptoms don't get in his/her way at all
Relapse of symptoms: When is the last time s/he had a relapse of symptoms (that is, when his/her symptoms have gotten much worse)?
Within the last month In the past 2 to 3 months
In the past 4 to 6 months
In the past 7 to 12 months
Hasn't had a relapse in the past year
Milestones of Recovery Scale (MORS)
1 item with 8 possible anchored responses
Assesses level of recovery/risk, engagement in treatment, and support needs
Administration Intake
Follow-up
Discharge
o 1. Extreme risko 2. Experiencing high risk/not engaged with mental health providerso 3. Experiencing high risk/engaged with mental health providerso 4. Not coping successfully/not engaged with mental health providerso 5. Not coping successfully/engaged with mental health providerso 6. Coping successfully/rehabilitating o 7. Early recoveryo 8. Advanced recoveryo Item not assessed
MORS
CHAMPSSS: Combined Health Assessment, Mental, Physical, Social, Substance, Strengths
Can be as short as one page with simple language
Includes PROMIS Global Health items
Data is comparable to a wide variety of state and national data being collected using NIH PROMIS
Screens and alerts for suicidality and relapse
Recovery-oriented and measures strengths
Optional substance abuse questions
Provides a general quality of life indicator
Supports cost effectiveness analysis using Quality Adjusted Life Years
CHAMPSSS Self-Report Domains
Strengths Mental Health
Physical Health
Social Health
Quality of Life Symptoms
Suicidality Substance Use
Family and Friend Version of the Illness Management and Recovery Scales
Only one page front and back with 15 items
Mirrors some questions on the clinician assessment
Recovery-oriented and measures strengths
Substance abuse questions and relapse alert from another view
Scales combine items to measure: Symptom Management
Participation in Wellness Activities
Substance Abuse
Sample IMR Items
Knowledge: How much do you feel s/he knows about symptoms, treatment, coping strategies (coping methods), and medication?
Not very much A little Some Quite a bit A great deal
Time in structured roles: How much time does s/he spend working, volunteering, being a student, being a parent, taking care of someone else or someone else’s house or apartment? That is, how much time does s/he spend in doing activities for or with another person that are expected of him/her? (This would not include self‐care or personal home maintenance.)
2 hours or less/week 3‐5 hours/week 6‐15 hours/week 16‐30 hours/week More than 30 hours/wk
Impairment of functioning: How much do symptoms get in the way of his/her doing things that s/he would like to do or need to do?
Symptoms really get in his/her way a lot
Symptoms get in his/her way quite a bit
Symptoms get in his/her way somewhat
Symptoms get in his/her way very little
Symptoms don't get in his/her way at all
Relapse of symptoms: When is the last time s/he had a relapse of symptoms (that is, when his/her symptoms have gotten much worse)?
Within the last month In the past 2 to 3 months
In the past 4 to 6 months
In the past 7 to 12 months
Hasn't had a relapse in the past year
Outcome Measures Decision Tool
Purpose of tool: Help counties/programs determine which measures and data elements to collect based on needs and program characteristics
Intended users: Teams or individuals from counties and programs who are deciding on the assessment measures
Clinician Assessment
Are program staff available to complete
assessments?
Yes
Are mental health staff available (e.g., clinician, social worker,
peer specialist)?
YesGo to IMR, MORS, Goals, and Key Outcomes
No Go to Key Outcomes
No Clinician Assessment not completed
MORS
Is the evaluation of the level of
recovery needed? (MORS)
Yes
Are staff trained in MORS or are
resources available for
MORS training?
YesMORS completed (after training)
No MORS not completed
No MORS not completed
Clinician Assessment Example Should the program complete ….? Please start on the left.
Toolkit Sections
Assessment Measures Integration into
Existing Data Collection Systems
Data Entry and Reporting
Preparation Promotional Materials
Electronic Data Capture (EDC) System Integration
The Toolkit provides technical assistance for integrating your EDC with Toolkit. Offers variations of integration and implementation of the
Toolkit Describes the steps for a data integration plan and
exporting data from your EDC Provides details for integrating the Toolkit into common
EDCs
Toolkit Integration FlowchartDecision-making tool to determine the best way to capture and integrate data
EDC Integration Pathways (Variation 1 of 4)
Direct Data Entry Best solution for
small counties Easiest set up Minimizes
paperwork
EDC Integration Pathways (Variation 2 of 4)
Entry Into Local EDC and Toolkit Spreadsheets Can use existing
EDC Use spreadsheet
for variables not captured in EDC
EDC Integration Pathways (Variation 3 of 4)
Using Existing EDC and Paper Forms Solution for systems that are unable to add new
data elements
EDC Integration Pathways (Variation 4 of 4)
Collection Using Forms and Entry into EDC and Toolkit Spreadsheets Option for
programs that must use paper forms
Great if computer access is limited
Integrating the Toolkit into Common EDC systems
Data Integration information available for: Anasazi & Cerner Community Behavioral Health Avatar Clinician’s Gateway CSI & DCR eBHS Echo mHOMS
Toolkit Sections
Assessment Measures Integration into
Existing Data Collection Systems
Data Entry and Reporting
Preparation Promotional Materials
Data Entry and Reporting
The Toolkit contains instruments designed to aid in the processing of data captured through assessment measures.
Data entry
Statistical guidance
Data dashboards
Basic Functionality Data entry and automated analysis in Excel
Build to support the Toolkit Assessments
Data Integration
Manually into Excel Spreadsheet
Data Entry Forms in Excel
Incorporating into EDC
Import From External Data Sources
Data Entry Option 1: Assessment Worksheet IMR Example
There is a separate tab in Excel for each of the 5 different assessment forms
Data Entry Option 1: Data ValidationIMR Example
Message box shows valid codes for variable
Warning message if code is invalid
Data Entry Option 2: Data Entry FormsIMR Example
Data can be entered directly into form
Data goes directly into Excel spreadsheet (Data Entry Option 1)
Data Entry Option 3: Importing data from Electronic Data Capture (EDC)
Working with main EDCs
Adding toolkit data elements to EDCs
Extracting data from EHR to fit spreadsheet format
Dashboards
Clinician Rated Outcomes
• Intake and Discharge Information• Demographics• Goals• Milestones of Recovery (MORS)• Illness Management and Recovery (IMR) Scale
Clinician Rated Outcomes – Key
Indicators
• Housing• Employment• Critical Events• Health
Client Rated Outcomes
• Combined Health Assessment, Mental, Physical, Social, Substance, Strengths (CHAMPSSS)
• Satisfaction ‐Mental Health Statistics Improvement Program (MHSIP)
Family or Friends Rated Outcomes
• Illness Management and Discovery (IMR) Scale (Family or Friends
Reports‐ Frequency Tables: Current Assessment IMR Example
Each variable has a frequency table
For categorical variables: data is also displayed in rank order
Reports‐ Frequency Tables: Previous and Current Assessment IMR Example
Compares the current assessment to the previous assessment using matched data.
Toolkit Sections
Assessment Measures Integration into
Existing Data Collection Systems
Data Entry and Reporting
Preparation Promotional Materials
Preparation
The Toolkit includes materials to inform planning and preparation for implementing a system.
Feasibility Checklist
Data Use Agreement Template
Plan
ImplementReview
Improve
Purpose Encourages counties and programs to consider
organizational characteristics and resources for implementing the Toolkit or other data collection and reporting system
Helps determine if obstacles prevent successful implementation
Received feedback from EAG
Feasibility Checklist Domains and Example Items
Leadership
Staff – Resources
Staff – Time
Training
Technical Requirements
County administration/Leadership prioritizes and values data collection for evaluation and quality improvement purposes.
Feasibility Checklist Domains and Example Items
Leadership
Staff – Resources
Staff – Time
Training
Technical Requirements
We have adequate staffing to oversee/manage the assessment completion process.
Feasibility Checklist Domains and Example Items
Leadership
Staff – Resources
Staff – Time
Training
Technical Requirements
Staff have time to complete a 10‐20 minute intake assessment for each client.
Feasibility Checklist Domains and Example Items
Leadership
Staff – Resources
Staff – Time
Training
Technical Requirements
Our staff members are already trained on/or have time to review training materials about the importance of data collection, quality, and completeness.
Feasibility Checklist Domains and Example Items
Leadership
Staff – Resources
Staff – Time
Training
Technical Requirements
Our staff members have routine access to data analysis and reporting software (e.g., Excel).
Scores range from 1 (strongly disagree) to 5 (strongly agree)
Instructions
Consider current and likely future situations (within one year)
Calculation of domain scores (higher scores – greater feasibility)
5 different domains (e.g.,
leadership)
Completed in Excel or paper form
Scoring and InterpretationScores: Domain scores and overall feasibility score
Decision making and planning: Scores help identify specific areas needing improvement.
Interpretation: Feasibility of implementing a system based on your program's or county’s existing situation (current score) or future situation (future score).
Data Use Agreement The assessments in this toolkit capture personally identifiable protected health information (PHI).
All appropriate security measures must be taken when storing, managing, and sharing data between organizations.
Benefits of Toolkit:
Includes DUA Templates to facilitate the sharing of data between organizations in order to carry out larger evaluations
This DUA Template is recommended for counties or programs that do not have an existing DUA form. Please adhere to local protocols for establishing DUAs.
Toolkit Sections
Assessment Measures Integration into
Existing Data Collection Systems
Data Entry and Reporting
Preparation Promotional Materials
Promotional Materials
Toolkit announcements flyers
Email templates
Communicating importance of assessments to clients
Questions
What are your overall impressions? Would you use certain measures/components from the Toolkit or would you be
using it as a whole? Would you incorporate Toolkit components in your own EHR? What other things would you like to see in the Toolkit (e.g., more validation) that
are not included? What toolkits for behavioral health evaluation have you used or seen in the
past? If you have used or seen a toolkit before, what did you like about it? If you have
seen a toolkit before, what did you not like about it? What challenges did you experience?
Do you have any recommendations for how to maximize the adoptability of a toolkit? What are some barriers to using a toolkit or challenges to adoptability?
How do you see yourself using such a toolkit? What else would you like to see in the Toolkit?
Breakout Group Instructions
Each table is assigned a facilitator responsible for taking notes
Spend first 5-10 minutes going over the questions on your handouts
After thinking about your responses, discuss your responses amongst your table.
Discussion Questions
How can we encourage people to use Toolkit?
What may be some challenges to implementing the Toolkit?
What do you think the next steps are?
Next Steps
Synthesize information from this meeting
Revise Toolkit based on feedback
Submit the final draft of the Toolkit
Contact Information
Andrew Sarkin, Ph.D.Director of Evaluation Research
UCSD Health Services Research Center