Improvement Research Carried Out Through Networked Improvement
Targets & Improvement Activities State Improvement Planning Carol Massanari, MPRRC Western Regional...
-
date post
19-Dec-2015 -
Category
Documents
-
view
214 -
download
0
Transcript of Targets & Improvement Activities State Improvement Planning Carol Massanari, MPRRC Western Regional...
Targets & Improvement Activities
State Improvement Planning
Carol Massanari, MPRRC
Western Regional Resource Center APR Clinic 2010 • November 1-3, 2010 • San Francisco,
California
It’s been six years already?
Old age arrives suddenly, as does the snow. One morning, on awakening, one realizes that everything is white.
Western Regional Resource Center APR Clinic 2010
A New Opportunity
Western Regional Resource Center APR Clinic 2010
• Using prior data to look at trends over time.
• Reviewing and evaluating what we’ve been doing.
• Shaping the form that is emerging from the chaos.
State Improvement Planning
• SPP/APR as a management tool
• Evaluating to document and report results
Using the SPP/APR as a Management Tool
Jeanna Mullins
Jim Leinen, WRRC
Western Regional Resource Center APR Clinic 2010 • November 1-3, 2010 • San Francisco,
California
Development
• Started with information from the November 2009 WRRC Think Tank on “Managing Improvement”
• Field tested with KY Part B staff
• Gathered input at Spring TA&D Conference and from RRC Program staff
Design
• TA Tool for TA providers at the national, regional, State or local levels
• Guide conversations on SPP process and management structures
• Serve as an informal formative evaluation
• Assist in identifying potential areas for improvement
Attributes of Promising PracticesTopical Areas• Leadership• Staff/Workforce Assignments• Strategic Planning• Technical Assistance and Improvement Activities• Measurement, Analysis and Knowledge Management• Customer Focus/Results
Western Regional Resource Center APR Clinic 2010
RubricRating of Attributes
•Fully Operational•Under Development•Under Consideration•Not Considered at This Time•Not Applicable
Western Regional Resource Center APR Clinic 2010
Needs Identification
• Strengths • Potential opportunities for change• Comparison to federal, State, regional or local priorities for targeting improvement• Resources available to support improvement
Western Regional Resource Center APR Clinic 2010
Using the SPP/APR as a Management Tool Activity:
• Select one of the state systems’ Attributes:
• Leadership, Staff/Work Force Assignments• Strategic Planning• Technical Assistance and Improvement Activities• Measurement• Analysis & Knowledge Management• Customer Focus/Results
• Answer the questions related to your selected Attribute
Western Regional Resource Center APR Clinic 2010
Activity #1 Team Report Out:
• What attribute did you select and why?
• Share1-2 highlights about what you discovered while applying this tool
• Any comments about the tool?
Western Regional Resource Center APR Clinic 2010
Guiding Evaluation and Improvement Planning
Logic Models & Evaluation PlansJeanna Mullins
Jim Leinen, WRRC
Western Regional Resource Center APR Clinic 2010 • November 1-3, 2010 • San Francisco,
California
Logic Model Elements Inputs (resources to be used)
Processes (activities and strategies to be used)
Outputs (immediate, countable results produced by activities), and
Outcomes (benefits and changes that are realized)
Logic Model Elements Inputs (resources to be used)
Processes (activities and strategies to be used)
Outputs (immediate, countable results produced by activities), and
Outcomes (benefits and changes that are realized)
A New Resource
EVALUATING SPP/APR IMPROVEMENT ACTIVITIES
Developed by members of the Systems and Improvement Planning Priority Team (NECTAC and RRCP)
December 2009
Evaluating SPP/APR Improvement Activities
Provides:
1. information about evaluation, improvement planning and strategic systems thinking
2. guidance on selecting an appropriate design for evaluating different types of improvement activities
3. additional resources and tools that support SPP/APR implementation and evaluation
Western Regional Resource Center APR Clinic 2010
Foundation: Systems Thinking• Systems are made up of interrelated,
interconnected components
• Systems change involves changing the capacity, interrelationships, and interdependencies among parts, levels and stakeholders
• Desired changes in one part or level of the system must be accompanied by changes in other parts or levels
Western Regional Resource Center APR Clinic 2010
Foundation: Logic Models
Inputs Process Outputs Outcomes
• ‘theory of change’ model to show the process by which change is expected to occur
• illustrated through a visual depiction or logic model
• shows how the inputs (resources) and processes (activities) connect to outputs (products or units produced, such as number of staff trained) and outcomes (intended results)
What is a Logic Model?
“A systematic and visual way to present and share your understanding of the relationships among the resources you have to operate your program, the activities you plan to do, and the changes or results you hope to achieve.”
WK Kellogg Foundation (2002)
Why Use a Logic Model?*
• Serves as a clear roadmap for the work
• Provides a framework for planning, implementation and evaluation
• Graphically illustrates key elements of an improvement activity and how they are interrelated
*Source: Evaluation of Improvement Activities: Considerations and Strategies. George Liacopoulos, Ph.D. Mid-South Regional Resource Center’s Systems Improvement Forum, Baltimore, MD. April, 2010.
Benefits of Logic Models*
• Ensures stakeholders understand the purpose of the activities, the resources needed, the activities to be conducted and their capacity to effect change
• Tracks what has worked and what has not worked well so that success can be replicated
• Identifies and prioritizes questions to be asked in an evaluation
*Source: Evaluation of Improvement Activities: Considerations and Strategies. George Liacopoulos, Ph.D. Mid-South Regional Resource Center’s Systems Improvement Forum, Baltimore, MD. April, 2010.
Foundations: SPP as long term plan for systems change• EI/ECSE are systems of complex, interrelated
components with goal of achieving outcomes for children and families
• Changes to EI/ECSE systems require a combination of improvement activities that are interconnected and support changes to infrastructure that work together to achieve the desired results
Selecting and Reviewing a Set of Improvement Activitiese.g. • Was a root cause analysis conducted to drive the activities?• Was a logical linkage established between the root cause(s)
and proposed activity outcomes? • Have specific action steps been developed (e.g., task,
activity, resources, timelines)? • Have personnel been identified who will develop, implement,
monitor and evaluate the improvement activity? • Have methods to collect the outcome data been identified? • Has a data collection timeline been developed?
(And more … see full paper)
Steps for Evaluating Individual Improvement Activities1. What is the goal or purpose?
2. What methods will you use for data collection?
3. What are your timelines for collecting, analyzing and reporting the data?
4. What is your baseline data that you will monitor for improvement?
5. How will the data be analyzed?
6. How will the evaluation results be used?
7. Who will be responsible for the various aspects of the evaluation?
Scenario: Child Outcomes DataSituation: • Considerable amount of missing data on C3/B7
• Local programs are unclear about when the data should be collected and reported, etc.
• Written policies and procedures are unclear, e.g. roles and responsibilities of Part C v. 619
• Not all local staff have been trained
State Improvement Activities to address missing data issue:
1. Collaborate across State Part C and Section 619 agencies to clarify roles and responsibilities
2. Revise written policies and procedures to clarify e.g. roles/responsibilities
3. Provide additional training on the child outcomes data collection and reporting process
Scenario: Child Outcomes Data
Short term goals of the 3 activities:• To clarify roles and responsibilities of Part C and Section 619
school district staff in the collection and reporting
• To improve the written policies and procedures so that local program staff find them clear and comprehensive
• • To increase local provider/program understanding of the child
outcomes data collection and reporting process
Long term goal of these 3 activities:
To increase statewide reporting of child outcomes data (and decrease the missing data in the child outcomes indicator)
Scenario: Child Outcomes Data
Logic Model for Scenario: Child Outcomes Data
Existing policies and procedures
Staff time
Training protocol
Trainer expertise
State data system
Meetings, communication, collaboration between Part C and 619 staff
Work on revising and clarifying policies and procedures
Survey providers on roles and responsibilities
Revise training to include updated policies and procedures
Provide training
Provide follow-up TA
Revised policies and procedures
Revised training protocol and materials
75% of Part C and 619 staff received training
90% of local staff report increased knowledge of roles and responsibilities related to collecting and reporting child outcomes data
Increase statewide reporting of child outcomes data (decreased missing data) e.g. 90% of all children exiting the program in year with at least 6 mo service are included in outcomes data
Improved reliability and validity of APR data for Indicator C3/B7
Inputs Activities Outputs Outcomes
Contact Information
Western Regional Resource Center APR Clinic 2010
Christina Kasprzak, NECTAC/[email protected]
Jeanna Mullins, [email protected]
Key ConsiderationsConsidering it may not be possible to evaluate all
improvement activities, you could develop criteria to determine which activities you would like to evaluate:
Where is the state investing most resources? “More bang for your buck” – What improvement
activities can be clustered and/or influence multiple Indicators?
Improvement activities associated with indicators for which no progress is being evidenced
Indicators for which you have no or little data Extent of implementation (e.g., statewide or more
targeted) Degree of confidence in the improvement activity
Western Regional Resource Center APR Clinic 2010
Evaluation Plan/Logic Model Team Activity Instructions:
• Select an improvement activity from your SPP
• Explain reasons for selecting this improvement activity
• Draft an evaluation plan or an evaluation logic model for this activity
Western Regional Resource Center APR Clinic 2010