Wraparound Fidelity Monitoring in the Massachusetts CBHI Summary of 2011 Results August 12, 2011
description
Transcript of Wraparound Fidelity Monitoring in the Massachusetts CBHI Summary of 2011 Results August 12, 2011
Wraparound Fidelity Wraparound Fidelity Monitoring in the Monitoring in the
Massachusetts CBHIMassachusetts CBHI
Summary of 2011 ResultsSummary of 2011 Results
August 12, 2011
Eric Bruns and April SatherUniversity of WashingtonWraparound Evaluation and Research [email protected] / [email protected] thanks to Andrea Gewirtz and Melissa King
Purpose for this MeetingPurpose for this Meeting
1.1. Very quick review of wraparound Very quick review of wraparound fidelity assessment and researchfidelity assessment and research
2.2. Review data from Massachusetts Review data from Massachusetts fidelity monitoringfidelity monitoring
3.3. Discuss what has been found and what Discuss what has been found and what it meansit means
Wraparound Principles
1. Family voice and choice2. Team-based3. Natural supports4. Collaboration5. Community-based6. Culturally competent7. Individualized8. Strengths based9. Persistence10.Outcome-based
Walker, Bruns, Adams, Miles, Osher et al., 2004
Wraparound Competence:What do we want to measure?
Implementing the practice model:
The Four Phases of Wraparound
Time
Engagement and Support
Team Preparation
Initial Plan Development
Implementation
Transition
Phase1A
Phase1B
Phase2
Phase3
Phase4
Wraparound Competence
Phase 1 : Engagement and Team Preparation
• Care Coordinator & Family Partner meets with the family to discuss the wraparound process and listen to the family’s story.
• Discuss concerns, needs, hopes, dreams, and strengths.
• Listen to the family’s vision for the future.
• Assess for safety and make a provisional safety plan if needed
• Identify people who care about the family as well as people the family have found helpful for each family member.
• Reach agreement about who will come to a meeting to develop a plan and where we should have that meeting.
Phase 1 A and BPhase 1 A and B
Phase 2: Initial Plan Development
• Conduct first Care Planning Team (CPT) meeting with people who are providing services to the family as well as people who are connected to the family in a supportive role.
• The team will:
– Review the family vision
– Develop a Mission Statement about what the team will be working on together
– Review the family’s needs
– Come up with several different ways to meet those needs that match up with the family’s strengths
• Different team members will take on different tasks that have been agreed to.
Phase 2Phase 2
Phase 3: Plan Implementation
• Based on the CPT meetings, the team has created a written plan of care.
• Action steps have been created, team members are committed to do the work, and our team comes together regularly.
• When the team meets, it:
– Reviews Accomplishments (what has been done and what’s been going well);
– Assesses whether the plan has been working to achieve the family’s goals;
– Adjusts things that aren’t working within the plan;
– Assigns new tasks to team members.
Phase 3Phase 3
Phase 4: Transition
• There is a point when the team will no longer need to meet regularly.
• Transition out of Wraparound may involve a final meeting of the whole team, a small celebration, or simply the family deciding they are ready to move on.
• The family we will get a record of what work was completed as well as list of what was accomplished.
• The team will also make a plan for the future, including who the family can call on if they need help or if they need to re-convene their team.
• Sometimes transition steps include the family and their supports practicing responses to crises or problems that may arise
Phase 4Phase 4
Wraparound Fidelity Index, v.4
• Items on the principles and core activities, organized by the 4 phases of wraparound
– Engagement: Did you select the people who would be on your youth and family team?
• Principle = Team based
– Planning: Does the plan include strategies for helping your child get involved with activities in the community?
• Principle = Community based
– Implementation: Does the team evaluate progress toward the goals of the plan at every team meeting?
• Principle = Outcome based
– Transition: Will some members of your team be there to support you when formal wraparound is complete?
• Principle = Persistence
Team Observation Measure
• Consists of 20 items, with two items dedicated to each of the 10 principles of wraparound.
• Each item consists of 3-5 indicators of high-quality wraparound practice as expressed during a care planning team meeting.
• Internal consistency very good
• Inter-rater reliability found to be adequate (Average 79% agreement for all indicators)
• Correlates well with WFI scores
– In previous studies and evaluations…
Document Review Measure
• Mass version consists of 26 items
• Each wraparound principle linked to several items
• Scale = 0-3, with criteria for each point on the scale
• Source material = documentation (electronic or paper) related to youth’s wraparound process
– Strengths, needs, culture discovery documentation
– Wraparound plan of care
– Safety plan
– Transition plan
– Progress notes
Measures of the WFAS seem to tap into the same constructs
50%
60%
70%
80%
90%
100%
WFI Combined 84% 77% 79% 79% 69% 72% 72% 81%
Team Observation 93% 84% 83% 78% 67% 56% 63% 78%
Site 1 Site 3 Site 5 Site 6 Site 7 Site 8 Site 9 Site 10
Mean WFI-4 and TOM total scores by site in one state’s QI project
WFAS Measures seem to be related to what we would expect
• Families who experience better outcomes have staff who score higher on fidelity tools (Bruns, Rast et al., 2006; Effland, McIntyre, & Walton, 2010)
• Wraparound initiatives with positive fidelity assessments demonstrate more positive outcomes (Bruns, Leverentz-Brady, & Suter, 2008)
• Wraparound initiatives with better support to implementation have higher fidelity scores
What does it take to get high fidelity scores?
• Training and coaching: Found to be associated with gains in fidelity and higher fidelity
• Community, Fiscal, and Service system support: Better developed supports = fidelity scores
• Sticking with it: Fidelity scores tend to increase over time and with persistence of effort
– But… they can also backslide…
Caregiver WFI Fidelity over time in NV
72%86%
73%64%
0%10%20%30%40%50%60%70%80%90%
100%
Pre Training(2001)
Training Only(2002)
Training andCoaching (2004)
After going toscale (2007-09)
Ave
rag
e W
FI
Fid
elit
y S
core
Bruns, Rast, Walker, Peterson, & Bosworth (2006). American Journal of Community Psychology.
What can we do with the data?
• Measurement as communication
– What are we attempting to do for families?
– How well are we stacking up against benchmarks? What is holding us back?
– How well are we doing over time?
– What is associated with positive outcomes in Massachusetts?
Measurement for quality improvement
• Examine data at the program level to discuss what is and is not happening consistently
• What can be reinforced in individual and group supervision?
• What are the team, program, and resource barriers?
• In what areas do we need additional training?
– At the community or system level by collaborative groups, along with reports of team, program, and resource barriers reported up from CSAs or programs
What kind of data get used?
• What kind of data?
– Item scores – Ns who endorse each response option (Yes, No, Somewhat)
– Members on teams
– Specific plan of care elements
– Qualitative feedback on open-ended items
Part 2: Results from Part 2: Results from MassachusettsMassachusetts
Scores on the WFI, TOM, and Mass-DRM
Total scores on the WFAS instruments
0
20
40
60
80
100
Percent of total fidelity
Massachusetts 2010 78 83 61
Massachusetts 2011 77 85 74
National mean 74 77 -
WFI TOM DRM
Correlations between WFI, TOM, and DRM across Massachusetts CSAs
• WFI – TOM = .121
• WFI – DRM = -.473 (p<.001)
• DRM – TOM = .012
• WFI – TOM = -.256
• WFI – DRM = .326 (p<.05)
• DRM – TOM =.168
2
0
1
0
2
0
1
1
WFI-DRM association at a site level
72.00 76.00 80.00 84.00
WFI
40.00
50.00
60.00
70.00
80.00
DR
M
2010 – DRM / WFI - 2011
WFI-DRM association at a site level
2011 – DRM/TOM
Wraparound Fidelity Index
Massachusetts Overall
WFI-4 Total Fidelity ScoresAll CSAs
Total Phase Principle
Mean Over-all
Eng Pln Imp Tr FVC TB NS Col CB CC Indiv
SB Per
OB
Mass 2010
78 86 82 79 64 88 84 53 89 74 95 70 83 83 63
Mass 2011
77 85 82 77 64 85 82 52 89 74 93 73 79 78 65
Nat Mean (CG only)
74 75 74 79 66 82 71 61 84 69 90 64 82 79 64
2010 N = 637 / 2011 N = 587
WFI Fidelity by PhaseAll CSAs 2010 N=637 / 2011 N = 587
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
2010 78% 86% 82% 79% 64%
2011 77% 85% 82% 77% 64%
Nat Mean 74% 75% 74% 79% 66%
Total Eng Plan Impl Trans
WFI Fidelity by PrincipleAll CSAs 2010 N=637 / 2011 N = 587
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
2010 78% 88% 84% 53% 89% 74% 95% 70% 83% 83% 63%
2011 77% 85% 82% 52% 89% 74% 93% 73% 79% 78% 65%
Nat Mean 74% 82% 71% 61% 84% 69% 90% 64% 82% 79% 64%
Total FVC TB NS COL CB CC IND SB PER OB
Item Scores: Engagement All CSAs
ITEMS 2010 2011 NAT MEAN
CG1.1 - When you first met ICC, were you given time to talk about strengths *and* Did this process help you appreciate? 1.78 1.74 1.65
CG1.2 - Before your 1st team meeting, did your ICC fully explain the WA process and the choices you could make? 1.78 1.82 1.68
CG1.3 - At beginning of wrap process, did you have a chance to tell WF what things have worked in the past? 1.83 1.78 1.75
CG1.4 - Did you select the people who would be on your WA team? 1.41 1.34 0.86
CG1.5 - Is it difficult to get team members to attend team meetings when they are needed? 1.66 1.65 1.57
CG1.6 - Before your 1st wrap team meeting, did you go through a process of i.d.ing what leads to crises for child and family?
1.81 1.90 1.52
Item Scores: Planning All CSAs
ITEMS 2010 2011NAT
MEANS
CG2.1 - Did you and your team create a written plan that describes how the team will meet your child's needs? *and* Do you have a copy?
1.83 1.87 1.64
CG2.2 - Did the team develop any kind of written statement about what it is working on with your child and family? *and* Can you describe what your team mission says? 1.76 1.76 1.56
CG2.3 - Does your wrap plan include mostly professional services? .99 1.10 0.61
CG2.4 - Are the supports and services in your WA plan connected to the strengths and abilities of your child and family? 1.80 1.72 1.74
CG2.5 - Does the wrap plan include strategies for helping your child get involved w/ activities in his/her community? 1.31 1.26 1.24
CG2.6 - Are there members of your wrap team who do not have a role in implementing your plan? 1.73 1.73 1.67
Item Scores: Planning All CSAs
ITEMS2010 2011 NAT
MEAN
CG2.7 - Does your team brainstorm many strategies to address your family's needs before selecting one? 1.79 1.79 1.73
CG2.8 - Is there a crisis plan? *and* does this plan specify how to prevent crisis? 1.48 1.57 1.43
CG2.9 - Do you feel confident that, in crisis your team can keep your child in the community? 1.57 1.58 1.5
CG2.10 - Do you feel like other people on your team have higher priority than you in designing your wrap plan?
1.83 1.73 1.53
CG2.11 - During planning process, did team make enough time to understand values? *and* Is your wrap plan in tune w/ family's values?
1.89 1.84 1.73
Item Scores: Implementation All CSAs
ITEMS 2010 2011 NAT MEAN
CG3.1 - Are important decisions made about your child or family when you are not there? 1.90 1.86 1.64
CG3.2 - When your wrap team has a good idea for support, can they find resources or make it happen? 1.58 1.54 1.7
CG3.3 - Does your wrap team get your child involved w/ activities they like and do well? 1.05 0.95 1.2
CG3.4 - Does the team find ways to increase the support you get from friends & family? 1.09 1.13 1.22
CG3.5 - Do the members of your team hold each another responsible for doing their part? 1.73 1.73 1.7
CG3.6 - Is there a friend or advocate of your child or family who actively participates in wrap team? .68 .66 0.95
CG3.7 - Does your team come up w/ new ideas? *and* Does your team come w/ ideas when something's not working?
1.75 1.70 1.74
Item Scores: Implementation All CSAs
ITEMS 2010 2011 NAT MEANS
CG3.8 - Are the services and supports in your wrap difficult for you family to access? 1.61 1.66 1.54
CG3.9 - Does the team assign specific tasks to all team members at end of mtng? *and* Does team review team member's follow-through at next mtng?
1.67 1.63 1.59
CG3.10 - Do members of your team always use language you can understand? 1.96 1.94 1.93
CG3.11 - Does your team create a positive atmosphere around successes and accomplishments at each team meeting? 1.88 1.91 1.86
CG3.12 - Does your team go out of its way to make sure all members present ideas and participate in decisions? 1.82 1.84 1.67
CG3.13 - Do you think your wrap process could be discontinued before you're ready? 1.48 1.32 1.35
CG3.14 - Do all the members of your team demonstrate respect for you and your family? 1.96 1.91 1.88
CG3.15 - Does your child have the opportunity to communicate their own ideas when it comes to decisions? 1.49 1.39 1.71
Item Scores: Transition All CSAs
ITEMS 2010 2011 NAT MEAN
CG4.1 - Has your team discussed a plan for how wrap will end *and* Does your team have a plan for when? .46 0.70 0.68
CG4.2 - Has the wrap process helped your child develop friendships w/ other youth .94 .92 1.2
CG4.3 - Has the wrap process helped your child to solve their own problems? 1.09 1.02 1.3
CG4.4 - Has your team helped you and your child prepare for major transitions? 1.50 1.64 1.35
CG4.5 - After formal wrap ends, do you think the process will be able to be 're-started' if you need it? 1.82 1.80 1.61
CG4.6 - Has the wrap process helped your family to develop or strengthen relationships that will support you when WA is finished?
1.45 1.42 1.49
CG4.7 - Do you feel like you and your family will be able to succeed on its own? 1.33 1.41 1.22
CG4.8 - Will some members of your team be there to support you when formal wrap is finished? 1.60 1.40 1.65
WFI – Summary
• Overall: Significantly higher than national mean
• Overall, no change from 2010
• Relative strengths:
– Engagement, Planning phases
– Family voice and choice, team based, collaborative, Individualized
– Initial engagement, family selecting team, crisis planning, vision/mission statements, balance of formal and informal supports
• Relative weaknesses:
– Natural supports, involvement of friends/advocates, involvement in the community
– Youth voice, youth friendships, youth involvement in community
WFI – Summary of trends from 2010
• Areas of improvement from 2010:
– Crisis identification and plans
– Transition planning• including prep for transitions from wrap and within wrap
• Areas of decline:
– Family voice and choice – selecting team, perception of family influence
– Youth friends, youth voice, youth problem solving
– Perception that wrap could be discontinued before family ready
Team Observation Measure
Mass CBHI overall
TOM Total Fidelity ScoresAll CSAs 2010 N=285 / 2011 N=658
Total PrincipleOverall Mean
FVC TB NS Col CB CC Ind SB Per OB
Mass CBHI Mean 2010
N = 285
83 95 84 43 88 91 92 83 88 89 73
Mass CBHI Mean 2011
N = 658
85 94 85 51 92 91 93 86 90 92 78
National Mean
77 86 72 58 81 84 82 72 79 83 68
TOM Fidelity by PrincipleAll CSAs 2010 N=285 / 2011 N=658
83
95
84
43
88 91 92
8388 89
7377
86
72
58
81 84 82
7279
83
68
85
94
85
51
92 91 9386
90 92
78
0
10
20
30
40
50
60
70
80
90
100
Mean 2010 83 95 84 43 88 91 92 83 88 89 73
National Mean 77 86 72 58 81 84 82 72 79 83 68
Mean 2011 85 94 85 51 92 91 93 86 90 92 78
Total Score
F V & C
TB NS Col CB CC Indiv SB Per OB
TOM Item & Indicator ScoresAll CSAs
TEAM BASED 2010 2011 Nat Mean
Item 1: Team Membership and Attendance 3.10 3.09 2.39a. Parent/caregiver is a team member and present at the meeting. .99 .99 .93b. Youth (over age 9) is a team member and present at the meeting. .61 .61 .75c. Key school or other public stakeholder agency representatives are present. .61 .60 .38
Item 2: Effective Team Process 3.61 3.71 3.4a. Team meeting attendees are oriented to the wraparound process and understand the purpose of the meeting. .88 .91 .86b. The facilitator assists the team to review and prioritize family and youth needs. .95 .95 .89
c. Tasks and strategies are explicitly linked to goals.* .91 .95 .85d. Potential barriers to the nominated strategy or option are discussed and problem-solved. .86 .91 .84
TOM Item & Indicator ScoresAll CSAs
COLLABORATIVE 2010 2011Nat
Mean
Item 3: Facilitator Preparation 3.50 3.68 3.36a. There is a clear agenda or outline for the meeting, which provides an understanding of the overall purpose of the meeting and the major sections of the meeting. .86 .91 .84b. The meeting follows an agenda or outline such that team members know the purpose of their activities at a given time. .84 .91 .84c. The facilitator has prepared needed documents and materials prior to the meeting. .90 .96 .86
d. A plan for the next meeting is presented, including time & date. .89 .91 .87
Item 4: Effective decision making 3.50 3.69 3.14a. Team members demonstrate consistent willingness to compromise or explore further options when there is disagreement. .96 .97 .91b. Team members reach shared agreement after having solicited information from several members or having generated several ideas. .87 .95 .81
c. The plan of care is agreed upon by all present at the meeting. .94 .98 .83d. The facilitator summarizes the content of the meeting at the end of the meeting, including next steps and responsibilities. .76 .82 .87
TOM Item & Indicator ScoresAll CSAs
INDIVIDUALIZED 2010 2011Nat
Mean
Item 5: Creative Brainstorming and Options 3.11 3.16 2.46a. The team considers several different strategies for meeting each need and achieving each goal that is discussed. .83 .87 .78
b. The team considers multiple options for tasks or action steps. .81 .86 .73c. The facilitator leads a robust brainstorming process to develop multiple options to meet priority needs. .67 .69 .40
Item 6: Individualized process 3.53 3.70 3.3a. Planning includes action steps or goals for other family members, not just identified youth. .85 .92 .79b. Facilitator and team members draw from knowledge about the community to generate strategies and action steps based on unique community supports. .85 .91 .78c. Team facilitates the creation of individualized supports or services to meet the unique needs of child and/or family.* .89 .96 .83d. Youth, caregiver, & family members give their opinions about potential services, supports, or strategies; including describing what has or has not worked in the past. .93 .95 .91
TOM Item & Indicator ScoresAll CSAs
NATURAL SUPPORTS 2010 2011Nat
Mean
Item 7: Natural and community supports 1.54 1.61 2.38a. Natural supports for the family are team members and are present. .27 .27 .26b. Team provides multiple opportunities for natural supports to participate in significant areas of discussion. .75 .80 .66c. Community team members and natural supports participate in decision-making. .72 .79 .66d. Community team members and natural supports have a clear role on the team.* .72 .81 .61
Item 8: Natural support plans 1.94 2.47 2.37a. Brainstorming of options and strategies include strategies to be implemented by natural and community supports. .70 .77 .66b. The plan of care represents a balance between formal services and informal supports. .45 .58 .36c. There are flexible resources available to the team to allow for creative services, supports, and strategies. .21 .58 .91
TOM Item & Indicator ScoresAll CSAs
UNCONDITIONAL/PERSISTANCE 2010 2011
Nat Mean
Item 9: Team mission and plans 3.44 3.61 3.23a. The team discusses or has produced a mission/vision statement. .84 .90 .75
b. The team creates or references a plan that guides its work. .91 .96 .87
c. The team has confirmed or is creating a crisis plan. .78 .90 .77d. The team plan contains specific goals that are linked to strategies and action steps. .92 .96 .87
Item 10: Shared Responsibility3.66 3.72 3.42
a. The team explicitly assigns responsibility for action steps that define who will do what, when, and how often.* .87 .91 .81b. There is a clear understanding of who is responsible for action steps and follow up on strategies in the plan. .92 .92 .86c. Providers and agency representatives at the meeting demonstrate that they are working for the family and not there to represent a different agenda or set of interests. .96 .98 .94
TOM Item & Indicator ScoresAll CSAs
CULTURAL COMPETENCE 2010 2011Nat
Mean
Item 11: Facilitation skills 3.55 3.62 2.97a. Facilitator is able to impart understanding about what the wraparound process is, how it will work for this family, and how individual team members will participate. .83 .88 .74b. Facilitator reflects, summarizes, and makes process-oriented comments. .89 .92 .77c. Facilitator is able to manage disagreement & conflict and elicit underlying interests, needs, and motivations of team members. .92 .92 .73d. Talk is well distributed across team members and each team member makes an extended or important contribution. .93 .92 .87
Item 12: Cultural and Linguistic Competence 3.76 3.86 3.56
a. The youth, caregiver, and family members are given time to talk about the family’s values, beliefs, and traditions. .87 .95 .86
b. The team demonstrates a clear and strong sense of respect for the family’s values, beliefs, and traditions. .95 .97 .88
c. Meetings and meeting materials are provided in the language the family is most comfortable with. .97 .98 .98
d. Members of the team use language the family can understand (i.e. no professional jargon/acronyms) .97 .98 .96
TOM Item & Indicator ScoresAll CSAs
OUTCOMES BASED 2010 2011Nat
Mean
Item 13: Outcomes Based Process 2.88 3.06 2.83
a. The team uses objective measurement strategies. .67 .76 .68
b. The team assesses goals/strategies using measures of progress. .72 .77 .62
c. The team revises the plan if progress toward goals is not evident. .84 .88 .87
Item 14: Evaluating Progress and Success 2.99 3.15 2.62
a. The team conducts a systematic review of members’ progress on assigned action steps. .78 .84 .71
b. The facilitator checks in with the team members about their comfort and satisfaction with the team process. .74 .79 .73
c. Objective or verifiable data is used as evidence of success, progress, or lack thereof. .72 .78 .61
TOM Item & Indicator ScoresAll CSAs
VOICE AND CHOICE 2010 2011Nat
Mean
Item 15: Youth and Family Voice 3.89 3.86 3.53a. The team provides extra opportunity for caregivers to speak and offer opinions, especially during decision making. .98 .98 .95b. The team provides extra opportunity for the youth to speak and offer opinions, especially during decision making. .93 .93 .91c. Caregivers, parents, and family members are afforded opportunities to speak in an open-ended way about current and past experiences and/or about hopes for the future. .98 .98 .95d. The youth is invited to speak in an open-ended way about current and past experiences and/or about hopes for the future. .96 .93 .88
Item 16: Youth and Family Choice 3.72 3.69 3.38a. The youth prioritizes life domains, goals, or needs on which he or she would like the team to work. .78 .79 .76b. The caregiver or parent prioritizes life domains goals, or needs on which he or she would like the team to work. .93 .96 .89
c. The family and youth have highest priority in decision making .97 .95 .90
TOM Item & Indicator ScoresAll CSAs
STRENGTH BASED 2010 2011Nat
Mean
Item 17: Focus on Strengths 3.31 3.47 2.89
a. Team members acknowledge or list caregiver/youth strengths. .92 .95 .82b. Team builds an understanding of how youth strengths contribute to the success of team mission or goals. .78 .85 .67c. In designing strategies, team members consider and build on strengths of the youth and family. .82 .89 .72d. Facilitator and team members analyze youth & family member perspectives and stories to identify functional strengths. .78 .87 .69
Item 18: Positive Team Culture 3.7 3.69 3.44a. The team focuses on improvements or accomplishments throughout the meeting. .91 .92 .82b. The facilitator directs a process that prevents blame or excessive focus on or discussion of negative events. .97 .95 .89c. The facilitator encourages team culture by celebrating successes since the last meeting .88 .93 .78
d. There is a sense of openness and trust among team members. .94 .93 .90
TOM Item & Indicator ScoresAll CSAs
COMMUNITY BASED 2010 2011Nat
Mean
Item 19: Community Focus 3.41 3.45 3.16a. The team is actively brainstorming and facilitating community activities for the youth and family.* .82 .85 .71
b. The team prioritizes services that are community-based. .82 .86 .79c. The team prioritizes access to services that are easily accessible to the youth and family. .93 .94 .90
Item 20: Least Restrictive Environment 3.92 3.86 3.59a. The team’s mission and/or identified needs support the youth’s integration into the least restrictive residential and educational environments possible.* .99 .97 .98
b. When residential placements are discussed, team chooses community placements for the child or youth rather than out-of-community placements, wherever possible. .87 .88 .94c. Serious challenges are discussed in terms of finding solutions, not placement in more restrictive residential or educational environments. .95 .94 .94
TOM – Summary
• Overall: Significantly higher than national mean
• Significant improvement from 2010
• Relative strengths:
– Nearly all principles. Particular strengths in:• School/Agency reps present, effective teams and facilitation, facilitator
prep, Creative/individualized services, services for all family members, crisis plans
• Relative weaknesses:
– Youth involvement/presence at meetings
– Natural supports present and involved
– Flex resources available and used when needed
TOM – Summary of trends from 2010
• Areas of improvement from 2010:
– Facilitator prep, team process
– Flexible resources available
– Balance of informal/formal services/supports
– Crisis planning and plans
– Use of objective measures and following up
– Tying strengths to planning
• Areas of decline:
– Youths being asked their priorities
– Youths present
Document Review Measure
Mass CBHI overall
Mass-DRM Principle Scores
Total Mass DRM improvement: 61% to 74%
DRM Item scores: Relative strengths 2010 2011
8 The family is present at all meetings. 2.83 2.914 There is a clearly articulated long range vision of the youth and family. 2.31 2.9
17There is evidence that the family and/or youth are making decisions about the direction and methods of the team. 2.53 2.84
11There is evidence that all members on the CPT are in regular communication, to ensure coordinated efforts on the youth and family's behalf. 2.71 2.74
15Services, supports and tasks are based on the needs of the youth and family, not availability of services or support that are needed. 2.34 2.73
23The RMSP identifies triggers, signs or behaviors that indicate the crisis is beginning and interventions to deescalate crisis situations. 2.62 2.68
2 The family and youth's needs or concerns are identified across life domains. 2.13 2.62
22The risk management/safety plan (RMSP) includes strengths of youth and family and strategies for preventing potential crises. 2.51 2.59
12There is evidence that the wraparound plan, including the RMSP, is updated at each team meeting or as needed. 2.25 2.56
18 There is documentation that progress toward goals and tasks has been monitored. 2.29 2.56
5Detailed examples of family and youth strengths, assets, resources, culture, values, and beliefs are included to support planning for areas of priority need. 2.07 2.52
25The RMSP identifies specific steps to be taken if the crisis occurs, people responsible for each step, including the youth's local ESP/MCI. 2.43 2.39
16 There is evidence that the team brainstorms more than 6 options to address goals. 1.99 2.33
DRM item scores: Needs for improvement
2010 2011
10
There is evidence that natural supports are actively involved in the planning and implementation process or there are ongoing and persistent efforts to identify and engage natural supports. 0.8 0.97
13
There is documentation that an attempt has been made to collaborate with the primary care physician (PCP) or primary care clinician (PCC).
1.72 1.21
20The wraparound plan includes opportunities for the youth to engage in community activities that he or she likes and does well. 1.05 1.34
21
There is documentation of transition and sustainability planning that identifies needs, services and support that will continue after formal wraparound ends or when the youth transitions to the adult service system. 1.16 1.4
26 The RMSP is proactively shared with the MCI team. 1.43 1.97
24The RMSP is updated as needed; always when a hub-dependent support joins the youth and family team. 1.94 2.09
6There is evidence that a strengths, needs, and culture discovery process was completed within 10 days of enrollment of the youth. 1.64 2.11
7The initial wraparound plan was developed within 35 days of enrollment with the youth and family. 2.47 2.16
14The plan includes strengths-based goals that are measurable with specific measurement strategies. 1.75 2.17
DRM Summary
• Greater correlation with WFI/TOM from 2010
• Large improvement from 2010, particularly:
– Vision statements
– Updated plans
– Needs ID’d across domains
– Examples of strengths, beliefs, culture
– SNCD within 10 days
• Declines in:
– Wrap plans w/in 35 days
– Collaboration with PCPs
DRM Psychometrics
• Total Alpha = .85
• Principal Components Analysis (w/Varimax Rotation) found Interpretable 8-Factor Solution explaining 60.6% of variance
• Correlation with WFI = .326 (p<.05)
– Certain items negatively correlate with WFI and/or have negligible correlation with DRM total
6 “Bad” DRM itemsDo not correlate with the WFI and have low item-total correlations
There is evidence that a CANS and full assessment were completed with the youth and family within 10 days of enrollment.The family is present at all meetings.
There is evidence that natural supports are actively involved in the planning and implementation process or there are ongoing and persistent efforts to identify and engage natural supports.
There is evidence that all members of the CPT are in regular communication, to ensure coordinated efforts on the youth and family's behalf.
There is documentation that an attempt has been made to collaborate with the primary care physician (PCP) or primary care clinician (PCC).The RMSP is proactively shared with the MCI team.
11 “Good” DRM ItemsSignificant correlation with WFI AND high item totals
Documentation is strengths based and identifies and prioritizes the needs for the youth and family.Detailed examples of family and youth strengths, assets, culture, values and resources are included to support planning for areas of priority need.The initial ICP was developed within 28 days of enrollment with the youth and family.The ICP specifies the goals that will help the family reach their long-range vision.There is evidence that the ICP is updated at each team meeting and as needed.Services, supports and tasks are based on the needs of the youth and family and not on the availability of services or support that are needed.There is evidence that the CPT brainstorms more than six options to address goals.There is documentation that progress toward goals and tasks has been monitored.There is evidence that the youth and family are making progress toward their goals.The RMSP identifies triggers, signs or behaviors that indicate the crisis is beginning and notes interventions to deescalate crisis situations.The RMSP identifies specific steps to be taken if the crisis occurs, people responsible for each step, including the youth's local ESP/MCI.
DRM Principal Components Analysis
• Goals and Strategies (5 items)
• Strengths and Needs (4 items)
• Progress monitoring and timely updates (4 items)
• Crisis and safety plans present and of quality (3 items)
• Natural supports and community based services evident (3 items)
• Families as primary decision makers (3 items)
• Meeting mandated timelines (2 items: CANS and ICP)
• RMSP is shared with MCI team (1 item)
DRM Principal Components Analysis
ItemStr and Needs
Goals and Strategies
Progress monitoring and timely updates
Crisis and safety
Natural supports and community based
Family de cision making
MCI sharing
Meet mandated timelines
There is evidence that a CANS and full assessment were completed with the youth and family within 10 days of enrollment. 0.098 -0.035 0.100 -0.075 0.055 0.040 0.069 0.830The family and youth's needs or concerns are identified across life domains. 0.840 0.066 0.042 -0.011 0.086 0.072 0.030 0.128Documentation is strengths based and identifies and prioritizes the needs for the youth and family. 0.745 0.125 0.135 0.156 -0.057 0.104 -0.080 -0.060There is a clearly articulated long range vision of the youth and family. 0.245 0.338 0.067 0.183 -0.237 0.435 -0.187 -0.002Detailed examples of family and youth strengths, assets, culture, values and resources are included to support planning for areas of priority need. 0.708 0.187 0.048 0.176 0.137 0.003 0.089 -0.051There is evidence that a strengths, needs, and culture discovery process was completed before the first CPT meeting. 0.614 -0.130 0.137 -0.124 0.176 0.042 0.195 0.359The initial ICP was developed within 28 days of enrollment with the youth and family. 0.026 0.214 0.110 0.208 -0.048 -0.076 -0.073 0.677The family is present at all meetings. 0.079 0.027 0.133 0.024 -0.004 0.776 0.145 0.022The ICP specifies the goals that will help the family reach their long-range vision. 0.169 0.675 0.386 -0.019 -0.051 0.098 0.006 0.020There is evidence that natural supports are actively involved in the planning and implementation process or there are ongoing and persistent efforts to identify and engage natural supports. 0.125 -0.027 0.064 0.085 0.601 0.145 -0.271 0.155There is evidence that all members of the CPT are in regular communication, to ensure coordinated efforts on the youth and family's behalf. 0.088 0.478 0.074 0.141 0.165 0.301 -0.078 -0.067There is evidence that the ICP is updated at each team meeting and as needed. 0.091 0.209 0.680 0.044 0.064 0.232 0.058 0.161There is documentation that an attempt has been made to collaborate with the primary care physician (PCP) or primary care clinician (PCC). 0.027 0.242 0.011 -0.155 -0.037 0.088 0.703 0.162The ICP includes strengths-based goals that are measurable with specific measurement strategies. 0.049 0.676 0.020 0.050 0.101 -0.035 0.319 0.085Services, supports and tasks are based on the needs of the youth and family and not on the availability of services or support that are needed. 0.253 0.600 0.403 0.018 -0.031 0.013 0.103 0.067There is evidence that the CPT brainstorms more than six options to address goals. -0.066 0.566 0.037 0.083 0.244 0.233 0.046 0.095There is evidence that the family and/or youth are making decisions about the direction and methods of the team. 0.040 0.226 0.128 0.108 0.172 0.754 0.063 -0.045There is documentation that progress toward goals and tasks has been monitored. 0.047 0.161 0.739 0.141 0.219 0.198 -0.003 0.058There is evidence that the youth and family are making progress toward their goals. 0.106 0.143 0.648 0.106 0.405 0.101 0.017 0.079The ICP includes opportunities for the youth to engage in community activities that he or she likes and does well. 0.080 0.183 0.174 -0.028 0.705 0.001 -0.051 -0.073There is documentation of transition and sustainability planning that identifies needs, services and supports that will continue after formal wraparound ends or when the youth transitions to the adult service system. 0.066 0.114 0.078 0.190 0.667 -0.014 0.390 -0.032The risk management/safety plan (RMSP) includes strengths of youth and family and strategies for preventing potential crises. 0.145 -0.019 0.204 0.777 -0.008 -0.011 0.002 0.044The RMSP identifies triggers, signs or behaviors that indicate the crisis is beginning and notes interventions to deescalate crisis situations. 0.105 0.070 0.160 0.800 0.048 0.026 0.049 0.015The RMSP is updated as needed, and reviewed at every CPT meeting. 0.119 0.029 0.538 0.222 -0.145 -0.213 0.316 0.029The RMSP identifies specific steps to be taken if the crisis occurs, people responsible for each step, including the youth's local ESP/MCI. -0.051 0.169 -0.077 0.689 0.189 0.291 -0.018 0.038The RMSP is proactively shared with the MCI team. 0.077 -0.001 0.147 0.137 -0.014 0.091 0.745 -0.109
Summary of Findings
Summary of findings
• Continued strong fidelity results, especially for a large statewide initiative
– WFI 3 points higher than national mean
• WFI scores stay stable, TOM shows small improvement, DRM show large improvement
• Big improvements noted for:
– Crisis planning
– Transition planning
– Team process / Facilitator prep
– Availability of flexible resources (but is this an artifact of TOM scoring clarification?)
– Documentation of strengths, needs, goals, crisis plans
Summary of findings
• Areas of strength continue to be:
– Managing the details of the wrap process BUT ALSO
– Family voice and choice
– Collaborative teaming
– Individualized plans with balance of formal svcs and informal supports
Looking deeper: Relative weaknesses
• Certain bigger ideas of wraparound not yet happening consistently
– Natural supports and natural support plans
– Connection to community activities and informal supports
– Full youth engagement
– Youth and family self-reliance and self-efficacy
– Youths present at meetings
• Better on some things, but still room for more improvement:
– Robust brainstorming
– Objective goal setting and monitoring of progress
– Transition planning
• Structural barriers:
– Flexible funds
– Local community collaboratives
– Time to develop more creative plans linked to community supports and natural supports
– Consistent understanding of how FPs and ICCs work together
Possible strategies: System level
• Fiscal supports
– Use of case rates
– Availability of flexible funds
– Methods to ensure less emphasis on productivity vs fidelity and outcomes?
• Community supports, collaboration, transition planning:
– Encouragement to taking a community-level system of care approach
– Hire resource development specialist(s)
– Training and support to community collaborative teams
– Continued social marketing of CBHI and cross agency training
– Collecting info on and reviewing barriers faced by teams and CSAs
• Youth engagement
– Use of youth peer to peer support partners?
Possible strategies: Organizational level
• Natural supports:
– Specific strategies (family finding?), training, time, supervision emphasis
– Role for family partner?
• Transition planning
– Clear understanding of interactions between medical necessity determination and Unconditional Care
• Ensure clear expectations are delivered to families about transition
• Outcomes based:
– Greater attention to consistent tracking of goal attainment and meeting needs
• Use a dashboard approach?
• Staff turnover/attrition:
– Training and guidance on staff recruitment and selection; compensation, caseloads, expectations (e.g., taking vacation to attend trainings)
Possible strategies: Practice level
• Invest in specific strategies for engaging youth
– E.g., Achieve My Plan! (AMP) from Portland State University
• Individualized care
– More attention to developing individualized, creative plans in supervision, QI
• Emphasis on training and supervision on:
– Transition planning
– Individualized care plans
– Natural support action steps
– Teaching self-efficacy
Measurement issues
• WFI: Start to interview youths
– As a way of modeling a new era of engagement of Youth
– To find out what is going on from their perspective
• How to use the TOM?
– Ceiling effect now evident
– Doesn’t correlate with WFI in Mass• Bias among raters/supervisors
– Recommend use of independent TOM raters for a subsample of selected teams
Measurement issues
• Continue to use the DRM?
– Now clearly demonstrating utility as a fidelity tool
• Could be evidence for the benefit of its continued use • OR…. Could mean don’t need to use it – duplicative
– Perhaps revise based on data and disseminate for use by supervisors / CSA QA?
Measurement issues
• Analytic questions: What influences fidelity?
– Established vs. new sites
– Caseload sizes
– Urban vs. rural
– Characteristics of families served
– Time enrolled in wraparound
– Agency culture
– Turnover
– Perception of Community supports
WHAT ARE THE OUTCOMES???
• Residential
• Medicaid expenditures/Fiscal
• CANS/Functional
Part 4: Massachusetts Data Part 4: Massachusetts Data by CSAby CSA
WFI CSA Results
0.86
0.82
0.82
0.82
0.82
0.81
0.81
0.81
0.81
0.8
0.77
0.7 0.75 0.8 0.85 0.9
18 Lawrence
29 Springfield
24 Park Street
20 Lynn
12 Gandara
2 Attleboro
23 N Central
22 New Bedford
5 Cape Ann
28 S Central
Nat Mean
CS
A
Percent Fidelity
CSA N Total Score
18 Lawrence 18 0.8629 Springfield 20 0.8224 Park Street
14 0.8220 Lynn 19 0.8212 Gandara 14 0.822 Attleboro 18 0.8123 N. Central 22 0.8122 New Bedford
21 0.815 Cape Ann 20 0.8128 S. Central 17 0.80
WFI CSA Results
0.8
0.8
0.79
0.79
0.79
0.78
0.78
0.77
0.77
0.76
0.76
0.77
0.74 0.76 0.78 0.8 0.82
10 Fall River
16 Holyoke
6 C and I
1 Arlington
25 Pittsfield
19 Lowell
32 Worcester W
8 Coastal
14 Harbor
31 Worcester E
11 Framingham
Nat Mean
CS
A
Percent Fidelity
CSA N Total Score
10 Fall River 19 0.80
16 Holyoke 16 0.80
6 Cape and Isl 19 0.79
1 Arlington 18 0.79
25 Pittsfield 21 0.79
19 Lowell 20 0.78
32 Worcester W 20 0.78
8 Coastal 21 0.77
14 Harbor 18 0.77
31 Worcester E 18 0.76
11 Framingham 20 0.76
WFI CSA Results
0.76
0.76
0.75
0.75
0.74
0.74
0.73
0.72
0.7
0.69
0.55
0.77
0 0.5 1
13 Greenfield
27 RVW
9 Dimock
21 Malden
15 Haverhill
7 CSR
3 Brockton
26 Plymouth
4 Cambridge
17 Hyde Park
30 Walden
Nat Mean
CS
A
Percent Fidelity
CSA N Total Score
13 Greenfield 19 0.76
27 RVW 20 0.76
9 Dimock 20 0.75
21 Malden 22 0.75
15 Haverhill 20 0.74
7 CSR 11 0.74
3 Brockton 18 0.73
26 Plymouth 20 0.72
4 Cambridge 22 0.70
17 Hyde Park 20 0.69
30 Walden 3 0.55
TOM CSA Results
0.93
0.93
0.93
0.92
0.92
0.91
0.91
0.91
0.91
0.9
0.77
0.7
8 Coastal
7 CSR
10 Fall River
5 Cape Ann
30 Walden
21 Malden
11 Framingham
26 Plymouth
22 New Bedford
20 Lynn
Nat Mean
CS
A
Percent Fidelity
CSA N Total Score
8 Coastal 14 0.93
7 CSR 58 0.93
10 Fall River 32 0.93
5 Cape Ann 38 0.92
30 Walden 4 0.92
21 Malden 44 0.91
11 Framingham
22 0.91
26 Plymouth 18 0.91
22 New Bedford
23 0.91
20 Lynn 19 0.90
TOM CSA Results
0.89
0.86
0.86
0.86
0.86
0.85
0.85
0.85
0.85
0.85
0.83
0.77
0.7 0.75 0.8 0.85 0.9
6 C and I
16 Holyoke
9 Dimock
2 Attleboro
31 Worcester E
18 Lawrence
1 Arlington
23 N Central
4 Cambridge
15 Haverhill
13 Greenfield
Nat Mean
CS
A
Percent Fidelity
CSA N Total Score
6 C and I 32 0.89
16 Holyoke 14 0.86
9 Dimock 13 0.86
2 Attleboro 10 0.86
31 Worcester E 23 0.86
18 Lawrence 10 0.85
1 Arlington 18 0.85
23 N Central 31 0.85
4 Cambridge 16 0.85
15 Haverhill 21 0.85
13 Greenfield 16 0.83
TOM CSA Results
0.83
0.83
0.82
0.81
0.79
0.79
0.79
0.77
0.77
0.75
0.73
0.77
0.65 0.7 0.75 0.8 0.85
28 S Central
17 Hyde Park
25 Pittsfield
3 Brockton
14 Harbor
27 RVW
19 Lowell
32 Worcester W
29 Springfield
24 Park Street
12 Gandara
Nat Mean
CS
A
Percent Fidelity
CSA N Total Score
28 S Central 14 0.83
17 Hyde Park 8 0.83
25 Pittsfield 25 0.82
3 Brockton 11 0.81
14 Harbor 18 0.79
27 RVW 25 0.79
19 Lowell 16 0.79
32 Worcester W 13 0.77
29 Springfield 31 0.77
24 Park Street 18 0.75
12 Gandara 6 0.73
DRM CSA Results
0.82
0.82
0.82
0.8
0.8
0.8
0.8
0.79
0.79
0.78
0.74
0.7 0.75 0.8 0.85
28 S Central
6 C and I
11 Framingham
20 Lynn
18 Lawrence
14 Harbor
31 Worcester E
4 Cambridge
1 Arlington
5 Cape Ann
MA Mean
CS
A
Percent Fidelity
CSA N Total Score
28 S Central 10 0.82
6 Cape and Isl 10 0.82
11 Framingham
10 0.82
20 Lynn 10 0.80
18 Lawrence 10 0.80
14 Harbor 10 0.80
31 Worcester E
10 0.80
4 Cambridge 10 0.79
1 Arlington 10 0.79
5 Cape Ann 10 0.78
DRM CSA Results
0.76
0.76
0.76
0.76
0.76
0.75
0.74
0.74
0.73
0.73
0.73
0.74
0.7 0.72 0.74 0.76 0.78
32 Worcester W
8 Coastal
3 Brockton
17 Hyde Park
23 N Central
10 Fall River
26 Plymouth
22 New Bedford
29 Springfield
2 Attleboro
7 CSR
MA Mean
CS
A
Percent Fidelity
CSA N Total Score
32 Worcester W 10 0.76
8 Coastal 10 0.76
3 Brockton 10 0.76
17 Hyde Park 10 0.76
23 N Central 10 0.76
10 Fall River 10 0.75
26 Plymouth 10 0.74
22 New Bedford 10 0.74
29 Springfield 10 0.73
2 Attleboro 10 0.73
7 CSR 10 0.73
DRM CSA Results
0.71
0.71
0.71
0.7
0.69
0.69
0.69
0.66
0.65
0.65
0.64
0.74
0.55 0.6 0.65 0.7 0.75
13 Greenfield
21 Malden
16 Holyoke
24 Park Street
15 Haverhill
25 Pittsfield
27 RVW
30 Walden
9 Dimock
19 Lowell
12 Gandara
MA Mean
CS
A
Percent Fidelity
CSA N Total Score
13 Greenfield 12 0.71
21 Malden 10 0.71
16 Holyoke 10 0.71
24 Park Street 10 0.70
15 Haverhill 10 0.69
25 Pittsfield 10 0.69
27 RVW 10 0.69
30 Walden 10 0.66
9 Dimock 10 0.65
19 Lowell 10 0.65
12 Gandara 10 0.64
Combined Scoresby CSA
0.4
0.5
0.6
0.7
0.8
0.9
1
WFI 0.82 0.86 0.81 0.79 0.76 0.8 0.77 0.8 0.81 0.79 0.77
TOM 0.9 0.85 0.92 0.89 0.91 0.93 0.93 0.83 0.91 0.85 0.77
DRM 0.8 0.8 0.78 0.82 0.82 0.75 0.76 0.82 0.74 0.79 0.74
20 Lynn18
Lawrence
5 Cape Ann
6 C and I11
Framingham
10 Fall River
8 Coastal28 S
Central22 New Bedford
1 Arlington
Nat Mean
Combined Scoresby CSA
0.4
0.5
0.6
0.7
0.8
0.9
1
WFI 0.76 0.81 0.81 0.74 0.72 0.75 0.8 0.77 0.7 0.82 0.78 0.77
TOM 0.86 0.85 0.86 0.93 0.91 0.91 0.86 0.79 0.85 0.77 0.77 0.77
DRM 0.8 0.76 0.73 0.73 0.74 0.71 0.71 0.8 0.79 0.73 0.76 0.74
31 Worcest
er E
23 N Central
2 Attlebor
o7 CSR
26 Plymout
h
21 Malden
16 Holyoke
14 Harbor
4 Cambrid
ge
29 Springfi
eld
32 Worcest
er W
Nat Mean
Combined Scoresby CSA
0.4
0.5
0.6
0.7
0.8
0.9
1
WFI 0.76 0.73 0.79 0.74 0.69 0.82 0.75 0.76 0.78 0.82 0.55 0.77
TOM 0.83 0.81 0.82 0.85 0.83 0.75 0.86 0.79 0.79 0.73 0.92 0.77
DRM 0.71 0.76 0.69 0.69 0.76 0.7 0.65 0.69 0.65 0.64 0.66 0.74
13 Greenfie
ld
3 Brockto
n
25 Pittsfiel
d
15 Haverhil
l
17 Hyde Park
24 Park Street
9 Dimock
27 RVW19
Lowell12
Gandara30
WaldenNat
mean
Percent improvement by CSA by tool
• WFI top 3 +
– 19 Lowell +.07
– 18 Lawrence +.05
– 25 Pittsfield and 22 New Bedford +.05
• WFI bottom 3 -
– 4 Cambridge -.09
– 17 Hyde Park -.14
– 30 Walden -.24 *only N = 3 in 2010
Percent improvement by CSA by tool
• TOM top 3 +
– 9 Dimcock +.34
– 8 Coastal + .12
– 5 Cape Ann + .12
– 25 Pittsfield +.12
– 6 C and I +.09
• TOM bottom 3 -
– 14 Harbor -.06
– 29 Springfield -.10
– 24 Park Street -.11
Percent improvement by CSA by tool
• DRM top 3 +
– 31 Worcester E +.43
– 32 Worcester W + .40
– 5 N. Cape Ann + .38
• DRM bottom 3 -
– 27 RVW -.10
– 19 Lowell -.13
– 30 Walden -.16