Multnomah Project LAUNCH€¦ · 15/12/2015 · Multnomah Project LAUNCH Year 5 Evaluation Report...

Multnomah Project LAUNCH

Year 5 Evaluation Report

October 1, 2010‐ September 30, 2015

Beth L. Green, Ph.D.

Callie H. Lambarth, M.S.W.

Center for Improvement of Child and Family Services

Portland State University

December 15, 2015

Center for Improvement of Child and Family Services Multnomah Project LAUNCH Year 5 Evaluation Report 2


Table of Contents Table of Contents ............................................................................................................................ 3

Executive Summary ......................................................................................................................... 7

Methodology ............................................................................................................................... 8 Key Findings & Lessons Learned ................................................................................................. 9 Systems Strengths & Key Achievements................................................................................. 9 System Challenges and Lessons Learned ................................................................................ 9 Service Delivery Implementation Strengths & Challenges ................................................... 10 Service Delivery Outcomes ................................................................................................... 11

Conclusions & Sustainability ..................................................................................................... 12 Project Description ....................................................................................................................... 14

Table 1. Multnomah County Project LAUNCH Evaluation Model Summary ........................ 16 Figure 1. Multnomah County Project High‐Level LAUNCH Logic Model .............................. 21

Systems Change ............................................................................................................................ 22

Collaboration ............................................................................................................................ 22 Research Questions .............................................................................................................. 22 Methods & Data .................................................................................................................... 22 Table 2. Characteristics of Key Stakeholder Systems Survey Respondents ......................... 23 Findings: Collaboration Implementation Evaluation ............................................................ 24 Table 3. New Collaborations Established Through LAUNCH ................................................ 24 Table 4. YCWC Membership ................................................................................................. 26 Findings: Collaboration Outcomes ........................................................................................ 26 Figure 2. Mean Ratings of Cross‐Sector Collaboration by Sector ......................................... 27 Figure 3. Mean Ratings of Systems Survey Areas ................................................................. 28 Figure 4. ECE Provider and HV Staff Perception of Service Linkages .................................... 30 Systems Challenges: Culturally Responsive and Culturally Specific Services ....................... 30 Figure 5. ECE Provider & HV Staff Confidence in Involving Families .................................... 32 Summary: Collaboration & Systems Coordination ............................................................... 32

Workforce Development .......................................................................................................... 32 Research Questions .............................................................................................................. 33 Methods & Data .................................................................................................................... 33 Findings: WFD Implementation ............................................................................................ 34 Table 5. Workforce Development Efforts & Number of Participants ................................... 34 Findings: WFD Outcomes ...................................................................................................... 35 Table 6. Participant Rating of Workforce Development Events ........................................... 35 Table 7. EC PBIS Coaching Group Impact .............................................................................. 36 Table 8. TPOT Training Post‐Evaluation & 12 Month Impact ............................................... 36 Findings: Communities of Practice ....................................................................................... 43 Findings: Overall Workforce Development Successes and Challenges ................................ 43

Public Awareness ...................................................................................................................... 43


Research Questions .............................................................................................................. 44 Methods & Data .................................................................................................................... 44 Findings: Implementation of Public Awareness Activities .................................................... 45 Table 11. 211infoFamily Community Awareness Activities .................................................. 45 Table 12. Number of 211infoFamily Promotion Efforts ....................................................... 45 Findings: Outcomes for Public Awareness Activities ............................................................ 46 Figure 8. ECE Provider Knowledge of 211infoFamily ............................................................ 46 Figure 9. Home Visitor Staff Knowledge of 211infoFamily ................................................... 46 Table 13. Primary Care Provider Expected Use of 211infoFamily ........................................ 47 Findings: Overall Successes and Challenges in Increasing Public Awareness ....................... 47

Sustainability ............................................................................................................................. 48 Research Questions .............................................................................................................. 48 Methods & Data .................................................................................................................... 48 Findings: Sustainability ......................................................................................................... 48

Core Strategies .............................................................................................................................. 50

Screening, Assessment & Behavioral Health Integration in Primary Care ............................... 50 Evaluation Questions ............................................................................................................ 50 Methods & Data .................................................................................................................... 50 Findings: Implementation of START Trainings ...................................................................... 52 Table 15. Primary Care Providers & Staff Trained by START Module Type .......................... 52 Findings: Outcomes of START Trainings ................................................................................ 53 Table 16. Change in Knowledge Reported by Primary Care Providers & Staff ..................... 53 Table 17. Screenings Completed by Providers ...................................................................... 54 Table 18. Referrals to Early Intervention with Screening Result by Primary Care Providers 55 Table 19. Behavioral Health Integration in Primary Care Readiness Checklist Domains ..... 56 Summary: Screening, Assessment, and Referral .................................................................. 57

Early Childhood Mental Health Consultation in Early Care & Education Settings ................... 58 Evaluation Questions ............................................................................................................ 58 Methods & Data .................................................................................................................... 58 Findings: Implementation of EC MHC in Early Care & Education ......................................... 59 Table 20. Characteristics of ECE Program Staff & Children .................................................. 59 Findings: Outcomes for Changes in ECE Provider Skills, Confidence, and Knowledge ......... 61 Table 21. LAUNCH ECE Staff Outcome Domains ................................................................... 61 Findings: Outcomes for ECE Classroom Environments ......................................................... 62 Figure 10. LAUNCH ECE Infant‐Toddler Classroom Staff Practices (TPITOS) ........................ 63 Figure 11. LAUNCH ECE Preschool Classroom Staff Practices (TPOT) .................................. 63 Findings: Outcomes for ECE Program Level Changes ........................................................... 63 Figure 12. LAUNCH ECE Program‐Level Implementation of EC PBIS (BOQ) ......................... 64 Summary: Early Childhood Mental Health Consultation in Child Care Findings .................. 66

Enhanced Home Visiting through Mental Health Consultation ............................................... 67 Evaluation Questions ............................................................................................................ 67 Methods & Data .................................................................................................................... 67 Table 24. Characteristics of HV Program Staff & Families Served ........................................ 68


Findings: Implementation Successes and Challenges ........................................................... 69 Findings: Outcomes for Home Visitors ................................................................................. 70 Figure 14. LAUNCH v Comparison HV Staff Parent Involvement Domain ............................ 71 Figure 16. LAUNCH v Comparison HV Staff Knowledge of EC PBIS ...................................... 72 Figure 17. LAUNCH v Comparison HV Staff Stress Domain .................................................. 72 Figure 18. LAUNCH v Comparison HV Staff Perception of Program Leadership Domain ..... 73 Table 25. LAUNCH HV Staff Outcome Domains .................................................................... 73 Summary: Early Childhood Mental Health Consultation in Home Visiting Findings ............ 73 Table 26. Characteristics of Families Served by LAUNCH & Comparison HV Programs July 1, 2011 through June 30, 2014 .......................................................................................................... 74 Table 27. LAUNCH & Comparison Group HV Program Child & Family Outcomes ................ 75

Family Strengthening: Parenting Education Series ................................................................... 75 Evaluation Questions ............................................................................................................ 75 Methods & Data .................................................................................................................... 76 Findings: Implementation of Parenting Groups ................................................................... 76 Table 28. Incredible Years Group Characteristics ................................................................. 76 Table 29. Characteristics of Incredible Years Families with Dosage & Pre‐Post Tests ......... 77 Table 30. Pre‐Test Characteristics of Dosage v Non‐Dosage Incredible Years Families ....... 77 Findings: Outcomes of IY and Parenting Groups .................................................................. 78 Table 31. Parent & Child Incredible Years Outcomes for Families with Dosage .................. 78 Table 32. Pre‐Post Test Characteristics of Incredible Years Families with Dosage by Child Race ............................................................................................................................................... 79

Family Strengthening: 211infoFamily ....................................................................................... 80 Evaluation Questions ............................................................................................................ 80 Methods & Data .................................................................................................................... 80 Table 33. 211infoFamily Callers ............................................................................................ 81 Table 34. 211infoFamily Caller & Target Child Characteristics ............................................. 81 Findings: 211info Family Implementation ............................................................................ 82 Findings: Outcomes of 211infoFamily .................................................................................. 83

Summary & Conclusions ............................................................................................................... 84

Key Stakeholder Report of LAUNCH Major Accomplishments ............................................. 84 System Challenges and Lessons Learned .............................................................................. 85 Service Delivery Implementation Strengths & Challenges ................................................... 86 Service Delivery Outcomes ................................................................................................... 87

Conclusions & Sustainability ..................................................................................................... 88 Appendices .................................................................................................................................... 90

Appendix A: Key Stakeholder Systems Survey .......................................................................... 90 Appendix B: Key Stakeholder Systems Interview Questions .................................................... 94 Appendix C: Summary Tables of Figures .................................................................................. 95 Figure 2. Reference: Mean Ratings of Cross‐Sector Collaboration by Sector ...................... 95 Figure 3. Reference: Mean Ratings of Systems Survey Areas .............................................. 95 Figure 4. Reference: ECE Provider & HV Staff Perception of Service Linkages .................... 95 Figure 5. Reference: ECE Provider & HV Staff Confidence in Involving Families .................. 95


Figure 8. Reference: ECE Provider Knowledge of 211infoFamily ......................................... 96 Figure 9. Reference: HV Staff Knowledge of 211infoFamily ................................................. 96 Figure 14. Reference: LAUNCH v Comparison HV Staff Parent Involvement Domain ......... 96 Figure 15. Reference: LAUNCH v Comparison HV Staff Knowledge of Children’s Mental Health Domain .................................................................................................................................. 96 Figure 16. Reference: LAUNCH v Comparison HV Staff Knowledge of EC PBIS .................... 96 Figure 17. Reference: LAUNCH v Comparison HV Staff Stress Domain ................................ 97 Figure 18. Reference: LAUNCH v Comparison HV Staff Perception of Program Leadership Domain ............................................................................................................................................... 97

Appendix D: Community of Practice Issue Briefs ..................................................................... 98 Home Visiting CoP ................................................................................................................. 98 EC PBIS CoP ......................................................................................................................... 102 Early Childhood Mental Health CoP .................................................................................... 106

Appendix E: EI Referral Practices Issue Brief .......................................................................... 110 Appendix F: Maternal Depression Screening Issue Brief ........................................................ 116 Appendix G: MHC in ECE Settings Issue Brief ......................................................................... 120 Appendix H: MHC in Home Visiting Logic Model .................................................................... 126 Appendix I: MHC in Home Visiting Issue Brief ........................................................................ 128 Appendix J: 211infoFamily Interview Questions .................................................................... 133 Appendix K: 211infoFamily Issue Brief ................................................................................... 135 Appendix L: Timeline of Evaluation Activities ......................................................................... 142


Executive Summary Multnomah Project LAUNCH had the following over‐arching goals:

1. To plan, align, and grow programs and partnerships that promote young child wellness;

2. To increase community awareness of the importance of young child wellness and the resources available to support early childhood physical, mental, and emotional health; and

3. To increase current screening, consultation, and training in primary health care, home visitation, and child care settings in order to promote young child wellness.

To accomplish these goals, LAUNCH implemented several core sets of activities:

A. Systems Improvement & Integration, including:

1. Collaborative work through the Young Child Wellness Council (YCWC) to bring together diverse program agencies, policy makers, service providers, and parents to address systems and service improvements;

2. Workforce development (WFD) focused on primary care physicians and staff, early childhood providers and home visitors.

3. Public awareness and social marketing of universal parenting support services (specifically through 211info);

4. Sustainability work to ensure ongoing support for LAUNCH strategies and sustained systems changes following the end of the LAUNCH grant.

B. Behavioral health in primary care, specifically by developing and implementing a live and web‐based training module for primary care doctors and staff on behavioral health for children ages 0‐8 years;

C. Increasing screening and assessment of young children, by training primary and pediatric care physicians to conduct the Ages and Stages Questionnaire (ASQ), ASQ‐Social Emotional screen, and providing key linkages for referral and follow up assessments;

D. Providing Early Childhood Mental Health Consultation (EC MHC) in early care and education settings using a unique model based on Early Childhood Positive Behavior Interventions and Supports (EC PBIS);

E. Enhancing home visiting by providing Early Childhood Mental Health Consultation to Healthy Families Oregon (HFO) home visitors;

F. Family strengthening services through implementing evidence‐based Incredible Years and other parenting education series, as well as by supporting a Parent Educator to respond to parenting‐related calls to the County’s 211infoFamily helpline.

This report describes the key implementation “lessons learned” and systems and services outcomes from the five‐year evaluation of Project LAUNCH conducted by Portland State University’s Center for the Improvement of Child and Family Services. The evaluation had several components. First, a


Systems Level Evaluation was conducted to describe and evaluate the systems changes support by Project LAUNCH and to address the following systems related research questions:

1. To what extent is LAUNCH successful in planning, aligning, and growing programs and partnerships that promote young child wellness?

2. To what extent does LAUNCH increase community awareness of the importance of young child wellness and the resources available to support early childhood physical, mental, and emotional health?

Second, a Service Delivery Evaluation was designed to answer the following key research questions:

1. To what extent does Project LAUNCH increase developmental/behavioral health screenings and integration of behavioral health in primary care settings?

2. To what extent does Project LAUNCH increase the likelihood that screenings result in appropriate referral and service linkages?

3. To what extent does Project LAUNCH support child, family, and staff outcomes by providing evidence‐based (a) early childhood mental health consultation; (b) Early Childhood Positive Behavioral Interventions and Supports; and (c) Incredible Years groups to parents and providers working with infants, toddlers, and young children?

4. To what extent does Project LAUNCH increase resources and coordination for workforce development and training for providers serving young children and their families?

Methodology

The LAUNCH evaluation used a mixed‐method longitudinal design to capture both qualitative and quantitative data to address the key research questions. The evaluation team worked closely with the YCWC and other LAUNCH partners to support continuous project improvement throughout the five‐year period.

Systems‐level evaluation included an annual quantitative assessment of key early childhood stakeholders’ perceptions of the extent and quality of inter‐agency collaborations focused on partnerships between early childhood and family service providers, behavioral health, and primary care (including indicators related to frequency and type of collaborative activities, extent of parent involvement, and attention to issues of cultural responsiveness), as well as annual qualitative interviews to gather information about buy‐in to LAUNCH goals, perceived successes and challenges facing LAUNCH, and to document the key barriers and facilitators of implementation.

Service delivery evaluation methods were tailored for each service delivery mechanism. Key types of data being collected include:

1. Tracking the utilization of the 211infoFamily service by community members by examining call databases, and assessing the quality of subsequent referral and follow‐up regarding needed services through a parent telephone interview;

2. Documenting child‐ and family‐level outcomes for those receiving individualized supports through Incredible Years, using standardized outcome measures of child behavior and social‐emotional development, parenting stress and parenting quality; and


3. Tracking changes in providers’ level of knowledge of children’s mental health ‘best practices’, Early Childhood Positive Behavior Intervention and Support concepts, and knowledge of community resources for referrals through regular surveys of participating providers.

Key Findings & Lessons Learned

Systems Strengths & Key Achievements

1. The YCWC, which included a number of community partners who had not previously worked together, was highly successful in sustaining key agency membership across primary care, mental health, and early childhood sectors. The group met monthly throughout the five‐year project to provide input into key LAUNCH related services and products and to identify and address gaps in the service system for young children and their families.

2. Project LAUNCH supported integration of Early Childhood Positive Behavior Support (EC PBIS) professional development into more and new settings, including supporting the modification of EC PBIS for domestic violence shelters; training for kindergarten teachers in EC PBIS and working with the Child Care Resource and Referral agency to create a joint EC PBIS training calendar and network of EC PBIS coaches.

3. Increased opportunities for early childhood, home visitation, and other providers to share training opportunities and to convene professional development Communities of Practice (CoPs), which are being sustained after project end.

4. Supporting a number of targeted, small‐scale innovations such as the emerging Adverse Childhood Experiences (ACEs) workgroup, START‐related efforts in doing perinatal depression screening, and support for training child care providers in efforts to reduce screen time.

5. Exceeding targeted goals for providing training and workforce development to a broad array of early childhood, mental health, physical health, and family support practitioners. Multnomah LAUNCH provided training to over 1,200 providers across the five year project.

System Challenges and Lessons Learned

Over the course of the five‐year project, a number of systems‐level challenges were encountered, and lessons learned about implementing the LAUNCH services and approach.

1. Despite a number of attempts to engage parents as partners in the YCWC and LAUNCH leadership, ultimately, this goal was never fully achieved. Each year of LAUNCH brought new strategies as partners attempted to engage parents most meaningfully. In retrospect, the original strategy, which involved providing funding to culturally specific organizations to mentor one or more parents to attend and participate in the YCWC meetings may have been more successful if it were implemented after the YCWC had a stronger sense of initial cohesion and purpose. The most successful parent engagement strategy involved convening a separate group of parents to provide input and guidance around specific issues (e.g., the best way to advertise the 211infoFamily supports). However, even this more focused group proved difficult to sustain. Final attempts to build in other systems‐level efforts to bring parent voice to community‐wide planning proved difficult as well.


2. Although progress was made to identify systemic “gaps” in the child screening, assessment and referral process, the goal of seamless, streamlined screening and referral systems proved difficult. At least part of the challenge was the lack of consensus across providers about the primary function of screening, as well as the lack of a central repository for screening results. Efforts to create systems for sharing information across system have continued to lag. This is no doubt due to both the complexity of the issues, as well as the critical need for better cross system leadership at the state level in developing key data systems that would facilitate information sharing more consistently.

Service Delivery Implementation Strengths & Challenges

Multnomah Project LAUNCH implemented four major service delivery activities: (1) trainings for primary care practitioners through START to increase the frequency of screening for children’s social‐emotional wellbeing and postpartum depression; (2) early childhood mental health consultation and Early Childhood Positive Behavior Intervention and Supports with child care and home visiting service providers; (3) parenting education, support, and referral through the 211infoFamily service; and (4) evidence‐based family strengthening in the form of Incredible Years groups. Implementation of planned service delivery activities for Project LAUNCH has in most cases exceeded initial targets. Specifically:

1. Project LAUNCH has exceeded its target of 60 PCPs and other primary care staff trained in ASQ, behavioral health, trauma‐informed care, and maternal depression screenings each year: Year 1 (152 primary care staff trained), Year 2 (212 primary care staff trained), Year 3 (196 primary care staff trained), Year 4 (188 primary care staff trained) and Year 5 (192 primary care staff trained).

2. Project LAUNCH has exceeded its cumulative target for 211info Family usage of 695 callers receiving referrals, with over 2,000 calls received (12 calls during Year 1; 184 calls during Year 2; 806 calls during Year 3, 579 calls during Year 4, and 676 calls in Year 5).

3. Incredible Years (IY) groups were implemented as planned, with fidelity, and included more than the originally targeted number of parents (91 participants attended at least one group; 57 parents completed 50% of more of the sessions (63% of participants).

4. Implementation of early childhood mental health consultation for home visitors, which was changed during Year 2 in order to provide this service to a different home visiting program, was highly successful during the last three years of the project. Home visitors indicated that the mental health consultant provided valuable support to them and their families, including training on EC PBIS strategies, parent coaching, wellness and stress reduction, and interpretation and supervision around social emotional screening results.

Challenges to implementation of Project LAUNCH services included the following:

1. Child care staff turnover. By far the biggest challenges in delivering the EC MHC services in child care settings have been related to uncontrollable (and unpredictably high) turnover in child care staff and program leadership; in the most extreme case, at the end of Year 4 one of the facilities that LAUNCH was supporting closed its doors entirely, with little notice. While turnover within


child care is not unexpected, the leadership turnover was particularly challenging, and no doubt related to a higher than expected rate of turnover within staff.

2. Need for follow up training for primary care. Significant numbers of START‐trained primary care physicians and practices needed additional support in order to fully implement the recommended START screenings, even for the basic developmental module. Follow‐up by evaluation and START staff indicated significant barriers to full implementation of screening protocols; thus, the decision was made at the end of Year 4 to fund a series of “tune‐ups” in Year 5, for previously trained providers during Years 1 through 4. Additionally, a follow‐up protocol will be implemented with these providers to provide ongoing coaching and technical assistance to support implementation.

3. Initial location of 211infoFamily outside of Multnomah County. During Year 1, LAUNCH contracted with a Parent Helpline provider whose primary experiences were in Lane County, Oregon, roughly 90 miles south of Multnomah County. Despite efforts to integrate resources and provide training to this provider, after a year of challenging implementation, the service was relocated to Multnomah County, with greater success.

4. Increasing public awareness and utilization of the 211infoFamily helpline. During Years 1 and 2 the number of callers to the 211infoFamily helpline was much lower than anticipated. This led to several efforts to “get the word out,” including public awareness and marketing campaigns, redoubled efforts by the 211infoFamily outreach specialist to make presentations to community providers, and integration of information about 211infoFamily into other LAUNCH strategies, specifically within START trainings and the work of the EC MHCs. Data suggest that these last two strategies may have been particularly effective.

Service Delivery Outcomes

1. Outcomes for child care providers and home visitors. Results of surveys completed by child care and home visiting staff receiving early childhood mental health consultation found some evidence for improvement in provider knowledge and confidence from baseline (pre‐consultation) to subsequent follow up assessments. Specifically, providers reported statistically significant increases in their knowledge of early childhood mental health, and in their confidence in working to involve parents to support children’s social emotional development and mental health. Further, observations conducted by consultants of child care providers in preschool aged classrooms also suggested substantial improvements in quality of teacher‐child interactions in classrooms receiving consultation.

2. Primary care providers. Primary care providers who received START training consistently reported improvements in their knowledge of developmental screening tools, community referral resources, and increased intention to conduct screenings. Additionally, there was a substantial increase in screenings and referrals to the local Early Intervention provider from primary care providers. During Years 3‐5, there were over 1,000 referrals received by Early Intervention/Early Childhood Special Education (EI/ECSE) from primary care, compared to only about 100 received during Years 1 and 2.


3. Developmental screenings. In a subset of primary care clinics for whom data were available, the number of developmental screenings provided by primary care providers increased by over 1,000 screens in the 12‐month period following receipt of START trainings.

4. 211infoFamily service users. The majority of users were highly satisfied with the services they received from 211infoFamily, and would recommend this service to others; however, a very small number of families could be successfully contacted for the evaluation. Follow up interviews conducted by PSU evaluation staff also documented the important role being played by the 211infoFamily Parent Educator in eliciting parenting concerns and providing appropriate support to families around a myriad of child‐related topics.

5. Incredible Years. 86 unique parents have participated in Incredible Years workshop series through Project LAUNCH through Year 4. Results of this service have been strong, with participating parents reporting reduced levels of child internalizing and externalizing behavior, and improved attachment, child initiative, and self‐control.

Conclusions & Sustainability

Over the five years of implementation, Multnomah Project LAUNCH had significant success in both systems improvement and service delivery. These successes are attributable at least in part to the commitment and diligence of the Young Child Wellness Council to working through early implementation challenges, and to using data for continuous program improvement. Changes in planned implementation were made as appropriate to help support continued progress towards LAUNCH goals.

Multnomah’s Project LAUNCH has been characterized by a close partnership between LAUNCH leadership and the Portland State University (PSU) evaluation team. The evaluation team was given many opportunities to share data and information with the YCWC and with key LAUNCH subcontractors. Further, the evaluation team worked to be responsive to emergent issues of particular interest or importance to LAUNCH stakeholders, for example, developing new ways to approach the evaluation of the START trainings; making sure mechanisms were in place to evaluate new aspects of the workforce development activities; and working to expand information gathering related to mental health consultation in home visiting. Contracted LAUNCH providers also worked closely with both the YCW coordinator, the YCWC, and the evaluation team to ensure that services were being delivered as planned, or, if challenges were encountered, to problem solve and make appropriate changes.

Beginning in Year 4, there was a focused effort to begin to create pathways to sustainability for LAUNCH system enhancements and service delivery. As of this writing (3 months after the end of project funding), the following activities continued to be ongoing:

The YCWC continues to meet, with staff support being provided by the local EI/ECSE provider. The success of the Council to continue was due to a strategic decision to transition the group to include the role of required advisory council for EI/ECSE, while continuing to play its current role in fostering cross‐sector partnerships;

EC PBIS Leadership Team continues to meet, with dedicated administrative staff support from one of the participating agencies;


Support for the EC PBIS model is being sustained both through state funding to train child care and kindergarten teachers in the model in a project being supported by Portland Public Schools as well as through training provided by the local Child Care Resource and Referral network;

The Home Visiting CoP continues to meet, with support from the state’s Maternal Infant Early Childhood Home Visiting (MIECHV) grants;

The Early Childhood Mental Health CoP also continues to meet, although less frequently and without administrative staff support;

The Parent Educator position is currently being maintained at 211infoFamily, supported by funding from one of the regional Coordinated Care Organizations;

Early Childhood Mental Health Consultants are being supported through County, City, and federal MIECHV funds;

START continues to support training for primary care physicians in a variety of behavioral health screening tools and how to incorporate behavioral health into primary care.

The success of Project LAUNCH in sustaining both systems work occurring through the YCWC as well as key elements of service enhancements is evidence of the value placed by the community on the services and enhancements supported during the five years of the LAUNCH grant.


Project Description Multnomah County Education Service District (MESD) Project LAUNCH had the following over‐arching goals:

1. To plan, align, and grow programs and partnerships that promote young child wellness; 2. To increase community awareness of the importance of young child wellness and the resources

available to support early childhood physical, mental, and emotional health; and 3. To increase current screening, consultation, and training in primary health care, home

visitation, and child care settings in order to promote young child wellness.

To accomplish these goals, Multnomah Project LAUNCH implemented several core sets of activities including:

1. Establishing and supporting a Young Child Wellness Council (YCWC) to implement LAUNCH’s mission and values at the policy and practice levels;

2. Ensuring that the YCWC includes diverse organizational and family perspectives, specifically including primary care, mental health, early childhood, and family support providers as well as organizations serving culturally specific populations;

3. Conducting social marketing to promote social norms around seeking help for parenting support, and promoting information and referral resources for parents;

4. Improving current information and referral systems by supporting a Parent Educator at the County’s 211info helpline (family‐ and child‐related services were known as 211infoFamily), and by supporting 211info to update service provider information, ensuring representation of the diverse array of services available in Multnomah County;

5. Expanding training for primary health care providers in conducting developmental and other wellness screenings for families and children;

6. Developing new training modules for primary health providers for screening for behavioral health issues;

7. Strengthening linkages between providers doing screening and referral of children to improve information sharing and reduce duplication of screening efforts;

8. Providing quality mental health consultation using the Early Childhood Positive Behavior Interventions and Support model (EC PBIS) to home visitors and child care centers serving children ages 0‐5;

9. Improving the system of training for professionals who work with families and young children by assessing training needs and gaps and coordination and providing additional workforce development activities to meet these needs;

10. Providing evidence‐based parenting education to strengthen families, using the Incredible Years program.

Multnomah Project LAUNCH was evaluated by researchers at Portland State University (PSU), who worked close with the Young Child Wellness Coordinator, the YCWC, and with contracted providers to collect and synthesize data and to facilitate utilization of data to improve Project LAUNCH and related activities. The evaluation team worked with LAUNCH partners to develop a Program Logic Model (see Table 1 and Figure 1, below) that guided both the YCWC’s work plan and the evaluation process. Table 1 represents a detailed logic model that was adapted over time to reflect changes in specific LAUNCH


activities, although the over‐arching goals and key strategies remained the same. These changes led to subsequent changes in some aspects of the evaluation as well. Figure 1 represents a “high level” logic model that was used largely as an internal and external communication tool to describe LAUNCH’s complex set of project activities at the systems and services level.

The evaluation had several components. First, a Systems Level Evaluation sought to document and describe the implementation and outcomes related to changes in the early childhood systems that were facilitated by Project LAUNCH. Second, a Service Delivery Evaluation was designed to document processes and outcomes related to planned service delivery activities. Evaluation results for both the Systems and Service Evaluation over the 5‐year projects are presented below, following the Logic Models.


Table 1. Multnomah County Project LAUNCH Evaluation Model Summary

Goal Activities Outputs Short‐term outcomes Long‐term Outcomes

Systems Level: Increase Coordination and collaboration among child‐serving providers, and Increase Community Awareness

Goal #1:

Plan, align, and grow programs and partnerships that promote young child wellness

Implement and sustain YCWC

Share information between YCWC members about research policy, etc.

Create opportunities for YCWC to advocate for and collaborate with the child‐serving and wellness promotion community

Establish Communities of Practice to support better alignment, integration, and shared professional development.

Inventory of child and family resources is developed and widely distributed

Inventory incorporated and used by 211info Family

Strategic plan implemented

4 Communities of Practice (CoPs) implemented

Providers, including primary health care providers, have improved knowledge of and access to available resources

Activities and priorities outlined in strategic plan are valued by the community

Activities outlined in strategic plan appropriately address community needs

Activities in strategic plan are implemented with fidelity to plan

Participants in CoPs have increased understanding of each other services and resources

More efficient use of services

High levels of community buy‐in for Project LAUNCH

Increased sustainability of LAUNCH

YCWC includes parents and providers from multiple organizations and family/cultural backgrounds

Parent representatives attend YCWC parent subcommittee

Activities, plans, and decisions made by the YCWC are more culturally responsive

Parent voices are included in decision‐making for the YCWC

Increased representation of communities of color on YCWC

Early childhood services and systems more culturally responsive/sensitive and family‐driven



Goal #2:

Increase community awareness of the importance of young child wellness and the resources available to support early childhood physical, mental, and emotional health

Promote the concept that “it’s ok to ask for parenting support,” young child wellness, and that resources are available through 211infoFamily system

Collaborate/do outreach to communities of color for public awareness activities

Conduct social marketing campaign to promote 211infoFamily

Conduct outreach to community organizations re: 211infoFamily

Collaborate with/do outreach to culturally‐specific organizations annually to do public awareness activities

Increased provider knowledge/awareness of LAUNCH, resources available (e.g., 211infoFamily, trainings)

Increased public/community awareness of LAUNCH, 211infoFamily and targeted early childhood messages

Increased knowledge amount culturally specific communities of early childhood mental health, social‐emotional development, and how to access resources (including 211infoFamily)

Greater public knowledge about early childhood mental health

Greater knowledge among provider community of resources, awareness of importance of early childhood mental health

Goal #2:

Increase community awareness of the importance of young child wellness and the resources available to support early childhood physical, mental, and emotional health

211infoFamily provides information and referral for calls regarding parenting/child development to the 211info system for children ages 0‐8

System tracks referrals and does follow up to ensure families get needed services

Community liaisons collect information and provide to 211infoFamily to ensure up‐to‐date and comprehensive information

Community liaisons do outreach and education to

Parents use 211infoFamily for parenting/child development issues

Parents from communities of color use 211infoFamily

Providers utilize 211infoFamily to find resources

Information in 211infoFamily is up‐to‐date

Increased use of 211infoFamily for parenting/child development issues

Increased use by communities of color of 211infoFamily

Increased number of providers use 211infoFamily for parenting‐related issues

100% of callers needing referral receive referral or referral information

Parents using 211infoFamily report satisfaction with services/information received

Decreased parent stress

Improved child wellness

Families’ needs met



communities of color and providers that serve these communities to inform/educate about 211infoFamily

Increased numbers of referrals for mental health/behavioral health for children

Service Delivery Level: Increase developmental screenings and integration of behavioral health in primary care settings

Goal #3:

Increase current screening, consultation, & training in primary health care, home visitation and child care setting to promote young child wellness

START provides primary care providers (PCPs) with training in conducting social‐emotional developmental assessments, maternal depression screening

New trainings on integration of behavioral health into primary care developed & implemented

Track State progress in recommending screening tool, and ensure alignment with LAUNCH

Work with local EI/ECSE to identify gaps in assessment, referral, and follow up

Train 60 PCPs/yr to do assessments & screenings

PCPs and community providers share ASQ results for 100 children.

PCPs get information back about referral outcomes

Develop and pilot integration of behavioral health module, develop on‐line version

Gaps in connecting families to needed EI/ECSE services identified

7,560 children screened for behavioral health in primary care.

Increased number of children with social‐emotional/behavioral delays are identified early and referred for appropriate follow up

Increased level of coordination between PCP and child‐serving agencies.

PCPs have a better understanding of early childhood mental health, atypical child development, and family risk

More children successfully connected to EI/ECSE services

More children show age‐appropriate development

Children enter school ready to learn



Service Delivery Level: Provide evidence‐based mental health consultation to home visitors, child care providers, parents

Goal #3:

Increase current screening, consultation, & training in primary health care, home visitation and child care setting to promote young child wellness

EC MHC and PBIS support provided in Healthy Start for home visitors, and childcare centers

Build the evidence base for EC MHC and PBIS for children 0‐3

7 home visitors (100 families) and 2 childcare centers (30 staff serving 100 children) receive mental health consultation (annually) and EC PBIS

Home visitors and child care providers increase confidence, skills and knowledge about infant‐toddler mental health (MH), MH best practices and EC PBIS

Child care classroom teachers provide higher‐quality environments for children

Children have fewer behavioral and social‐emotional issues/concerns; more positive social behaviors; are successfully linked to appropriate referrals

Improved parent‐child interactions

Parents have less parenting stress; less depressed; provide more developmentally appropriate environments for children

Staff less stressed; higher retention

More children enter kindergarten ready to learn

Fewer children removed from child care because of behavioral issues Train two Incredible Years

group facilitators

Provide Incredible Years parent training to parents of children enrolled in targeted Healthy Start or child care centers

Train child care providers and Healthy Start workers in EC PBIS model

15 parents receive Incredible Years annually

30 early childhood education (ECE) and child care providers trained in EC PBIS

2 child care centers implementing EC PBIS

Service Delivery Level: Increase resources and coordination for workforce development and training

Goal #3:

Increase current screening, consultation, & training in primary health care, home

Identify training needs among child‐serving professionals

Provide recommendations regarding workforce development needs

Increased number of providers trained in MH‐related issues

Creation of home visiting roundtable with

Providers more knowledgeable about children’s mental health and related issues

Workforce serving young children more knowledgeable about children’s social emotional development,



visitation and child care setting to promote young child wellness

Coordinate training for child wellness professionals

Facilitate and support training and workforce development opportunities for emerging communities of practice (EC PBIS and home visiting)

diverse program membership

More providers trained in evidence‐based early childhood program models

Home visiting community aligns goals and identifies gaps in services

Increased sharing of resources around training across agencies/organizations

mental health, and related issues

EC PBIS workforce development and training system is sustained

Home visiting system has improved coordination and communication within early childhood service system


Figure 1. Multnomah County Project High‐Level LAUNCH Logic Model

Young Child Wellness Council• Promote cross agency communication, planning, collaboration• Engage parent voice• Involve culturally specific organizations

Partnerships to improve referral & information sharing processes, leverage & share resources

Social MarketingSupport help‐seeking for parentsPromote concept of child wellnessPromote 211/Parent Help Line

211 Info/Parent Help LineWarm line parenting support

Mental health referrals

Pediatric Primary Care TrainingDevelopmental & Social‐Emotional

ScreeningFamily Risk & Maternal Depression

Mental Health Consultation and Positive Behavior Intervention &

Supports0‐3 home visiting

Child care

Workforce Development Training & Coordination

System

s Change

Service Enhancement

Activities & Interventions Outputs Short Term Outcomes

Long Term Outcomes

• Resource Inventory• Strategic plan • Family & cultural communities engaged

• Outreach to key organizations , providers, communities• Social marketing activities, materials

• Increased calls to 211/PHL• Increased referrals for child/family MH/well‐being

• 60 Pediatric Primary Care Providers trained/year• New risk & mental health modules developed; expand developmental and social‐emotional module

• 7 home visitors, 30 child care staff, 150 families/children’s receive MHC/PBIS/year• 15 parents/year receive Incredible Years

• Available trainings and training gaps identified• Training provided to fill unmet needs

+ Use of common referral forms+ Sharing of screening results

+ Community Buy‐In+ Increased agency collaboration+ Stakeholder knowledge of

services+ Community awareness of

parenting resources

+ children screened for development and social‐emotional well being

+ early identification of children at risk

+ referrals of children to needed services

+ provider knowledge, skills, support

+ culturally responsive, developmentally appropriate practices

+ job satisfaction‐ job stress

+ child development+ parent‐child interactions‐ parenting stress+ parent satisfaction with

services+ parents know where to get help & are confident in knowing how to meet child’s needs

System shows:

Reduced service duplication

Better ability to meet child/family

needs

More efficient use of resources

More culturally appropriate services

Sustainability

Children & Families:

Have more access to needed services

Are empowered

Are identified and supported early

Are school ready

Have improved child & family

wellness

Multnomah Project LAUNCH Logic Model

8/11/2011


Systems Change The systems level evaluation of Multnomah Project LAUNCH included both process and implementation evaluation as well as documentation of key outcomes. Below we present the key questions, methods, and results for the following key areas of systems work for the project. Following this, we present a summary of the overall systems‐level strengths and challenges both in terms of implementation of planned activities, as well as in achieving systems level outcomes.

Collaboration

Research Questions

Based on the Project LAUNCH logic model, the evaluation addressed the following key research questions related to systems collaboration and coordination:

1. Collaboration Implementation Research Questions:

a. To what extent were key systems‐level activities implemented as planned and outputs achieved?

b. What supported successful implementation of system‐level activities? c. What barriers were encountered to implementation?

2. Collaboration Outcome Research Questions: a. In what ways did LAUNCH increase understanding, communication, coordination, and

collaboration between providers in different sectors? b. What changes were there in the screening, assessment, and referral processes between key

partners? c. To what extent did LAUNCH successfully involve parents in planning, decision‐making, and

problem‐solving?

Methods & Data

Systems Evaluation methods were centered on collecting qualitative (Key Stakeholder Interview) and quantitative (Key Stakeholder Survey) data from key partners involved with LAUNCH. Additionally, information was gathered through participant observation of LAUNCH partner and YCWC meetings by members of the evaluation team, and through reviewing key LAUNCH documents, such as meeting minutes, deliverables (Environmental Scan, Strategic Plan, etc.).

In addition to the information collected through the YCWC and systems stakeholders, we also included questions on surveys collected from child care providers and home visitors who were participating in LAUNCH‐sponsored Early Childhood Mental Health Consultation (EC MHC) about the perceptions of collaboration and coordination with other community service providers, in particular, mental health providers. This provided another perspective on how LAUNCH was influencing cross‐sector collaboration at the provider level, as a key function of the EC MHC work was to help early childhood care providers and home visitors build their understanding of how to better connect families with mental health resources. These surveys also included questions about providers’ confidence in


involving parents in supporting their children’s mental health and development. Results from these surveys are also summarized below.

Key Stakeholder Systems Survey. Quantitative Key Stakeholder System Surveys (see Appendix A for instrument) were conducted with all members of the Young Child Wellness Committee and contracted partners four times during the LAUNCH project. The purpose of the survey was to assess changes in cross‐sector collaboration among key LAUNCH partners. The survey was based on an adaptation of the Early Childhood System Assessment developed for Arizona’s “First Things First” Systems Assessment. It included a variety of questions related to key aspects of collaborative group functioning (e.g., quality of leadership, communication, representativeness of membership, etc.) as well as specific questions related to early childhood systems change (e.g., quality of collaboration between specific sectors such as primary care, mental health, and early childhood; quality and availability of services for children and families, etc.).

At baseline, a total of 47 individuals (95% of the YCWC members) completed the survey, Members of Multnomah County’s Early Childhood Council (ECC), a larger policy making body that predated LAUNCH (n=20), and new YCWC members also completed the 12‐month survey (n=8). However, the ECC was undergoing membership changes during the survey time period, resulting in lower participation among its membership, compared to baseline. A total of 42 individuals completed the 12‐month survey, including 32 individuals who had also completed a baseline the year before. A third round of surveys was collected in Year 4, with 36 respondents. Seventeen participants responded to the survey at all three time points. In summer 2015, at the close of the project, a final round of surveys was collected, with 27 partners completing the survey. See Table 2 for sample characteristics.

Table 2. Characteristics of Key Stakeholder Systems Survey Respondents

Baseline Summer 2011

(N=47)

T2 Summer 2012

(N=42)

T3 Fall 2013 (N=36)

T4 Summer 2015

(N=27)

Sector

Primary Care 11% (5) 14% (6) 19% (7) 30% (8)

Mental Health 11% (5) 14% (6) 17% (6) 22% (6)

Family Support 38% (18) 31% (13) 25% (9) 11% (3)

Early Care & Education (ECE) 17% (8) 21% (9) 19% (7) 19% (5)

Other 19% (9) 19% (8) 17% (6) 15% (4)

Parent 4% (2) 0 3% (1) 4% (1)

Race/Ethnicity

African American/Black 6% (3) 2% (1) 6% (2) 0

Latino 2% (1) 2% (1) 0 7% (2)

Asian American/Pacific Islander 2% (1) 0 3% (1) 0

American Indian/Alaska Native 2% (1) 2% (1) 3% (1) 0

Multiracial 9% (4) 88% (37) 83% (30) 93% (25)

White 79% (37) 5% (2) 6% (2) 0

Matched Surveys BL to T4 na N=32 N=18 N=8

Key Stakeholder Interviews. In addition to the more quantitative Key Stakeholder Survey, qualitative telephone interviews were completed at the end of Year 1 (Fall 2011) with 34 YCWC members and


contractors, in Year 3 (late Fall 2012) with 23 YCWC members, and again with 19 stakeholders in Year 4 (Fall 2013 and Winter 2014). These interviews were designed to gather more in‐depth information about systems barriers, facilitators, strengths, and challenges. Surveys were modified each year to assess various systems‐level issues that emerged throughout the work. The Year 4 survey protocol is included in Appendix B as an example.

The last round of key stakeholder interviews was conducted during Year 4 of the project. These included 19 key stakeholders, and were content coded and analyzed to better understand the barriers and facilitators of systems‐level changes through LAUNCH. Results were shared with the YCWC to provide background/context for specific, topical discussions at several meetings. Qualitative interviews during this round of data collection focused on understanding the key successes of LAUNCH at the systems level during Years 1‐4, areas that had proved challenging over the course of the project, and on identifying strategies for sustainability moving into the final year of LAUNCH activities. Key findings are summarized below.

Findings: Collaboration Implementation Evaluation

The foundation for LAUNCH’s YCWC was the network of community collaborations that were supported by the grant. As such, we documented the number of new collaborative relationships that were developed each year through the YCWC. This information is summarized in Table 3 below. As can be seen, LAUNCH exceeded its targeted number of new collaborations each year. Qualitative data (see below) supported the notion that LAUNCH was successful in bringing new partners together in a collaborative forum that was unlikely existing collaborative structures.

Table 3. New Collaborations Established Through LAUNCH Y1 Y2 Y3 Y4 Y5

New collaborations established through LAUNCH systems and service partners 50 29 29 27 13

Target number of new collaborations 40 10 10 20 10

Successes: Developing Partnerships & Increasing Collaboration Survey respondents were asked about whether, and to what extent, they had developed new partnerships, or strengthened existing inter‐organizational connections, through Project LAUNCH. Most answered affirmatively, indicating an increase in the size of their partnership networks due to Project LAUNCH. A few also discussed how participating in LAUNCH had helped to deepen existing partnerships.

The most frequently mentioned new connections included those that are most central to the mission of Project LAUNCH, including:

Representatives from the primary health care sector citing new partnerships with early childhood providers and other community agencies

Early childhood providers noting new partnerships with, and increasing understanding of, the medical community

Greater understanding of, and value for, the parent helpline through 211infoFamily


Greater understanding of EC PBIS and increased partnerships between child care, home visiting and mental health through mental health consultation

Stakeholders described the key role that has been played by the Young Child Wellness Council in providing a forum for sharing information about these various cross‐sector providers and generating discussion across sectors. Several also mentioned specific workgroups, such as the Adverse Childhood Experiences (ACEs) workgroup, as being important in strengthening these partnerships.

Systems Successes: Linking to New Systems Change Initiatives at the State & Local Levels In the previous round of systems interviews, one of the key issues that was mentioned as critically important for Multnomah Project LAUNCH in the upcoming years was to integrate with emerging systems changes happening at the state level through the restructuring of Oregon’s Health Care (through Coordinated Care Organizations, or “CCOs”) and Early Childhood systems (through the new Early Learning Hubs, or “Hubs”), as well as through emerging new local collaborative efforts focused on strengthening school readiness and other key child and family outcomes.

Respondents this year were asked to comment on how well LAUNCH was coordinating with these other efforts, as well as to describe the benefits to LAUNCH in strengthening these connections. In terms of how coordination was happening, by far the most frequent responses focused on the active work of the Young Child Wellness Coordinator in serving as a liaison between these other efforts and the YCWC. Stakeholders also mentioned the opportunities for mutual learning created by having some of the key individuals involved in these other initiatives share information and updates at YCWC meetings.

Respondents strongly and unambiguously felt that LAUNCH has important lessons to share with these other efforts in terms of specific strategies. In particular, stakeholders felt that lessons learned from LAUNCH work in mental health consultation in home visiting, integration of mental health consultation and ECBIS, START trainings and other workforce development efforts, and implementation and marketing of the 211infoFamily service were mentioned, as well as LAUNCH’s overall approach in terms of specifically targeting and working to connect with primary care. One clear success in terms of building on LAUNCH success was mentioned: the Early Learning Hub’s decision to establish (with LAUNCH support) Communities of Practice to support workforce development in early childhood mental health and EC PBIS.

Stakeholders also felt that there were lessons to be shared in terms of the multiple, albeit mostly unsuccessful, attempts at parent involvement implemented by LAUNCH. That is, sharing what hadn’t

“First and foremost and I can’t underscore this enough, I think that what LAUNCH has done and what any type of grant like this does, it helps build and connect or illuminate relationships and we don’t give enough credence to fostering relationships and that’s what affects being able to do further initiatives like this. We really need to have healthy relationships and so I think although those relationships might have been there from the ECC, from the Commission anyways, LAUNCH continued to nourish those relationships in a time where there was a breakdown in other places that couldn’t continue to nourish them .”

– Systems Interview Participant,


worked as an important way to help these other efforts be more successful in this area. Finally, respondents also felt that linking to these other initiatives was critical to LAUNCH sustainability, both in terms of maintenance of collaborative partnerships developed by LAUNCH, as well as in terms of sustaining key LAUNCH investments such as mental health consultation and 211infoFamily (see below).

Systems Implementation Challenges: Parent Involvement A key goal for Project LAUNCH was to actively engage parents in the YCWC for the purpose of providing parent “voice” in decision making around how to better support families. Thus, collaboration and partnership with parents, and the extent to which LAUNCH and LAUNCH‐funded partners were involving parents was a key aspect of the evaluation. Table 4 below summaries the number of parent participants in YCWC meetings.

Table 4. YCWC Membership Y1 Y2 Y3 Y4 Y5

Annual average number of YCWC members 21 18 17 19 21

Annual number of YCWC or work group members who were parent representatives 2 2 12 1 0

Annual percent of YCWC or work group members who were parent representatives 10% 8% 25% 5% 0

Target percent of YCWC or work group members who were parent representatives 15% 20% 20% 20% 20%

As can be seen, the number of parents involved in the YCWC was quite low, except during Year 3. This was an ongoing challenge for Multnomah Launch, as described below. The relatively large number of parents involved during Year 3 was due to a shift in strategy. After two years of struggling to involve parents, Project LAUNCH acknowledged that its original approach to involving parents representing communities of color was not working and piloted a new approach in Year 3. The original strategy had been to provide funding to agencies representing communities of color to participate on the council, and to identify and mentor a parent representative from their agency. However, continued efforts to involve parents have not been successful. In Year 3, a separate parent advisory group was formed, with its own dedicated facilitator. This group met several times during year 3 to provide input on particularly LAUNCH data and plans. It was disbanded at the beginning of Year 4, however, in order to link with the emerging Early Learning Hub’s parent advisory council. However, due to the slow start up of the Multnomah Early Learning Hub, little progress was made in this area during Years 4 and 5.

Findings: Collaboration Outcomes

Key systems outcomes for Multnomah County LAUNCH that were examined included:

1. Increased communication, collaboration, and coordination between agencies and partners; 2. Changes in the screening, assessment, and referral process between partners; and 3. Changes in family services availability, access, and ability to involve parents in service delivery

decisions, especially related to children’s mental health issues.

Results of the Key Stakeholder Systems Survey are shown in Figures 2 and 3 below, and address stakeholders’ perceptions of the changes in inter‐organization, cross‐sector communication, collaboration and coordination. Because of turnover among key stakeholders in the YCWC, we used


the data to compare stakeholder’s perceptions at baseline and Time 4 at the aggregate systems level. T‐tests comparing the average stakeholder perceptions at baseline to the average perceptions at follow‐up were conducted. Detailed results (means, standard deviations, etc.) for each of the four primary domains are shown in Appendix C.

As can be seen, representatives from various sectors had different perceptions of the overall quality of relationships across sectors. Representatives from Mental Health, Family Support, and ECE who participated in LAUNCH generally reported increased quality of collaboration with the Primary Care sector over the four‐year period. There were fewer changes in terms of the quality of collaboration with the Mental Health Sector, and in fact, Early Childhood representatives rated these relationships less favorable in Year 4, compared to baseline. Representatives from ECE and Mental Health agencies reported the largest improvements in collaboration with other sectors (with the exception of Mental Health).

These results may speak to the opportunity provide through Project LAUNCH to facilitate direct communication and relationship building between ECE, MH, and Family Support and the primary care sector, something that many noted was unique to Project LAUNCH. Other child and family serving sectors in Multnomah County had a relatively long shared history of collaboration, while direct work with the Primary Care sector was new to many partners.

Figure 2. Mean Ratings of Cross‐Sector Collaboration by Sector

3.02.8

3.0

2.4

3.5

3.8

2.7 2.7

3.33.3 3.3

1

2

3

4

BL T4 BL T4 BL T4 BL T4

Primary Care Mental Health Family Support ECE

Collaboration with Primary Care

Collaboration with Mental Health

Collaboration with Family Support

Collaboration with ECE

“I have a greater understanding of the range of support organizations and their roles in early childhood development and family support.”

– Systems Survey Respondent, Primary Care sector


Figure 3 below shows the overall ratings across the five years in terms of the level of functioning of the YCWC; the project progress in achieving LAUNCH goals, overall cross‐sector collaboration, and quality of services for families. Results comparing baseline to Year 4 showed significant increases only for the quality of services for families, which were seen as improving significantly during the 5 years of Project LAUNCH. There was also a marked increase in the quality of cross‐sector collaboration from baseline to Year 4, although this did not reach statistical significance. Ratings of the functioning of the YCWC, as well as its ability to advance LAUNCH goals, were relatively high across all four years of the project.

Figure 3. Mean Ratings of Systems Survey Areas

*Difference in mean rating between BL and T4 was statistically significant (p <.05) for Quality of Services for Families.

Successes: LAUNCH Influences on Practice Stakeholders were asked to describe whether, or to what extent, their own work, or the work of their agency, had changed because of Project LAUNCH. Most frequent responses included:

An increased emphasis in their work on understanding children's mental health issues (especially for medical providers);

Greater attention to issues of trauma, including using a trauma informed perspective, participating in trainings related to understanding early childhood trauma and its consequences, and working on changing approaches to screening through the ACEs workgroup;

Increased fidelity to the EC PBIS model, and expansion of the EC PBIS model to new settings such as school, domestic violence shelters, and home visiting;

Expansion and prioritization of funding for mental health consultation in home visiting;

4.2

3.6

2.4

4.4

3.9

4.2

2.6

3.8 3.9

4.3

3.1

4.3 4.2 4.1

3.1

1

2

3

4

5

YCWCFunctioning

Ability to AchieveLAUNCH Goals

Collaboration betweenOrganizations

Quality of Servicesfor Families*

Baseline T2 T3 T4

“I think [the YCWC] has been an excellent collaboration. Time, relationships, joint work improves practice across our agencies for families in our community.”

– Systems Survey Respondent, ECE Sector


Integration of Child Care Resource and Referral information into trainings for primary care physicians.

In terms of changes within these organizations, most described the role of LAUNCH as one of helping them to gain a deeper understanding of mental health, trauma, physical health, and related issues and how the work occurring in different sectors needs to be integrated:

Findings: Improving Coordination of Screening and Referral Although a major goal for Multnomah LAUNCH was to improve the coordination of developmental screening, assessment and referral processes, it was an ongoing challenge throughout the life of the grant. Considerable information‐sharing and discussion around ways to improve these processes occurred during YCWC meetings. For example, during Year 4, several YCWC meetings focused on discussion of what it would take to develop more integrated systems and to improve the current system.

A group of interested partners then explored and shared data that had been collected by the local EI/ECSE provider regarding the current screening and referral system between primary care and Early Childhood Special Education/Early Intervention. Data suggested that many families were “falling through the cracks” between referral, evaluation, and engagement in services. The YCWC identified key questions about how to dig deeper into these data to identify ways that the system could be improved and held a series of meetings between the EI/ECSE provider, primary care physicians, and other partners to discuss the data and brainstorm ideas for improving the system. Further, the YCWC was invited to provide feedback to health care providers who are involved in developing what could be a universal “family well‐being” screening used by primary care to link families to needed community resources.

Moreover, the Child Health Director of the Oregon Health Authority participated in a YCWC meeting on this topic and has had her staff participate in YCWC meeting since then, helping connect LAUNCH with state and local early childhood and health care systems to increase rates of developmental screening. A number of YCWC members also participate in various groups working on screening‐related issues. During Year 5, a conscious decision was made by the YCWC to facilitate these conversations and to ensure that other existing collaborative groups were “taking up the cause” of referral and screening issues.

Findings: Home Visitor and Child Care Providers Perceptions of Community Linkages Another way of exploring systems change outcomes was through the work of the home visiting and early childhood providers receiving EC MHC. A key function of the EC MHC services was to increase

“I’ve seen a move for a more holistic understanding of how kids and families move through these early childhood systems; understanding that pediatricians and preschool they’re not all kind of separated, that we need to have each area of a child’s life in 0‐5 age range understand how their systems and policies affect other areas; I think bringing other people to the table, to make it easier for families, to get more information in more places so parents don’t have to hunt to find the things they need.”

– Systems Interview Participant


knowledge of community resources and to support better connections with mental health services. Thus, the survey of home visitors and child care providers also included questions about their confidence and knowledge of community resources and linkages to mental health and primary care providers. Data related to changes in these indicators were collected at 6, 12, 24, and 36 months. Figure 4 below shows the home visitors’ and ECE providers’ reports of their current community linkages at baseline and at each follow up time point (note that these figures represent ‘snapshots’ of providers who were surveyed at each time point, rather than individual changes over time). As can be seen, over time, these staff generally felt more knowledgeable and connect to mental health services.

Figure 4. ECE Provider and HV Staff Perception of Service Linkages

*Difference in mean rating between BL and 12/24/36 months was statistically significant (p <.05) for ECE provider knowledge.

Systems Challenges: Culturally Responsive and Culturally Specific Services

A key challenge at the systems level for Multnomah Project LAUNCH has been how to improve access to and availability of culturally responsive and specific services. During the last round of systems interviews, this emerged as one area in which stakeholders felt LAUNCH had made less progress than originally hoped, while at the same time noting the challenges in making progress in this area. This year, stakeholders again described this as a challenging area, although some important successes were also noted:

3.3

2.7

3.5

2.9

3.7

3.1

4.0

3.5

4.3

3.7

1.0

2.0

3.0

4.0

5.0

HV Staff ratings ofcommunity services linkages

ECE Provider knowledge ofmental/behavioral health services*

Baseline 6 Mo 12 Mo 24 Mo 36 Mo

“It’s such a complicated problem we all face, even in the most basic ways there needs to be shared power and leadership with diverse communities, really creating a shared power structure, not just with people of color, but many specific populations and having a role in decision‐making and leadership, not thinking about culture and identity as a separate add‐on thing, but that they need to be more central from the ground up…I think about the YCWC and while retention is always an issue, it seemed that it was the population‐specific people who fell off.”



Stakeholders also made a number of suggestions in terms of ways that LAUNCH partners could take steps towards improving this area, including:

Work with LAUNCH partners to promote issues of equity and accessibility.

Bring data related to health disparities and differences in outcomes and services provided for culturally diverse groups “to the table” for discussion.

Evaluate effectiveness of culturally‐specific services and build the evidence base for these services.

Work with 211info Family to enhance this aspect of services by:

o Hiring more diverse parent education specialists at 211infoFamily,

o Continue to improve the 211infoFamily inventory in terms of information for culturally specific programs and supports, and

o Make 211infoFamily look‐up easier to access information about culturally‐specific services.

Overall, partners continued to emphasize that making progress in this are requires persistent, repeated efforts, and hoped that LAUNCH and LAUNCH partners would continue to bring this issue to the forefront. Focused outreach to culturally diverse networks, nontraditional avenues for engaging families, and other approaches were seen as important for moving the dial in this area:

Successes: Parent Involvement in Service Delivery We included questions on the Key Stakeholder Survey about the extent to which YCWC and other key stakeholders’ own agencies involved parents in service delivery and planning. There were no statistically significant changes in the extent to which key stakeholders reported involving parents actively from baseline to follow‐up, but only 13% reported not doing this at all, 25% reported just starting to do this, and a 63% reported somewhat or actively doing this at 24 months.

Parent involvement is also a key value in the mental health consultation model. Therefore, we asked home visitors and child care providers on the provider survey (see Workforce Development section for description) at baseline, 6, 12, and 24 month follow‐up, to report on the extent to which they involve parents in partnerships to address children’s challenging behaviors and mental health issues. Both groups of providers reported significant increases in their overall level of confidence in involving

“I would ask if they are thinking about their work with an equity lens and are the services they are supporting, do they have true cultural relevance.” – Systems Interview Participant

“Effectively addressing equity and health disparities is our biggest challenge. I think we just have to keep talking about it. It has to continuously come up on the agenda and be a persistent area that we pay attention to. If it’s a standing part of discussions on where things are going or how results differ by communities, like looking at data for a health equity perspective…but it’s not a tag on thing at the end, it really needs to be at every stage in the process, more woven in.”



parents around these issues as shown in Figure 5. Note, however, that these data present a “snapshot” of providers’ levels of confidence at each time point, rather than changes over time within the same group of providers.

Figure 5. ECE Provider & HV Staff Confidence in Involving Families

*Difference in mean rating between BL and 24/36 months was statistically significant (p <.05) for HV staff. **Difference in mean rating between BL and 24 months was statistically significant (p <.05) for ECE providers.

Summary: Collaboration & Systems Coordination

In sum, Multnomah Project LAUNCH had several major successes in terms of increasing the number of organizational partnerships, creating new partnerships between primary care, behavioral health, and early childhood sectors, and building a foundation for ongoing cross‐sector collaborative work. While some goals, including making measureable and clear changes in the system for developmental screening, assessment, and referral, were not accomplished, partners had multiple opportunities to learn from each other about how they approached developmental screening with families, and examined data that helped to identify places where the system could be strengthened. LAUNCH also successfully connected primary care and child care and early childhood care and education partners in ways that may have created a foundation for making further systems changes as the evolving landscape of health care reform and early childhood system coordination in the County and State move forward.

Workforce Development Multnomah Project LAUNCH implemented several strategies that focused on workforce development (WFD) among primary care providers, early childhood, and mental health sectors; several of these strategies overlap considerably with service delivery activities as well. For example, there were significant investments by LAUNCH in the START program for training pediatric and primary care staff to implement developmental screenings through the START program. Therefore, the number of providers trained, and the changes those providers attributed to the training, is included in the

3.7 3.6

4.2

3.7

4.23.9

4.6

4.1

4.7

4.0

1.0

2.0

3.0

4.0

5.0

HV Staff* ECE Providers**



systems‐level section below. Outcomes of the workforce development in terms of actually conducting screens, are included in the “service delivery” section of the report.

Overall, LAUNCH’s approach to WFD changed and adapted over the five years, in response to emerging needs of the community as well as to the changing nature of the Early Childhood workforce developmental and training context. Originally, the project work focused on large‐scale trainings and convenings, and linking, through LAUNCH, providers’ ability to share these common resources. The Young Child Wellness Summit in December 2011 drew more than 300 cross‐sector participants and build connections among providers, while also delivering high‐quality training. While a few large‐scale training events were supported by LAUNCH in later years, the work was primarily focused on building Communities of Practice (CoPs), and on supporting specific networks for providers and trainings related to key LAUNCH activities. Specifically, CoPs related to home visitation, Early Childhood Mental Health, and an EC PBIS coaches network were directly supported by LAUNCH.

In the final two years of the project, LAUNCH‐coordinated trainings primarily focused on providing peer support, networking, and training opportunities for a group of mental health consultants in Multnomah County who also serve as EC PBIS coaches in early childhood settings. Feedback on the value and utility of these Coaches Group sessions was assessed through brief surveys in Fall of 2013 and Spring 2014. Workforce development efforts also included eliciting feedback from an expanded EC PBIS Stakeholder Group of early childhood program administrators and staff implementing EC PBIS in their settings. These results, and the associated sample sizes, are reported in the appropriate section of “Findings,” below. Workforce also completed the adaptation of the EC PBIS Module 1 for implementation in domestic violence shelters and facilitated the development of communities of practice, focused on EC PBIS and home visiting professionals.

Research Questions

1. WFD Implementation Research Questions: a. To what extent were key WFD activities implemented as planned and outputs achieved? b. What supported successful implementation of WFD activities? c. What barriers were encountered to implementation?

Methods & Data

Data for the process and systems‐level evaluation of the workforce development component were collected from several sources: (1) training activities provided by mental health consultants reported on consultant Monthly Activity Reports; and (2) perceptions of key systems stakeholders of the availability, accessibility, and extent of coordination of early childhood/mental health/behavioral health‐related trainings in Multnomah County (collected by the Key Stakeholder Systems Survey); and (3) surveys of training participants conducted for each Launch‐sponsored training event (note that training provided to primary care providers for supporting developmental screening in primary care are not reported here, but included in the Screening and Assessment Service section of the report).

Outcomes of workforce development activities (such as changes in provider knowledge, attitudes, behaviors, and work settings) are primarily embedded in evaluation tools used to collect other outcomes related to core service strategies (START training surveys, 211infoFamily, MHC Provider Outcomes/Mental Health Consultation Survey for Home Visitors and Child Care Providers. These surveys are described in more detail in the specific Core Strategies (Service Delivery) section of this


report. In addition, post‐training surveys are collected from participants who attend trainings coordinated through LAUNCH.

Additionally, a series of qualitative interviews as conducted in Year 5 with members of the LAUNCH‐supported “Communities of Practice” (CoPs). These comprised a more in‐depth examination of this aspect of the WFD strategy to gauge their success and sustainability.

Because of the large number of different WFD activities provided, we present the outcome findings for each separately below. Several WFD activities had embedded process evaluation data collection; for these training activities, we include that information within the activity‐specific section below. Following this, we summarize key overall WFD implementation and outcome successes.

Findings: WFD Implementation

Table 5 below summarizes the number of providers who participated in various types of trainings supported through Project Launch. As can be seen, with the exception of Year 1, LAUNCH greatly exceeded the targeted number of providers trained.

Table 5. Workforce Development Efforts & Number of Participants1 Training Topics & Events Y1 Y2 Y3 Y4 Y5

MHC & EC PBIS Total includes: Reflective group supervision (Y1‐5) Infant‐Toddler modules 1‐2 (Y2‐5) Preschool modules 1‐2 (Y2‐5) Home Visiting Family Coaching modules 1‐2 (Y3‐5) Coaches group (Y3‐5) EC PBIS Module 1 adapted for domestic violence shelter staff (Y4) TPOT reliability training (Y4) TPITOS reliability training (Y5) EC PBIS Toolkit training (Y5) 25 153 362 427 444

Young Child Wellness Summit (Y2) Total includes: 12 community trainings, including (Y2): Trauma‐Informed Care Motivational Interviewing Intersectional Identity Social‐Emotional Development of Children Early Literacy Reflective Supervision ‐‐ 851 ‐‐ ‐‐ ‐‐

ASQ training for family child care (Y4) ‐‐ ‐‐ ‐‐ 160 ‐‐

Communities of Practice Total Includes: Home visiting (Y4‐5) EC PBIS (Y4‐5) Early childhood mental health (Y5) ‐‐ ‐‐ ‐‐ 39 97

3 community trainings, including (Y5): ‐‐ ‐‐ ‐‐ ‐‐ 705

1 These numbers include an unknown number of duplicated individuals. Provider‐level information and participation was tracked when possible with focused workforce development efforts or training series, for example, but the level of duplication within and across years is unknown for one‐time events.


Training Topics & Events Y1 Y2 Y3 Y4 Y5

Diagnosis & assessment in early childhood, infant mental health Reflective practice Trauma‐informed care with a cultural lens

Total number of participants 25 1,004 362 626 1,246

Target number of provider participants 250 1,188 400 400 400

During Years 1 and 2, LAUNCH WFD focused on providing large‐scale community training conferences on a variety of topics, as well as cross‐sector convenings and workshops. However, during Years 3 through 5, the workforce development efforts took a different form, and focused less on one‐time trainings and events, and more on focused areas of growth for smaller groups of providers over time. Further, recognizing the need for WFD that providing ongoing support for peer learning, LAUNCH supported three CoPs at the County level in the areas of home visiting, mental health consultation and EC PBIS.

As the CoP work as emerged, the WFD team had to significantly shift their role and work related to LAUNCH, with an increasing emphasis on supporting the systems that support the workforce in Multnomah County (e.g., the EC PBIS leadership team and the CoPs). Further, the WFD emphasis began to more proactively work towards building sustainability of EC PBIS with early learning programs. For example, LAUNCH supported a Teaching Pyramid Observation Tool (TPOT) reliability training for mental health consultants and early learning program staff who serve (or hope to serve) as EC PBIS coaches. Of 46 participants in the 2‐day training, 100% became certified as a reliable TPOT observers. In an effort to expand the implementation of EC PBIS, the WFD supported work that adapted the EC PBIS Module 1 for use in domestic violence shelters. This adaptation also made EC PBIS more trauma‐informed. The adapted module was piloted with staff from two domestic violence shelters, revised, then delivered again and video‐recorded in order to make the module available online.

Findings: WFD Outcomes

Training Outcomes: EC PBIS Module Trainings Table 6 below shows the average ratings for participants’ responses on post‐training surveys conducted ECPBIS trainings held with ECE providers. As can be seen, responses generally indicated that participants felt they had increased their knowledge, skills, and confidence after participating in these trainings (average between “Very Good” and “Excellent).

Table 6. Participant Rating of Workforce Development Events Mean (sd) Higher score indicates strength Y1 Y2 Y3 Y4 Y5

EC PBIS Module Trainings (N=98) (N=184) (N=239) (N=325)

EC PBIS module post‐training mean rating of increased knowledge, skills and confidence (4‐pt scale) ‐‐ 3.8 (.4) 3.8 (.7) 3.7 (.4) 3.8 (.4)

EC PBIS Community Trainings (N=86) (N=8) (N=70) (N=51)

EC PBIS community training post‐event mean rating of increased knowledge, skills & confidence (5‐pt scale) ‐‐ 4.5 (.7) 4.4 (.4) 4.4 (1.0) 4.4 (.5)


Mean (sd) Higher score indicates strength Y1 Y2 Y3 Y4 Y5

Other Community Trainings (N=540) (N=315) (N=303)

Other community training post‐event mean rating of increased knowledge, skills & confidence (5‐pt scale) ‐‐ 4.3 (.6) 4.7 (.5) ‐‐ ‐‐

Other community training post‐event mean rating of usefulness of content (5‐pt scale) ‐‐ ‐‐ ‐‐ ‐‐ 4.2 (.9)

Training Outcomes: EC PBIS Coaches Group Table 7 and 8 below show the results from participant surveys for the ECPBIS coaches group and related TPOT trainings conducted during Years 3‐5. Again, these ratings indicated that participants felt these LAUNCH‐sponsored training opportunities helped to increase their confidence, skills, and knowledge. It also appears that participants felt more confident in their skills in terms of working with early care and education (ECE) providers in preschool aged settings, as compared to those focused on infants and toddlers. Thus, the EC PBIS coaches group appears to have been a worthwhile investment in supporting training for new and emerging EC PBIS coaches.

Table 7. EC PBIS Coaching Group Impact Post‐training evaluation mean (sd) 5‐pt scale, Higher score indicates strength

T1 (N=12)

T2 (N=14)

T3 (N=10)

I am confident I know how to help early childhood providers implement EC PBIS in settings with 0‐3 year olds. 3.7 (.5) 3.0 (.8) 3.5 (.5)

I am confident I know how to help early childhood providers implement EC PBIS in settings with 4‐5 year olds. 4.0 (.6) 3.6 (1.2) 4.1 (1.4)

I am confident in my role in helping program administrators buy‐in, implement, and sustain EC PBIS. 3.8 (.6) 3.5 (1.2) 3.5 (.5)

I am more connected with other EC PBIS coaches. 4.3 (.5) 4.3 (.5) 4.1 (.6)

I am a more effective EC PBIS coach. 3.9 (.7) 3.8 (.9) 3.8 (1.0)

Table 8. TPOT Training Post‐Evaluation & 12 Month Impact Post‐training (N=43) 5‐pt scale, Higher score indicates strength

Mean (sd)

I can use the new TPOT in classroom settings for program improvement 4.0 (.4)

I understand how to use the new TPOT scores in goal setting with teachers 3.6 (.7)

I understand how to score the new TPOT 3.9 (.5)

12 Month follow‐up (N=13) Count

At 12‐month follow‐up, how many TPOT observations did you complete in:

Winter 2014 39

Spring 2015 27

“Great networking. Learning what is happening at other EC sites helps inspire and motivate my work.”

– Coaches Group Participant on the

benefits of the group


When asked how the group has made a difference for them in Spring 2014, respondents also shared:

Training Outcomes: EC PBIS Module 1 Adapted for Domestic Violence Shelter Staff In Year 4 Multnomah LAUNCH supported the development of the adaptation of the EC PBIS Module 1 training for staff working in domestic violence (DV) shelters. In collaboration with the local Defending Childhood Initiative (DCI) in Multnomah County. The module was piloted with staff from two DV shelters in November 2013, and offered again in June 2014 for new staff. The second training was also videotaped for future use with new hires, and for disseminating with other DCI grants nationwide. In November 2013, a total of 44 staff attended at least 1 of 2 Module 1 sessions. The majority (72%) attended both sessions. Post‐tests were collected at the end of the second session, participants were asked to rate their level of comfort and confidence in EC PBIS skill areas. These results are presented in Figure 6 below.

Figure 6. EC PBIS Module 1 in DV Shelters Training Feedback

EC PBIS Module in DV Shelters: Process & Implementation Feedback In April‐June 2014 the DCI evaluation team also conducted on‐site observation of two participating domestic violence shelters, focus groups conducted with shelter direct‐service staff, and comments received to open‐ended questions on post‐training follow‐up surveys:

23%

17%

30%

13%

70%

77%

57%

77%

0% 20% 40% 60% 80% 100%

I increased my understanding forsupporting children's ability to learn rules

and routines

I can identify strategies that can be usedto build positive relationships with

children

I have increased my comfort andconfidence in working with children with

challenging behaviors

I understand the important role all staffplay in using positive adult‐child

interactions to support children in shelter

Somewhat Agree Strongly Agree

“More confident in coaching PBIS as opposed to just implementing PBIS. I am relatively new to this position and need all the help I can get.”

– Coaches Group Participant on the benefits of the group

“Collaboration was the most important aspect of the coaches’ group meetings. Connecting with others who are encountering similar struggles and brainstorming best practices.”

– Coaches Group Participant on the benefits of the group


Since the [EC PBIS/DV Shelter] pilot training, some staff report there has been a “180o shift in staff attitude” to using a more trauma‐informed approach. Staff are more comfortable and positive working with families: they have “more tools in the toolkit.”

DV shelter staff like the inherent structure of EC PBIS:

o Use of consistent language (e.g., reinforcing the message of the “3 B’s: Be safe. Be friendly. Be a hard worker.”). They now talk in staff meetings about developing common phrases and ways to re‐frame how they talk to and about kids. [What else do they need? Relevant language and approaches to use with older children and teens!]

o Establishing physical boundaries and routines using visual cues (signs, charts, etc.). Continuously reassessing the environment to identify what is (and is not) accessible to children, and adapting the space instead of trying to re‐shape the behavior. [Example: Some kids liked pushing the hand sanitizer dispenser to watch the gel come out. So staff moved the dispenser higher up on the wall where it was harder to reach.]

“Calling out” kids’ good behavior is becoming the norm. Staff routinely use “praising phrases” and stickers to reinforce children’s pro‐social behavior. [“Giving out stickers each time we see a child doing one of the 3 B's has been very effective. Many of the children are excited by this and want to earn more stickers every day!”]

Some staff regularly observe how youth and moms are interacting, then intentionally check in with each other to make sure they are viewing the family’s goals as a unit. Some also actively engage parents to explain what approaches they are using with children and why.

Setting aside a quiet play area for children in advocate/counselor meeting rooms makes meeting with a parent more effective and efficient. Using different lighting options in a room noticeably changes the mood.

Use of EC PBIS in shelter is consistent with PBIS being used in nearby schools (mutually reinforces in both settings).

Hiring processes for direct‐service shelter staff now include candidate questions about comfort level and experience in working with children/youth.

Parents are modeling pro‐social, relationship‐building strategies used by staff: e.g., reflective listening, positive language, re‐direction, narration, etc. Some staff believe the EC PBIS strategies are more effective when used in the common areas where other moms are present.

From follow‐up surveys: Of shelter staff who participated in the training and responded to a 6‐month follow‐up survey (38% of attendees):

More than 73% indicated they “mainly work with” families (vs. only with adults or only with children).

More than 80% reported that in working with young children they use EC PBIS strategies several times a week or on a daily basis. Specific strategies used by more than 75% of respondents included: o Established or maintained behavior rules, schedules and routines


o Used visual cues to establish boundaries o Re‐organized space or storage in a room to make it more orderly.

Strategies used by more than 85% of respondents to build positive relationships with young children included: o Greeted each child by name o Interacted with children at eye level [100%] o Focused attention on the child during interactions with them: e.g., eye contact, facial

expression, body language [100%] o “Reframed” their perception to take a more positive view of children’s challenging

behaviors o Followed a child’s lead in a conversation or interaction o Used positive language to characterize a child’s behavior o Used “narration” to describe back to a child what they are doing at the moment o Used empathetic statements to model positive behaviors.

Strategies used by more than 85% of respondents to respond to young children’s challenging behaviors included: o Recognized a child’s pro‐social behavior, efforts, or accomplishments o Provided children with clear expectations and examples of appropriate behavior.

[One respondent said their co‐workers “Developed a ‘catch phrase’ matrix to help staff more consistently respond to challenging behaviors.”]

More than 90% of respondents: o Developed a Rules Matrix with co‐workers (e.g., “3 Bs: Be Safe, Be Friendly, Be a Hard

Worker”) o Discussed with co‐workers ways to enforce rules consistently o Discussed EC PBIS concepts at staff meetings.

The most effective techniques identified by staff included: o Verbal communication skills; common language/phrases (3B’s); narration o Visual cues o Timers and solution cards o Reflecting a child’s emotions back to them o Redirecting challenging behaviors and using positive language o Getting down to eye level.

The least effective techniques identified by staff included: o After a brief learning curve, “all techniques…have been extremely effective.” o [Note: A few respondents who do not interact often with children in shelter were uncertain

or tentative about their skill in using verbal techniques such as emotional reflection. “I simply have to practice this more.”]

The most challenging situations for staff involving young children include: o “Assisting parents when they are frustrated with their children, in a way that doesn't feel

like I am judging their parenting styles.” [Note: This sentiment was expressed by several staff on more than one occasion.]


o “It continues to be challenging to support [children] who are struggling with trauma when their moms are not present and supportive, physically or emotionally, in part due to their own trauma.”

o Helping some moms stay engaged with their children. o Getting children to comply with guidelines they are least happy with…. o Keeping children from exiting doors or gates by themselves despite signage. o Environmental changes or external disruptions beyond staff control. o Implementing these strategies across a wide range of age groups at the same time. o Running in hallways despite frequent admonitions to “please use your walking feet.” o Preparing children for transitions. o Getting children to listen to directions.

ASQ Training for Child Care Providers In Year 4, Multnomah LAUNCH supported the ASQ Research Project at Child Care Resource & Referral of Multnomah County (CCRR‐MC) to assess outcomes of the training efforts as well as provider perspective regarding the tool. More specifically, after providers attended the trainings were they able to implement the ASQ‐3 as a developmental screening tool in their child care setting. Additionally, questions such as how many screenings did they do and how often, barriers to using the screening tool, positive outcomes to the implementation and technology use in their programs was surveyed. Approximately 160 child care providers have been trained by CCRR‐MC staff since February of 2013. The class, “Using the ASQ‐3 and ASQ‐SE in your Child Care Program” was offered on site at the CCRR‐MC as well as via private trainings held at child care centers or other venues, upon request. Each training offered an overview of the ASQ‐3, outlined the need for and benefits of screening and how to use the screening tool. Participants had an opportunity to practice using a screen as well as scoring.

A series of interview questions were developed by an external researcher with input from CCRR‐MC Director and staff as well as LAUNCH Coordinator. Questions were divided into two sections: one for providers who had started using the tool and one for those who had not yet implemented the screening tool. Providers who reported that they had been using the ASQ‐3 in their programs prior to or following the training were asked questions regarding the number of screenings completed and the screening process in their sites. Only providers who were currently using the screening were asked about Parent Involvement, though all providers were asked whether they knew where to go if a concern was identified or if they had a concern about a child in their care. All providers were asked to respond to questions about how they liked the ASQ‐3, the training, and their understanding of the benefits vs. possible negative impact on their work. Also queried was whether they had the right support to implement use of the ASQ‐3 in their setting and if they felt they needed additional training. All providers answered questions about Technology as well as the final more general questions. The two directors interviewed were asked specific “director questions.” Four parents were interviewed regarding how they experienced use of the ASQ‐3 in the child care setting. Their comments follow at the end of the director comments. As directed by the study, the researcher interviewed 20 providers from a variety of programs across the Multnomah County who had attended at least one of the trainings. Interviewees were chosen randomly 3 from class lists. Interviews were conducted by phone with providers of all types of child care: Registered Family Care (RF), Certified Family Care (CF) and Center Care (CC). Staff interviewed included: owner/provider and/or staff, center staff and center directors. In all, there were 8 RF care respondents, 8 CF care respondents (one of these was staff in a


CF home) 2 CC directors and 2 CC staff. Three of the providers were Spanish speaking only (all RF).

The implementation of the ASQ‐3 in child care settings following training had solidly begun: 60% of those surveyed have used it in their centers and are planning to continue its use while 40% had not yet begun to use it. Providers who serve smaller numbers of children have completed screens on a smaller proportion of the children in their care while one larger program has screened all of the children in their care. Providers who are using the ASQ‐3 like it and find its use beneficial for their program. They cite knowledge of development, ideas for activities to support children’s development, increase in communication with parents, professionalism, and referral to Early Intervention services for children as some of the positive benefits. Providers who are not yet using it found the training they received helpful and feel ready to implement the tool as they begin their journey on the QRIS qualification process.

The majority of providers would like additional training in use of the ASQ‐3. Some of their comments reflect a need for more support around logistics such as choosing the right screening tool and scoring. Others cite their concerns around talking to parents about a delay in development. Additionally, use of the information from the screening tool is incorporated generally but opportunities may exist for it to be fine‐tuned for teachers/providers to use with individual children i.e. fine or gross motor activities, social or behavioral strategies. Some providers cited funding as a barrier to implementing the ASQ‐3. Additionally, smaller programs have screened only one or two children by asking parents to use the free online version. Another program may be using an older version of the ASQ. To address this concern and expand use of the tool, a plan for supporting providers to purchase the ASQ‐3 may help. Scholarships for RF and CF providers may provide an incentive to purchase.

One provider commented:

“We need to understand development as a priority. I think trainings should be more available. I am a strong advocate for the ASQ‐3. It was a huge expense and I may not recoup it. But I understand that providers are so influential to children (some providers do not understand this). It really helps in planning for where you are headed and what you want to do with kids.”

Training Outcomes: Screen Time Reduction Module for Family Child Care Providers In Year 2 Multnomah LAUNCH supported the development of a Screen Time Reduction (STR) training module that was piloted with a group of 7 registered family child care providers. The training began with an orientation, based on raising awareness for registered home providers and the families of the children they serve; identifying alternatives to television and other electronics that are rooted in the reality of their settings; and developing written policies. The initial training was followed up with in‐home coaching from the trainer and facilitated peer‐to‐peer support. Three weeks later, a follow‐up training provided group processing and additional reflection of strategies, challenges, and successes.

Six providers completed pre‐ and post‐surveys for the evaluation of the pilot. Providers reported changes in screen time exposure for the children in their care, as shown in Figure 7. In addition, 6 providers agreed that they have reduced screen time for children in their care, as well as in their own lives. Five providers also agreed that they had shared information on screen time with parents of children in their care. Although the STR training has been expanded to include additional providers


with the help of grant funds from a local foundation, LAUNCH evaluation of these efforts are not included beyond the pilot in Year 2.

Figure 7. Presence of Children & Television, Pre and Post STR Training

Training Outcomes: Inclusion of Culturally Diverse Staff and Differences in Outcomes by Race/Ethnicity Because a stated goal of Multnomah LAUNCH’s workforce development work was to engage and support providers representing culturally‐specific organizations and others who work with families from diverse cultural and ethnic backgrounds, we compared the levels of participation in key LAUNCH training activities by provider race/ethnicity, as well as comparing several key outcomes for these participants. Results are shown in Table 9 and 10, below.

Among providers attending community or EC PBIS trainings through Year 3 who were asked and provided information about their race/ethnicity, 43% identified as Providers of Color as shown in Table 9. Compared to overall Multnomah County population statistics, this suggests that these providers are representative of the percentage of African Americans in the community, but that Latino/as are slightly under‐represented in this group of providers, while Asian Americans are overrepresented. Providers attending EC PBIS‐specific trainings are much more likely to be White, with only 15% reporting they identify as a provider of color, mirroring the current make‐up of mental health consultants and EC PBIS coaches in the area, who are by and large White.

Table 9. Race/Ethnicity Categories of Training Participants

Community Trainings

(N=398) EC PBIS Trainings

(N=75)

White 57% (228) 85% (64)

Black/African American 6% (24) 0

Latino/a 10% (39) 4% (3)

Asian American 18% (72) 3% (2)

American Indian/Alaska Native 2% (9) 0

Other: “Other”, multiracial 7% (26) 8% (6)

There were no significant differences in the percentage of respondents who “agreed” or “strongly agreed”, comparing White providers and Providers of Color as shown in Table 10.

67%

17%

33%

67%

17%

0%

20%

40%

60%

80%

100%

Pre‐Training Post‐Training

No TV Present

Rarely/Never

Sometimes


Table 10. Percentage of Participants who “Agree” or “Strongly Agree” by Race/Ethnicity of Participants Community & EC PBIS Trainings

Provider of Color

(N=177) White (N=289)

Plan to apply knowledge, skills, and principles 95% (168) 92% (266)

Training increased knowledge of children’s social emotional behavior and/or development 90% (159) 83% (240)

Training increased knowledge of available follow‐up services 78% (138) 63% (182)

Findings: Communities of Practice

To better understand the process of developing the Communities of Practice, their outcomes for participants, and their sustainability plans, a series of interviews was conducted to explore this aspect of WFD in more depth. These results were summarized in a series of “Research Briefing Papers” and shared with the Multnomah LAUNCH team. These research briefs are included in Appendix D.

Findings: Overall Workforce Development Successes and Challenges

Major successes for workforce development over the five years included: (1) Successfully meeting targeted training objectives including training at least 2,000 early, family support, primary care, and mental health professionals in a variety of topics related to children’s mental health, social‐emotional development, family trauma, and related topics; (2) recognizing and responding to the shifting needs of Multnomah County’s early childhood environment by supporting more intensive communities of practice which are likely to continue to be sustained post‐LAUNCH; (3) supporting fidelity of implementation of evidence based models specifically Early Childhood Positive Behavior Interventions and Supports; and (4) providing a platform for expanding the EC PBIS model into other family‐serving organizations, specifically Domestic Violence Shelters.

Major challenges for the WFD efforts were primarily related to the shifting early childhood context, specifically the development of the Early Learning Multnomah Hub and emerging questions about the role that the hub would play in WFD; the diverse and widespread training needs of the early childhood community; the lack of a coordinated system of calendaring and sharing information about early childhood trainings (while originally it was a goal of the LAUNCH project to create and maintain such a system, it became apparent very early that there were not sufficient resources available to develop and implement this type of coordinated training system); and the challenge, early in the project, related to developing more focused and sustained mechanisms for training. The shift towards more focused training, training and support for coaches, and developing sustainable CoPs was seen as a positive response to these challenges, with greater potential for creating long‐term practice changes.

Public Awareness Multnomah Project LAUNCH engaged in several key activities designed to increase community awareness of the importance of early childhood and, more specifically, to promote the use of the 211infoFamily helpline. These activities included supporting a staff person at 211infoFamily to do outreach to, and collect information from, a wide variety of family and child‐serving agencies as well as supporting marketing activities in collaboration with 211infoFamily partners. Moreover, there was a


collective effort on the part of YCWC members and contracted LAUNCH partners to share information about the 211infoFamily resource and to utilize these agency partnerships to share information with families and other child‐serving professionals. For example, multiple presentations were made at YCWC meetings from 211infoFamily staff about their services and promotional information was widely distributed through these partnerships.

Research Questions

1. Public Awareness Implementation Research Questions a. To what extent are community outreach and awareness activities implemented that

promote 211infoFamily and communicate information about children’s mental health and related issues?

b. To what extent are these activities culturally relevant and responsive?

c. What barriers were encountered during implementation?

2. Public Awareness Outcome Research Questions a. To what extent did LAUNCH increase knowledge and awareness among the broader

community of LAUNCH, 211infoFamily, and issues related to early childhood health and mental health?

b. To what extent were these efforts successful at reaching communities of color?

Methods & Data

Data related to implementation of community public awareness activities were tracked by the Young Child Wellness Coordinator and by staff at 211infoFamily. Outcomes of public awareness efforts were collected through surveys conducted as part of staff outcome surveys collected from the following target audiences who were thought to be likely to be reached through YCWC collaborative networks and information sharing:

1. Child care providers receiving EC MHC services

2. Home visitors receiving EC MHC services

3. START‐trained primary Care providers and staff

For child care and home visitors, we collected baseline, 6, 12, and 24 month follow‐up data for providers receiving mental health consultation to assess changes in their knowledge of 211infoFamily as a key resource for families. Primary care participants in START trainings were also asked questions in their post‐training follow up survey about their knowledge and intentions to use 211infoFamily.

Additionally, the LAUNCH‐funded 211infoFamily outreach coordinator made numerous presentations to community groups and organizations, and included questions about the outcomes of these presentations and trainings. Participant responses in terms of their knowledge and intended use of, 211infoFamily were also used to assess the success of these awareness building activities. Note also, that actual use of the 211Family Info constitutes a key outcome of the success of these activities; however, these results are included under the Service Delivery section focused on the 211infoFamily service in subsequent sections of the report.


Findings: Implementation of Public Awareness Activities

Key outputs for the Public Awareness activities are summarized in Tables 11 and 12 below. A wide variety of different marketing and outreach strategies were used over the course of the project.

Table 11. 211infoFamily Community Awareness Activities Y1 Y2 Y3 Y4 Y5 TOTAL

Number of Events

Community Fairs 4 17 22 14 23 80

School Events 1 10 23 11 7 52

Presentations to Community Groups/Orgs 1 30 56 35 31 153

Number of Culturally‐Specific Groups

Working in partnership to promote 211infoFamily 5 9 7 7 3 36

Number of Participants

Community Fairs 800 1,787 7,347 1,660 2,690 14,284

School Events 75 1,055 1,150 1,054 780 4,114

Presentations to Community Groups/Orgs 25 685 1,090 709 572 3,081

Table 12. Number of 211infoFamily Promotion Efforts Y1 Y2 Y3 Y4 Y5

211infoFamily Facebook posts na na na 30 26

Magazine articles na na 2 0 0

Newspaper ads na 6 1 0 2

Newspaper articles na na 3 0 3

Television pieces aired na na 72 0 4

Estimated number of partners providing materials

Child care center & family providers na 1,087 unk unk unk

County library branches na 19 unk unk unk

Parks & recreation programs na 11 unk unk unk

Primary care provider clinics na na 309 unk unk

Estimated number of materials distributed

211infoFamily brochures na na na 1,180 unk

211infoFamily magnets in back‐to‐school packets na 14,350 779 unk unk

211infoFamily rack cards na na 2,850 12,600 8,500

211infoFamily flyers na na 18,900 2,205 1,850

211infoFamily posters na na 329 714 530

211infoFamily wallet cards na na 1,250 2,800 1,150

Based on Facebook performance analytics for Year 5 placement of six (6) 211info ads placed in Facebook newsfeeds, information about 211info reached 95,236 Facebook users in Multnomah County.

2 Two of seven television pieces in Y3 were with a Spanish‐language television station.


Findings: Outcomes for Public Awareness Activities

Although the most important outcome of public awareness activities is actual utilization of the 211infoFamily services (see Service Delivery, Family Strengthening, 211infoFamily, below), we also tracked whether providers trained by and involved with LAUNCH were learning about the availability of the 211infoFamily service.

As shown in Figures 8 and 9 below, ECE providers and home visitors generally reported improvements in their awareness, understanding, and knowledge of the 211infoFamily services. The largest increases appear after 12 months of consultation, suggesting that this information is shared somewhat later in the process of developing a relationships with the mental health consultant (MHC). Note that these data represent “snapshots” of staff knowledge after 6, 12, 24, and 36 months of working with the consultant, not changes over time within the same group of providers.

Figure 8. ECE Provider Knowledge of 211infoFamily

Figure 9. Home Visitor Staff Knowledge of 211infoFamily

1.9 1.8 1.8 1.7

2.2 2.3 2.3

2.0

3.93.7

2.6 2.5

3.4

3.0 3.12.9

3.33.1

2.9 2.8

1

2

3

4

I have heard about 211infoFamily

I understand the services211info Family provides

I know how to access 211infoFamily

I am satisfied with thesupport received from

211info Family


2.32.1 2.1 2.2

3.7

3.1 3.23.4

3.73.4 3.5

3.3

3.9 3.84.0

3.4

4.03.8 3.9

3.1

1

2

3

4

I have heard about 211infoFamily

I understand the services211info Family provides

I know how to access 211infoFamily

I am satisfied with thesupport received from

211info Family



We also collected data from primary care providers (PCPs) who participated in LAUNCH‐funded START trainings about their awareness of the 211infoFamily service. These results are shown in Table 13 below. Because of the problems noted previously with collecting follow‐up surveys, we report only the information collected at the post‐training survey regarding participating PCP’s intention to use the 211infoFamily resource. As can be seen, almost all START‐trained PCPS are aware of the 211infoFamily services by the end of the training, and intend to provide information about this resource to parents. As can be seen, providers were mostly positive about their intentions to provide information about 211InfoFamily to parents, as well as to use the service themselves.

Table 13. Primary Care Provider Expected Use of 211infoFamily

To assess change in knowledge and awareness of 211info in the broader community, the 211infoFamily community outreach coordinator administered post‐presentation surveys to participants to gauge the extent to which they knew about or had used 211info prior to the presentation. In Year 4, this question was modified to ask about participants’ knowledge of 211infoFamily. As shown in Table 14, most participants had heard about or used 211info prior to the presentation they attended. However, 96% of participants agreed they would apply the information they received. Further, the “reach” of these community presentations increasingly focused on providers that were, in fact, not as aware of 211infoFamily services, suggesting that this work was successfully reaching those who can benefit from the presentations.

Table 14. Knowledge & Use of 211info & 211infoFamily by Community Presentation Participants

Y1

(N=19) Y2

(N=87) Y3

(N=128) Y4

(N=198)

I knew about 211info before attending this presentation 90% (17) 89% (77) 76% (97) na

I used 211info before attending this presentation 63% (12) 63% (55) 62% (79) na

I knew about 211infoFamily before attending this presentation na na na 56% (111)

I used 211infoFamily before attending this presentation na na na 31% (61)

Findings: Overall Successes and Challenges in Increasing Public Awareness

Multnomah LAUNCH used a variety of strategies to increase public awareness of 211infoFamily specifically and early childhood mental health issues more broadly. Through both marketing efforts, as well as by integrating information about the 211Family Info resource into a variety of WFD strategies,

Post START Training% Strongly or

Somewhat Agree

Item (Years 1‐3)

I plan to provide information about 211info Family with parents (N=100) 97% (97)

I plan to use 211info Family to help identify local resources for referrals (N=98) 97% (94)

Item (Years 4‐5)

My referrals of families to 211info Family will increase as a result of this training (N=282) 83% (236)

My use of 211info Family will increase as a result of this training (N=281) 82% (230)


LAUNCH appeared to be quite successful in “getting the word out” about the 211infoFamily service (and see Service Delivery Findings, below). However, it was difficult to assess the extent to which information about 211infoFamily did reach a culturally diverse audience, or the extent to which primary care and early childhood providers, did pass the 211infoFamily materials and information to their clients.

Sustainability

Research Questions

There were no specific a priori research questions related to sustainability. However, key questions that emerged and were addressed by the evaluation were:

1. What were the key changes and outcomes that were achieved that are likely to have a longer term impact on the service system due to Project LAUNCH?

2. What components of Multnomah Project LAUNCH were likely to be sustained after the end of the grant?

3. What factors were related to supporting sustainability of these activities?

Methods & Data

Information about sustainability was collected through the key stakeholder interviews. Additionally, at the end of Year 5, the evaluation team reviewed the status of key activities, YCW Coordinator documentation, and other documents to assess which strategies had an active plan for sustainability following the end of the project. These results are summarized below.

Findings: Sustainability

At the end of Year 4, key stakeholder interview respondents were asked to discuss what LAUNCH needed to do to ensure the sustainability of both the system‐level collaborations that have been developed, as well as the specific interventions being funded through LAUNCH. Respondents discussed the importance of making sure that the accomplishments and results of LAUNCH were communicated to ELM and to the state legislature in order to create support for the most effective strategies. Overall, the most frequently mentioned route to sustainability was building on existing connections to the Early Learning Multnomah hub as well as doing focused advocacy and outreach to state and local funders for ongoing support of particular intervention strategies. As of this writing (3 months after the end of project funding), the following activities were ongoing as of the end of the project:


EC PBIS Leadership Team continues to meet, although without administrative staff support;



The Home Visiting CoP continues to meet, with support from the state’s MIECHV grants;

The Early Childhood Mental Health CoP also continues to meet although less frequently and without administrative staff support;

The Parent Educator position is currently being maintained at 211infoFamily, supported by funding from a regional Coordinated Care Organization.


Core Strategies Screening, Assessment & Behavioral Health Integration in Primary Care

Evaluation Questions

1. Screening & Assessment: Implementation Research Questions a. To what extent were key START services implemented as planned? b. What supported successful implementation? c. What barriers were encountered?

2. Screening & Assessment: Outcome Research Questions a. Provider Outcomes:

i. What changes, if any, occurred in START training participants’ knowledge, skills, and understanding of how to conduct, score, and make referrals for developmental delay, behavioral health, and maternal depression?

2. Child & Family Outcomes: i. Were more children screened/assessed for developmental, social‐emotional, and

behavioral health issues by primary care physicians after receiving START training? ii. How many children screened by LAUNCH trained providers were identified and referred

for early intervention services?

Methods & Data

Process and outcome evaluation for the START trainings utilized data already being collected by the START program, including the following:

Registration information. START collects detailed information about training participants through its registration process (type of provider, trainings attended, and basic demographic characteristics). The evaluation team has set up a system for receiving regular (quarterly) reports of required information from the START data system. All individuals who participate in LAUNCH‐funded START trainings will be included in this data.

START participant surveys collected during trainings. At the beginning of Project LAUNCH, START providers had in place a system for collecting surveys from each participant prior to, during, and 4‐months following, participation in a training. However due to problems in collecting the pre‐ and post‐surveys, the process has changed to collecting an in‐training survey at the end of each session as the sole method of evaluating training outcomes. Survey content was reviewed by LAUNCH evaluators and questions added to address key LAUNCH‐related outcomes that were not already covered by existing items. Surveys included questions such as clinicians’ perceptions of frequency of screenings administered, changes in knowledge and confidence, and progress in implementing social‐emotional screenings in primary care settings.

The original evaluation plan relied on these surveys to evaluate key outcomes related to START trainings; however, the START program struggled to collect the information. START also experienced


significant staff turnover during Years 1, 2 and 3, including the loss during Year 3 of the long‐time executive director. This resulted in ongoing challenges related to data collection for this component of LAUNCH. During Year 4, we worked with the START staff to revise, again, the data collection protocols, recognizing that not only were follow up data not being regularly collected, but that implementation of changes in physical practices required more sustained follow up and support than had previously been provided by START. The revised strategy met with mixed success, as obtaining additional follow up information from primary care providers proved an ongoing struggle. Response rates for trainings varied from just over half of participants completing the follow up survey (58%) to over 75% completing the survey.

Clinic case studies. During Year 4, we identified a small number of clinics from whom we obtained information about screenings provided, in addition to learning more about their implementation process specifically related to the new module developed to train providers in integrating behavioral health into primary pediatric care settings, as well as for learning about integration of post‐partum depression screening in a medical setting. We conducted two case studies clinics that piloted the behavioral integration module, including qualitative and quantitative data collection. We also conducted two qualitative interviews, and gathered multiple kinds of materials and resources, from the lead primary care physician at the Neonatal Intensive Care Unit (NICU) site that is doing cutting‐edge work in integrating depression screening. By adopting a more qualitative case study approach we focused on documented the “lessons learned” in these medical settings about what it takes to implement changes related to supporting integration of behavioral health screening into medical environments.

Number of referrals to Early Intervention. In order to capture the number of referrals from START‐trained primary care providers to Early Intervention (EI), referral data was obtained directly from the EI service provider in Multnomah County. The data was selected based on criteria that the referral came from a primary care provider, and identified the type of developmental screen that accompanied the referral to EI, e.g., ASQ, M‐CHAT, PEDS. While it was not possible to select data for specific names of providers trained by START, due to limitations of START‐data sharing and EI‐data system processes, referrals from primary care providers with an accompanying screen were “counted” for LAUNCH‐funded START‐trained providers to serve as an estimate of START training impact at the county level.

Number of screens conducted. While we originally planned to use Medicaid billing records to document the number of screens provided by LAUNCH‐trained physicians, over the five years of LAUNCH, obtaining reliable medical billing information related to billing for conducting developmental screenings proved quite challenging.

Data during Years 1 and 2, however, were estimates of developmental screenings was obtained through post‐tests with LAUNCH‐funded START‐trained primary care providers who were asked to estimate the number of screenings completed each week prior to training and since training. This number was then extrapolated out to include the 3‐month follow‐up period post‐training. The accuracy of these estimates is unknown.

However, during Years 3‐ 5 we were able to access electronic health record data from a subset of participating providers. Using this information, we were able to compare the number of screens 12 months prior to training to the 12 month interval post‐training and examine changes in the number of


screenings provided. Thus, the number of screenings documented during these years does reflect changes in screening behavior following START trainings.

Analysis of Early Intervention/Early Childhood Special Education data. During Year 5, the evaluation team worked with the EI/ECSE provider to conduct a more in depth analysis of patterns of screening, referral, and assessment for developmental delay. This was in response to information shared with the YCWC that suggested that there were concerns that families were “falling through the cracks” and that, perhaps, referrals from primary care might be more likely to be successfully connected with assessment and, if needed, additional services. PSU received from the EI/ECSE an administrative data set including referral information (dates, source, and reason), family characteristics, information about attempts to contact families, and whether subsequent assessments were ever completed.

Findings: Implementation of START Trainings

Data related to the number of providers trained is reported in Table 15 below. As can be seen, substantially more providers were trained through Project LAUNCH than originally anticipated. In particular, LAUNCH supported expanded developmental screening, training 528 providers, training in conducted maternal depression, and in trauma‐informed practices.

Table 15. Primary Care Providers & Staff Trained by START Module Type Number of Staff Trained Number of

Post Surveys Completed Y1 Y2 Y3 Y4 Y5 Total Target

Basic developmental screening (ASQ, PEDS) 97 142 116 138 35 528

300

409 (77%)

Maternal depression screening (Edinburgh) 55 70 47 0 0 172 132 (77%)

Autism screening (M‐CHAT) 0 0 33 0 0 33 19 (58%)

Behavioral health in primary care (PSC) na na na 50 8 58 27 (47%)

Adverse childhood experiences/ Trauma‐informed care (no screening tool) na na na na 159 159 119 (75%)

Implementation Success: Developing and Implementing the ACEs/TIC Training Module In Year 5, START worked in partnership with YCWC members and other stakeholders to respond to the need among primary care providers for their practice to be increasingly informed by Trauma‐Informed Care (TIC) principles, in order to better serve children under their care who have been impacted by Adverse Childhood Experiences (ACEs). The ACEs/TIC training module introduces TIC principles and provides information to primary care providers on how to conduct a clinic self‐assessment of TIC responsiveness. There had been great demand by practices to participate in the module but as of the writing of this report, training follow‐up data collection and a more robust screening tool recommendation has yet to be developed.


Implementation Challenges: Developing and Implementing the Integration of Behavioral Health in Primary Care Training Module At the start of the LAUNCH project, the goals related to training primary care providers included both expanding basic START training on the Ages and Stages Questionnaire (ASQ) to more primary and pediatric care clinics and providers as well as developing new training modules related to behavioral health. Additionally, based on emerging project needs, LAUNCH provided supports for trainings related to maternal depression, Autism, and trauma‐informed practice.

Significant efforts were made during Years 3 and 4 to develop and pilot a web‐based training module to support the integration of behavioral health into primary care settings. While this took longer to develop than originally planned (due in part to a need to completely revamp the original version of the training to better meet the needs of pediatric primary care providers), the module was successfully completed and launched as a web‐based training resource. Two clinics piloted the module and began to implement enhanced behavioral health screening processes. Data from these clinics are presented in outcomes, below.

Another challenge that was recognized after the first two years was the difficulty that START‐trained primary care physicians and practices had in fully implement the recommended START screenings, even for the basic developmental module. Follow up by evaluation and START staff indicated significant barriers to full implementation of screening protocols; thus, the decision was made at the end of Year 4 to fund a series of “tune‐ups” for previously trained providers during year 5. Additionally, a follow‐up protocol will be implemented with these providers to provide ongoing coaching and technical assistance to support implementation.

Findings: Outcomes of START Trainings

Training Outcomes: Changes in Knowledge, Skills, and Confidence for START Participants The Basic Developmental, Autism Spectrum Disorder, and Behavioral Health Integration in Primary Care modules do not include learning objectives related to changing providers’ knowledge or confidence about early childhood mental health, typical and atypical development, and family risk. The Postpartum Depression module does include a focus on changing providers’ knowledge of maternal mental/behavioral health. A goal of all modules is to change providers’ knowledge of applicable screening tools. Thus, surveys were modified as appropriate to be consistent with the stated training objectives. Participant survey results collected immediately following participation in the trainings are summarized in Table 16 below. As can be seen, the overwhelming majority of participants either “strongly” or “somewhat” agreed that participating in the trainings increased their knowledge of screening processes, screening tools, and related topics.

Table 16. Change in Knowledge Reported by Primary Care Providers & Staff

Post START Training % Strongly or

Somewhat Agree

This training increased my knowledge of (developmental, ASD, maternal depression, behavioral health, family risk) screening. (N=427) 97% (414)

This training increased my knowledge of (developmental, ASD, maternal depression, behavioral health, family risk) screening tools. (N=698) 97% (677)

This training increased my knowledge of maternal mental/behavioral health. 100% (39)


Post START Training % Strongly or

Somewhat Agree

(N=39)

Table 17 below shows the number of screens conducted by LAUNCH trained providers. During Year 2, these numbers are based on estimates provided on post‐training surveys so should be interpreted with caution. However, during Years 3 through 5 we were able to obtain electronic health record (EHR) data and calculate the increase in screenings conducted, comparing one year prior to START training and one year following training (these numbers are shown in Table 17). Note that this table also includes the number of screens completed by Mental Health Consultants in early childhood settings (see Mental Health Consultation Outcome section, below).

As can be seen, START‐trained providers substantially increased the number of screenings conducted in the one‐year period following START training, especially during Year 5. During Years 3 through 5 of Project LAUNCH, there was an estimated increase of 1,619 screenings conducted by primary care providers, compared to the one year prior to training, exceeding the targeted number for this period of 1,000 additional screenings. Interestingly, the biggest increase in screenings provided was in the area of perinatal depression.

Table 17. Screenings Completed by Providers Y1 Y2 Y33 Y4 Y5

Annual number of developmental screenings completed by primary care providers ‐‐ 5044 90 85 373

Annual number of maternal depression screenings completed by primary care providers ‐‐ 605 ‐‐ ‐‐ 1,006

Annual number of screenings completed by MHCs in ECE settings ‐‐ 2 9 33 19

Annual number of screenings completed by MHCs in home visiting ‐‐ ‐‐ 3 1 ‐‐

Total number of screenings completed by LAUNCH‐trained or ‐supported providers 0 566 102 119 1,398

Target number of screenings completed by LAUNCH‐trained or ‐supported providers 378 1,795 500 250 250

3 Y3‐Y5 screening numbers for primary care providers was obtained directly from their clinic electronic health record, comparing the number of screens 12 months prior to training to the 12 month post‐training. The difference in number of screens is the number reported. 4 The Y2 estimate of developmental screenings was obtained through post‐tests with LAUNCH‐funded START‐trained primary care providers who were asked to estimate the number of screenings completed each week prior to training and since training. This number was then extrapolated out to include the 3‐month follow‐up period post‐training. (N=48 providers) 5 The Y2 estimate of developmental screenings was obtained through post‐tests with LAUNCH‐funded START‐trained primary care providers who were asked to estimate the number of screenings completed each week prior to training and since training. This number was then extrapolated out to include the 3‐month follow‐up period post‐training. (N=29 providers).


In order to examine the number of referrals received by the local Early Intervention/Early Childhood Special Education provider, we received data from this agency that indicated the number of referrals received during each Year of Project LAUNCH from primary or pediatric care providers. This information is shown in Table 18. Although this does not necessarily reflect changes caused by participation in START trainings, there is a steady and substantial increase over the five years of Project LAUNCH in the number of referrals from primary care sources. Increases from Year 1 to Year 2 should interpreted with caution, however, as we were working closely with the EI/ECSE agency during that time to improve their ability to track referral sources.

Table 18. Referrals to Early Intervention with Screening Result by Primary Care Providers Number of Referrals Y1 Y2 Y3 Y4 Y5 Totals

Annual number of referrals to EI from primary care providers 15 85 223 345 440 1,108

Target number of referrals to EI from primary care providers 39 117 117 217 217 707

Analysis of EI/ECSE Referral Data Results of the analysis of referral and follow up contact and assessment is included in the Research Brief in Appendix E. These data were shared with primary care providers, early childhood providers, and other stakeholders in a series of facilitated conversations. From these conversations and follow up interviews, the following strengths and challenges of the screening and referral system at the end of Project LAUNCH were noted:

Young Child Wellness Council members, pediatric primary care providers (PPCPs), and developmental pediatricians (DPs), who participated in data conversations, described strengths in the early intervention and identification system:

The developmental screening training module offered through the START program and the Assuring Better Child Health and Development (ABCD) initiative have been effective in promoting practice change related to regular developmental screening within Multnomah County pediatric health care settings.

The total number of referrals to EI/ECSE by physicians continues to increase over time. From October 2012 through June 2013, 159 referrals to EI/ECSE came from this referral source. For the same time period in 2014‐2015, nearly twice as many (n=336) referrals to EI/ECSE came from physicians.

The development and use of a Universal Referral Form has improved communication between referring physicians and EI/ECSE on the results of developmental screenings and outcome of referrals.

There is agreement that health care providers seek to improve universal screening and referral practices in order to continue to facilitate access to early intervention services for families at the earliest possible age, in culturally‐relevant and responsive ways.


Challenges Reviewing the EI/ECSE data by referral source and race/ethnicity raised additional questions and concerns about the effectiveness and cultural responsiveness of referrals for Black children overall, and from physician sources in particular for children of all race/ethnic groups.

PPCPs and DPs voiced a need for additional tools or training to talk with families in culturally‐relevant and responsive ways about the purpose of screening and referral, what families can expect in the referral and evaluation process, and what services can be available to families if their child is eligible.

Providers suspected that stronger, positive relationships with families would contribute to families understanding and trusting the referral and evaluation process, but providers also were frustrated that building relationships with families in short, periodic well‐child visits was difficult.

Providers acknowledged that some families, and especially for families of color and immigrant families, there is a stigma attached to early intervention and special education services, and that this label can carry forward in the K‐12 education system.

Providers noted that some families may face significant barriers that interfere with their ability to get their child evaluated, e.g., transportation, child care and work demands, or unstable housing.

Despite improvements to the communication process between EI/ECSE and PPCPs on the outcome of a referral to EI/ECSE, there is a recognition that PPCPs still too often do not know whether a family followed‐through with an evaluation.

Clinic Case Studies: Behavioral Health Integration & Perinatal Depression Screening Results of the clinic‐specific case study of maternal depression screening were summarized in a Research Brief (see Appendix F). Additionally, we summarized and shared information collected through interviews and survey data regarding efforts to implement Behavioral Health screening processes into primary care.

Two clinics agreed to serve as pilot sites for the Behavioral Integration in Primary Care (BHI in PC) training module. As a participating pilot site, the “clinic champion”, or primary care provider responsible for leading implementation, was asked to participate in a pre‐ and post‐training survey and a 3‐ and 12‐months post training phone interview to understand the successes and challenges of implementation. Readiness Checklist survey items were selected from the Mental Health Practices Readiness Inventory, representing five domains.6 The mean scores from two clinic champions at pre‐ and 12‐months post training are presented in Table 19.

Table 19. Behavioral Health Integration in Primary Care Readiness Checklist Domains Mean Ratings (N=2) 3‐pt scale, Higher score indicates strength

Pre‐Training(sd)

12 Months Post‐Training (sd)

Provider knowledge of community behavioral health care resources 2.5 (.6) 2.5 (.6)

6 American Academy of Pediatrics. (2010). Mental health practices readiness inventory. In Addressing Mental Health

Concerns in Primary Care: A Clinician’s Toolkit. Washington, DC: Author.


Mean Ratings (N=2) 3‐pt scale, Higher score indicates strength

Pre‐Training(sd)

12 Months Post‐Training (sd)

Health care financing of behavioral health services 1.8 (1.0) 2.0 (1.2)

Behavioral health services support for children and families, patient‐centered care 2.5 (.5) 2.5 (.5)

Clinical information systems, behavioral health services delivery system redesign 2.3 (.8) 2.5 (.8)

Behavioral health services decision‐support for clinicians 2.5 (.6) 2.5 (.6)

On average, these ratings changed very little due to the circumstances facing the two pilot site clinics. Prior to the BHI in PC training, Clinic 1 already had co‐located behavioral health services, but participated in the training to learn if there were other recommendations for moving their practice even further along. Longer‐term, Clinic 1 is interested in developing a series of short interventions (10‐15 minutes) that behavioral health staff could teach primary care providers to deliver with families around a set of specific challenges. At the 12‐month interview, Clinic 1 had improved elements of their screening process within the clinic by clarifying timelines, staffing, and expectations for families facing behavioral health challenges.

Pilot Clinic 2 initially was interested in the BHI in PC training in order to adopt a behavioral health screening tool and establish a robust referral system. However, because Clinic 2 is a community clinic serving primarily low‐income families and staffed by a rotating team of physicians who largely volunteer their time, it was difficult for Clinic 2 to find adequate time and sufficient resources to address needed changes in procedures and staffing in order to begin building stronger referral systems with behavioral health care providers. Clinic 2 had not been able to make any strides toward their screening and referral system goal at the 12‐month interview.

Summary: Screening, Assessment, and Referral

Overall, these findings suggest considerable success for Project LAUNCH in terms of implementing training to primary care providers focused on developmental and behavioral screening. Further, these trainings appear to be correlated with steady and substantial increases both in the number of screenings being conducted by primary care providers, as well as the number of referrals being made for additional assessment and services. The success in terms of screening and referral during Years 3 and 5 likely reflects the concerted effort on the part of the START provider to increase their implementation support to providers, as well as improvements in the training process itself and to ensure that providers conducting screens feel confident that they have the resources they need to make referrals and connect families with resources if needed (an issue identified early in the project as a barrier to providers implementing screenings). Further, START providers worked closely with the evaluation team and with the EI/ECSE provider to identify challenges in the screening and referral process, which may have contributed to the project’s ultimate success in increasing the number of children screened and referred.


Early Childhood Mental Health Consultation in Early Care & Education Settings


1. Early Childhood Mental Health Consultation (MHC) in Child Care: Implementation Research Questions a. To what extent were key EC MHC in child care services implemented as planned? b. What supported successful implementation? c. What barriers were encountered?

2. Early Childhood Mental Health Consultation in Child Care: Outcome Research Questions a. Provider Outcomes:

i. Do child care staff who receive EC MHC support show increased skills, confidence, and knowledge about early childhood mental health and mental health best practices?

ii. Do child care staff who receive EC MHC support provide higher quality classroom environments to support children compared to before receiving this service?

iii. Do child care center staff who receive EC MHC support show decreased work‐related stress and reduced turnover?

b. Child Outcomes: i. Are fewer children expelled/asked to leave child care because of behavioral issues after

receiving EC MHC supports, compared to before?

Methods & Data

Several methods were used to evaluate the implementation and outcomes of the MHC and EC PBIS services in three child care centers.

Mental Health Consultant Contact Notes & Activity Summaries. These forms, completed by the MHC and the EC PBIS specialist, provide detailed process and ‘output’ information about the type, frequency, and participants in consultation activities. The LAUNCH evaluation team developed a database for these forms, which were sent to PSU on a quarterly basis.

Provider Outcome Measures. The Mental Health Services Survey (MHSS)7 was used to measure the child care providers’ perceptions of the type and quality of consultation services being received, and their own knowledge and confidence in meeting children’s social‐emotional needs. The MHSS has been used in several studies of mental health consultation, and is recommended by the National Center for Effective Early Childhood Mental Health Consultation. Providers will also complete a subset of items from the Teacher Opinion Survey8, a measure of child care teacher job‐related stress and support. All child care staff at sites that receive consultation are included in the evaluation. Surveys are completed at baseline (prior to consultation), 6 months after that, and annually thereafter.

7 Green, B. L., Everhart, M., Gordon, L., & Gettman, M. G. (2006). Characteristics of effective mental health consultation in early childhood settings: Multilevel analysis of a national survey. Topics in Early Childhood Special Education, 26(3), 142‐152. 8 Geller, S., & Lynch, K. (1999). Teacher Opinion Survey‐Revised. Richmond: Virginia Commonwealth University Intellectual Property Foundation and Wingspan, LLC.


Classroom Environment Quality. The Teaching Pyramid Infant and Toddler Observation Scale (TPITOS) and Teaching Pyramid Observation Tool (preschool‐aged TPOT) classroom assessments were completed by the Mental Health Consultant/EC PBIS specialist for each classroom in the centers receiving EC PBIS/MHC services. These measures were developed as part of the research and evaluation of the EC PBIS model and are designed to assess the social‐emotional climate and quality of relationships between teachers and children in the classroom. The original plan included assessments being completed at baseline (prior to delivery of EC PBIS/MHC services) and every 12 months thereafter. At the start of Year 4, this was changed to completion in the Fall (September/October) and Spring (April/May) in order to better measure teacher change over the course of a typical “school year” for providers.

EC PBIS Implementation Fidelity. The Benchmarks of Quality (BOQ) fidelity tool was completed by the EC PBIS team for each child care site receiving EC PBIS services. This measure was developed as part of the EC PBIS model and is designed to assess the degree to which a child care site or other program is implementing EC PBIS. Assessments will be completed at baseline (prior to delivery of EC PBIS services) and every 12 months thereafter.

Findings: Implementation of EC MHC in Early Care & Education

LAUNCH funded MHCs provided support to 3 child care programs over the course of the project. Two providers were not‐for‐profit child care, while a third was a for‐profit child care center. During Year 3, one of the non‐profit centers unexpectedly closed down, ending data collection at this site. A total of 160 child care providers completed at least one LAUNCH survey. As show in Table 20, the majority of providers were White/Caucasian, although 12% were Latino (serving 15% Latino children). Most had worked at the ECE program for only about 6 months at the time of the baseline survey (note that new incoming staff received a baseline survey within 30 days of hire). A full 40% of providers had left the program by the 12 month follow‐up, reflecting the frequent turnover

Table 20. Characteristics of ECE Program Staff & Children ECE Programs9

Number of staff 198

Staff represented in survey data 160

Mean length of time with program at baseline survey in months (sd) 17.3 (22.3)

Median length of time with program at baseline survey in months 6.0

Mean length of time working in early childhood services at baseline survey in months (sd) 74.9 (77.51)

Median length of time working in early childhood services at baseline survey in months 54.0

Staff leaving program prior to 12 month follow‐up survey 40% (60)

Staff age

Mean staff age in years at baseline survey (sd) 28.6 (9.0)

Staff race/ethnicity

African American/Black 3% (5)

Asian American/Pacific Islander 7% (10)

Latino 12% (18)

9 In Y2‐4, three ECE programs received MHC; one closed at the end of Y4, leaving 2 ECE programs served by MHC in Y5.


ECE Programs9

White 68% (105)

Other (unspecified) 5% (7)

Multiracial 6% (9)

Number of children served 944

Number of children dismissed due to behavioral concerns (%) 13 (.013%)

Children attending with subsides10 25% (905)

Child race/ethnicity11

African American/Black 7% (55)

American Indian/Alaska Native 1% (9)

Asian American/Pacific Islander 7% (54)

Latino 15% (122)

White 69% (567)

“Other” unspecified <1% (2)

Qualitative data were collected quarterly from LAUNCH providers related to implementation success and challenges as a part of required reporting. A Research Brief (see Appendix G) summarizes the key success, challenges, and outcomes for ECMHC in child care for Multnomah Project LAUNCH. Briefly, the following were the key strengths and challenges reported over the course of the project:

Mental Health Consultation in Child Care: Successes After initial start‐up challenges, ECMHC supports in child care were highly successful. Key factors related to this success, from the perspective of the consultants and child care directors, included:

Partnering with child care directors and staff to establish clear structure and parameters for consultation, including scheduling regular staff training times and maintaining regular, consistent meetings between MHCS and program leadership

Establishing successful, long‐term relationships between LAUNCH‐funded consultants and child care programs

Using the ECPBIS module and providing training to child care staff on this framework.

Providing reflective time for staff and leadership to meet with each other and the consultant, and providing group coaching time

Mental Health Consultation in Child Care: Challenges By far the biggest challenges in delivering the ECMHC services in child care settings were related to uncontrollable (and unpredictably high ) turnover in child care staff and program leadership; in the most extreme case, at the end of Year 4 one of the facilities that LAUNCH was supporting closed its doors entirely, with little notice. While turnover within child care is not unexpected, the leadership turnover was particularly challenging, and no doubt related to a higher than expected rate of turnover within staff. However, it appears that the new program leadership is now fully “on board” with the

10 Subsidy status was requested once per year based on current enrollment. 11 Child race/ethnicity was requested once per year based on current enrollment.


MHC model and providing strong leadership to ensure continued program‐wide implementation of EC PBIS. Despite this support, maintaining an EC PBIS leadership team at each site remains a challenge.

Staffing challenges also include lack of staff time to meet with consultants outside of class time; LAUNCH has considered paying for substitute time to facilitate this process, and also implemented group coaching time as an alternative, and more successful, strategy.

While using the TIPITOS and TPOT assessments were seen as valuable for guiding the work, the assessments take considerable time, and reduced the time that consultants are available for coaching and observation. However, when pressed, consultants agreed that they would not “give up” doing this more structured assessments given the information they provided to guide the work with classroom staff.

Findings: Outcomes for Changes in ECE Provider Skills, Confidence, and Knowledge

Surveys were collected from child care providers at baseline (start of Project LAUNCH or within 30 days of new staff hire), 6, 12, 24, and 36 months post‐baseline. These surveys included questions regarding staff skills, confidence, and knowledge of early childhood mental health and mental health best practices. The results of these surveys are shown in Tables 21 and 22. Table 22 reflects means for “snapshots” of staff at each point in time, rather than changes within the same providers over time. As can be seen, providers at the 6 month follow up reported significantly higher knowledge of children’s mental health. At 24‐month follow up, providers reported significantly more confidence in involving parents in conversations about children’s mental health, and also reported a stronger sense of shared program vision related to support children’s mental health. No other changes over time were statistically significant, although clearly those staff who remained in the study at 24 and 36 months showed a general pattern of reporting higher knowledge and confidence; it may be that they started off with higher scores generally and that these feelings of confidence and efficacy were related to their longevity.

Table 21. LAUNCH ECE Staff Outcome Domains Mean (sd) 5‐pt scale, Higher score indicates strength

Baseline(N=160)

6 Mo (N=109)

12 Mo (N=89)

24 Mo (N=45)

36 Mo (N=15)

Parent involvement 3.6 (1.0) 3.7 (1.0) 3.9 (.8) 4.1 (.8)* 4.0 (.7)*

Knowledge of children’s mental health 3.8 (.7) 4.1 (.7)* 4.2 (.5)* 4.4 (.6)* 4.4 (.6)*

Efficacy in children’s mental health 4.2 (.6) 4.3 (.5) 4.5 (.5) 4.6 (.6) 4.4 (.5)

Knowledge and comfort using EC PBIS ‐‐ 4.4 (.5) 4.5 (.5) 4.7 (1.0) 4.6 (.7)

Staff stress 2.3 (.8) 2.4 (.8) 2.5 (.8) 2.5 (.7) 2.4 (.8)

Program leadership 4.1 (1.0) 3.8 (1.2) 4.1 (1.1) 4.1 (1.0) 3.9 (1.0)

Program vision 3.8 (1.0) 4.0 (1.1) 4.2 (.7) 4.4 (.7)* 4.2 (.8)

Program support 3.9 (.9) 3.7 (1.0) 3.8 (.9) 3.6 (1.0) 3.6 (.9)*Significant difference in means (p<.05) appears at this time point from baseline within the domain.

Table 22 shows the small subsample of providers who remained in their position through the 24 month follow up. Within subjects analyses were conducted to compare each follow up time point with baseline. For those providers who remained employed at these facilities for two years there were significant increases over time in terms of the knowledge of children’s mental health and confidence in involving parents in mental health related conversations and challenges.


Table 22. Child Care Provider Knowledge, Comfort & Skills Mean (N=33) 5‐pt scale, Higher score indicates strength

Baseline(sd)

6 Mo (sd)

12 Mo (sd)

24 Mo (sd)

Parent involvement 3.5 (.9) 3.7 (.9) 3.8 (.9) 4.2** (.9)

Knowledge of children’s mental health 3.8 (.8) 4.0 (.7) 4.3 (.4) 4.5** (.5)

Efficacy in children’s mental health 4.2 (.5) 4.3 (.5) 4.5 (.4) 4.5 (.5)

Knowledge and comfort using EC PBIS ‐‐ 4.5 (.5) 4.6 (.4) 4.9 (1.2)

Program support 4.0 (.7) 3.8 (1.0) 4.0 (.8) 3.7 (1.0) **Statistical significance of change within this group of providers from baseline (p<.01).

Differences in Outcomes for Providers from Different Racial/Ethnic Backgrounds To explore whether consultation was differentially effective for providers from different racial/ethnic backgrounds, we examined results of the Provider Survey, comparing white providers to providers of color (POC). These results are shown in Table 23 below (sample sizes were too small to examine differences for specific racial/ethnic groups).

Table 23. Mean Scores by Race/Ethnicity of Home Visitors and Child Care Providers

Mean (sd) 5‐pt scale

Knowledge of Children’s Mental Health Domain

Involving Parents Domain

Feeling Supported by Program Domain

Baseline 24 Mo Baseline 24 Mo Baseline 24 Mo

Providers of Color (N=19) 3.5 (.9) 4.4** (.5) 3.5 (.9) 4.1** (.9) 4.0 (.7) 4.0 (.9)

White Providers (N=22) 3.9 (.7) 4.5** (.5) 3.7 (.7) 4.3* (.7) 4.0 (.7) 3.8 (1.1)**Change over time within group significant, (p<.01) *(p<.05)

Providers of color showed a larger increase from baseline to 24‐month follow up in terms of their levels of knowledge of children’s mental health but both groups increased significantly over time. Providers of color rated themselves somewhat lower in knowledge at baseline, compared to White providers; at follow‐up both groups rated themselves similarly. In terms of involving parents, both providers of color and White providers significantly increased their level of confidence in involving parents from baseline to 24‐month follow‐up. The race‐by‐time interactions were not significant, however, likely due to small sample sizes.

Findings: Outcomes for ECE Classroom Environments

Mental Health Consultants collected classroom observations using the TPOT (for 3‐5 year old classrooms) and the TPITOS (0‐2 year old classrooms) at the child care centers in Fall 2012 and completed 12‐month follow‐up observations by Fall 2013, using an updated version of the tools. As a result of discussion on how to make the results of these observations more useful to the child care providers and mental health consultants, the observation schedule was changed to a Fall‐Spring schedule (Spring 2014, Fall 2014, and Spring 2015). Figures 10 and 11 summarize the baseline (Fall 2013) and Spring 2015 follow‐up data collected on the classroom environment quality as measured by these tools for those with data at both time points. Note that items are scored as 1=yes/present; 0=no/not present.


Figure 10. LAUNCH ECE Infant‐Toddler Classroom Staff Practices (TPITOS)

Figure 11. LAUNCH ECE Preschool Classroom Staff Practices (TPOT)

Findings: Outcomes for ECE Program Level Changes

Benchmarks of Quality (BOQ) In addition to the classroom observations and ratings conducted by the mental health consultants annually, the ECPBIS model includes a programmatic assessment of the level of implementation of EC PBIS at the program level. The BOQ is completed annually with the PBIS Leadership Team at each site. Baseline, 12‐ and 48‐month follow‐ups were completed at 2 of the child care sites.

80%

95%

93%

94%

93%

53%

61%

85%

83%

70%

67%

31%

0% 20% 40% 60% 80% 100%

TPITOS Mean Implementation Rate

Collaborates with colleagues

Demonstrates warmth & responsivity

Uses strategies for children w/disabilities ordual language learners

Creates opportunities for communication &relationship‐building

Is responsive to emotions & teaches aboutfeelings

Fall 2013(N=12)

Spring 2015(N=12)

69%

75%

82%

74%

71%

55%

51%

64%

62%

57%

49%

31%

0% 20% 40% 60% 80% 100%

TPOT Mean Implementation Rate

Uses supportive conversations

Provides transitions between activities

Provides directions

Utilizes schedules, routines, & activitiesvisuals

Teaches social skills & emotionalcompetencies

Fall 2013(N=8)

Spring 2015(N=8)


The Benchmarks of Quality (BOQ) tool was used by the MHC with each program’s EC PBIS Leadership Team, which was the group responsible for decision‐making within each ECE program around program policies and supports for EC PBIS implementation. The Leadership Team completed the BOQ annually to self‐assess programmatic changes and EC PBIS implementation over time.

Figure 12 shows the percent of elements within each domain that were fully in place at project start, compared to those fully in place four years later for two ECE programs. Although some EC PBIS elements were in place prior to Multnomah Project LAUNCH, these ECE programs had very few behavioral supports fully implemented program‐wide. At the end of their work with the MHC, most BOQ domains showed dramatic growth in the rate of EC PBIS program‐wide implementation, measured by the percent of elements fully in place.

Four domains had 50% or more of elements fully in place at the ECE program level at the end of the project, including staff support of the EC PBIS implementation plan (56%), staff buy‐in to EC PBIS implementation in the ECE program, establishment of EC PBIS leadership team (50%), and monitoring EC PBIS implementation and outcomes (50%).

Figure 12. LAUNCH ECE Program‐Level Implementation of EC PBIS (BOQ)

Findings: Behavior Incident Recording System (BIRS) with Child Care Providers Another component of the ECPBIS system involves tracking and monitoring detailed information about the frequency and type of behavioral incidents for specific children. One of the three child care sites opted to pilot test this system, known as the Behavior Incident Recording System (BIRS). The system is not designed to be used as a method of outcome evaluation, but rather as a way to use data and information to inform practices in relationship to both overall classroom events and strategies, as well as for individual children (especially children with more persistent behavioral problems). Therefore we

13%

17%

42%

42%

42%

50%

50%

50%

56%

0%

0%

0%

8%

8%

0%

0%

17%

6%

0% 20% 40% 60% 80% 100%

Families are involved in planning

Teach program‐wide expectations

Classrooms demonstrate EC PBISimplementation

Create procedures for responding tochallenging behavior

Establish program‐wide expectations

Monitor EC PBIS implmentation &outcomes

Staff buy‐in to EC PBIS implementation

Establish EC PBIS Leadership Team

Staff support EC PBIS implementation plan

2012(N=2programs)

2015(N=2programs)


present this information descriptively, to illustrate the types of data that were collected, and how data were used to inform practice.

Implementation of BIRS in the ECE program first consisted of work done by the ECPBIS leadership team at the site to set criteria for when Behavior Incident Reports (BIRs) would be completed and by whom. The leadership team agreed to train all staff on the use of BIRs but that lead teachers would be responsible for submitting BIRs and that two target classrooms would be asked to pilot the consistent use of BIRS. One administrative staff person was designated to be responsible for entering BIRs weekly, and BIRs were reviewed at least weekly by program administrators and lead teachers in order to plan for specific behavior supports for particularly intrusive and challenging behaviors, e.g., running away and biting.

Initial use of the BIRs was described as successful by program staff, and particularly useful in documenting challenging behaviors in order to have ongoing conversations with families about strategies employed at the ECE program, and to work together for the success of the child. The use of BIRS graphs were particularly helpful to teachers, in being able to describe the intensity and frequency of challenging behaviors, and to identify patterns, i.e., determine if behaviors occurred during similar types of activities or times throughout the day.

The main implementation challenge cited by ECE program staff was lack of time to avoid lapses in completing, submitting, and entering BIRs in a timely manner. When a program leadership transition occurred in Summer 2014 and administrative staff had a break due to family leave, the use of BIRs dropped off markedly, until leadership stabilized and re‐committed to regular use of the BIRs in early 2015. In addition, it was during this same time that the LAUNCH evaluation team worked with the program to build an Excel version of the online BIRS in order for the program to continue to use the system after grant funding ended. The annual cost of the BIRS was considered prohibitive for this large ECE program ($100 annually per classroom for 12 classrooms).

In total, between July 31, 2013 and May 31, 2014, a total of 419 incidents were logged by one of the LAUNCH child care sites in the pilot test of the use of the BIRS through Multnomah Project LAUNCH. Incidents were attributed to 19 teachers in 9 classrooms and involved 31 children. The number of incidents logged per child ranged from 1 to 78, with an average of 13.5 over the 10 month period.

Figure 13 below shows the total number of incidents by type for the five children who had 20 or more incidents, accounting for 60% (n=253) of the total incidents logged. Overall, the majority of behaviors prompting an incident report were due to physical aggression and biting, which was categorized separately from other forms of physical aggression. There were also instances of the child running away and other unsafe behaviors. Generally, the number of incidents fell over time for each child, suggesting improved supports and behavior.


Figure 13. Total Incident Types for 5 Children in Time Period (n=253)

Summary: Early Childhood Mental Health Consultation in Child Care Findings

Results overall indicate implementation success in terms of successfully providing EC MHC supports to child care providers in Multnomah County. Provider outcomes show modest support for the effects of the consultation on staff knowledge, skills, and confidence in supporting children’s mental health. In particular, results found significant improvements in terms of knowledge of children’s mental health issues, as well as increased confidence in working with parents to address behavioral and related challenges. There were no significant reductions in staff stress, and no evidence that rates of turnover were positively impacted by consultation (Table 21).

Observational assessment conducted by MHCs also supported the success of the consultation and training in improving the quality of infant/toddler and preschool classrooms as related to supporting children’s social emotional development. Outcomes at the child level were not within the scope of this evaluation, although limited BIRS data suggested that staff were using a variety of strategies to support children with fairly significant behavioral problems. Further, overall, there were very few children dismissed from these child care settings because of behavior concerns (about .04%). This rate is low compared to some documented rates that have been found elsewhere (with 2‐3% of preschoolers being expelled at least once).12

12 Gilliam, W., & Shahar, G. (2006). Preschool and child care expulsion and suspension: Rates and predictors in one state. Infants & Young Children, 19(3), 228‐245.

0

10

20

30

40

50

60

70 Other: Self‐injury,disruption, non‐compliance, tantrums,property damage, verbalaggressionRunning away

Biting

Physical aggression


Enhanced Home Visiting through Mental Health Consultation


1. Mental Health Consultation in Home Visiting: Implementation Research Questions a. To what extent were key EC MHC in home visiting supports implemented as planned? b. What supported successful implementation? c. What barriers were encountered?

2. Mental Health Consultation in Home Visiting: Outcome Research Questions a. Provider Outcomes:

i. Do home visitors who receive EC MHC support show increased skills, confidence, and knowledge about early childhood mental health and mental health best practices?

ii. Do home visitors who receive EC MHC support provide higher quality classroom environments to support children compared to before receiving this service?

iii. Do home visitors who receive EC MHC support show decreased work‐related stress and reduced turnover?

b. Child & Family Outcomes: i. Do parents and children in the Healthy Families Oregon (HFO) site receiving EC MHC

support have more positive parenting‐related outcomes compared to parents and children in sites not receiving consultation?

Methods & Data

Evaluation of the EC MHC in home visiting used a similar approach to evaluating the impacts on home visiting staff as were described for child care providers (see previous). Consultation is currently being provided to one Healthy Families America program that includes 13 home visitors. In addition, an exploration of potential effects on parent and child outcomes using a secondary dataset was conducted.

Mental health consultation in home visiting logic model development. In addition to evaluation activities planned in the original evaluation design, the evaluation team collected additional information from MHCs and home visitors in order to develop a better understanding of the nature and process of Mental Health Consultation in home visiting. Specifically, we conducted two focus groups with home visitors, as well as interviews with supervisors and the mental health consultant to collect information about: (1) the types of activities and supports being provided; (2) their perceptions of the kinds of changes they were seeing either in their own work with families or with families themselves because of the supports provided by the MHC; (3) successful strategies for partnering with the consultants; and (4) barriers and challenges in working with the consultants. This information was compiled into a preliminary logic model for MHC in home visiting (see Appendix H). This information was also shared with the consultant team for program improvement purposes.

Mental health consultant contact notes and activity summaries. These forms, completed by the MHC and the EC PBIS specialist, provide detailed process and ‘output’ information about the type, frequency, and participants in consultation activities. The LAUNCH evaluation team developed a database for these forms, which were sent to PSU on a quarterly basis.


Provider outcome measures. The Mental Health Services Survey (MHSS)13 was used being used to measure the child care providers’ perceptions of the type and quality of consultation services being received, and their own knowledge and confidence in meeting children’s social‐emotional needs. The MHSS has been used in several studies of mental health consultation, and is recommended by the National Center for Effective Early Childhood Mental Health Consultation. Providers will also complete a subset of items from the Teacher Opinion Survey14, a measure of job‐related stress and support. All home visitors who received consultation are included in the evaluation. Surveys were completed at baseline (prior to consultation), 6 months after that, and annually thereafter. At the end of Year 5, 17 home visitors had completed baseline surveys 16 had completed 6‐month follow‐ups, 14 completed 12‐month follow‐ups, and 12 had completed 24‐month follow‐ups. By the end of Year 5, only 8 staff remained who had completed the baseline survey. Additionally, we collected baseline data from two control groups of home visitors (n=26): 18 from a neighboring county and 8 from another county that was not currently receiving mental health consultation. Table 24 below shows the demographics of these groups at baseline time periods.

Table 24. Characteristics of HV Program Staff & Families Served

LAUNCH HV Programs

(Y1‐5)

Comparison HV Program 1

(Y3‐4)


(Y4‐5)

Number of staff 3915 17 8

Staff represented in survey data 17 17 8

Mean length of time with program at baseline survey in months (sd) 16.7 (17.3)16 70.6 (55.5) 85.4 (67.4)

Median length of time with program at baseline survey in months 6.0 60.0 56.5

Staff leaving program prior to 12 month follow‐up survey 6% (1) 29% (5) 0

Staff age

Mean staff age in years at baseline survey (sd) 40.8 (13.3) 38.3 (11.9) 43.3 (10.5)

Staff race/ethnicity

African American/Black 12% (2) 6% (1) 0

Latino 29% (5) 53% (9) 0

White 59% (10) 35% (6) 100% (8)

Multiracial 0 6% (1) 0

Number of children served 857 ‐‐ ‐‐

13 Green, B. L., Everhart, M., Gordon, L., & Gettman, M. G. (2006). Characteristics of effective mental health consultation in early childhood settings: Multilevel analysis of a national survey. Topics in Early Childhood Special Education, 26(3), 142‐152. 14 Geller, S., & Lynch, K. (1999). Teacher Opinion Survey‐Revised. Richmond: Virginia Commonwealth University Intellectual Property Foundation and Wingspan, LLC. 15 One LAUNCH home visiting program received MHC services in years 1‐5 and are reflected in survey data. An additional LAUNCH home visiting program began receiving services in Y5 but are not reflected in survey data due to the lack of sufficient follow‐up period to draw comparisons. 16 Statistically significant difference in baseline mean length of time with program at baseline (p <.01), due to new staff with LAUNCH HV program being enrolled in the study, contrasted to comparison groups which only enrolled staff in the study at one point in time.


Program‐wide child & family outcomes. To assess the overall influence of consultation across families in the home visitation program we used a quasi‐experimental design that allowed us to compare selected outcomes for families and children in the Healthy Families Oregon (HFO) site receiving consultation compared to a HFO program not receiving consultation. We established a data sharing agreement to allow us to receive de‐identified data collected by the HFO statewide evaluation. Using this data, we examined demographic and risk characteristics at baseline, and compared parent‐reported outcomes at 6‐ and 12‐months around child maltreatment risks (depression, stress), parenting skills, and developmentally supportive parenting practices. Outcome analyses controlled for pre‐existing differences at program enrollment, including mother’s race and mean number of risk factors.

HFO data collection tools used by NPC Research in their statewide evaluation include the following measures, collected from parents at family intake and every 6 months thereafter: (1) the short form of the Parenting Stress Index17; (2) a two‐item depression screener, the PHQ‐218; and the Parent‐Child Activities scale19, a validated measure of the frequency of developmentally supportive parent‐child interactions. Additionally, we will use the Home Observation Measure of the Environment20, a measure of the quality of developmentally supportive materials and interactions provided by parents. The HOME is collected when children are 12, 24, and 36 months of age by a trained home visitor. Rate of identification and referral for social‐emotional problems is also tracked by all HFO clients, and will be compared for consultation vs. non‐consultation teen parent clients. These measures comprised our key outcomes for this component of the study. We examined outcomes at 12 months post‐HFO enrollment.

Findings: Implementation Successes and Challenges

Mental Health Consultation in Home Visiting: Successes The consultation being provided to the Healthy Families Oregon (HFO) team was well‐received by home visitors, whose primary feedback regarding how the service could be improved was to request more hours of consultation time. The success of this effort was attributed in large part to the time invested in developing positive, long‐term relationships between the MHC and the home visitors, as well as working collaboratively with HFO supervisors to ensure seamless integration of the ECPBIS framework into existing HFO curricula and models, and continually adjusting the MHC activities and schedule to respond to emerging needs of the home visiting team.

Results of interviews and focus groups with staff suggested other successes in terms of specific components of the model. Program supervisors described the process of using the Benchmarks of

17 Abidin, R. R. (1995). Parenting Stress Index: Professional Manual (3rd Ed.). Odessa, FL: Psychological Assessment Resources, Inc. 18 Gjerdingen, D., Crow, S., McGovern, P., Miner, M. (2009). Validity of the two‐question screen and PHQ‐9. Annals of Family Medicine, 7(1), 63‐70. 19 Love, J., Kisker, E., Ross, C. M., Schochet, P. Z., Vogel, C., Fuligni, A. S., & Brady‐Smith, C. (2002). Making a difference in the lives of infants and toddlers and their families. U. S. Department of Health and Human Services. 20 Bradley, R. H., & Caldwell, B. (1984). 174 children: A study of the relationships between the home environment and cognitive development during the first five years. In A. W. Gottfried (Ed.), Home environment and early cognitive development. Orlando, FL: Academic Press.


Quality (BOQ) tool with the MHC and discovering that their screening and referral process needed strengthening. As a result of using the BOQ and working with the MHC, the program made the screening process a stronger piece of relationship‐building with families, and has built‐in additional follow‐up steps and referral connections for home visitors to do in partnership with families.

Reflective group supervision was described as an invaluable service provided by the MHC. The team appreciated being able to raise training topics of interest, and have dedicated time to debrief and strategize for specific cases.

Another implementation success for the MHC in home visiting was the development of materials for ECPBIS tailored to home visiting; these were subsequently used to support training and work with home visitors. In particular, the BOQ was adapted for a home visiting environment and has proved valuable for supporting the program, as noted above. The MHC is also working with the program to support a new system for maternal depression screening using the PHQ‐9.

Mental Health Consultation in Home Visiting: Challenges Over the course of the 5 years, a number of challenges were encountered, and most were overcome. By far the biggest challenge was the initial implementation of consultation at a different HFO program site during Year 1. After several months of attempting to work with this site, it became apparent that the supervisor and staff were not ready to work with a consultant. The supervisor in particular proved challenging, and made it difficult for the consultant to partner with the home visitor teams. Eventually, the decision was made to relocate the consultant to a different program site, which actively solicited the support of the consultant. Consultation at this site went much more smoothly, although some challenges were encountered mostly related to the amount of time needed for the MHC to build relationship with the team, and clarifying how the MHC can support the program; new home visitors take time to figure out how to build a relationship with the consultant and overall workload among home visitors was noted as a barrier to some home visitors seeking out one‐on‐one consultation time. There was also a perceived need for more consultant hours to provide more one‐on‐one support to both home visitors and families

Findings: Outcomes for Home Visitors

Below we present the key outcomes related to changes in home visitor knowledge, skills, and confidence. In addition to this information, a Research Brief focused on summarizing the implementation and outcomes for the consultation in home visiting was developed and is included in Appendix I.

Figures 14 through 18 below show the change over time on key dimensions for home visitors at baseline, 6 and 12 month follow up, for both the HFO site receiving consultation as well as the two quasi‐experimental control groups. As can be seen in Figure 14, the HFO group that received consultation rated themselves significantly lower in their confidence in involving parents in conversations and work related to children’s mental health, compared to comparison site 1. While there was a significant improvement over time, site 1 also showed improvement over time. Site 2 staff did not significantly improve over time. A similar pattern can be seen in terms of knowledge of children’s mental health (Figure 15). HFO staff receiving consultation significantly improved their knowledge in this area; however, staff at site 1 who did not receive consultation also showed improvements. No group‐by‐time interactions were significant.


Figure 14. LAUNCH v Comparison HV Staff Parent Involvement Domain

*Statistically significant difference (p<.05) between LAUNCH and Comparison 1 HV groups at baseline; within LAUNCH groups from Baseline to 6/12 months; and within Comparison 1 groups from Baseline to 12 months. **Statistically significant difference (p<.01) between LAUNCH and Comparison 1 HV groups at 12 months.

Figure 15. LAUNCH v Comparison HV Staff Knowledge of Children’s Mental Health Domain

*Statistically significant difference (p<.05) between LAUNCH and Comparison 1 HV groups at 6 months; within LAUNCH groups from Baseline to 12 months.

Figure 16 shows improvements in knowledge of EC PBIS. Not surprisingly, only the LAUNCH supported home visitors showed increases in knowledge of EC PBIS.

3.74.2

4.24.4

4.54.7

4.2 4.3 4.4

1.0

2.0

3.0

4.0

5.0

Baseline* 6 Mo 12 Mo**

LAUNCH HV(N=15)*

Comparison 1 HV(N=12)*

Comparison 2 HV(N=8)

3.4 3.7

4.24.1

4.44.5

44.2 4.2

1.0

2.0

3.0

4.0

5.0

Baseline 6 Mo* 12 Mo

LAUNCH HV(N=15)*




Figure 16. LAUNCH v Comparison HV Staff Knowledge of EC PBIS

Figure 17 shows changes over time in staff stress. Counter to expectations, the team receiving consultation actually reported slightly more work‐related stress at the 12 month follow up, while there were no changes in stress for the other two groups. The LAUNCH group was not significant different from the comparison groups at either time point, however.

Figure 17. LAUNCH v Comparison HV Staff Stress Domain

*Statistically significant difference (p<.05) within LAUNCH group from Baseline to 12 months.

Figure 18 shows a similar pattern of results, with significant change over time only for the LAUNCH home visitor group, which reported an increase in the quality of program leadership related to children’s mental health over time. There was no change in the comparison groups for this domain.

2.5

2.73.2

2.12.4 2.42.3

2.32.1

1.0

2.0

3.0

4.0

5.0

Baseline 6 Mo 12 Mo

LAUNCH HV (N=14)



1.9

1.8

2.3

1.9

1.9

1.9

2.5

2.5 2.8

1.0

2.0

3.0

4.0

5.0

Baseline 6 Mo 12 Mo

LAUNCH HV (N=15)*




Figure 18. LAUNCH v Comparison HV Staff Perception of Program Leadership Domain

*Statistically significant difference (p<.05) within LAUNCH group from Baseline to 12 months.

Three other domains did not show either growth over time nor differences between the consultation group and the comparison home visitors: (1) Efficacy in children’s mental health; (2) Knowledge of adult mental health issues; and (3) efficacy in adult mental health.

Table 25 below shows the data for all home visitors who responded at each follow up time point (i.e., not change over time). These results show differences between those providers survey at baseline and those surveyed at 12 months (increased knowledge of children’s mental health) and 24 months (confidence involving parents, knowledge of adult mental health, and quality of program leadership specific to mental health issues).

Table 25. LAUNCH HV Staff Outcome Domains Mean (sd) 5‐pt scale, Higher score indicates strength

Baseline(N=17)

6 Mo(N=15)

12 Mo(N=16)

24 Mo (N=12)

36 Mo(N=9)

Parent involvement 3.7 (.8) 4.2 (.6) 4.2 (.5) 4.5 (.5)* 4.7 (.4)*

Knowledge of children’s mental health 3.4 (.9) 3.7 (.6) 4.1 (.5)* 4.5 (.4) * 4.6 (.5) *

Efficacy in children’s mental health 4.2 (.8) 4.3 (.5) 4.3 (.4) 4.3 (1.1) 4.7 (.5)

Knowledge and comfort using EC PBIS 2.4 (1.3) 2.7 (1.3) 3.1 (.9) 3.5 (.9) 3.2 (.4)

Knowledge of adult mental health 3.2 (1.1) 3.5 (.8) 3.8 (.9) 4.3 (.6)* 4.6 (.4) *

Efficacy in adult mental health 4.4 (.6) 4.4 (.6) 4.7 (.5) 4.6 (.5) 5.0 (0)*

Staff stress 1.9 (.9) 1.8 (.8) 2.3 (.9) 2.2 (.8) 2.0 (1.0)

Program leadership 3.6 (.9) 3.9 (.9) 4.2 (.5) 4.5 (.5)* 4.7 (.7) *

Program vision ‐‐ 3.3 (.9) 3.7 (.5) 3.8 (.6) 4.0 (0)

Program support 3.9 (.6) 4.1 (.8) 4.2 (.5) 4.6 (.5)* 4.7 (.4) **Significant difference in means (p<.05) appears at this time point from baseline within the domain.

Summary: Early Childhood Mental Health Consultation in Home Visiting Findings

The data collected from comparison teams of home visitors in two different communities suggests that these two teams started out feeling more knowledgeable about, and comfortable involvement parents in talking about, children’s mental health, compared to the home visiting team receiving the LAUNCH

3.73.9

4.2

4.24.1

4.1

3.33.4

3.6

1.0

2.0

3.0

4.0

5.0

Baseline 6 Mo 12 Mo

LAUNCH HV(N=15)*




intervention. One area in which the LAUNCH team scored higher than the comparison group was in terms of their knowledge of EC PBIS; this is not surprising, as the EC PBIS was a unique aspect of the consultation that would not be expected to be part of general supervision or training provided to other home visiting teams. The overall pattern of findings suggest that consultation in Project LAUNCH is being provided to a team of home visitors who may be particularly in need of support in this area, something to consider when making future decisions about where to place needed consultation resources. Moreover, these data suggest that this consultation may have been associated with increases in some areas of skills and knowledge for home visitors.

Outcome data for families and children served by comparison site 2 and those served by the home visiting program receiving MHC are presented below. Comparing demographics of families served by the two programs revealed that the LAUNCH home visiting program served a significantly higher proportion of families who were African American/Black, and families overall who had a significantly lower mean number of risk factors at screening for program eligibility. The LAUNCH home visiting team served virtually no teen mothers due to the presence of an age‐specific home visiting program serving teen mothers in Multnomah County. LAUNCH home visiting families also were less likely to be less than a high school graduate and have relationship problems, but more likely to report substance abuse issues.

Table 26. Characteristics of Families Served by LAUNCH & Comparison HV Programs July 1, 2011 through June 30, 2014

LAUNCH

HV Program Comparison HV Program 2

Number of children served 235 164

Mother’s Race/Ethnicity

African American/Black** 12% (28) 2% (4)

American Indian/Alaska Native 1% (2) 2% (3)

Asian American 2% (4) 1% (1)

Native Hawaiian/Pacific Islander 0 1% (1)

Latino 17% (41) 18% (29)

White 60% (141) 69% (113)

Multiracial 6% (13) 6% (9)

“Other” unspecified 1% (3) 1% (2)

Unreported 1% (3) 1% (2)

Family Risk Factors

Mean number of risk factors present at screening (sd)* 3.3 (1.3) 3.6 (1.3)

Risk 1: Teen mother** 0.4% (1) 11% (16)

Risk 2: Single mother 75% (177) 81% (133)

Risk 3: Late prenatal care 15% (31) 20% (33)

Risk 4: Inadequate prenatal care 3% (7) 0

Risk 5: Less than high school graduate* 16% (37) 25% (40)

Risk 6: Parent/s are unemployed 64% (148) 57% (94)

Risk 7: Difficulties paying for living expenses 91% (213) 89% (144)

Risk 8: Depressive symptoms 35% (80) 39% (64)

Risk 9: Relationship problems** 25% (58) 39% (64)

Risk 10: Substance abuse issues** 10% (24) 3% (4)


*Significant difference in proportions between groups (p < .05). *Significant difference in proportions between groups (p < .001)

Table 27 shows comparisons between LAUNCH and comparison home visiting groups on the child maltreatment risk, parenting skills, and supportive parenting practices at 6 and 12 months, controlling for mean number of risks at screening and mother’s race, resulted in no significant differences between groups.

Significantly more change over a 6‐month period within the LAUNCH group, however, was observed in the area of reading books more frequently.

Table 27. LAUNCH & Comparison Group HV Program Child & Family Outcomes

Mean (sd) Higher scores indicate strength

LAUNCH HV Program


6 Months 12 Months 6 Months 12 Months

LAUNCH HV Program N=132 N=114 N=97 N=95

Child maltreatment risks

Parental depression 2.1 (.8) 2.0 (.7) 2.3 (.7) 2.2 (.8)

Parenting stress 2.1 (.9) 2.0 (.8) 2.2 (.8) 2.2 (.8)

HOME total mean ‐‐ 40.9 (3.6) ‐‐ 41.1 (4.6)

Parenting skills ladder

Retrospective mean 1.0 (.7) 1.0 (.7) 1.1 (.7) 1.0 (.6)

Current mean 1.5 (.4) 1.6 (.4) 1.6 (.4) 1.5 (.4)

Supportive parenting practices

Sing songs 5.5 (1.0) 5.5 (.8) 5.2 (1.1) 5.4 (1.0)

Tell stories 5.6 (.9) 5.6 (.7) 5.6 (.9) 5.6 (.8)

Read books 4.9 (1.1) 5.3 (.8)** 4.8 (1.0) 5.1 (1.1)

Play games 5.5 (.8) 5.7 (.6) 5.5 (1.0) 5.6 (.8)*Significant difference within group over time, paired samples t‐test (p < .01).

Family Strengthening: Parenting Education Series


1. Parent Education Series: Implementation Research Questions a. To what extent were Evidence‐Based Parenting Education Series implemented as planned

and to fidelity? b. What supported successful implementation? c. What barriers were encountered?

2. Parent Education Series: Outcome Research Questions a. Parent Outcomes:

i. Do parents report improved parenting skills and decreased parent‐related stress following participation in the Parenting Education Series, compared to before participating?

b. Child Outcomes:


i. Do parents report fewer child behavioral challenges following participation in the Parenting Education Services, compared to before participating?

Methods & Data

Incredible Years outcomes. To evaluate the effectiveness of the IY programs implemented through LAUNCH, the following measures were used at the beginning (baseline) and end (posttest) of each group.

Child outcomes: (1) the Eyberg Child Behavior Inventory (ECBI) for children ages 24 months and older; and the (2) Devereux Early Childhood Assessment (DECA).

Parent outcomes: (1) the short form of the Parent Practice Interview a validated measure of parenting used in evaluations of the Incredible Years program (only the warmth/responsive subscale, and the harsh/inconsistent discipline subscales were used); the short form of the Parenting Stress Index (PSI), and the Patient Health Questionnaire‐9 (PHQ‐9), a widely used brief depression screening instrument.

All measures were completed by the parent at the beginning of the IY parenting group (baseline) and then at the end of the group 10‐14 weeks later (post‐test). Consent for sharing child and family‐specific information was obtained from parents by the MHC at the point of enrollment in the group. All families participating in this service were included in the evaluation if they consented to participate. Five IY groups had been implemented and evaluated, including 86 unique parents attending at least one group session (5 parents participated in 2 series); 57 completed half or more sessions (considered full dosage by program developers); and 46 of these provided both complete pre‐ and post‐tests. For series that occurred in Years 1‐4 of the grant, the groups were offered as stand‐alone Incredible Years series (2 preschool, 1 infant‐toddler, and 1 Attentive Parenting series). In Year 5, the IY series was structured with an initial 6‐weeks of “Balancing Lessons” (BL) programming based on cognitive‐behavioral principles to strengthen families’ skills in stress management and self‐care first. This series set (BL followed by 12 weeks of IY infant‐toddler groups) was designed to address families’ pressing needs for stress management and support, and was implemented, in part, to help increase family engagement and participation in the subsequent IY parenting skills group.

Incredible Years fidelity. In addition, model fidelity is being monitored using the Incredible Years fidelity checklists. These are a part of the established intervention processes. An IY supervisor collects the fidelity checklists for each session in the Incredible Years series, and provided this information to the evaluation team. The checklist is comprised of a variety of items assessing whether or not IY workshop leaders cover specific topics and follow established IY protocols.

Findings: Implementation of Parenting Groups

Table 28 below shows characteristics of preschool and infant‐toddler Incredible Years groups offered in Years 2 through 5 of LAUNCH.

Table 28. Incredible Years Group Characteristics Y2 Y3 Y4 Y4 Y5 PreK IY PreK IY I‐T IY AP BL & I‐T IY

Number of sessions in series 14 14 9 6 18


Y2 Y3 Y4 Y4 Y5 PreK IY PreK IY I‐T IY AP BL & I‐T IY

Number of parents attending at least one session 30 24 11 13 13

Number of parents receiving full “dosage”, e.g., half or more of sessions 21 9 7 9 11

Number of parents with full dosage and pre‐post tests 16 8 6 8 8

% of parents receiving full dosage (target 75%) 76% 38% 64% 69% 85%

Table 29 below shows characteristics of participants and children in IY groups who received 50% of more of the IY sessions and who completed pre and post‐test surveys. Series were offered in connection with different programs involved in the LAUNCH project, which may account for some of the differences we see in child race/ethnicity across series.

Table 29. Characteristics of Incredible Years Families with Dosage & Pre‐Post Tests Y2 Y3 Y4 Y4 Y5 PreK IY PreK IY I‐T IY AP BL & I‐T IY Totals

Number of parents 15 8 6 8 8 45

Number of children 12 7 5 8 7 39

Relationship to child

Mother 73% (11) 88% (7) 83% (5) 88% (7) 88% (7) 37 (82%)

Father 27% (4) 13% (1) 17% (1) 13% (1) 13% (1) 8 (18%)

Child mean age in years (sd) 3.6 (1.0) 3.7 (.8) .3 (.2) 2.2 (.5) 1.9 (.7)

Child gender

Male 67% (8) 43% (3) 40% (2) 38% (3) 29% (2) 18(46%)

Female 33% (4) 57% (4) 60% (3) 63% (5) 71% (5) 21 (54%)

Child race/ethnicity

White 92% (11) 43% (3) 60% (3) 50% (4) 57% (4) 32 (77%)

Black/African American 0 14% (1) 0 0 14% (1) 2 (5%)

Latino 0 14% (1) 20% (1) 13% (1) 29% (2) 5 (13%)

Multiracial 8% (1) 29% (2) 20% (1) 38% (3) 0 6 (15%)

Table 30 below shows the differences in baseline characteristics between families who received the full dosage of IY session to those who did not. As can be seen, there were no significant differences in any of the baseline characteristics or pretest scores, indicating no differential attrition, at least based on the characteristics measured.

Table 30. Pre‐Test Characteristics of Dosage v Non‐Dosage Incredible Years Families

Dosage Non‐Dosage

Level of Sig.

Child mean age in years (sd) 2.5 (1.3) 2.8 (1.6) ns

Child gender

Male 47% (20) 53% (10) ns

Female 53% (23) 47% (9) ns


Dosage Non‐Dosage

Level of Sig.

Child race/ethnicity

White 68% (30) 50% (9) ns

Child of Color 32% (14) 50% (9) ns

Lower score indicates strength

PSI‐SF parent general distress mean 16.3 19.2 ns

PSI‐SF parent‐child dyadic interaction mean 11.9 13.0 ns

PHQ‐9 parent depression mean 4.3 5.5 ns

PPI parent use of harsh discipline mean 1.5 1.5 ns

ECBI child problem behavior frequency mean 117.1 118.7 ns

ECBI child problem behavior intensity mean 9.2 13.5 ns

DECA child behavioral concerns t‐score mean 60.4 59.4 ns

Higher score indicates strength

DECA child initiative t‐score mean 46.8 47.3 ns

DECA child attachment/relationships t‐score mean 46.6 46.5 ns

DECA child self‐control t‐score mean 47.3 45.3 ns

DECA total child protective factors t‐score mean 43.4 43.4 ns

Findings: Outcomes of IY and Parenting Groups

Table 31 below shows outcomes for the key domains of parent and child outcomes pre‐ and post‐ participation in the IY series. As can be seen, there were no significant improvements in parenting stress, depression, or use of harsh discipline. However, there were significant improvements in parents’ perceptions of the frequency and intensity of child problem behaviors as well as improvements in child positive behaviors and self‐regulation. This suggests that although parents may not report or perceive changes in their own behavior, they do perceive their children differently at the end of the IY group, perhaps reflecting changes in their own parenting abilities or ability to deal with children’s behavior that were not measured or captured in the parenting measures used for evaluation.

Table 31. Parent & Child Incredible Years Outcomes for Families with Dosage

Pre‐Test (sd)

Post‐Test (sd)

Level of Sig.

Decreased score indicates improvement

PSI‐SF parent general distress mean (N=44) 16.8 (5.9) 16.9 (7.7) ns

PSI‐SF parent‐child dyadic interaction mean (N=44) 12.6 (7.2) 11.9 (7.2) ns

PHQ‐9 parent depression mean (N=43) 4.5 (4.8) 4.3 (4.4) ns

PPI parent use of harsh discipline mean (N=23) 1.6 (.3) 1.4 (.2) ns

ECBI child problem behavior frequency mean (N=30) 117.3 (25.4) 107.2 (15.8) p<.05

ECBI child problem behavior intensity mean (N=30) 10.5 (6.4) 5.5 (4.6) p<.001

DECA child behavioral concerns t‐score mean (N=22) 61.1 (9.1) 58.7 (7.4) ns

“This class changed my life! It made me much more confident as a parent and gave me some great, practical tools. I’m very grateful I had the opportunity to learn from [the MHCs].”

– IY Series Participant


Pre‐Test (sd)

Post‐Test (sd)

Level of Sig.

Increased score indicates improvement

DECA child initiative t‐score mean (N=37) 47.2 (7.9) 53.9 (9.3) p<.001

DECA child attachment/relationships t‐score mean (N=37) 46.9 (9.3) 53.2 (11.1) p<.001

DECA child self‐control t‐score mean (N=31) 46.8 (9.7) 52.0 (9.3) p<.01

DECA total child protective factors t‐score mean (N=37) 43.3 (9.6) 48.9 (9.1) p<.001

Table 32 below examines these outcomes for children of different racial/ethnic background. Because of the small sample sizes we group all non‐white children (primarily multi‐racial) together and compared their outcomes to those of white children. As can be seen, while there were baseline (pretest) differences such that parents of children of color reported higher levels of harsh discipline, there were no significant race‐by‐group interactions. While overall both groups of parents reported improvements in child behavior, the change over time was significant only for White parents and children (although this may be due to the small sample sizes).

Table 32. Pre‐Post Test Characteristics of Incredible Years Families with Dosage by Child Race

Mean (sd)

White Children (N=24)

Children of Color (N=14)

Level of Sig.Between Groups Pre Post Pre Post

Lower score indicates strength

PSI‐SF parent general distress mean 15.0 (5.9) 15.0 (7.2) 18.6 (6.0) 18.2 (7.8) ns

PSI‐SF parent‐child dyadic interaction mean 11.7 (7.1) 10.6 (6.4) 13.4 (8.4) 13.7 (8.2) ns

PHQ‐9 parent depression mean 3.8 (3.7) 3.5 (3.5) 6.8 (3.6) 5.7 (3.0) ns

PPI parent use of harsh discipline mean 1.5 (.2) 1.4 (.2) 1.8 (.3) 1.6 (.3)

p < .05 at pre

ECBI child problem behavior frequency mean 115.5 (19.4) 107.2 (15.0 120.7 (37.7) 102.7 (12.2) ns

ECBI child problem behavior intensity mean 10.6 (4.2) 5.7** (3.7) 8.6 (6.6) 4.6 (2.8) ns

DECA child behavioral concerns t‐score mean 57.8 (8.9) 58.0** (5.1) 65.8 (7.3) 59.2 (11.4)

p < .05 at pre

Higher score indicates strength

DECA child initiative t‐score mean 50.9 (6.5) 57.6 (9.5) 42.8 (8.2) 51.0*** (8.2)

p < .05 at pre

DECA child attachment/relationships t‐score mean 49.8 (8.1) 56.5** (9.7) 44.5 (10.2) 49.2 (8.4)

p < .05 at post

DECA child self‐control t‐score mean 49.9 (7.0) 55.0* (7.5) 45.2 (12.8) 46.8* (11.9) ns

DECA total child protective factors t‐score mean 46.1 (7.6) 51.7** (7.4) 40.8 (12.1) 46.8*** (10.7) ns


* Statistically significant pre‐post difference in means p < .05. ** Statistically significant pre‐post difference in means p < .01. ***Statistically significant pre‐post difference in means p < .001.

Family Strengthening: 211infoFamily


1. 211infoFamily: Implementation Research Questions a. To what extent were 211infoFamily services implemented as planned?

i. To what extent did families of color access 211infoFamily services? b. What supported successful implementation? c. What barriers were encountered?

2. 211infoFamily: Outcome Research Questions a. Provider Outcomes: (Note these questions are addressed in the Systems Evaluation section

under “Public Awareness”) i. Are child‐serving providers (physicians, home visitors, and child care providers)

knowledgeable about 211infoFamily? ii. To what extent do child‐serving providers utilize 211infoFamily to help families

connected with needed services? b. Parent Outcomes:

i. Are parents who contact 211infoFamily successfully connected to needed services? ii. Are parents satisfied with the supports received from 211infoFamily?

Methods & Data

Evaluation of the 211infoFamily services involves two primary data sources: the electronic caller database maintained by 211infoFamily, and a System User telephone survey conducted by the LAUNCH evaluation team.

Electronic Caller Database. Process and outcome evaluation of the 211infoFamily service (former also the Parent Helpline) included pre‐post analysis of the types of calls made to the 211info Family, the number of referrals for social‐emotional issues, and the rate of successful call resolution/referral. This information is tracked through the 211infoFamily electronic call database. The evaluation team set up a system for receiving regular (quarterly) reports of required information from the 211infoFamily data system. This data included all callers and calls to the 211infoFamily.

System User Telephone Survey. Additionally, PSU conducted a telephone survey to collect process and outcome information from a targeted sample of 211infoFamily callers. The Telephone Interview (see Appendix J for Year 5 version) includes questions about the users’ perception of the ease and accessibility of the 211infoFamily, the extent to which their needs were met, and whether or not there was follow‐up from 211infoFamily after the call to ensure that needed services were received.

Obtaining consent to follow up with 211infoFamily callers was challenging throughout the life of the project, and various strategies were implemented to increase the rate of consents. A primary challenge was the reluctance of the 211infoFamily Parent Educator to ask for this additional consent, especially for callers whose issues were perceived as more instrumental/basic needs and with whom less direct


support and advice was provided. Further, there were significant staff turnovers during Years 4 and 5 that led to fewer families being asked to consent to participate.

The original evaluation plan estimated that the evaluation follow‐up would include a total of 200 system users (100 parent callers and 100 provider callers) who agreed to be contacted by the evaluation team. Ultimately, we were able to contact and interview 174 callers (primarily parents, due to relatively low volume of provider calls), representing 42% of those who provided consent to be contacted for follow up. Table 33 below shows a description of the number of 211info Family callers/users, and how many received referrals, provided consent, and were interviewed by the evaluation team.

Table 33. 211infoFamily Callers Y1 Y2 Y3 Y4 Y5 Total

Number of 211infoFamily callers 12 184 806 579 676 2,257

Number of 211infoFamily email users ‐‐ ‐‐ 8 25 76 109

Number of 211infoFamily text users ‐‐ ‐‐ 28 89 123 240

Number of 211infoFamily callers receiving referrals 12 184 703 477 651 2267

Number of 211infoFamily callers consenting to PSU follow‐up interview 1 22 145 142 106

416

Number of 211infoFamily callers completing PSU follow‐up interview 0 9 38 47 80

174 (42%)

Table 34 below shows the demographic characteristics of all 211infoFamily callers as compared to those who were interviewed for the evaluation, as well as overall Multnomah County demographics. As can be seen, there were no significant differences between those interviewed and the general population of 211infoFamily callers in terms of basic demographic characteristics.

Table 34. 211infoFamily Caller & Target Child Characteristics21

Interviewed

Callers All

Callers Multnomah County22

Target Child Race/Ethnicity N=14323 N=1,571 N=90,358

African American/Black 31% (44) 22% (348) 12%

American Indian/Alaska Native 8% (11) 4% (67) 4%

Asian American 3% (5) 3% (44) 11%

Latino 20% (29) 20% (318) 20%

Native Hawaiian 6% (8) 2% (24) 2%

White 51% (73) 38% (603) 44%

“Other” (unspecified) 10% (14) 14% (227) 17%

Target Child Age N=157 N=1,386

0‐3 y/o 50% (79) 62% (863) ‐‐

>3‐5 y/o 29% (46) 17% (234) ‐‐

21 Totals may not equal 100% due to callers endorsing multiple categories. 22 Based on U.S. Census estimated, retrieved September 27, 2013 from www.census.gov/popest/data/counties/asrh/2012/CC‐EST2012‐ALLDATA.html 23 Race/ethnicity data on target child of interviewees were not available for Y2 and 22 interviewees in Y3‐5 did not provide this data.


Interviewed

Callers All

Callers Multnomah County22

>5‐8 y/o 16% (25) 18% (248) ‐‐

>8 y/o 4% (7) 3% (41) ‐‐

Type of Caller N=165 N=1,767

Parent 84% (139) 78% (1,385) ‐‐

Mother 76% (125) ‐‐ ‐‐

Father 8% (14) ‐‐ ‐‐

Grandparent or other relative 6% (10) 4% (73) ‐‐

“Concerned party” 2% (4) 11% (201) ‐‐

Service provider 7% (12) 6% (108) ‐‐

Referral Source N=1,370

Health care provider ‐‐ 2% (32) ‐‐

Other service provider ‐‐ 38% (518) ‐‐

Friend or family ‐‐ 12% (159) ‐‐

211info ‐‐ 22% (299) ‐‐

Self, prior caller ‐‐ 20% (271) ‐‐

Advertisement, directory, Internet ‐‐ 7% (91) ‐‐

Findings: 211info Family Implementation

In terms of implementation, over the course of the 5 years, 211infoFamily received almost 3,000 calls from parents seeking information, advice, and resources for their family. The majority of these calls occurred during Years 3 through 5, which corresponds to two major shifts within this strategy for Project LAUNCH. First, was the decision during Year 2 to shift the 211infoFamily service provider organization to the 211info organization located in Multnomah County. Originally the service was being offered by an organization that had successfully supported a Parent Helpline in Lane County (Eugene), Oregon; however, after the first year it was clear that there were implementation problems related to this provider, primarily related to their lack of knowledge of the complex network of services in Multnomah County. Second, in response to the low rate of calls during Years 1 and 2 there were redoubled efforts on the part of Multnomah LAUNCH to strengthen public awareness activities, both broad‐scale activities based on marketing approaches and community presentations to service providers, as well as by more systematically integrating dissemination of 211infoFamily into the service delivery strands being supported by the grant (e.g., START trainings, EC MHC, home visiting, etc.). Including only years 3‐5, there was an average of 4.4 calls per day (assuming a 365 day period per year). Assuming a 5‐day workweek (more realistic given the FTE allocated to the 211infoFamily Parent Educator) this equates to about 9 calls per day, likely approaching the maximum that could be handled by a single call Parent Educator, given the time often needed for follow up and research related to finding appropriate referral information.

The table above (Table 34) also shows the demographic characteristics of callers compared to Multnomah County population demographics. These data suggest that 211infoFamily services are used by a relatively greater proportion of African American/Black families than would be expected based on the general population. More than half of callers were parents of at least one child who was age 0‐3 years, and the large majority (78%) were the child’s mother. Families reported a variety of ways that they had learned about 211infoFamily. The most common was from another service provider,


suggesting that the public awareness and information sharing supported by LAUNCH that encouraged community presentations and information sharing by 211infoFamily staff directly to service providers may have been an effective method. Relatively few, however, were referred by primary care (2%) or because they had seen advertisements (7%).

Findings: Outcomes of 211infoFamily

Results from the evaluation of the 211Info Family were summarized in a Research Brief (see Appendix K). This report summarizes the results of interviews conducted with 211infoFamily users between 2012 and 2015. Overall, 211infoFamily callers were overwhelmingly positive about their experience with the service. The information, referrals, and over‐the‐phone parenting strategies were seen as valuable supports. Callers’ needs were often met and in some cases, expectations were far exceeded. At the same time, interviewees shared insights to areas for continued attention and improvement both within 211info as well as the broader social service system in Multnomah County.


Summary & Conclusions Key Stakeholder Report of LAUNCH Major Accomplishments

At the end of the last round of qualitative key stakeholder interviews, LAUNCH partners were asked to describe what the major successes of LAUNCH had been, specifically, “what is different” because of Project LAUNCH? Major successes that were described included:

1. The expansion and improvement of cross‐sector collaborations and relationships, especially across medical and early childhood organizations, and in particular that primary care and pediatric providers are working more closely with other early childhood providers, have a better understanding of each other’s cultures, demands, and priorities, and that LAUNCH created a space and place for conversations across these sectors to facilitate these changes.

2. LAUNCH’s success at promoting, expanding, and integrating EC PBIS into more and new settings, including supporting the modification of EC PBIS for domestic violence shelters; training for kindergarten teachers in EC PBIS, and working with the Child Care Resource and Referral agency to create a joint EC PBIS training calendar and network of EC PBIS coaches.

3. Investments in Mental Health Consultation in home visiting and associated evaluation and sharing of ‘lessons learned”. Respondents mentioned that LAUNCH’s support for this model has increased local understanding of the importance of MHC in home visiting and created a pathway for sustainability through changes in mental health providers’ understanding of how the model can work in home visiting, as well as through linkages to MIECHV and other home visiting initiatives statewide and locally.

“LAUNCH is an interesting place where a bunch of disparate people come together, it’s a nice coming together place where we’re all really clear we’re paying attention to young children.”


“I think that’s definitely helped us improve our efforts to sustain and grow EC PBIS in our system; that’s where I’ve had the most involvement and so I think that some things that are different are we have the TPOT, an instrument we were exploring but we really didn’t have before LAUNCH came on and some of the dollars we brought in training on that and also consultation to see that it’s a really good instrument for taking a look at environments and specifically how those environments support social and emotional development; before that it was based on individuals with expertise making observations which is good but this particular instrument helps us get more continuity about what we’re looking for so I think that’s different for us and LAUNCH has been a big part of that.”



4. Expansion of 211info Family, and work to continually improve these services, and doing mid‐course corrections as needed.

5. LAUNCH’s impact on the new Multnomah County Early Learning HUB activities, specifically through LAUNCH support for workforce development via implementing Communities of Practice in Early Childhood Mental Health and EC PBIS.

6. LAUNCH’s role in supporting innovation through small, focused efforts, such as the Oregon Public Health Initiative work to developing trainings for reducing screen time in child care, and in supporting the emerging ACES workgroup.

System Challenges and Lessons Learned

Over the course of the five year project, a number of systems‐level challenges were encountered, and lessons learned about implementing the LAUNCH services and approach.

“I definitely feel like the consultation piece in home visiting is pretty big – from the perspective of this being a big shift in the consultants’ model and how that co‐located service with mental health has improved home visiting; this is the one I pay most attention to – I think because of LAUNCH that some of the main drivers/players in MHC services have had shifts in their thinking – they now see how well the model works or can work in Home Visiting so that’s key to driving the service forward.”


“Connection with 211FamilyInfo, rolling it out beyond Multnomah County has been a huge success.”


“Because EC PBIS has now become a formal Community of Practice in the new Hub it is a huge win for sustainability and LAUNCH has been a partner in making that happen. That’s going to continue to grow, we have work to do but that’s going to continue because it’s getting put into some formal structures, like in the hub.”


“What I also think LAUNCH has done well is provide a platform to experiment with innovation; it is a platform for innovation opportunities especially in a world where everything has to be an evidence‐based practice, which limits you to IY and NFP and those programs that have had enough funding to become an evidence based practice.”



1. Despite a number of attempts to engage parents as partners in the YCWC and LAUNCH leadership, ultimately, this goal was never fully achieved. Each year of LAUNCH brought new strategies as partners attempted to engage parents most meaningfully. In retrospect, the original strategy, which involved providing funding to culturally specific organizations to mentor one or more parents to attend and participate in the YCWC meetings may have been more successful if it were implemented after the YWCW had a stronger sense of initial cohesion and purpose. The most successful parent engagement strategy involved convening a separate group of parents to provide input and guidance around specific issues (e.g., the best way to advertise the 211Fmaily Info supports). However, even this more focused group proved difficult to sustain. Final attempts to build in other systems‐level efforts to bring parent voice to community‐wide planning provided difficult as well.

2. Although progress was made to identify systemic “gaps” in the child screening, assessment and referral process, the goal of seamless, streamlined screening and referral systems provide difficult. At least part of the challenge was the lack of consensus across providers about the primary function of screening, as well as the lack of a central repository for screening results. Efforts to create hard‐wired systems for sharing information across system have continued to lag. This is no doubt due to both the complexity of the issues, as well as the critical need for better cross system leadership at the state level in developing key data systems that would facilitate information sharing more consistently.

Service Delivery Implementation Strengths & Challenges

Multnomah Project LAUNCH implemented four major service delivery activities: (1) trainings for primary care practitioners through START to increase the frequency of screening for children’s social‐emotional wellbeing and postpartum depression; (2) early childhood mental health consultation and Early Childhood Positive Behavior Intervention and Supports with child care and home visiting service providers; (3) parenting education, support, and referral through the 211infoFamily service; and (4) evidence‐based family strengthening in the form of Incredible Years groups. Implementation of planned service delivery activities for Project LAUNCH has in most cases exceeded initial targets. Specifically:

1. Project LAUNCH has exceeded its target of 60 PCPs and other primary care staff trained in ASQ, behavioral health, trauma‐informed care, and maternal depression screenings each year: Year 1 (152 primary care staff trained), Year 2 (212 primary care staff trained), Year 3 (196 primary care staff trained), Year 4 (188 primary care staff trained) and Year 5 (192 primary care staff trained).

2. Project LAUNCH has exceeded its cumulative target for 211info Family usage of 695 callers receiving referrals, with over 2,000 calls received (12 calls during Year 1; 184 calls during Year 2; 806 calls during Year 3, 579 calls during Year 4, and 676 calls in Year 5).

3. Incredible Years (IY) groups were implemented as planned, with fidelity, and included more than the originally targeted number of parents (91 participants attended at least one group; 57 parents completed 50% of more of the sessions (63% of participants).

4. Implementation of early childhood mental health consultation for home visitors, which was changed during Year 2 in order to provide this service to a different home visiting program, was


highly successful during the last three years of the project. Home visitors indicated that the mental health consultant provided valuable support to them and their families, including training on EC PBIS strategies, parent coaching, wellness and stress reduction, and interpretation and supervision around social emotional screening results.

Challenges to implementation of Project LAUNCH services included the following:

1. Child care staff turnover. By far the biggest challenges in delivering the EC MHC services in child care settings have been related to uncontrollable (and unpredictably high) turnover in child care staff and program leadership; in the most extreme case, at the end of Year 4 one of the facilities that LAUNCH was supporting closed its doors entirely, with little notice. While turnover within child care is not unexpected, the leadership turnover was particularly challenging, and no doubt related to a higher than expected rate of turnover within staff.

2. Need for follow up training for primary care. Significant numbers of START‐trained primary care physicians and practices needed additional support in order to fully implement the recommended START screenings, even for the basic developmental module. Follow‐up by evaluation and START staff indicated significant barriers to full implementation of screening protocols; thus, the decision was made at the end of Year 4 to fund a series of “tune‐ups” in Year 5, for previously trained providers during Years 1 through 4. Additionally, a follow‐up protocol will be implemented with these providers to provide ongoing coaching and technical assistance to support implementation.

3. Initial location of 211infoFamily outside of Multnomah County. During Year 1, LAUNCH contracted with a Parent Helpline provider whose primary experiences were in Lane County, Oregon, roughly 90 miles south of Multnomah County. Despite efforts to integrate resources and provide training to this provider, after a year of challenging implementation, the service was relocated to Multnomah County, with greater success.

4. Increasing public awareness and utilization of the 211infoFamily helpline. During Years 1 and 2 the number of callers to the 211infoFamily helpline was much lower than anticipated. This led to several efforts to “get the word out,” including public awareness and marketing campaigns, redoubled efforts by the 211infoFamily outreach specialist to make presentations to community providers, and integration of information about 211infoFamily into other LAUNCH strategies, specifically within START trainings and the work of the EC MHCs. Data suggest that these last two strategies may have been particularly effective.

Service Delivery Outcomes

1. Outcomes for child care providers and home visitors. Results of surveys completed by child care and home visiting staff receiving early childhood mental health consultation found some evidence for improvement in provider knowledge and confidence from baseline (pre‐consultation) to subsequent follow up assessments. Specifically, providers reported statistically significant increases in their knowledge of early childhood mental health, and in their confidence in working to involve parents to support children’s social emotional development and mental health. Further, observations conducted by consultants of child care providers in preschool aged classrooms also suggested substantial improvements in quality of teacher‐child interactions in classrooms receiving consultation.


2. Primary care providers. Primary care providers who received START training consistently reported improvements in their knowledge of developmental screening tools, community referral resources, and increased intention to conduct screenings. Additionally, there was a substantial increase in screenings and referrals to the local Early Intervention provider from primary care providers. During Years 3‐5, there were over 1,000 referrals received by Early Intervention/Early Childhood Special Education (EI/ECSE) from primary care, compared to only about 100 received during Years 1 and 2.

3. Developmental screenings. In a subset of primary care clinics for whom data were available, the number of developmental screenings provided by primary care providers increased by over 1,000 screens in the 12‐month period following receipt of START trainings.

4. 211infoFamily service users. The majority of users were highly satisfied with the services they received from 211infoFamily, and would recommend this service to others; however, a very small number of families could be successfully contacted for the evaluation. Follow up interviews conducted by PSU evaluation staff also documented the important role being played by the 211infoFamily Parent Educator in eliciting parenting concerns and providing appropriate support to families around a myriad of child‐related topics.

5. Incredible Years. 86 unique parents have participated in Incredible Years workshop series through Project LAUNCH through Year 4. Results of this service have been strong, with participating parents reporting reduced levels of child internalizing and externalizing behavior, and improved attachment, child initiative, and self‐control.

Conclusions & Sustainability Over the five years of implementation, Multnomah Project LAUNCH had significant success in both systems improvement and service delivery. These successes are attributable at least in part to the commitment and diligence of the Young Child Wellness Council to working through early implementation challenges, and to using data for continuous program improvement. Changes in planned implementation were made as appropriate to help support continued progress towards LAUNCH goals.

Multnomah’s Project LAUNCH has been characterized by a close partnership between LAUNCH leadership and the Portland State University (PSU) evaluation team. The evaluation team was given many opportunities to share data and information with the YCWC and with key LAUNCH subcontractors. Further, the evaluation team worked to be responsive to emergent issues of particular interest or importance to LAUNCH stakeholders, for example, developing new ways to approach the evaluation of the START trainings; making sure mechanisms were in place to evaluate new aspects of the workforce development activities; and working to expand information gathering related to mental health consultation in home visiting. Contracted LAUNCH providers also worked closely with both the YCW coordinator, the YCWC, and the evaluation team to ensure that services were being delivered as planned, or, if challenges were encountered, to problem solve and make appropriate changes.

Beginning in Year 4, there was a focused effort to begin to create pathways to sustainability for LAUNCH system enhancements and service delivery. As of this writing (3 months after the end of project funding), the following activities continued to be ongoing:



EC PBIS Leadership Team continues to meet, with dedicated administrative staff support from one of the participating agencies;


The Home Visiting CoP continues to meet, with support from the state’s Maternal Infant Early Childhood Home Visiting (MIECHV) grants;

The Early Childhood Mental Health CoP also continues to meet, although less frequently and without administrative staff support;

The Parent Educator position is currently being maintained at 211infoFamily, supported by funding from one of the regional Coordinated Care Organizations;

Early Childhood Mental Health Consultants are being supported through County, City, and federal MIECHV funds;

START continues to support training for primary care physicians in a variety of behavioral health screening tools and how to incorporate behavioral health into primary care.

The success of Project LAUNCH in sustaining both systems work occurring through the YCWC as well as key elements of service enhancements is evidence of the value placed by the community on the services and enhancements supported during the five years of the LAUNCH grant.


Appendices

Appendix A: Key Stakeholder Systems Survey


Appendix B: Key Stakeholder Systems Interview Questions

Multnomah Project LAUNCH Key Stakeholder Systems Change Interview – Year 4

1. Have you made new (or strengthened) connections with individuals or organizations as a result of your participation in LAUNCH?

a. What is an example? b. How did this connection come about? c. What has resulted from this connection?

2. Are you or your organization doing work differently as a result of LAUNCH? a. How? b. What is an example?

3. What lessons learned or other information should LAUNCH be sharing with other local or statewide early childhood system integration efforts (e.g., Early Learning Hubs, Parenting Hubs, Coordinated Care Organizations)?

a. What can these entities learn from LAUNCH? 4. How can LAUNCH benefit from coordinating with these local and statewide efforts?

a. What should LAUNCH be learning from them? Click here to enter text. 5. How should LAUNCH be working to ensure the sustainability of Early Childhood PBIS, mental

health consultation in home visiting, START, workforce development and training, and 211infoFamily in Multnomah County?

6. Where do you think LAUNCH has been most successful in improving early childhood system integration?

a. What is different as a result of LAUNCH? 7. Are there things that you wished LAUNCH would have accomplished by now that it hasn’t yet?

a. What do you think would help LAUNCH work toward this in the next year and a half? 8. One area that the key stakeholder surveys showed as an area for improvement is the

accessibility of information about culturally‐specific services. a. What should LAUNCH be doing to address this gap or to strengthen this area?


Appendix C: Summary Tables of Figures

Figure 2. Reference: Mean Ratings of Cross‐Sector Collaboration by Sector

Mean (sd) 4‐pt scale, Higher score indicates strength

Collaboration w/ Primary

Care

Collaboration w/ Mental Health

Collaboration w/ Family Support

Collaboration w/ ECE

Baseline

Primary Care (N=4) na 3.0 (1.2) 2.8 (1.0) 2.8 (1.0)

Mental Health (N=4) 2.3 (.5) na 3.0 (.8) 3.0 (1.4)

Family Support (N=15) 2.5 (.8) 2.5 (.9) na 3.3 (.6)

ECE (N=7) 2.9 (.4) 3.6 (.5) 2.6 (.5) na

T4

Primary Care (N=8) na 3.0 (.8) 2.8 (.7) 3.0 (.8)

Mental Health (N=6) 2.4 (.5) na 3.5 (.5) 3.8 (.4)

Family Support (N=3) 2.7 (.6) 2.7 (.6) na 3.3 (1.2)

ECE (N=4) 3.3 (.5) 3.0 (.8) 3.3 (.5) na

Figure 3. Reference: Mean Ratings of Systems Survey Areas Mean (sd) 5‐pt scale, Higher score indicates strength

Baseline (N=44)

T2 (N=42)

T3 (N=31)

T4 (N=26)

YCWC Functioning na 4.4 (1.8) 3.8 (.7) 4.3 (.5)

Ability to achieve LAUNCH goals 4.2 (.6) 3.9 (.5) 3.9 (.5) 4.2 (1.3)

Collaboration between organizations 3.6 (1.1) 4.2 (.8) 4.3 (.8) 4.1 (.8)

Quality of services for families 2.4 (.7) 2.6 (.7) 3.1 (.8) 3.1 (.9)

Figure 4. Reference: ECE Provider & HV Staff Perception of Service Linkages Mean (sd) 5‐pt scale, Higher score indicates strength Baseline 6 Mo 12 Mo 24 Mo 36 Mo

Home Visitors (N=17) (N=15) (N=15) (N=12) (N=9)

3.3 (1.1) 3.5 (.7) 3.7 (.6) 4.0 (.5) 4.3 (.8)

ECE Providers (N=157) (N=107) (N=87) (N=43) (N=15)

2.7 (1.0) 2.9 (1.1) 3.1 (1.1) 3.5 (1.1) 3.7 (1.1)

Figure 5. Reference: ECE Provider & HV Staff Confidence in Involving Families Mean (sd) 5‐pt scale, Higher score indicates strength Baseline 6 Mo 12 Mo 24 Mo 36 Mo

Home Visitors (N=17) (N=15) (N=16) (N=12) (N=9)

3.7 (.8) 4.2 (.6) 4.2 (.5) 4.6 (.5) 4.7 (.4)

ECE Providers (N=159) (N=109) (N=88) (N=45) (N=15)

3.6 (1.0) 3.7 (1.0) 3.9 (.8) 4.1 (.8) 4.0 (.7)


Figure 8. Reference: ECE Provider Knowledge of 211infoFamily Mean (sd) 4‐pt scale, Higher score indicates strength

Baseline (N=146)

6 Mo (N=100)

12 Mo (N=85)

24 Mo (N=44)

36 Mo (N=14)

I have heard about 211infoFamily 1.9 (1.2) 2.2 (1.3) 3.9 (1.5) 3.4 (1.1) 3.3 (.9)

I understand the services 211infoFamily provides 1.8 (1.0) 2.3 (1.2) 3.7 (1.5) 3.0 (1.1) 3.1 (.9)

I know how to access 211infoFamily 1.8 (1.0) 2.3 (1.2) 2.6 (1.1) 3.1 (1.2) 2.9 (1.2)

(N=67) (N=45) (N=36) (N=15) (N=5)

I am satisfied with the support received from 211info Family 1.7 (1.1) 2.0 (1.2) 2.5 (1.2) 2.9 (1.2) 2.8 (1.1)

Figure 9. Reference: HV Staff Knowledge of 211infoFamily Mean (sd) 4‐pt scale, Higher score indicates strength

Baseline (N=16)

6 Mo (N=15)

12 Mo (N=16)

24 Mo (N=12)

36 Mo (N=9)

I have heard about 211infoFamily 2.3 (1.3) 3.7 (.6) 3.7 (.6) 3.9 (.3) 4.0 (0)

I understand the services 211infoFamily provides 2.1 (1.3) 3.1 (.8) 3.4 (.9) 3.8 (.5) 3.8 (.4)

I know how to access 211infoFamily 2.1 (1.1) 3.2 (.8) 3.5 (.7) 4.0 (0) 3.9 (.3)

(N=5) (N=7) (N=6) (N=10) (N=7)

I am satisfied with the support received from 211info Family 2.2 (1.3) 3.4 (1.1) 3.3 (.8) 3.4 (.5) 3.1 (.7)

Figure 14. Reference: LAUNCH v Comparison HV Staff Parent Involvement Domain Mean (sd) 5‐pt scale, Higher score indicates strength Baseline* 6 Mo 12 Mo**

LAUNCH HV (N=15)* 3.7 (.9) 4.2 (.6) 4.2 (.5)

Comparison 1 HV (N=12)* 4.4 (.6) 4.5 (.5) 4.7 (.4)

Comparison 2 HV (N=8) 4.2 (.5) 4.3 (.3) 4.4 (.3)*Statistically significant difference (p<.05) between LAUNCH and Comparison 1 HV groups at baseline; within LAUNCH groups from Baseline to 6/12 months; and within Comparison 1 groups from Baseline to 12 months. **Statistically significant difference (p<.01) between LAUNCH and Comparison 1 HV groups at 12 months.

Figure 15. Reference: LAUNCH v Comparison HV Staff Knowledge of Children’s Mental Health Domain Mean (sd) 5‐pt scale, Higher score indicates strength Baseline 6 Mo* 12 Mo

LAUNCH HV (N=15)* 3.4 (.8) 3.7 (.6) 4.2 (.5)

Comparison 1 HV (N=12) 4.1 (.8) 4.4 (.5) 4.5 (.4)

Comparison 2 HV (N=8) 4.0 (.5) 4.2 (.7) 4.2 (.5)*Statistically significant difference (p<.05) between LAUNCH and Comparison 1 HV groups at 6 months; within LAUNCH groups from Baseline to 12 months.

Figure 16. Reference: LAUNCH v Comparison HV Staff Knowledge of EC PBIS Mean (sd) 5‐pt scale, Higher score indicates strength Baseline 6 Mo 12 Mo

LAUNCH HV (N=14) 2.5 (1.3) 2.7 (1.3) 3.2 (.7)


Mean (sd) 5‐pt scale, Higher score indicates strength Baseline 6 Mo 12 Mo

Comparison 1 HV (N=11) 2.1 (1.3) 2.4 (1.3) 2.4 (1.4)

Comparison 2 HV (N=8) 2.3 (1.3) 2.3 (1.4) 2.1 (1.0)

Figure 17. Reference: LAUNCH v Comparison HV Staff Stress Domain Mean (sd) 5‐pt scale, Higher score indicates strength Baseline 6 Mo 12 Mo

LAUNCH HV (N=14)* 1.9 (.9) 1.8 (.8) 2.3 (.9)

Comparison 1 HV (N=11) 1.9 (.8) 1.9 (1.0) 1.9 (1.0)

Comparison 2 HV (N=8) 2.5 (1.3) 2.5 (1.5) 2.8 (1.3)*Statistically significant difference (p<.05) within LAUNCH group from Baseline to 12 months.

Figure 18. Reference: LAUNCH v Comparison HV Staff Perception of Program Leadership Domain Mean (sd) 5‐pt scale, Higher score indicates strength Baseline 6 Mo 12 Mo

LAUNCH HV (N=15)* 3.7 (.8) 3.9 (.9) 4.2 (.6)

Comparison 1 HV (N=12) 4.2 (.6) 4.1 (.6) 4.1 (.9)

Comparison 2 HV (N=8) 3.3 (1.0) 3.4 (.5) 3.6 (.6)*Statistically significant difference (p<.05) within LAUNCH group from Baseline to 12 months.


Appendix D: Community of Practice Issue Briefs

Home Visiting CoP


EC PBIS CoP


Early Childhood Mental Health CoP


Appendix E: EI Referral Practices Issue Brief


Appendix F: Maternal Depression Screening Issue Brief


Appendix G: MHC in ECE Settings Issue Brief


Appendix H: MHC in Home Visiting Logic Model


Appendix I: MHC in Home Visiting Issue Brief


Appendix J: 211infoFamily Interview Questions Multnomah Project LAUNCH 211FamilyInfo User Interview

Rev: 8/14/2014

1. First, can you remember how you first learned about the 211info Family line? [Mark all that apply and probe for more details if it’s not clear how or where they heard about the service.] a. Heard about it from:

i. 211info ii. Friend/neighbor iii. Family member iv. Child care provider v. Primary care provider vi. Called 211FamilyInfo before (repeat caller) vii. Other, please describe:

b. Saw an advertisement for it at: i. Child care provider ii. Primary care provider iii. Child’s school iv. Neighborhood paper v. Other, please describe:

c. Other, please describe: 2. Can you remember why you called the 211info Family line or wanted to talk with someone

about parenting or early childhood issues? [Probe, if needed] Were you looking for any specific types of information for you or your child/ren?

a. What were you looking for? If caller received referrals (based on info provided from 211FamilyInfo), complete #3, else skip to #5 3. Were you able to access the referrals you received, for example, [you can reference some of

the referrals listed in the 211info Family raw data report]? Yes No a. If yes, Did the referrals you received help you? Yes No

4. If yes, How did they help you? 5. If no, Why did they not help?

b. If no, What prevented you from accessing the referrals? c. [Probe] What got in the way?

4. Did the referrals meet your expectations based on your family’s culture? a. If no, what would have helped better?

5. Did you talk with the 211info Family parenting specialist about anything else, like advice or suggestions she gave, or ways you’d use the information in your family or with your child? Yes No [If no, skip to #6] [If yes, continue]


a. What did you talk about? b. Was the information helpful? Yes No

i. If yes, How did it help you? ii. If no, Why wasn’t it helpful?

c. Did the information or suggestions that the Specialist provided meet your expectations based on your family’s culture?

i. If no, what would have helped better? 6. Are there any services or supports that weren’t available to you when you called 211info

Family that you wished existed? a. If yes, what would have been helpful?

7. How would you respond to the statement, “I would recommend 211info Family to a friend”? Would you:

[Offer these response categories to the caller] a. Strongly agree b. Somewhat agree c. Somewhat disagree d. Strongly disagree


Appendix K: 211infoFamily Issue Brief


Appendix L: Timeline of Evaluation Activities

Y1: 2010‐11

Y2: 2011‐12

Y3: 2012‐13

Y4: 2013‐14

Y5: 2014‐15

1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4

SYSTEMS CHANGE

Collaboration

Develop, select collaboration measures X X X X

Administer key stakeholder surveys X X X X

Conduct key stakeholder interviews X X X X

Workforce Development

Develop workforce development measures X X X X X X X X X X X X X X X X X

Collect post‐training surveys X X X X X X X X X X X X X X X X X

Public Awareness

Develop public awareness measures X X X X

Collect public awareness data X X X X X X X X X X X X X X X X X

CORE STRATEGIES

Screening, Assessment & BHI in Primary Care

Develop START measures X X X X X X X

Collect START training & screening data X X X X X X X X X X X X X X X X X X

Develop EI/ECSE data sharing agreement X

Collect EI/ECSE referral data X X X X X X X X X X X X X X X X X X

EC MHC in ECE Settings

Develop, select ECE setting measures X X X X X X

Collect EC MHC reports X X X X X X X X X X X X X X X X X

Collect ECE provider surveys X X X X X X X X X X X X X X

Collect TPOT/TIPITOS observational data X X X X X X

Collect BOQ data X X X X

Collect BIRs data X X X X X X X X

EC MHC in HV

Develop, select HV measures X X X X X X

Collect EC MHC reports X X X X X X X X X X X X X X X X X

Collect HV provider surveys X X X X X X X X X X X X X X X

Collect HFO program data X

Family Strengthening

Develop, select IY series measures X X

Collect pre‐post IY series data X X X X X X X X X

Develop 211infoFamily data collection tools X X X X X X

Collect 211infoFamily data X X X X X X X X X X X X X X X X X

Reporting

YCWC, LAUNCH‐funded partners X X X X X X X X X X X X X X X X X X X X

TRAC X X X X X X X X X X X X X X X X X X X X

Cross‐site evaluation X X X X X X X X X

Annual report X X X X X

Multnomah Project LAUNCH€¦ · 15/12/2015 · Multnomah Project LAUNCH Year 5 Evaluation Report...

Documents

Transcript of Multnomah Project LAUNCH€¦ · 15/12/2015 · Multnomah Project LAUNCH Year 5 Evaluation Report...