Maine Title IV-E Demonstration Project Final Evaluation Report

zjft`233 Maine Title IV-E Demonstration Project

Final Evaluation Report PREPARED FOR Maine Department of Health and Human Services Office of Child and Family Services

BY Public Consulting Group, Inc. 373 Broadway South Portland, ME 04106

November 2019

Public Consulting Group, Inc. ME Title IV-E Final Report

Maine Title IV-E Demonstration Project Final Evaluation Report PREPARED FOR Maine Department of Health and Human Services Office of Child and Family Services BY Public Consulting Group, Inc. 373 Broadway South Portland, ME 04106 November 2019


Contents

Executive Summary ....................................................................................................... i

Introduction and Overview ........................................................................................... 1

Evaluation Framework .................................................................................................. 5

Process Evaluation ..................................................................................................... 13

Child Welfare Outcomes ............................................................................................. 45

Cost Outcomes ............................................................................................................ 69

Recommendations ...................................................................................................... 73

Appendices .................................................................................................................. 79


List of Figures Figure 1. Map of Service Providers as of December 31, 2018 Figure 2. MEPP Participant Eligibility by Region Figure 3. Number of Weeks and Sessions by Active, Completed and Incomplete Figure 4. Percent of Cases Without New Appropriate Reports at Six and Twelve

Months Figure 5. Percent of Cases with Children Age 0–5 Remaining in Home After Six

Months Figure 6. Percent of Cases with Children Age 0–5 Remaining in Home After Twelve

Months Figure 7. Percent of Children Age 0–5 Reunified Within 12 Months Figure 8. Percent of Children Age 0–5 Enrolled in Trial Placement Within Twelve

Months of Removal Figure 9. Average Number of Days Until Reunification Figure 10. Comparison of CANS Item Score Percentages from Initial to Follow-up

CANS Figure 11. CANS Domain Changes between First and Second Assessment Figure 12. CANS Domain Rates of Positive and Negative Change by Domain Figure 13. CANS Caregiver Domain Item Changes Figure 14. CANS All Other Domain Item Changes Figure 15. Percent of Cases with Children with Improvement in Health by Domain

Based on Case Review Figure 16. Percent of Cases with Children with Improvement in Mental Health by

Domain Based on Case Review Figure 17. Percent of Cases with Children with Improvement in Education by Domain

Based on Case Review Figure 18. Initial and Follow-up DASS Scores by Subdomain Figure 19. MEPP Participant Agreement with Parenting Statements on Parenting

Survey Figure 20. Parent Perceptions of Staff Attitudes


List of Tables Table 1. Total Number of MEPP Participants by Region and Cohort Table 2. Number Enrolled in MEPP by Cohort Table 3. MEPP Participants’ Status Table 4. Age of Children of MEPP Participants by Region Table 5. Gender of Children Ages 0–5 of MEPP Participants by Region Table 6. Race/Ethnicity of Children Ages 0–5 by Region Table 7. MEPP Provider Staff Training Table 8. Percent of Graduated Participants by Completion of Fidelity Component Table 9. When Participants Start Triple P by Completions and Discharges Table 10. When Participants Were Discharged from MEPP Before Starting Triple P Table 11. Compliance with Urine Screening When Participants Attend At Least One

Session per Week Table 12. Compliance with Urine Screening When Participants Attend At Least

Three Sessions per Week Table 13. Factors Impacting MEPP Completion and Fidelity Outcomes Table 14. Measurable MEPP Cohort Family Count Table 15. Factors Impacting Child Welfare Outcomes at Twelve Months Table 16. Average Initial and Follow-up Scores on Parent Practice Scales of MEPP

Participants Table 17. Average DASS Scores by Subdomain among MEPP Participants Table 18. MEPP Responsivity to Need Table 19. Total Costs of Services to Treatment and Comparison Groups (both

Children and Parents) Table 20. Costs for Services to Treatment and Comparison Groups Table 21. MaineCare Payments to Treatment and Comparison Group Participants Table A1. Significance Values of Comparison Group Matching Criteria

Public Consulting Group, Inc. ME Title IV-E Final Report i | P a g e

Executive Summary In late 2015, the Maine Department of Health and Human Services (DHHS) was granted a Title IV-E Waiver for a Demonstration Project to help the Maine Office of Child and Family Services (OCFS) improve the stability, health, well-being and quality of permanent connections for parents and children involved with child protection services, targeting families with children ages zero to five who are assessed at moderate to high risk, particularly for substance abuse. This project became known as the Maine Enhanced Parenting Program (MEPP) and was intended to provide intensive outpatient (IOP) substance use treatment as well as parenting courses to the Waiver population, both concurrently and at the same location. Prior to the Waiver these services were offered separately from one another and in multiple locations, making completion of both services challenging. DHHS contracted with Public Consulting Group, Inc. to perform an evaluation of the project. The primary goals of the project are to keep children safe in their own homes, reduce the number of new maltreatment reports, and increase the rate and speed at which children are reunified. Families enrolled in the program may have an open, in-home child welfare case or may already have had a child removed from the home. Implemented April 1, 2016, MEPP was designed to achieve several outcomes addressing child safety, permanency, well-being, and reunification as well as parent well-being and functioning.

• Increase the number of children who remain safely in their homes

• Reduce repeat maltreatment

• Increase the rate of reunification and the timeliness of reunification

• Improve child and family well-being, and

• Improve specific outcomes related to substance abuse. This report covers two and three-quarter years of the project’s operation, which accounts for the entire Waiver period. For the families who engage in MEPP, both child and parent well-being are considered to be improved as measured by changes in health, mental health, education, and parenting practices. These are supported by reductions in depression, anxiety, and stress and improvements in parenting practices on pre- and post-parental self-assessments. It is more difficult to detect differences in traditional child welfare outcomes identified above, although if a parent completes MEPP to fidelity he or she is more likely to be reunified with his or her child, however, the relationship is not statistically significant. Clients enrolled in MEPP are less likely to have a new appropriate report at 12 months than comparison group clients; however, for families with children in the home at the time of enrollment, a smaller percentage of MEPP families have the children remaining safely in the home than comparison group families.

Public Consulting Group, Inc. ME Title IV-E Final Report ii | P a g e

Because MEPP is only one element of a family’s case plan and success of the program is not statistically significant in terms of reunification, it is not clear that completing the program has a definitive impact on the child welfare outcome. Caseworkers are looking for all elements of the plan to be met, including parent/caregiver sobriety, behavioral changes that signify an ability to sustain recovery, a realistic safety plan if relapse should occur, development of natural supports, and the demonstration of appropriate parenting skills. There also appears to be discrepancies as to what a “harm reduction” model means among Matrix IOP (MIOP) providers, child welfare workers, and judges. There needs to be greater clarity of policy and consistency in expectations among the parties. The cost evaluation shows higher costs for services to MEPP participants compared to families receiving traditional services, which are largely due to the contracted costs paid to the providers of substance abuse treatment and parenting services. Room and board costs are lower for children of MEPP clients than comparison group children, but not by enough to offset the service costs. Because enrollment numbers are still relatively low and provider costs are high, and given feedback of all parties, if the state were to continue MEPP, it makes sense to expand eligibility to a families of a somewhat older age group of children, perhaps to age eight. This will make the Positive Parenting Program (Triple P) curriculum more readily applicable as well. Not surprisingly, the data suggests that MEPP is most successful when the program is implemented to fidelity. Providers made clear improvements over the lifetime of the Waiver to address fidelity short comings. However, due to the novelty of the program concept, to deliver substance use treatment and parenting concurrently, stakeholder interviewees suggested that less than three years of implementation may not have been long enough to work out all the kinks in the system and show the full potential of this complex program. Key fidelity assessment findings are presented below.

1) The program was most successful for the population most in need. Parents having greatest need for development of new parenting skills and parents having higher depression scores were more likely to complete the program than parents reporting less need in these two areas. MEPP was also successful for parents who had previous substantiated reports with the Department and who received additional substance use services.

2) Program fidelity was consistently an issue for various reasons. a) Initial definitions of graduation from MEPP were inconsistent across

providers. Thus, not all participants who are listed as MEPP graduates met all of the fidelity requirements. This improved over time through provider education and the addition of fidelity reporting capabilities in the provider database.

b) The biggest impediment to fidelity was completion of all 48 Matrix IOP sessions. Part of this gap appears to be attributable to variation in how sessions were counted by providers, as well as lack of

Public Consulting Group, Inc. ME Title IV-E Final Report iii | P a g e

provider buy-in that completion of 48 sessions is necessary for program success or graduation.

c) While participants were required to complete only one level of Triple P to fidelity to complete MEPP, at no point during the Demonstration Project was there 100 percent compliance. There was, however, a higher rate of completion for participants who started Triple P during the first eight weeks of MEPP.

d) Though providers in Regions One and Two consistently ensured that most participants received at least one random urine screen each week when they attended three sessions per week, providers in Region Three did not have the same success. Region Three providers consistently observed fewer drug screens over the course of the Demonstration Project.

3) Participants consistently moved through MEPP at a slower pace than the program originally predicted. The majority of clients who completed MEPP spent 17 to 34 weeks in the program. Providers reported making efforts to help parents be accountable and maintain attendance, but participants still struggled with various situations, like illness, lack of transportation, incarceration, and conflict with court dates and visitation, which providers could not control to decrease time spent in the program.

As a result of the political and economic factors incurred by OCFS in 2017 and 2018, in March 2019, OCFS leadership made a formal request to end the Demonstration Project early, as of December 31, 2018. The Leadership Committee discussed continuation of MEPP services post-Demonstration over the course of several months. When interviewed, nearly all OCFS caseworkers and supervisors expressed a desire to see MEPP services continue, but they admitted to seeing very little success from the program for families on their caseloads or in their units. Eagerness for program continuation appears to derive from the fact that many districts have limited substance use or parenting providers to begin with. Thus, elimination of any program, regardless of effectiveness, is seen as detrimental to the community. Matrix IOP has consistently been reimbursable through the state Medicaid plan, MaineCare. Providers and OCFS agreed that Matrix IOP could continue to be provided for MaineCare-eligible clients after the Demonstration Period if providers wished to continue the programming with no impact on billing. While there is current legislation pending to incorporate Triple P as a covered service under MaineCare as well, this has not yet happened. Providers have the option of continuing Triple P services with the understanding OCFS will not be able to reimburse them after September 30, 2019. OCFS leadership confirmed plans to continue to refer families to MEPP provider agencies for IOP and parenting education services as appropriate, although providers vary in their plans for continuing to deliver the service.

Public Consulting Group, Inc. ME Title IV-E Final Report iv | P a g e

The report concludes with the following recommendations: Launch a Multi-disciplinary Process to Unify and Clarify Policy about Marijuana, Medication Assisted Treatment and Harm Reduction Strategies Use an existing mechanism such as the Court Improvement Program or Children’s Cabinet to address the issues raised here about the role of marijuana use and medication-assisted treatments in child welfare cases. These issues include the goals of drug treatment (i.e., harm reduction or abstinence) that will be acceptable to child welfare and the courts; what, if anything, is considered acceptable marijuana use in child protection; and what is the role of medication-assisted treatment. OCFS should translate the results into policy and practice guides for caseworkers. The guiding principle in modifying and clarifying policy should be the achievement of safety and permanency for the child. For instance, the selection of Matrix IOP as a key component of the project suggests that a policy or practice requiring total abstinence is not appropriate, because that is not what Matrix IOP seeks to achieve. In addition, case plan provisions that require parents to get and maintain jobs do not necessarily enhance the children’s safety and may be counterproductive, given the difficulties parents have in participating in the services and simultaneously holding down jobs. In designing new policies and practice, each proposal needs to be tested against the questions of the degree to which it is necessary for the safety and permanency of children and how it contributes to those outcomes. Determine how Medicaid can Help Financially to Support Treatment and Parenting Programs for Child Welfare Families including Expansion of Treatment Options Medicaid has the potential to provide federal matching funds for evidence-based treatment and parenting programs. One of the major sticking points to the continuation of MEPP programming is funding. Parenting education in Maine is currently viewed as a supportive service; thus, it is not reimbursable via traditional Medicaid. However, parenting education is often recommended as part of permanency planning. It is understood that OCFS is already working closely with the Maine Legislature and MaineCare office to explore opportunities to expand Medicaid to cover Triple P. At the same time, other evidence-based parenting programs should be considered, particularly for parents of younger children or those whose children have been removed. Providers and OCFS staff recognized the importance of participants practicing parenting skills with their children. While a large percentage of clients already had their children removed from the home at the time of program enrollment, the idea of incorporating parenting education into visitation arose several times in stakeholder interviews. Generally, it was expressed that this method of parenting education could be more effective than traditional classroom style learning. Providers and caseworkers suggested that having hands-on practice to demonstrate skills and work through issues with a parenting clinician present would be useful not only to increase parent education but also to document concrete progress for permanency planning. This time could also be used

Public Consulting Group, Inc. ME Title IV-E Final Report v | P a g e

to address specific issues OCFS staff identified during the investigation or when working with the families. Stakeholders suggested that this hands-on parenting education could be completed during supervised visitation or arranged playgroups. Though Matrix IOP is already a covered Medicaid service, providers report billing issues due to the length of the program. While many substance use programs are four weeks in duration, the fact that Matrix IOP is four times as long can sometimes cause issues with reimbursement, requiring extra justification and more frequent reauthorization of services. Therefore, OCFS should continue to work closely with MaineCare to remove obstacles for billing. One of the features of MEPP that made this program inherently more valuable than traditional substance use and parenting referrals was that Matrix and Triple P clinicians were reimbursed for time spent collaborating. Traditionally, behavioral health services are strictly fee-for-service; time spent discussing strategies with other clinicians, in attendance at client meetings, or returning phone calls or emails is not reimbursable. In one region with superior collaboration the clients experienced more success. Future programming should strategize a way to fund or reimburse provider collaboration as a mechanism to improve participant outcomes. Increase Communication Mechanisms Between Caseworkers and Providers Communication was a common theme throughout the interviews, focus groups, committee meetings, and staff surveys. Providers reported a desire to have faster delivery of collateral information (including relevant history and case goals) from OCFS and more timely responses to phone calls and emails, whereas OCFS staff emphasized the importance of frequent, transparent updates from all providers about client progress and setbacks with recovery. All parties generally agreed that provider involvement in Family Team Meetings was helpful for permanency planning. Additionally, while collaboration between OCFS and providers for aftercare or discharge planning did not seem to be a common occurrence, both parties recognized that better coordination in this area would be helpful in supporting sustained recovery and setting clear expectations for clients. A mechanism for increasing communication was an effective MEPP champion. Each region designated MEPP liaisons and coordinators for providers and OCFS offices. However, the degree of their engagement ranged widely. Generally, regions with strong MEPP champions appeared to have better integration of programs, better communication between OCFS and providers, and more alignment with fidelity requirements. Assignment of a liaison or coordinator title was not enough. Future programming should recognize the importance of this role and take care to ensure that the person chosen has the time, effort, and energy to actively pursue cross-site collaboration for maximum effect. This includes regular provider presence in local OCFS offices to educate and answer caseworker questions about programming and referrals.

Public Consulting Group, Inc. ME Title IV-E Final Report vi | P a g e

Increase Training Across the Board Implementation of any new program requires training for OCFS staff and participating or impacted providers at the start and ongoing refreshers for veteran and newly hired staff. Interviewees reported that initial training prepared them for implementation, but follow up interviews revealed ongoing training needs, especially in light of staff turnover and growth of a less experienced workforce. Future programming should include ongoing training for OCFS staff about program fidelity as well as substance use, specifically around recovery, relapse, abstinence, and best practice. In turn, program providers should also receive training on the child protection system and how cases are moved through the system to give them a better understanding of OCFS requirements, limitations, and their role in helping participants navigate the system. Additionally, the Department should consider trainings for the community particularly with the court system, including drug court, the Bar Association, and defense attorneys for any new initiatives as court buy-in often impacts program success. Expand Eligibility When asked about eligibility criteria, both OCFS and providers frequently suggested that the age range of children be broadened to at least seven or eight. They cited two primary reasons; first, child age is the element of the eligibility criteria that most often disqualified a family from participation. Second, the Triple P curriculum does not clearly apply to children ages zero to three, thus the curriculum may not be as effective for parents who only have younger children. To the contrary, many families enrolled in MEPP also had older children to which new parenting skills could be applied. Interviewees suggested that expanding the age criteria may be especially important for districts with smaller, more rural populations so that the program is more widely applicable to OCFS’ families. Family First legislation already allows any family with substance abuse issues that has a child who is a candidate for foster care to receive subsidized treatment when participating in a Title IV-E Clearinghouse approved program. Therefore, we recommend expanding future programming efforts to include all families with children under age 18. Continue to Improve Transportation Transportation has repeatedly risen in conversation with participants, providers, and OCFS staff as a barrier to participant success. Focus group participants reported that transportation can be inconsistent and unreliable, which makes it difficult for individuals to get to group on time or arrive there at all. Some participants also reported travelling great distances, over an hour one way, to attend Matrix or Triple P groups. While OCFS can arrange transportation for clients in the MEPP program, caseworkers and supervisors acknowledge that the transportation system is flawed as it is often volunteer-operated and organizing rides for participants can be very time consuming. One supervisor reflected that a caseworker could spend an entire day arranging rides. Steps need to be taken to identify alternative, more reliable transportation sources and vendors.

Public Consulting Group, Inc ME Title IV-E Final Report 1 | P a g e

Introduction and Overview Background and Context In late 2015, the Maine Department of Health and Human Services (DHHS) was granted a Title IV-E Waiver to support a Demonstration Project to help the Maine Office of Child and Family Services (OCFS) improve the stability, health, well-being and quality of permanent connections for parents and children involved with child protection services. This project became known as the Maine Enhanced Parenting Program (MEPP). Maine chose to target a high-risk group, families with children ages zero to five who were assessed at moderate to high risk with a substance abuse issue. Some families had an open in-home case and others had at least one child removed from the home. The goal of MEPP was to keep children in their homes, reduce the number of new maltreatment reports, and increase the rate and speed in which children become reunified when placed into substitute care. The innovation was to provide two types of evidence-based and coordinated services, one which focused on substance abuse treatment and the other on parenting education. Specifically, the program allowed families to access evidence-based parenting education during their substance abuse treatment instead of requiring treatment first. Before parents could not be “using” and attending parenting; now they could. In collaboration with the Office of Substance Abuse and Mental Health (SAMHS), OCFS considered five substance use interventions: Matrix Model Intensive Outpatient Program (MIOP), Seeking Safety, Motivational Interviewing, Community Reinforcement and Vouchers, and Sobriety and Treatment Recovery Teams. Each program was weighed in terms of cost, relevance to the target population, degree to which the intervention is already provided in Maine, and evidence of effectiveness. Matrix IOP won out as it is a Medicaid-funded, cost-effective, evidence-based intervention that was already available in Maine and applicable to the target population. Motivational Interviewing did not have a mechanism for monitoring fidelity, so it was less ideal for a proof-of-concept Demonstration Project. The other three programs presented substantial challenges in statewide training and start-up costs. Participants were to attend MIOP treatment at least three hours a day for three days per week, for at least 16 weeks, with weekly urinalysis to monitor decreasing drug use. Similar to the selection process for the substance use component, OCFS considered four parenting education interventions: Positive Parenting Program Level 4 and Level 5 (Triple P), Parent Child Interaction, Child-Parent Psychotherapy, and Attachment and Biobehavioral Catch-up. Programs were again weighed for cost, relevance to the target population, degree to which the intervention is already provided in Maine, and evidence of effectiveness. Triple P was chosen because it is an evidence-based intervention, relevant to the target population, available statewide, and geared for children zero to twelve (over a larger duration of childhood).


The other three programs presented more limited age applicability for children and/or were more expensive due to the duration of programming. Triple P is delivered in a group format, consisting of eight group sessions, involving no more than 12 parents each, with the option of substituting up to three group sessions with follow‐up phone calls. Level 4 and Level 5 Triple P were selected because they are more intense, are designed to help families learn skills to manage their children’s moderate-to-severe behavioral and/or emotional difficulties, and broadly promote positive parenting skills among young or inexperienced parents of young children. OCFS contracted with community agencies to provide MIOP and Triple P services, i.e., MEPP, at the same location in a common timeframe. Historically, parents completed substance abuse services prior to participation in parenting education classes, which made for an extended treatment period. There were also accessibility issues for families for both parenting and substance abuse services, particularly in rural areas where public transportation is limited or nonexistent. Because the Waiver offered both substance use treatment and parenting education concurrently and at the same location, OCFS staff and families could plan transportation more effectively and made more efficient use of time for parents to demonstrate engagement and progress so that reunification could be considered sooner. The Demonstration Project was originally intended to end on September 30, 2019. However, in the summer of 2017 OCFS experienced two tragic deaths of children involved with DHHS. The resulting public response prompted the Commissioner and leadership to act quickly to change policy and practice to immediately increase child protections and the safety of the children in care. These changes ultimately led to an increase of investigations, the number of children in state custody, funding required to support children in custody and staffing, and workload at all levels of the organization. Further, the increased workload prompted substantial staff turnover, prompting the Department to recognize and address staff retention needs by applying a blanket $5-per-hour raise to all casework-level staff. These unanticipated political and economic factors led OCFS leadership to make a formal request to end the Demonstration Project early, as of December 31, 2018. While federal partners agreed to end the contract prematurely, OCFS leadership recognized the extensive fiscal and personnel investment providers made over the last two and three-quarter years to implement MEPP; therefore, OCFS elected to continue to fund current providers through September 30, 2019 as originally planned. Providers, in turn, agreed to continue to serve parents through the end of the originally planned contract period. This final report describes participant enrollments completed between April 1, 2016 and December 31, 2018, allowing time for follow-up analysis. However, since the request to end the program was not made until March 2019, where indicated, session information and interviews completed between January 1, 2019 and March 31, 2019 are also included.


Purpose of Waiver Demonstration The primary goals of the project were to keep children safely in their own homes, reduce the number of repeat maltreatment reports, and increase the rate and speed at which children were reunified when placed into out-of-home care as well as improve child and family well-being and reduce the rate of re-entry into foster care. The project targeted families who were assessed at moderate to high risk, particularly for substance abuse, with children ages zero to five. These families could have had an open, in-home child welfare case or could already have had a child removed from the home. MEPP was implemented April 1, 2016.


Evaluation Framework Overview This evaluation consists of three components: a process evaluation, an outcome evaluation, and a cost evaluation. The overarching plan was to determine what factors influenced the effectiveness of the implementation; assess the effectiveness of the intervention on improving health, well-being, functional and safety outcomes for children and families; and calculate the cost effectiveness of the Waiver services. To accomplish these goals, in 2015 OCFS engaged Hornby Zeller Associates, Inc. (HZA) to provide objective, third-party evaluation services. In March 2018, HZA was acquired by Public Consulting Group, Inc. (PCG), a national expert in child welfare that brings extensive knowledge of program operations and policy to comprehensive evaluations. A seamless transition of HZA staff to PCG maintained continuity of the evaluation throughout the completion of the Demonstration Period. An historical comparison group was selected from parents with children ages zero to five with substance abuse risk factors to compare the impact of MEPP in achieving positive outcomes. A detailed description of the comparison group is found in the outcome evaluation component of this report. Theory of Change/Logic Model To illustrate the conceptual linkages between the Waiver demonstration activities and the measurable short-term, intermediate and long-term outcomes, PCG developed the following logic model to illustrate the theory of change.


MEPP Logic Model

Parents with children 0–5 with substance abuse risk factors Flexible

funding under Title IV-E Waiver CANS

Assessment tool Caseworkers Providers

specializing in IOP treatment and parenting PAFAS and

DASS Assessment tools

Matrix IOP substance abuse treatment Triple P

parenting classes delivered concurrently

Number of families engaged in MEPP while children remain at home Number of

families engaged in MEPP while in out-of-home placement

Delivery of IOP and parenting services to address parent needs Completion of

services

Reduce new appropriate reports of maltreatment Increase safety

of children in the home Increase

reunification rates for children removed Increase

timeliness to reunification

Data Sources and Data Collection Methodology The MEPP evaluation utilizes a mixed-methods approach to inform the process, outcome, and cost pieces. Qualitative data were collected through key-informant interviews, focus groups, and surveys of OCFS staff and parents. Quantitative data were collected through parent and child assessment tools, the state case management systems, and provider contracts. A summary of the data collection sources and methods is provided below.

Child and Adolescent Needs and Strengths Assessment The Child and Adolescent Needs and Strengths (CANS) assessment was administered to each child in the program upon entry and every three months thereafter until program exit. These assessments informed the child well-being outcomes and parenting growth of caregivers in the program. Initially, OCFS caseworkers conducted the CANS assessments for children when a parent enrolled in MEPP and entered the results into

Outcome Linkages

System Outcomes Outputs Interventions Inputs


the Maine CANS database. The administration of this assessment switched from OCFS caseworkers to Triple P providers in October 2017 due to concern that parents involved with DHHS may not be as forthcoming about their parenting knowledge and skills with caseworkers as they would be with a therapeutic provider. As of December 31, 2018, 31 children (roughly seven percent) had both an initial assessment and at least one follow-up CANS with which to measure changes in needs and strengths over the course of their parents’ participation in the program. Providers reported two barriers to achieving higher follow-up rates: the first was finding the time to administer the follow-up assessment, and second was the ability to sit down with the family after they were discharged from the program.

Depression Anxiety and Stress Scales The Depression Anxiety and Stress Scales (DASS) is a self-administered tool assessing the core symptoms of depression, anxiety and stress experienced by clients in the past week. Assessments were performed at the beginning and end of the clients’ involvement with MEPP and were used to measure changes in parents’ mental health and well-being during the course of the program. Initially, the DASS was administered at the beginning and end of participation in Triple P, but it was changed in April 2017 to be administered at the start and completion of MEPP. This change was made for two reasons. Due to high participant dropout within the first month of the program, there was concern that few participants were staying long enough to complete the first DASS assessment. Also, moving the assessment to the start of MEPP instead of the start of Triple P allowed the evaluators to obtain a more accurate baseline. As of March 31, 2019, a total of 81 MEPP participants (23.5 percent) completed pre and post DASS assessments.

Parent and Family Adjustment Scales The Parent and Family Adjustment Scales (PAFAS) is a self-administered assessment change in parenting practices and family adjustment. Similar to the DASS, the PAFAS was initially administered at the beginning and end of Triple P and changed to the start and completion of MEPP in April 2017. The PAFAS assesses parenting behaviors associated with risk factors of child emotional and behavioral problems (e.g., coercive parenting, lack of parental adjustment). These behaviors are categorized into subdomains and changes in average subdomain scores are analyzed to determine if parental behaviors change as a result of their participation in MEPP. As of March 31, 2019, 88 clients (25.5 percent) completed a pre- and post-PAFAS survey.

Parent Survey

The parent survey was initiated in January 2017 for all parents exiting MEPP. It queries their experiences, satisfaction with specific aspects of the program, and recommendations for program improvement. PCG trained the Matrix IOP providers on how to administer the survey, which is performed either during their last week in MEPP or by mail to those no longer participating. To protect parents’ confidentiality and encourage open feedback, all respondents were provided with a postage-paid envelope to mail the survey directly to PCG. They were also invited to enter a monthly drawing for


a $25 gift certificate to encourage participation. As of March 31, 2019, 24 parent surveys (7%) were completed.

MACWIS PCG used data from Maine’s Child Welfare Information System (MACWIS) to describe the population served, measure client outcomes (e.g., number of children who remain safely at home, rates and timeliness of reunification, and child and parent well-being) and inform the evaluation’s cost analysis. PCG worked with OCFS to identify the appropriate tables to measure costs for maintenance (costs associated with foster care such as room and board) and services for families in the treatment and comparison groups for the cost evaluation.

MEPP Provider Database

The MEPP Provider Database, developed and maintained by PCG, captured information on MEPP client involvement and staff qualifications and training. Client profiles included assessments (e.g., PAFAS and DASS), service start and end dates, discharge information, and session records. Staff profiles included agency name, education, and certification information as well as dates for the completed two-day, intensive Matrix IOP training and/or Level 4 and Level 5 Triple P training as required. Data were entered by MEPP providers and monitored by the evaluators; the information was utilized for both fidelity monitoring and the outcomes evaluation.

MEPP Provider Contracts

MEPP provider contracts were reviewed by PCG and used to calculate the costs to implement the MEPP services.

Focus Groups Four focus groups were conducted with MEPP clients in the Spring of 2018, with at least one having been completed in each region. In total, there were 28 participants, four in Scarborough, seven in Augusta, seven in Kennebunk, and 10 in Bangor. Each group lasted approximately one and a half hours. Participants were asked specifically about their engagement, experiences, and expectations about MEPP and related involvement with OCFS. Facilitators started each group by gaining participants’ informed consent and explaining the purpose and how information would be collected, stored and reported. All participants received a $15 gift certificate to Walmart or Hannaford grocery store.

Staff Survey PCG administered an online survey of OCFS staff each year from 2016 to 2018 to gather information about their perceptions of the program and recommendations for improvement and future staff trainings. Response rates ranged from 35 percent of OCFS staff surveyed in 2016 to 42 percent in 2017 and 18 percent in 2018. The fluctuation may be attributable to other social, economic, and political factors influencing OCFS staff engagement discussed at length later in the report. No OCFS staff survey was conducted for 2019 due to early termination of the Waiver.


Stakeholder Interviews PCG staff conducted two rounds of key stakeholder interviews in each region: Regions One and Three in the Summer of 2017 and 2018 and Region Two in the Spring of 2018 and 2019. Stakeholders included Matrix and Triple P providers, OCFS central office staff, and OCFS district staff (encompassing program administrators/ assistant administrators, supervisors, and caseworkers). Among other topics, interviewees were asked for details about their role in program implementation; how their agency has changed as a result of the program; their perspective on specific aspects of the program such as referrals, intakes, and assessments; communication within and between agencies; and recommendations for program improvements. In total, 95 interviews were conducted statewide consisting of 15 central office staff, 17 program administrators / assistant administrators, 28 caseworkers and supervisors, and 35 Matrix and Triple P provider staff. In some cases, the same person was interviewed in both rounds to discuss changes over time.

Case Record Reviews The case record reviews gathered information on changes in the well-being of children and families and services utilized by both MEPP and comparison group clients. PCG developed an online tool for reviewers to input and store the data found in the case records. Reviewers read through narrative log entries in MACWIS for each case and recorded the services clients received in addition to well-being information such as medical appointments and education status. In total, PCG reviewed and analyzed data from 408 treatment and comparison group cases over the course of the project. Stakeholder Collaboration Evaluation Committee Meeting The Evaluation Committee was formed in mid-2016 to include members from the PCG evaluation team, OCFS staff and others from outside the agency. While originally there was a single provider representative, it was determined that all providers should be updated on evaluation activities as a form of continuous quality improvement. Therefore, the Leadership Committee (see below) was founded. The Evaluation Committee continued to meet quarterly to discuss implementation of the evaluation, fidelity of the evaluation methodology, outcomes achieved, project challenges and recommendations for improvement. Leadership Committee Meetings The Leadership Committee included providers from each district, OCFS central office staff, and PCG evaluation staff. This committee met monthly to discuss and strategize client participation and implementation at the ground level. PCG also provided ongoing technical assistance, reports on evaluation findings, and gathered feedback and perspectives on the results.


Sampling and Data Analysis Plan The Sampling Plan below summarizes the data collection processes based on the method used, the source of the information, the frequency with which it was collected and the sample for whom data were collected. Data were typically collected semi-annually with the exception of Medicaid claims, which were assessed through annual extracts, and MACWIS data, which were received monthly.

Method Source Frequency Sample

Document Review OCFS, SAMHS, Providers Semi-annually All state and local providers in Project

Observation/ Site Visit

District offices and community providers Semi-annually Implementation counties/districts

Key Informant Interviews

OCFS, SAMHS, Providers, Parents, Judges Semi-annually Implementation counties/districts

MACWIS OCFS Monthly Statewide with data segmentation

Case Record Reviews OCFS Semi-annually Treatment and comparison groups

Medicaid Claims MaineCare Annually Treatment and comparison groups

Parent Survey Parents served by OCFS Semi-annually Treatment and comparison group

Standardized Assessments Parents served by OCFS Semi-annually Treatment group families; others if

available Data, in all its forms, were analyzed on an ongoing basis, to answer the research questions for each of the three evaluation components. Providers had access to a data dashboard where they created their own reports to monitor rates of participation and fidelity of clients to the model. Each method had multiple information sources, as shown in the table above. PCG began by reviewing documents from OCFS, SAMHS and providers, such as provider contracts, the original Request for Proposal from SAMHS, and OCFS policy information. We also considered stakeholder interviews, focus group data, care record reviews, parent surveys, and observation notes from OCFS and provider site visits. Next, we used quantitative data to analyze relevant data points for each of the research questions. Then, we analyzed the qualitative information organized into the research questions to expand on the information from the other sources. Tools for use in the analysis are SPSS statistical software for the parent survey, MaineCare extracts, and MACWIS records, and Excel for the qualitative components, such as interview responses. A regression analysis was used to evaluate data for significant relationships.


Limitations As the real world rarely matches the controlled scientific design, limitations of methodology and logistics were encountered throughout the project, particularly methodological and logistical. Methodological Quantitative data derived from sources like MACWIS, the provider database, case record reviews, PAFAS, DASS, and CANS were limited to the quality and quantity of information contained in the respective data sets. There was no way to hypothesize or analyze data that were missing. To the contrary, much of the qualitative information in this report was derived from staff surveys, parent surveys, focus groups and interviews, all of which were self-reported and limited to the experience of the specific participant. PCG utilized both types of data in tandem to clearly define quantitative outcomes and offer an explanation of trends through qualitative refinement. Logistical Enrollment never reached the magnitude originally planned, approximately 125 families per six-month timeframe.1 Evaluators posit multiple reasons for the limited enrollment; first, MEPP was a voluntary program that required an extensive amount of time commitment. Additionally, surveyed OCFS staff reported the top three factors preventing more referrals to MEPP were clients’ failure to meet eligibility criteria (i.e., no child under five, no open case, or not qualified for IOP level of treatment), participation in another IOP, and parental employment prohibitive of program attendance. More than a third of OCFS staff (35%) also stated that the requirement of clients to participate nine to 10 hours per week in the program for 16 weeks was not a realistic time commitment for families with children still in the home.

1 This is outlined in OCFS’s Initial Design and Implementation Report.


Process Evaluation Key Questions Five key research questions guided the evaluation of program fidelity. Process Research Questions

1. How extensive and inclusive was the planning process?

2. Was the Demonstration Project implemented as planned? If not, what changed and why? 3. What in the policy, organizational, service delivery and cross-agency collaboration

context supported the Demonstration Project? 4. Were the evidence-based services, Matrix Intensive Outpatient Treatment and Triple P-

Positive Parenting Program, meeting fidelity criteria as originally designed or intended?

5. What factors influenced the adoption and spread of model enhancements?

Data Sources and Data Collection Data were collected annually from DHHS provider staff, youth and their families using interview and survey protocols developed by the evaluator. A case review tool also assisted in assessing child and parent well-being and service utilization. Data from the State’s case management system, MACWIS, were used to describe the characteristics of MEPP’s population. Results Characteristics of Families Served The following sections describe the families served by MEPP, including the needs, parenting skills and mental health status of parents at enrollment and the demographic characteristics of children. This profile provides context for the evaluation results by highlighting common family member characteristics. Data from the MEPP provider and CANS databases, as well as the data extracted from MACWIS, current as of March 31, 2019 were used to describe the characteristics of MEPP participants who enrolled as of December 31, 2018. Throughout this report, data are presented by region and by cohort. Regions refer to the three areas of Maine in which MEPP providers were active with services offered in 10 locations. The location of MEPP services providers is illustrated in Figure 1.


Figure 1. Map of Service Providers as of December 31, 2018


Cohorts refer to all clients who enrolled in MEPP in a given six-month period since project inception. Table 1 shows the breakout of the six cohorts by region. Note that MEPP had a delayed start and was not rolled out in Region Two until October 2017. Table 1. Number of MEPP Participants by Region and Cohort

Number of Families Served Table 2 shows the number of families enrolled in MEPP during each cohort and the number of parents and children represented. The number of parents is higher than the number of families in each cohort as more than one caregiver within a family could participate in MEPP. Table 2. Number Enrolled in MEPP by Cohort

Families Parents Children 0–5 Cohort One 44 48 53 Cohort Two 46 51 65 Cohort Three 45 53 63 Cohort Four 70 82 98 Cohort Five 76 80 122 Cohort Six 29 30 38

TOTAL 310 344 439 In total, 344 parents received MEPP services during the Demonstration Project, representing 310 families and 439 children ages zero to five years. As of the close of the program, 47 clients completed MEPP to fidelity. Table 3 shows the status of all participants as of December 31, 2018.

Region One Region Two Region Three Total Cohort One (4/1/2016 – 9/30/2016) 18 – 30 48 Cohort Two (10/1/2016 – 3/31/2016) 26 – 25 51 Cohort Three (4/1/2017 – 9/30/2017) 24 – 29 53 Cohort Four (10/1/2017 – 3/31/2018) 18 42 22 82 Cohort Five (4/1/2018 – 9/30/2018) 21 36 23 80 Cohort Six (10/1/2018 – 12/31/2018) 4 14 12 30

TOTAL 111 92 141 344


Table 3. MEPP Participants’ Status

Parent Characteristics Parent characteristics were derived from three assessment tools: Child and Adolescent Needs and Strengths (CANS), Parent and Family Adjustment Scales (PAFAS), and Depression Anxiety Stress Scales (DASS). Based on the 172 initial CANS completed, the domain items in which caregivers had the most “actionable needs” were substance abuse (n=154), collaboration with other parents/caregivers (n=116), financial status (n=105), transportation (n=85), and job functioning (n=79). Based on the DASS assessment, while many parents did not report symptoms of stress at enrollment, most prevalent were symptoms of depression and anxiety, which were reported by 105 and 100 of the 190 parents, respectively. In contrast, scores on the PAFAS show that parents reported frequent use of positive parenting skills as opposed to punitive methods. However, providers have expressed concern that at the time of enrollment clients were inclined to present a more positive picture of themselves than after they had become engaged in the service.

Child Characteristics

On average, MEPP families had at least two children. A total of 543 children were in the enrolled cases when all children in the case, including those older than five years of age, are considered. Table 4 shows the ages of all the MEPP children in Regions One, Two and Three. There were few differences across the regions in the distribution of the children’s ages. For example, in all regions the highest proportion of children were between the ages of zero and two (51%) at the time of referral. However, there were also numerous families who had older children in addition to those in the target age range (0–5 years); roughly one in five of all children were between the ages of six and seventeen (19%) Table 4. Age of Children of MEPP Participants by Region

Age < 1 year

1 year old

2 years

old

3 years

old

4 years

old

5 years

old

6–8 years

old

9–13 years

old

14–17 years

old

Region One (n=164) 30% 16% 12% 9% 12% 9% 10% 1% 0%

Region Two (n=174) 25% 16% 8% 11% 13% 7% 10% 6% 2%

Region Three (n=205) 25% 12% 8% 13% 6% 11% 10% 10% 5%

TOTAL 27% 15% 9% 11% 10% 9% 10% 6% 3%

Region One Region Two Region Three Total

Active clients (as of 12/31/2018) 3 24 20 47 Completed to Fidelity 25 14 8 47 Discharged Before Graduation or Completion 83 54 113 250

TOTAL 111 92 141 344


Tables 5 and 6 describe the gender and race of children between the ages of zero and five in Regions One, Two and Three. A slightly higher percentage of children in Region Three were males when compared to those in the other two regions. The overwhelming majority of the children ages zero to five served by MEPP (86 percent) identified as white, however, this follows closely with the general population of children ages zero to five who are involved with OCFS (80 percent). Table 5. Gender of Children Ages 0–5 of MEPP Participants by Region

Gender Male Female

Region One (n=145) 50% 50%

Region Two (n=141) 50% 50%

Region Three (n=153) 56% 44% TOTAL 52% 48%

Table 6. Race/Ethnicity of Children Ages 0–5 by Region

Race/Ethnicity White Black Asian American

Native American

Pacific Islander Unknown

Region One (n=145) 80% 7% 3% 1% 0% 9%

Region Two (n=141) 92% 2% 0% 1% 0% 5%

Region Three (n=153) 87% 4% 0% 6% 2% 1% TOTAL 86% 5% 1% 3% 1% 5%

*More than one race/ethnicity can be selected so percentages may not sum to 100%. The top actionable needs among children ages zero to five identified on the 172 initial CANS completed at the time children’s parents started MEPP were substance exposure (n=91), abuse and neglect (n=68) and relationship permanence (n=68). These are among the primary issues that MEPP was designed to address, ensuring that child safety, ability to reunify in a timely manner, and overall well-being improved. Q1. How extensive and inclusive was the planning process? Under a contract separate from this evaluation and prior to the Hornby Zeller Associates acquisition, DHHS contracted with PCG to assist in planning for the implementation of Maine’s Waiver initiative. DHHS and PCG used data from MACWIS to identify a service gap for families with children between the ages of zero and five who had concurrent needs for parent education and substance use treatment. According to the Adverse Childhood Experience (ACE) study,2 prolonged, persistent neglect and toxic stress can impede brain

2 Felitti, V. J., Anda, R. F., Nordenberg, D., Williamson, D. F., Spitz, A. M., Edwards, V., ... & Marks, J. S. (2019). Relationship of childhood abuse and household dysfunction to many of the leading causes of death in adults: The Adverse Childhood Experiences (ACE) Study. American journal of preventive medicine, 56(6), 774–786. Abstract available at: https://www.ajpmonline.org/article/S0749-3797(98)00017-8/abstract

https://www.ajpmonline.org/article/S0749-3797(98)00017-8/abstract

https://www.ajpmonline.org/article/S0749-3797(98)00017-8/abstract


development and negatively impact cognitive, social, and emotional functioning. As of state fiscal year 2014, substance use was a risk factor for 61 percent of families substantiated for abuse or neglect of children between zero and five years of age. Therefore, the target population was defined as families with at least one child between the ages zero and five, who were involved in the child welfare system with an open case, where one or more parents also met substance use assessment criteria. Traditionally, OCFS recommended that parents complete substance use services prior to receiving parenting education, but accessibility and transportation were often noted as barriers. This Demonstration Project aimed to offer both services concurrently and at the same location to increase accessibility and expedite reunification when children were removed from the home. The planning process was informed by data collected from participants of several focus groups, which included Native American tribes, service providers and parents receiving substance abuse treatment. Discussions in these groups supported the state’s identification of the need for the services and the groups made suggestions as to what the services should look like. OCFS partnered with the Department’s Substance Abuse and Mental Health Services (SAMHS) regarding the search for coordinated parenting education and substance abuse treatment services through a request for proposals. OCFS also recognized that SAMHS had information on the quality and integrity of the applicants since it had contracted with them as well. District staff (Program Administrators, supervisors, and caseworkers) were not involved with the planning process or provider selection, nor were the courts. In addition to the partnership with SAMHS, OCFS has made substantial efforts to increase awareness and promote the implementation of the Waiver demonstration both inside and outside of the agency. MEPP was discussed at district meetings, town hall calls, in an OCFS/CBHS informational newsletter, and at a Judicial Symposium. PCG performed numerous evaluation activities in late-2015 to plan and prepare for MEPP implementation. Building on the work completed to respond to the Department’s RFP to complete the evaluation, PCG used a literature review, input from OCFS and SAMHS on existing measures and instruments, and consulted with DHHS’ Office of Quality Improvement Services to develop a comprehensive process, outcome and cost evaluation. Through a series of meetings with OCFS, a comprehensive evaluation plan, including a logic model, research methodology and target population/sampling plan, was developed. Included within the evaluation plan, were the initial quantitative and qualitative data collection instruments. Those materials were submitted for federal review and approval.


Q2. Was the Demonstration Project implemented as planned? If not, what changed and why?

Implementation was designed to roll out in three phases beginning in Region Three (which encompasses Districts 6, 7 and 8); followed by Region Two, encompassing Districts 3, 4 and 5; then concluding with Region One, encompassing Districts 1 and 2. Region Two providers, however, experienced considerable delays in securing offices, achieving licensure and setting up referral processes. Therefore, programs across the state were not implemented in the order first planned. With the staggered roll-out of Region Two months after implementation of the program in Regions One and Three, providers were better prepared to implement the program, learning from the challenges encountered from the other regions. Consequently, the delay in Region Two created different challenges for staff, such as remembering how to input provider database information, as the launch happened at least two months after the staff were trained. In Regions One and Three, providers struggled with low volumes of client enrollment making it difficult to run classes and low participation in CANS assessments. As mentioned, initially, CANS assessments were conducted by OCFS case workers, but the administration of this assessment switched to Triple P providers in October 2017 after concerns that parents involved with DHHS may not be as forthcoming about their parenting knowledge and skills with caseworkers as they would with therapeutic providers. As MEPP enrollment increased, the provider database was updated to add demographic reports, the tracking of fidelity measures, and improved navigation. These updates and refinements were critical for accurate data collection and fidelity monitoring by providers and evaluators but required additional training and technical assistance for providers on efficient use of the database. Implementation continued to evolve over the life of the Waiver. As of September 17, 2018, the Triple P provider in Region Two, Community Health & Counseling Services, chose not to continue its contract. As a result, MaineGeneral Medical Center assumed the role of Triple P provider at all four sites (Auburn, Rockland, Skowhegan, and Augusta) in Region Two (see Figure 1). Additionally, Maine faced other challenges including interagency collaboration at the state level, client identification, program referrals, and participant accountability. Each is described below.


State-Level Coordination The program was meant to be a joint offering between OCFS and SAMHS; however, staffing turnover within SAMHS during the planning phase created a barrier to strong cross-agency collaboration. After nearly a year of implementation, OCFS staff cited a critical need for staff education from SAMHS about Medication Assisted Treatment (MAT) and how to incorporate clients on MAT into the MEPP program. It was clarified that clients could be involved with MAT and, also enroll in MEPP. However, providers reported that MAT prescribers were not always willing to work closely with them to support MEPP clients through their IOP treatment. Despite efforts to re-engage SAMHS staff in the Demonstration Project, due to organizational turnover and changes to internal funding allocation priorities, collaboration never rematerialized. Client Identification The target population included parents with moderate or high risk factors for alcohol and/or substance abuse, who met the substance abuse assessment criteria of the Matrix Model Intensive Outpatient Program, consistent with American Society of Addiction Medicine (ASAM) Level II. Identifying eligible MEPP clients also evolved over the course of implementation. Initially, OCFS caseworkers had primary responsibility for identifying those participants eligible for MEPP on their respective caseloads. To help identify qualifying clients, caseworkers began using a report of children who had recently come into care, also known as the “kids in care” report. This report was developed by central office staff after concern was expressed that caseworkers may be missing referral opportunities. A representative from the central office sent this report weekly to each district’s MEPP liaison, who then reviewed the list, researched the case status and performed outreach to the client’s caseworker to determine whether the family was eligible. This addition to the referral process offered a second check for caseworkers, but the process relied heavily on central office staff to send the report and a MEPP liaison at each site to follow up. In March of 2018, the State of Maine enacted several changes in law for child protective services which placed greater demands on OCFS staff. As a result, how clients were identified as potential candidates for MEPP changed, and so too did the referral process. The responsibility for identifying clients shifted more to the providers. During late 2018, it

“The MAT provider requires people attend a group once a month, but that's not really treatment. They also consider

someone clean and sober if they are smoking marijuana and drinking. We

don't. We need better treatment resources. People get confused about

what treatment is when clinics have different standards. It’s harm reduction

versus abstinence. There's really no community education around this.”

– MEPP provider


became an unofficial policy for caseworkers to urge clients with substance use concerns to obtain a Level of Care (LOC) assessment. Often, the provider conducting the LOC were also the district’s MEPP provider. While the state’s original intention was to have the LOC administered by a provider that was not providing MEPP services to avoid potential conflict of interest, this was not possible due to the limited number of substance abuse provider agencies in the state. However, this precaution was not justified as all clients prospectively eligible for the program receive a standard bio-psycho-social assessment using the ASAM level for care checklist. As such, caseworkers began to work more closely and rely on MEPP providers to help them identify clients who were eligible for MEPP by way of the assessment. Through interviews conducted in March 2019, it became evident there was a lack of ongoing staff training for MEPP and the MEPP referral process slowed in Region Two. There was no longer the statewide OCFS push to have people assessed for a Level of Care. Additionally, significant staff turnover at most OCFS district offices and the lack of a formal MEPP training plan left new caseworkers to rely on their supervisor or co-workers to help with identifying and making appropriate referrals. Program Referrals When the program began in mid-2016, OCFS district staff did not immediately receive formal training on the referral criteria. This led to confusion among staff which resulted in incomplete referrals (i.e., those with missing documentation or referrals not going to the right person within an agency), and the referral of some clients who did not meet the eligibility criteria (e.g., who did not have a child under the age of five). As a result, PCG conducted statewide trainings in the latter portion of the first year of implementation to educate OCFS staff and providers more thoroughly on the criteria clients need to satisfy and the referral process for prospective clients. At the end of year one, providers in Regions One and Three revised their intake procedures. One provider developed Matrix IOP policy and procedure manuals for providers. Two organizations created a new admissions packet for their MEPP clients to allow for common policies and procedures across both organizations. Other innovations included a new referral form and revisions to intake procedures to account for the administration of the PAFAS and DASS tools and use of the MEPP provider database. Providers also offered regular “office hours” in OCFS district offices to allow caseworkers to meet with them and ask questions about the services. Though this was recognized by OCFS as a best practice, several providers were not able to continue this practice on a consistent basis during the third and fourth years of implementation due primarily to increased caseload demands of clinicians, lack of time to dedicate to out-of-office activities, and a general belief that the face time with caseworkers was no longer needed. Follow-up site visits by PCG, conducted in the of Spring 2018, were used to provide additional technical assistance to clarify the qualifying criteria. Interviews with staff in the Summer of 2018 from Regions One and Three revealed that worker turnover necessitated regular, ongoing training of the MEPP program. Region Two caseworkers and


supervisors in Spring 2019 interviews anecdotally reported that having providers in the office was also still necessary. These interactions served to foster relationships between providers and OCFS staff, educate a high-turnover workforce that generally has less than five years of experience in case management, and promote collaboration between substance use treatment goals and OCFS case goals. Annual staff surveys consistently showed that more than 60 percent of staff wanted more training and information on substance use, Matrix IOP, and Triple P programming to help with making appropriate referrals. Concomitantly, OCFS interviewees suggested that there was a lack of provider knowledge about child protection services. This was confirmed by provider interviewees who expressed an interest in learning more about the child welfare system. The referral process was updated in year three. Originally, caseworkers made two separate referrals during MEPP intake. The first was for the client to complete a LOC assessment; then, if the client qualified for IOP, the OCFS caseworker made a referral to MEPP itself. While effective, it was not efficient and often resulted in substantial delays for getting people into treatment. Collaboration between OCFS central office and providers resulted in a revised process by allowing caseworkers to submit both LOC and IOP referrals at the same time so that services could start immediately if the client qualified. Even with the discontinuation of Waiver funding as of December 31, 2018, OCFS plans to use this process to offer dual prior approval for LOC and services through September 30, 2019. Regardless, low referral rates were of concern throughout the project. Some providers suggested that caseworkers were simply not making the referrals. According to OCFS staff surveyed in 2018, the top three factors preventing more referrals to MEPP were clients’ failure to meet eligibility criteria (i.e., no child under five, no open case, or not qualified for IOP), participation in another IOP, and parental employment prohibitive of program attendance. More than a third of OCFS staff (35%) also stated that the requirement of clients to participate nine to 10 hours per week for 16 weeks in MEPP was not a realistic time commitment for families with children still in the home. Staff were not surveyed in 2019. However, stakeholder interviews within Region Two during 2019 suggest similar existing barriers. Interviewees in all three Regions also suggested that timing of a referral in relation to the OCFS case also impacted the success of program participants. In other words, if families were referred to MEPP later in their case as a last-ditch effort to reunify with their children, they were less likely to be successful in completing the program or reunifying. If they are referred earlier for treatment, they have a longer period of time during which they can achieve and demonstrate sustained sobriety and behavioral change to OCFS, which could influence timelines for reunification. Providers reported sometimes feeling that families were referred too late to give MEPP an adequate amount of time to really work and impact a case trajectory or outcome.


Many clients who participated in MEPP already had their child(ren) removed from the home at the time of referral. In fact, over half of all clients had their child(ren) removed from the home prior to beginning the program. Considering the program was originally intended primarily for families with all children in the home at referral, this is one piece of fidelity to the original model that was not adhered to. Participant Accountability Implementation as it relates to the circumstance and degree to which participants were held accountable to the program by the provider was a struggle on many levels. Some districts in Regions One and Three reported that they had a “false start” in the beginning. Unclear expectations in terms of attendance and drug use resulted in confusion and lack of participant accountability. Region Two had the benefit of taking what was learned in the other Regions to develop better protocols around these issues. However, discrepancies in participant accountability between districts and regions persisted. For example, some providers were of the mindset that as long as a client was willing to continue to engage, even to a very limited extent, they did not want to discharge the client from services, with the belief that some engagement is better than no engagement. Conversely, other providers developed strict organizational policies around excused and unexcused absences, contending that lapsing attendance and lax accountability affected other members of the group. Clients who participated in the focus groups indicated a desire to see agencies more clearly outline attendance requirements to reinforce consistency in group member attendance. Since Matrix is a harm-reduction model, not an abstinence-based model, prescribed substance use was a grey area. Guidelines around drug use took many conversations with OCFS and the Matrix IOP trainer to outline. It was not until year two that the Evaluation Committee determined that it was okay for clients to participate in Medication Assisted Treatment during Matrix, but medical marijuana should not be permitted past week five. However, it was never clarified whether banning the use of medical marijuana was an explicit Matrix requirement or a preference of the Matrix trainer. This resulted in continued confusion, with some providers refusing to discharge a client for any continued marijuana use (recreational or medical), and others discharging or placing clients on treatment holds to ensure re-commitment to the program.

“Very few complete in 16 weeks because of the pre-contemplative

nature of most of the clients. They’re under a lot of pressure to reunify,

and it can seem daunting and clients don’t engage because of that.

Oftentimes, cases are further into the process into the reunification time

fames before the referral is made.”

– MEPP provider


There were also conflicting opinions about medical marijuana use among providers, staff, and courts. While all interviewed OCFS staff stated that sobriety was one of, if not the primary factor for determining the trajectory of a case, medical marijuana use seemed to be determined as appropriate or inappropriate on a case-by-case basis, taken into consideration with the client’s drug of choice, original reason for DHHS involvement, and behavioral changes (as demonstrated by the attitude and actions of the clients). Furthermore, the Leadership Committee discussed that while staff seemed to have differing opinions on continued use, courts were reportedly more conservative, often seeing any use as a potential risk. Therefore, providers were encouraged to discuss drug use with clients in this way. That is, despite whatever the program allows or their caseworker states, any continued marijuana use could be seen as a risk by the court and could, in turn, impact keeping all children in the home or reunifying with if a child was already removed. Q3. What in the policy, organizational, service delivery and cross-agency

collaboration context supports the Demonstration Project? The Demonstration Project required a significant amount of collaboration and interdependence of various agencies acting together to support recovery and reunification of participants with their children. This collaboration did not come naturally as provider and DHHS systems are generally organized to operate in silos. During interviews with both OCFS staff and providers, it was observed that the objectives of both parties to help clients meet their goals were directly related to the service each agency provided. The goals for reunification and recovery, while complementary, did not always overlap in terms of objectives. For example, consistently in all Regions, providers saw their role as helping people to become sober or manage their addiction, while caseworkers viewed their own role as monitoring success of families in creating safe environments for children. While caseworkers stated that sobriety was an important factor when making decisions about case trajectories, they did not take responsibility in helping clients to become sober. Conversely, when providers were asked about their role in helping clients meet their case goals, roughly half related their position to creating safe environments for children. Furthermore, child safety was viewed as a secondary outcome to sobriety or addiction management. Several clinicians were not aware of client goals, had never received a copy of the goals, or were aware only of what the client had told them about his or her case goals. Several providers indicated interest in receiving a copy of client reunification plans so that case goals could be more broadly addressed while a client was in treatment. However, some OCFS interviewees reported that they do provide information about reunification plans and goals to the providers and were uncertain as to where information breakdown occurred.


Policy Changes As mentioned earlier, in March 2018, OCFS enacted several changes in law and policy for child protective services which placed greater demands on OCFS staff. Consequently, changes to intake and assessment practices resulted in more investigations, which was complicated by a shortage in staffing as a result of significant worker turnover. As of the Spring of 2019, the Department reported a 20 percent increase of children in care. Some provider agencies also experienced noticeable staff turnover throughout the project with some sites having to train multiple new clinicians and providers to run Triple P groups and MIOP sessions. Feedback received from key informant interviews, focus groups, and staff surveys explored changes including staffing, program oversight, service delivery, funding allocation, and courts for each region. Changes within each area are discussed in the following sections. Staffing To streamline communication between OCFS workers and the providers, OCFS district offices appointed a MEPP liaison and providers appointed a MEPP coordinator. Both positions acted as a point person to which other OCFS or provider staff could reach out. These staffing changes, in fact, helped to clarify responsibilities, reduce referral/intake errors, and increase efficiency of communication between OCFS and providers. However, there were unintended consequences to this streamlining as well. Interviewed clinicians and caseworkers from all three regions consistently expressed that the lack of individual relationships between OCFS and the providers was a hindrance in timely communication and building a trusting rapport with one another. Caseworkers reported that the streamlined communication segmented the Triple P and Matrix providers since MEPP coordinators are Matrix clinicians in all districts. Therefore, most caseworkers reported little to no contact with Triple P providers. As a result, some Triple P providers expressed feeling disconnected from the case or client goals. Program Oversight Central office responsibilities for MEPP included program oversight, contract management and program improvement. Program oversight was accomplished through monthly leadership meetings and provision of training for Matrix IOP and Triple P. Providers cited the monthly leadership meetings as a forum to promote collaboration and assist with program delivery. It also strengthened the relationship between OCFS and community providers as they worked collectively to ensure successful delivery of substance use treatment. OCFS staff identified four setbacks which hindered team building capacity. First, the departure of two high-level OCFS central office staff responsible for Demonstration Project oversight left others to fill in and quickly familiarize themselves with the nuances of the program since its inception. Second, state policy changes enacted in March 2018 necessitated a massive shift in organizational priorities, increased workload at all levels of OCFS, and pulled focus away from the Demonstration Project. Caseworkers reported being forced to choose between completing activities that had to be done, and those that should be done to deliver quality case management, like making timely MEPP referrals.


Third, there was large-scale caseworker turnover within OCFS, which led to a lack of awareness and “greening” of the workforce, especially in familiarity with substance use and recovery communities. As responsibilities also increased for OCFS management, the lack of time for formal training of new staff was compounded. Key stakeholders reported relying on the assistance of peers and supervisors to help new staff complete referrals; office facetime with providers was also used to educate staff on the MEPP program over the last year. Fourth, while OCFS leadership of the Waiver stabilized, a change in political climate resulted in a change in the DHHS commissioner and two changes to the director of OCFS in the last year. Thus, the department was forced to realign priorities, like maintaining traditional Title IV-E funding to provide for the influx of children coming into care. As a result, the Demonstration Project never regained full leadership support or the momentum it needed to sustain and grow. Service Delivery While the evidence-based programs, Triple P and Matrix, did not change, service delivery evolved over time. As MEPP referral and enrollment waxed and waned, provider agencies saw the need to integrate clients from other referral sources to keep the groups going. Providers added groups to accommodate more clients and started to incorporate drug court clients into Matrix groups with MEPP clients. While clinicians expressed some initial hesitancy with this idea, the response was overwhelmingly positive. One provider stated that while drug court clients might have more severe addiction issues to overcome, as they are court-ordered to treatment, their attendance was more reliable and offered a level of consistency and accountability to MEPP group members. Another provider reported asking group members about the combination of groups and found that MEPP participants did not see the incorporation of drug court clients as an issue. In fact, focus group participants told facilitators they believed the program should be open to more people throughout Maine, not just those involved with DHHS. Funding Allocation Under a capped Title IV-E allocation, OCFS pledged to use a portion of the funding for children in foster care as well as to provide Demonstration Project services to children at home to help keep them safely in the home. Waiver funding was used to train contracted providers for Matrix IOP and Triple P and support the cost of parenting education and other services necessary for successful implementation, like transportation. Unfortunately, due to the substantial increase of children coming into care, increase in re-assessment of low-risk reports, and a blanket $5 per hour raise for all caseworker positions within the Department, this balance with traditional Title IV-E funding was

Focus group participants told facilitators they

believed the program should be open to more

people throughout Maine, not just those involved

with DHHS.


unsustainable and required DHHS to ask federal partners for permission to end the grant early, as of December 31, 2018. Court Involvement Originally, the evaluation was designed to solicit feedback from Maine court staff to gauge court relations. While MEPP involvement was not specifically dictated by court order, it was reasoned that judges could impact referrals by requiring families to participate in substance use treatment and parenting programs, which could be MEPP. It was theorized that different treatment strategies work better or worse for different populations of people. The thought was that if judges were educated in substance use disorder, they would naturally see treatment programs as more or less effective and parent participation in those programs as more or less a commitment to sobriety, which could impact the trajectory of court-involved cases (i.e., reunification). In year one, the evaluation team found it difficult to obtain commitment from judges to participate in the Demonstration Project. In subsequent years, interviewed OCFS staff and providers stated that the role of the courts in MEPP was not as prominent as originally thought. While neither providers nor OCFS staff necessarily agreed that treatment should be mandated by a court, they did state that courts should be better educated as to why MEPP is important and more effective than traditional IOP. OCFS staff in Region Three also offered information sessions about MEPP and substance use disorders to court staff, including judges and attorneys, but there was limited participation. This lack of court participation invariably limited project buy-in and thus prompted the evaluation team to revise the evaluation plan by removing court education as part of the plan. Instead, OCFS leadership worked to establish more formal collaboration venues with the court in general through development of the court improvement plan. Q4. Were the evidence-based services, Matrix Intensive Outpatient Treatment and

Triple P-Positive Parenting Program, meeting fidelity criteria as originally designed or intended?

The fidelity assessment of MEPP examined the administration of the evidence-based practices, Matrix IOP and Triple P, by looking at implementation in the following areas:

1) Qualifications of the Providers 2) Eligibility of the Population 3) Service Completion and Attendance 4) Service Content, and 5) Duration and Frequency of the Service.


Qualifications for providers and eligibility requirements for participants remained consistent throughout MEPP. Matrix and Triple P clinicians were required to be trained in the model in which they provided services. Participants were required to have at least one child between the ages of zero and five, an open case with OCFS, and at least one parent qualifying for an IOP level of care to be eligible for MEPP. Qualifications of Providers This fidelity requirement was met consistently each reporting period with all provider staff members trained in either Triple P or MIOP, despite the high turnover rates and extensive time commitments required to train new staff. Although chosen for the positive attributes mentioned previously, Triple P required significant time commitments of clinicians to attend initial, week-long trainings, offered sporadically around the country, and did not offer a train-the-trainer model. Table 7 reflects these results by agency and training type. Note that Community Health and Counseling chose not to renew its contract for service provision in Region Two during the start of Year Four. Table 7. MEPP Provider Staff Training

Trained Staff Matrix IOP Triple P Aroostook Mental Health Center 10 1

Crisis and Counseling 4 –

Crossroads 8 4

Maine General Community Care 3 –

Maine General Medical Center 1 5

Wellspring 5 –

Community Health and Counseling (Region Two) – 3

Community Health and Counseling (Region Three) – 2 TOTAL 31 15


Eligibility of the Population Figure 2 shows that most participants served through MEPP met the eligibility criteria. That is, they had at least one child who was between the ages of zero and five at the time of MEPP enrollment, an open case with OCFS, and at least one parent was qualified for IOP level of care. Only three parents without an open case received services, with two of those having open assessments when they were referred. Among the families in which a parent was referred and all children were over the age of five, the youngest child in the household ranged in age from six to 15. Although such circumstances were more common in Region Three, both Regions One and Three had issues with eligibility around the age of children in cohort five which were resolved in cohort six. Figure 2. MEPP Participant Eligibility by Region

Service Attendance and Duration MEPP was shaped by participants receiving concurrent intensive outpatient treatment and parenting services. Hence, one focus for the evaluation was to assess whether participants met model attendance and duration requirements. Fidelity measures included attending 48 Matrix sessions over 16 weeks consisting of eight early recovery Matrix groups, 32 relapse prevention Matrix groups, and 12 family education Matrix groups, as well as the completion of either Triple P Level 4 or Level 5 topics. Participants were required to complete only one of the two Triple P levels to fidelity to complete the MEPP program. Triple P providers indicated that they utilized Level 4 and Level 5 curricula for different clientele. Where Level 4 is geared toward parents of children with severe behavioral difficulties, Level 5 is focused more on providing intensive support for families with complex concerns usually around partner conflict, stress, or mental health issues. Providers reported making decisions about needed treatment based on individual circumstances. Thus, there may never be 100 percent compliance for both levels.

95%100%

96%100% 100% 100% 100% 100% 100%

97% 100% 100% 100% 100% 100%

90%97%

100%95%

79%84%

90% 87%

Cohort1

Cohort2

Cohort3

Cohort4

Cohort5

Cohort6

Cohort1

Cohort2

Cohort3

Cohort4

Cohort5

Cohort6

Cohort1

Cohort2

Cohort3

Cohort4

Cohort5

Cohort6

Region1 Region2 Region3

Open service case or child entering state custody Youngest child between 0 and 5 years


Most fidelity measures looked specifically at clients who successfully completed the program; however, the discussion presented here also includes information about clients who did not complete the program and the extent to which fidelity had an impact on client engagement. As noted, the fundamental change in how fidelity was determined (by tracking independent components rather than clinician-assigned status) drastically decreased the number of individuals who successfully completed the program in all regions. Table 8. Percent of Graduated Participants by Completion of Fidelity Component

Cohort 1 (n=19)

Cohort 2 (n=16)

Cohort 3 (n=18)

Cohort 4 (n=22)

Cohort 5 (n=28)

Total (n=103)

16 Weeks in MEPP 94% 100% 83% 95% 96% 94% 48 Matrix Sessions 61% 69% 53% 68% 73% 65%

8 Early Recovery Groups 94% 100% 100% 100% 100% 99% 32 Relapse Prevention Groups 72% 94% 65% 82% 95% 82% 12 Family Education Groups 28% 50% 48% 73% 91% 60%

Triple P Level 4 56% 56% 71% 91% 78% 72% Triple P Level 5 45% 44% 35% 46% 86% 53%

Total Completed to Fidelity 22% 25% 29% 55% 64% 41% While the areas of service completion, attendance, content, duration, and frequency remained unchanged, some of the ways they were tracked by providers and monitored within the database were modified. Historically, there were circumstances in which providers determined that clients were eligible to graduate from Matrix before all components were completed. Providers and clinicians considered a number of factors, such as full commitment to MEPP and compliance with its requirements, before identifying participants as ready to graduate. However, the Leadership and Evaluation Committees agreed that this method of identifying program completion allowed too much variation among agencies, districts, and regions. Therefore, to increase fidelity and program consistency, beginning in July 2018 fidelity of the program began to be monitored more specifically in terms of meeting a required number of sessions, groups, topics, and weeks. Only participants who satisfied all requirements were counted as having completed the model to fidelity. Clinicians were still asked to denote whether someone had graduated from the MEPP program, but this was no longer the primary basis for determining fidelity. Table 8 shows the percent of parents who completed each aspect of the MEPP model to fidelity. Completion increased comparatively from Cohort One to Cohort Five, increasing from 22 percent to 64 percent, respectively. While it is clear that providers strived to implement MEPP with integrity, refinements to the provider database found that a number of clients who were previously counted as having completed the program had not met the program’s complete set of fidelity


requirements. Three additional fidelity components were also tracked outside of MEPP program completion: urine screen compliance, participant substance use, and aftercare planning. The availability of new fidelity tracking reports assisted clinicians with increasing program component completion and fidelity compliance for Cohorts Four and Five. Despite these changes, providers continued to struggle in obtaining 100 percent fidelity. When interviewed about the ease of meeting the program’s fidelity requirements, providers in all three regions stated that meeting fidelity was not hard since changes were made to the database to help with session tracking. However, participant attendance was an uncontrollable variable. Intermittent attendance and absences still made it challenging for providers to offer nonduplicative curriculum. A few providers talked about the need for flexibility in the timing and duration of services, suggesting that clinicians be allotted more control of the duration of treatment required for individuals to be successful. Participants who were unable to complete the program or graduate reported facing challenges affecting their success in MEPP and impacting their reunification with their child in all three regions. Reliable transportation was consistently a struggle for clients to get to and from appointments and sessions. For OCFS to consider reunification, parents also needed to demonstrate a consistent behavioral change and sobriety. Additionally, parents needed to be employed to support themselves and to find stable housing, a necessity for reunification and maintaining children in their own home; however, attendance at a 16-week program, three hours a day, three times per week proved to be a strain on finding or maintaining employment. Matrix Attendance. Table 8 shows that overall 65 percent of the participants who graduated from MEPP attended at least 48 Matrix sessions or days of treatment. One day of participation in MEPP was equal to attending one session. In a single session, clients could participate in one or more groups, which were areas of focus covered in a Matrix session; however, the whole day was still only counted as one session. Nearly all of the clients participated in the eight Early Recovery Groups. The most dramatic improvement observed since first implementing the Waiver program was client attendance in all 12 of the Family Education Groups, increasing from 28 percent in Cohort One to 91 percent in Cohort Five. Duration. Attendance data over the lifetime of the Waiver demonstrated that most clients progressed through MEPP at a slower rate than intended. MEPP was designed for

“The MEPP program is a long, long time for IOP. Clients can't really work while

participating. They lose their kids, lose their benefits and can't work. It's a cycle.

Perhaps courts could mandate 12 weeks instead of 16 and it wouldn't be such a

hardship. The last month could be aftercare.

Also, spouses can't do IOP together at the same place, but it's too far to go

somewhere else. Twelve weeks could allow them to cycle through faster.”

– MEPP provider


participants to attend three Matrix sessions per week for 16 weeks for a total of 48 Matrix sessions (or days of service). Yet, due to illness, issues with childcare, transportation, inclement weather, lack of engagement and other challenges, nearly all participants moved through the program at a slower pace. Overall, 85 percent of the clients who graduated and 96 percent who completed MEPP to fidelity did so in more than 16 weeks, with the longest requiring 34 weeks. It is not entirely clear how spending more time in MEPP impacted participant engagement. During stakeholder interviews, providers speculated that the longer clients were in the program, the more motivation to finish could wane; but, time in the program was not always a bad thing when clients were committed to change. Caseworkers expressed that client engagement could be more important than the actual completion of MEPP. A lack of engagement could demonstrate that someone was just going through the motions and, therefore, lacked commitment to sobriety. Two caseworkers also mentioned that if participants failed to complete MEPP, they were not likely to complete any other goals on their case plan. In this way, MEPP was used as a barometer for client commitment and behavioral change. Figure 3 shows the number of sessions and weeks completed by clients who were active, completed MEPP to fidelity or were discharged before they met fidelity requirements (i.e., were incomplete) as of December 31, 2018. As indicated by the black line, if participants moved through the program at the pace in which it was designed, they would have completed 21 sessions by the end of week seven, 30 sessions by the end of week 10, and 48 sessions by the end of week 16. Ultimately, participants were most likely to drop out of the program within the first two and a half weeks. Additionally, the vast majority of those who did complete the program to fidelity took more than 17 weeks to do so. Figure 3. Number of Weeks and Sessions by Active, Completed and Incomplete

The black line represents the number of sessions completed if a client always attended three sessions per week.

05

101520253035404550556065

0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48

Sess

ions

WeeksActive Complete Incomplete


Triple P Initiation. OCFS and MEPP providers jointly made a clinical recommendation that participants start Triple P approximately four weeks after Matrix. In theory, this delayed start would allow participants an opportunity to engage in IOP and give them time to focus on their recovery. However, Table 9 shows that participants who started Triple P within the first three weeks of MEPP have slightly higher completion rates (33 percent) than those who began between four and eight weeks of enrollment (25 percent). In fact, more than half of participants who completed the program started within the first three weeks of MEPP. It can be concluded that a more delayed start in Triple P programming likely hinders program completion. There is a considerable decrease in the percentage of families who complete the program when beginning Triple P after eight weeks of enrollment. Table 9. When Participants Start Triple P by Completions and Discharges

Weeks of MEPP participation at the time of Triple P start

Participants who completed MEPP

(n=51)

Participants who did not complete MEPP

(n=147)

Total (n=198*)

0–3 30 (33%) 62 (67%) 46% 4–8 17 (25%) 51 (75%) 34% 9–12 2 (10%) 19 (90%) 11% 13–16 1 (10%) 9 (90%) 5% 17+ 1 (14%) 6 (86%) 4%

* Includes four clients who have met fidelity requirements to complete the program but have not yet graduated and are still actively attending sessions. Table 10 shows the number of participants who discharged from MEPP before starting Triple P. Nearly three-quarters left before the ninth week in the program. Table 10. When Participants Were Discharged from MEPP Before Starting Triple P

Weeks of MEPP Participation Among clients discharged from MEPP before starting Triple P

Number of Clients Percentage of Clients

0–3 47 42% 4–8 36 32% 9–12 16 14% 13–16 5 5% 17+ 8 7%

TOTAL 112 100%


Drug Screenings. Matrix IOP requires weekly urine screenings to increase participants’ accountability and monitor their drug use. While consistent sobriety was the goal, caseworkers used the screenings to monitor behavioral change in substance use as well. Tables 11 and 12 show the percentages of participant compliance with urine screenings by cohort and region. Table 11 includes participants who attended at least one Matrix session per week; Table 12 includes those who attended three sessions per week. As the program is designed for participants to attend three times per week, providers and caseworkers viewed decreased attendance as partially reflective of a lack of behavioral commitment to change and reduced opportunities for participants to prove sobriety via urine screens. Table 11. Compliance with Urine Screening When Participants Attend At least One Session per Week

Statewide, compliance with urine screening remained similar across all cohorts, varying within three percentage points from Cohort One to Cohort Five. There was in increase in urine screening compliance in Region One over the lifetime of the Waiver, but a decrease in Region Three. Region Two compliance remained similar between Cohorts Four and Five. Clients who attended at least three sessions per week were more consistent in completing the urine screens than those who attended only once or twice per week. However, overall compliance declined over time, especially for participants in Region Three. Region Three which also experienced a decline in compliance for participants who attended less than three sessions per week.

Total Participant Weeks Participant Weeks in Compliance

Region Cohort

1 Cohort

2 Cohort

3 Cohort

4 Cohort

5 Cohort

1 Cohort

2 Cohort

3 Cohort

4 Cohort

5 Region One 184 283 327 170 248 86% 90% 90% 91% 92%

Region Two – – – 540 465 – – – 85% 83%

Region Three 320 211 245 150 263 79% 79% 70% 65% 64%

TOTAL 504 494 572 860 976 82% 85% 80% 82% 79%


Table 12. Compliance with Urine Screening When Participants Attend At Least Three Sessions per Week Total Participant Weeks Participant Weeks in Compliance

Region Cohort

1 Cohort

2 Cohort

3 Cohort

4 Cohort

5 Cohort

1 Cohort

2 Cohort

3 Cohort

4 Cohort

5 Region One 83 118 150 64 136 99% 98% 97% 98% 97%

Region Two – – – 232 205 – – – 99% 96%

Region Three 160 93 81 54 88 90% 86% 80% 76% 67%

TOTAL 243 211 231 350 429 93% 93% 90% 94% 87% *One parent in Cohort One did not have region information available and is omitted from the regional counts. Participants’ Use of Matrix Materials. Providers regularly asked participants to complete worksheets during their group sessions. Participant binders and workbooks also were observed by evaluators during site visits. Participants reported using binders to collect the worksheets throughout the program, serving as a journal to document their progress. Regression Analysis A stepwise linear regression analysis was performed on the fidelity criteria to determine which combination of presenting client characteristics and service utilization are correlated with higher levels of fidelity to the MEPP model. In brief, a stepwise regression first runs a regression on a set of independent variables (i.e., factors) and determines if the resulting model would be improved by eliminating variables or adding back variables eliminated in previous iterations. Ultimately, the analysis produces a list of factors found to be correlated with the outcome. Table 13 displays factors that the regression found to impact at least one outcome measure. Factors in green are positive results (e.g., more likely to spend weeks in MEPP) with red used to note negative results. Darker green- or red-shaded factors are significantly correlated (p < 0.05) while lighter green- or red-shaded factors show correlation, but not at the significance threshold. It should be noted that the sample sizes available for these outcomes are large enough to make causation claims for factors on the outcomes at a power of 95 percent.


Table 13. Factors Impacting MEPP Completion and Fidelity Outcomes

Intermediate Outcomes

Factor

Weeks of MEPP

(N = 177)

Number of Matrix IOP Sessions

Completed (N = 177)

Number of Triple P

Sessions Completed (N = 177)

Completed MEPP to Fidelity

(N = 177) Region One Region Two Region Three Removal Prior to MEPP Previous Substantiated Report Previous Unfounded Report Drug of Choice: Marijuana Employed Full Time Received Counseling Services Received Substance Abuse Inpatient / Detox Services

Received Other Substance Abuse IOP Services

Received Other Parenting Services PAFAS Parenting Score PAFAS Family Score DASS Depression Score DASS Stress Score

Significant findings of the regression analysis are discussed below.

• Region. Participants from Region One were significantly less likely to spend more weeks in MEPP than the other two regions. Similarly, Region One was slightly less likely to complete more sessions of Triple P than the other two regions.

• Services. Clients who received other IOP supports besides Matrix were more likely to complete MEPP to fidelity. Additionally, parents who received other parenting services were more likely to spend more weeks in MEPP and complete more sessions of Matrix and Triple P. Clients receiving counseling services during MEPP were slightly less likely to complete more weeks of MEPP.

• Parenting Skills. Parents with a higher initial PAFAS family score (meaning parents showed fewer positive family practices; e.g., worse family relationships and parental teamwork) were slightly more likely to complete more Triple P topics. Parents with higher PAFAS parenting scores (meaning parents showed fewer positive parenting practices; e.g., more coercive parenting and lower parental consistency) were slightly less likely to complete MEPP to fidelity.


• Depression, Anxiety and Stress. Parents with a higher DASS depression scores (meaning the client shows more depression symptoms) were less likely to attend more weeks in MEPP than parents with lower scores. Parents with a higher DASS stress score were less likely to complete more Matrix sessions or to complete MEPP to fidelity.

• Removal Prior to MEPP. Clients with children removed prior to the initiation of MEPP were significantly less likely to remain enrolled in MEPP for longer periods of time and slightly less likely to complete sessions of Matrix or Triple P.

• Employment. Clients who were employed full time were more likely to complete MEPP to fidelity.

• Previous Reports. Parents who had reports prior to the case with MEPP involvement, whether they were substantiated or unsubstantiated, were more likely to complete MEPP to fidelity.

Q5. What factors influenced the adoption and spread of model enhancements? Throughout the Demonstration Period, several agency, departmental, and judicial factors influenced the ability to grow the program. On an agency level, providers noted several times that the cost of Triple P training was very expensive and limited how many staff could be trained, resulting in a consistent lack of qualified providers for some areas, especially the more rural districts. Interviews with providers also revealed that intentional cooperation between Matrix IOP and Triple P providers was critical to success of the program. Each district did this differently to more or less the same effect. For example, in some districts the clinicians had regular communication, which was made a priority; in other districts, providers rarely interacted or spoke about client progress unless issues arose. Focus groups reveals that the coordination and interaction appeared to have a direct impact on client retention and graduation rates. Within OCFS, several changes in leadership, including the OCFS director and DHHS commissioner, shifted departmental priorities away from MEPP to other more pressing department initiatives, like re-evaluation of assessments, increased children in care, and immediate policy overhaul to ensure child safety. Additionally, the lack of formal, ongoing training for current and new staff in combination with the lack of consistent provider presence in some district offices heavily influenced program referrals. As previously noted, OCFS also initially offered training to court staff; however, there did not appear to be much interest by judges or lawyers in learning more about MEPP. According to OCFS staff, this lack of education worked against the dissemination of MEPP in a few ways. First, some caseworkers reported that attorneys would dissuade clients from completing a Level of Care assessment and attending MEPP or any other


treatment program, so that a substance use disorder was not documented or acknowledged. Second, providers and OCFS staff in all regions consistently expressed belief that MEPP requirements for length of time and duration were not as enticing to participants as shorter, less involved programs. They also said that if families were required to attend programs, attorneys would naturally steer clients to shorter programs or programs that were abstinence-based to present a clearer picture to the court. Finally, judges were not always aware of the importance or benefits of different types of programming, thus they would not necessarily encourage participation in longer, more intensive programming. However, both providers and caseworkers saw value in having clients participate in a longer program, understanding that substance use recovery takes time and is often not a linear progression. Program Perceptions Focus groups were conducted in the Spring of 2018 to capture client perceptions of the program. In all four groups, clients emphasized learning about addiction and recovery as a positive experience of MEPP. Specifically, participants stated they learned about relapse prevention strategies, coping skills, ways to channel emotions into healthy activities, time management, seriousness of addiction, and effects of addiction on the brain structure. Participants also consistently stated that they achieved some level of personal growth and development in self-confidence, the desire to succeed, reclaiming a purpose in life, reducing depression and/or sense of isolation, and increasing stability and personal care. More than 80 percent of OCFS staff who have referred clients to MEPP indicated in the 2018 survey that families are satisfied with Matrix IOP and Triple P and like receiving substance use treatment and parenting services concurrently. However, only three-quarters of the staff surveyed believe MEPP assisted clients to achieve sobriety and behavioral changes, which were the highest priority factors for determining reunification.

Both providers and caseworkers saw value in having clients

participate in a longer program, understanding that substance use

recovery takes time and is often not a linear progression.


The majority of OCFS staff surveyed agreed or strongly agreed with the following:

► Delivering Matrix IOP and Triple P concurrently is more effective than delivering them one at a time (96%)

► MEPP improves outcomes for families by helping parents meet case goals (85%)

► MEPP helps to prevent removal of children from their homes (63%)

► MEPP is effective in achieving reunification (84%)

► MEPP helps to reunify parents with their children more quickly (71%) Caseworkers want to see a fundamental shift in how caregivers make decisions about their lives and their children. They saw MEPP completion as a factor only in that it demonstrated engagement in recovery and corresponding behavioral change. Other important factors weighed by the department are what friends and family are saying about the person, their social group, their ability to take responsibility for past actions and demonstrate a plan for ongoing recovery. Most OCFS staff were unsure of how the Triple P and Matrix curricula overlapped or if overlap was intended. Providers who stated that the programs were strongly integrated made intentional efforts to relate the Triple P curriculum to the Matrix curriculum or vice versa and had regular, detailed communication with the other provider. However, several suggested that Triple P was often seen as an afterthought by OCFS. Clinicians suggested that having two different agencies provide Matrix and Triple P complicated the relationship and required more effort to communicate effectively about program enrollment, attendance, progress, relapses, and discharge.

“We have similar material, but it doesn't really line up. We don't

know much about what curriculum the other program is covering

on a given day.”

– MEPP provider


This lack of integration was also evidenced in aftercare planning. While discharge and aftercare planning were requirements for both OCFS and clinicians, there was little evidence of collaboration between the two entities. All interviewed Matrix and Triple P providers stated that the responsibility for developing the aftercare plan fell to the Matrix clinicians with little to no input from the Triple P clinician. Caseworkers and providers also often reported little to no communication about MEPP completion or discharge. More robust planning between caseworkers and providers around transitional or step-down care may be more effective in supporting parents to sustain their recovery. Communication, Collaboration and Relationship-Building While stakeholders reported that communication started to wane in year three as a result of provider and OCFS staff turnover, all parties agreed that the project, in general, gave agencies and DHHS more reasons to interact. This fostered more trust between the entities. However, OCFS did not think providers were completely forthcoming about client relapses. A small number of caseworkers stated that they had not invited a provider to a team meeting despite that providers may have valuable insight into a client’s recovery. Alternatively, providers recognized that caseworkers often had detailed history about the clients that is valuable for treatment, though they did not always receive that background information timely from caseworkers. Nearly all providers stated that they had been to at least one client’s team meeting, but only about half indicated that they attended team meetings with any regularity. Providers who did not consistently attend meetings most often stated that they were not invited, nor given enough notice, and/or could not bill for time outside of the office. However, all providers agreed that their knowledge of a client’s treatment progress was valuable information and should have been considered when caseworkers made decisions about case trajectories. When asked about relationships between MEPP participants and providers, both parties reported positive experiences. During focus group discussions, participants stated that providers made the group feel safe and non-judgmental, that they could be open, honest and disclose issues that they could not do elsewhere due to a safe environment. Participants also reported viewing MEPP providers as advocates in and outside of the groups. In addition, the providers appear to be viewed as mediators, fostering the relationship between parents and OCFS staff. Further, some participants viewed the MEPP provider as having a significant role in supporting their success in MEPP. During the Spring 2018 interviews, Regions One and Three providers also commented on their dichotomous relationship with clients. They reported having to be professional and hold people accountable for their substance use, while also creating a respectful, trusting relationship wherein people feel comfortable sharing their struggles. However,

“Most parents have the same recommendations because that's all

we have. We have to hope that parents will do what they need to do

and do things on their own.”

– OCFS staff


the relationship between participants and Triple P providers was not as strong as the relationship between Matrix providers and participants. This was most often attributed to the fact that Triple P providers had comparatively less facetime and interaction with participants. Even so, while all Matrix providers said they offered continued services to clients beyond MEPP, less than half stated that the relationship continued beyond MEPP discharge. By contrast, when asked about the relationships between MEPP participants and OCFS staff, focus group attendees stated the relationships vary. Participants reported that the relationship was heavily dependent upon the individual caseworker, his or her experience with substance use and recovery, longevity with DHHS, and the frequency and quality of communication between the participant and caseworker. Some participants said that they had successes and/or worked toward case goals rather quickly, while others reported facing obstacle after obstacle with DHHS. Some felt excluded, disempowered, or undermined by their OCFS caseworker. Focus group participants described concerns about lack of communication. Challenges included lack of contact with caseworkers and updates on what was happening in their case or with their children, issues setting up transportation, and a general lack of interest in the parents. Participants did not indicate whether they felt the relationship improved over time; however, the OCFS staff surveys for 2018 indicate that 74 percent of staff agree or strongly agree that parents’ attitudes toward OCFS improved during their time in the program, which is consistent with staff perceptions from the 2017 survey (71%). Sustainability As a result of the aforementioned political and economic factors incurred by OCFS in 2017 and 2018, in March 2019, OCFS leadership made a formal request to end the Demonstration Project early as of December 31, 2018. The Leadership Committee discussed continuation of MEPP services post the Demonstration Period over the course of several months of conversations. When interviewed, nearly all OCFS caseworkers and supervisors expressed a desire to see MEPP services continue, but they admitted to seeing very little success from the program with families on their caseloads or in their units. Eagerness for program continuation appears to derive from many districts that have limited substance use or parenting providers to begin with. Thus, elimination of any program, regardless of effectiveness, is seen as detrimental to the community. Matrix IOP has consistently been reimbursable through the state Medicaid plan, MaineCare. Providers and OCFS agreed that Matrix IOP could continue to be provided for MaineCare-eligible clients beyond the Waiver if providers wished to continue the programming. While there is current legislation pending to incorporate Triple P as a covered service under MaineCare, this has not yet happened. Providers have the option of continuing

“Clients know this program has our back. We are going to be there even

if things aren't going well.”

– MEPP provider


Triple P services with the understanding that OCFS will not be able to reimburse them after September 30, 2019. OCFS leadership confirmed plans to continue to refer families to MEPP provider agencies for IOP and parenting education services as appropriate; though, providers vary in their plans for continued service delivery. As of April 2019:

• Crossroads decided not to continue Matrix IOP or Triple P at either location after September. In the meantime, they will offer both programs to Department referrals and open enrollment to community members to sustain program attendance.

• Wellspring plans to continue to offer Matrix IOP to Department referrals, drug court clients, and the community. They are also working to increase partnership with another agency, Penquis, around the provision of parenting education. Penquis currently does not offer Triple P programming but does provide other types of parenting education.

• Aroostook Mental Health Clinics states that they will discontinue both Matrix IOP and Triple P offerings. They report a consistent struggle to maintain IOP referrals and find Triple P providers for the northernmost sites, in partnership with Community Health & Counseling Services.

• Crisis & Counseling has not made a final decision about continuation of services; however, due to lack of referral interest and the cost of clinician coordination outside of billable hours, the provider reports it is unlikely that it will offer Matrix IOP after the Waiver ends.

• MaineGeneral plans to continue offering Matrix IOP at the Augusta site. It has also taken steps to write Triple P into a grant for interim funding in hopes that it will eventually become MaineCare reimbursable as well.

Regardless of whether providers continue providing Matrix IOP or Triple P sessions, MEPP will end as of September 30, 2019. That is, OCFS will no longer pay for the co-location of substance use and parenting providers, nor will the Department pay for non-billable time for clinicians to collaborate with caseworkers, either remotely or in-person at client meetings.


Fidelity Assessment Summary and Key Findings Not surprisingly, the data suggest that MEPP is most successful when the program is implemented to fidelity. Providers made clear improvements over the lifetime of the Waiver to address fidelity short comings. However, due to the novelty of the program concept, stakeholder interviewees suggested that less than three years of implementation may not have been long enough to work out the challenges experienced in implementing the system and show the full potential of this complex program. The list below provides a detailed summary of the fidelity assessment findings and data pertaining to participants who graduated the program.

1) The program was most successful for the population most in need: parents having greatest need for development of new parenting skills and parents with higher depression scores were more likely to complete the program than parents reporting less need in these two areas. MEPP was also successful for parents who had previous substantiated reports with the department and for those who received additional substance use services.

2) Program fidelity was consistently an issue for multiple reasons. a) Initial definitions of graduation from MEPP were inconsistent across

providers. Thus, not all participants who are listed as MEPP graduates met all of the fidelity requirements. This improved over time with provider education and the addition of fidelity reporting capabilities in the provider database.

b) The biggest impediment to fidelity was completion of all 48 Matrix IOP sessions. Part of this gap appears to be attributable to variation in how sessions were counted by providers, as well as lack of provider buy-in that completion of 48 sessions is necessary for program success or graduation.

c) While participants were required to complete only one level of Triple P to fidelity to complete MEPP, at no point during the Demonstration Project was there 100 percent compliance. There was, however, higher rates of completion for participants who started Triple P during the first eight weeks of MEPP.

d) Though providers in Regions One and Two consistently ensured that most participants received at least one random urine screen each week when they attended three sessions per week, providers in Region Three did not have the same success. Region Three providers consistently observed fewer drug screens over the course of the Demonstration Project.

3) Participants consistently moved through MEPP at a slower pace than the program originally predicted. The majority of clients who completed MEPP spent 17 to 34 weeks in the program. Providers reported making efforts to help parents be accountable and maintain attendance, but participants still


struggled with various situations, like illness, lack of transportation, incarceration, and conflict with court dates, visitation, which providers could not control to decrease time spent in the program.


Child Welfare Outcomes Key Questions The following sections describe the findings of the outcome component of the evaluation of MEPP. The outcome evaluation is designed to address the following nine research questions. Outcome Research Questions

To what extent does MEPP… 1. reduce repeat maltreatment?

2. increase the number of children who remain safely in the home?

3. increase rates of reunification?

4. increase timeliness of reunification?

5. improve the well-being and functioning of children?

6. improve the functioning and well-being of family members?

7. lead to improved parental perceptions of child welfare services?

8. reduce risky behaviors in families?

9. affect system, practice, and child and family level factors associated with the demonstration effectiveness and outcomes?

Sample Treatment group cohorts are composed of parents who enrolled in MEPP during a given six-month period beginning on April 1, 2016. Outcomes are reported where enough time has passed to measure an outcome, i.e., six-month outcomes are reported for Cohorts One, Two, Three, Four, and Five,3 and twelve-month outcomes are reported for Cohorts One through Four. Additionally, a combined cohort of all MEPP participants from Cohorts One through Five and Cohorts One through Four are reported to determine overall effectiveness of the program within six and 12 months, respectively. Where possible, outcomes for each of the cohorts are compared with those of a matched comparison group, with the results tested for statistical significance. For well-being data collected via the case reviews, data are available for Cohorts One, Two, Three, and Four.

3 Cohort Six is not included here as not enough time has passed to measure outcomes for this group.


For the treatment groups, the six- and twelve-month time periods for which most outcomes are measured start with the date of first participation in MEPP (based on the date participants began Matrix IOP), while the six-month period for the comparison group is measured based on a “pseudo-enrollment” date. The pseudo-enrollment date estimates a similar case start date as that of MEPP participants, who often entered the program after their OCFS case opened. To find the pseudo-enrollment date, the average number of days from case open date to the MEPP start date is calculated for each treatment cohort (190 days for Cohort One, 214 days for Cohort Two, 121 days for Cohort Three, 152 for Cohort Four, and 161 for Cohort Five). That number of days is then added to each comparison group’s original case open date. Data used for the outcome analyses come from the MEPP provider database and CANS, PAFAS, and DASS assessments, using data which were current as of May 22, 2019; data from MACWIS were extracted on May 1, 2019. Limitations As the real world rarely matches the controlled scientific design, methodological and logistical limitations were encountered throughout the project, as described below. Methodological Ideally, a Randomized Control Trial would have been used for the evaluation. However, due to the already-limited sample size and the desire to serve as many clients as possible, PCG utilized a quasi-experimental design with a historical comparison group to evaluate MEPP (see below for a complete description). While the matching criteria required each case in the comparison pool to have a substance abuse risk factor, the level of risk was not able to be determined, preventing the ability to identify if the comparison group clients satisfied the IOP level of service. Therefore, conclusions about outcome causality between the two groups cannot be definitive. Logistical As described in the process evaluation section, the demanding nature of MEPP resulted in a high participant dropout, especially within the first month of a client’s enrollment into the program. Because of this, only seven percent of the treatment group had both an initial and follow up CANS assessment, hindering the ability to draw definitive conclusions on improved well-being. Additionally, roughly a quarter of treatment group participants completed pre- and post-tests for the DASS and PAFAS assessments, therefore, only a sub-population of MEPP participants who remained in the program long enough to receive both assessments can be studied. Finally, the research question around reduction of risky behaviors was unable to be fully investigated due to challenges following up with clients after the completion of MEPP services.


Comparison Group A pool of comparison group members was selected from cases whose pseudo-enrollment date was between October 1, 2014 and September 30, 2015, i.e., for the year prior to the implementation of MEPP. Additionally, comparison group cases needed to contain a risk factor of parental drug or alcohol misuse, a child age five or younger in the household, and have remained open for at least one month after the pseudo-enrollment date. Once the pool of cases meeting these criteria was established, a unique comparison group was created for each treatment group member using the Propensity Score Matching technique. Propensity scores were constructed based on the following data elements for each case: district, number of children, number of adults, prior OCFS involvement, and placement type (e.g., residential, relative) of children ages zero to five at the MEPP start date or pseudo-enrollment date. A nearest neighbor algorithm was used to match comparison group pool members to treatment group members. See Appendix A for statistical similarities among matching criteria between treatment and comparison group cases. The comparison group for the combined treatment group is the combination of all comparison groups for each respective cohort.4 Table 14 shows the number of participants in the treatment and comparison groups for each six-month cohort as well as the percentage of those cases where all children aged 0–5 were in the home at referral. It should be noted that families were still referred to, and participated in, the program if their child(ren) had already been removed from the home and placed in care at the time of referral. Table 14. Measurable MEPP Cohort Family Count

Timeframe Number of MEPP Families

Percentage of MEPP Families

Referred In Home

Number of Comparison

Group Families

Percentage of Comparison

Group Families Referred In Home

Cohort 1 (4/1/2016 – 9/30/2016) 44 50% 44 55%

Cohort 2 (10/1/2016 – 3/31/2017) 46 52% 46 50%

Cohort 3 (4/1/2017 – 9/30/2017) 45 51% 45 56%

Cohort 4 (10/1/2017 – 3/31/2018) 70 44% 70 45%

Cohort 5 (4/1/2018 – (9/30/2018) 88 39% 88 42%

Cohort 6 (10/1/2018 – (12/31/2018) 30 23% 30 23%

4 It is possible one case could be selected for multiple comparison groups, however, the pseudo-enrollment date (and thus measurable timeframe) will be different for each instance of the case.


Q1. To what extent does MEPP reduce repeat maltreatment? One of the goals of MEPP is to prevent families from having future involvement with OCFS. PCG investigated repeat maltreatment by finding the number of cases with a new appropriate report occurring between one and six or twelve months after enrollment in MEPP (or in the case of the comparison group, after the pseudo-enrollment date). Updates to OCFS’ policy in May of 2018 added two new criteria by which a family could be considered appropriate for an assessment: (1) upon receipt of a third inappropriate report on a family, the family should be considered appropriate for an assessment; and (2) new reports of abuse or neglect on an already open case are to be treated as unique reports rather than being incorporated into the case without a new report being created. To consistently measure reports appropriate for assessment, if the third inappropriate report on the family occurred within the outcome timeframe but before the policy changed, the report was considered an appropriate report; however, the other new avenue for how a possible appropriate report is handled is not measurable for historical timeframes. This last criteria will systematically lower the number of appropriate reports qualifying for an assessment for the comparison group. Overall, the treatment group has a slightly lower percentage of cases without a new appropriate report with six months of referral to MEPP than the comparison group (Figure 4). The treatment groups in the first three and final reporting periods (i.e., Cohorts One, Two, Three, and Six) contain a slightly higher percentage of cases without a new appropriate report than the comparison groups; however, this trend reverses for Cohorts Four and Five. This large drop in the percentage of cases without a new appropriate report is likely due to the change in policy around the criteria of when reports should be deemed appropriate for assessment. Overall, only the six-month results for Cohort Five are statistically significant (p>0.05). Twelve months have elapsed for Cohorts One, Two, Three, and Four providing additional opportunity to measure the extent to which MEPP participants were involved in repeat maltreatment. Treatment group cases for Cohorts One, Two, and Three have a higher percentage of cases without appropriate reports than the comparison group. Similar to the six-month outcomes, Cohort Four shows a lower percentage of treatment group cases without a new appropriate report within 12 months of referral than the comparison group. The differences between treatment and comparison groups are not statistically significant (p>0.05). Therefore, it is inconclusive that participation in MEPP reduces repeat maltreatment for parents involved in the program.


Figure 4. Percent of Cases Without New Appropriate Reports at Six and Twelve Months

Q2. To what extent does MEPP increase the number of children who remain safely

in the home? Another aim of MEPP is increasing the number of children between the ages of zero and five who remain safely in the home, as defined by no removals and no new appropriate reports. In Maine, children of that age group make up the highest proportion of those who enter state custody. For this reason, this measure looks at families in which a child between the ages of zero and five was in the home at MEPP enrollment or the pseudo-enrollment date (for comparison group cases) and whether those children had a new appropriate report and/or had resulting removals following that report. Policy changes in May of 2018 no longer allow safety plan removal for placement of children with relatives not in OCFS custody. Safety planning is now limited to identifying the services and steps families must take to keep the child(ren) in the home. This will inevitably increase the number of removals reported for cohorts whose outcome timeframe is during or after implementation of the new policy (i.e., Cohort Three and later). Six-Month Outcomes Overall, among MEPP families with children ages zero to five in the home, slightly over half had children who remained in the home without any new appropriate reports within six months of enrollment (Figure 5). Comparison group cases, however, were nearly 15 percentage points more likely to keep a child safely in the home. Overall results are statically significant in favor of the comparison group (p<0.01); however, variation was observed within the cohorts. For example, differences between the treatment group and comparison group were only found to be significant for Cohort Three where the children of MEPP parents entered care over twice as often as comparison group children (p<0.05).

76%

61%

80%

68%

85%

70%

87%

69% 71%

51%

65%

80%79%

63%

75%

61%

74%

59%

82%

64%

81%

66%

80% 77%

6 Months 12 Months 6 Months 12 Months 6 Months 12 Months 6 Months 12 Months 6 Months 12 Months 6 Months 6 Months

Combined(N = 323)

Cohort 1(N = 44)

Cohort 2(N = 46)

Cohort 3(N = 45)

Cohort 4(N = 70)

Cohort 5(N = 88)

Cohort 6(N = 30)

Treatment Comparison


Figure 5. Percent of Cases with Children Age 0–5 Remaining in Home After Six Months

When examining removals, PCG identified whether a child was removed from the home following a subsequent report of maltreatment. In most cases, removals did not occur after a new report, which would likely include safety plan removals which allowed the child to be placed with another relative while remaining in custody of the parent. When removals did occur, the difference in the proportion of cases with a removal between the overall treatment and comparison groups is statistically significant (p<0.01), with the treatment group being more than twice as likely to have a child removed (13% vs 30%). Moreover, for Cohorts Two through Four children in the treatment group are twice as likely to be removed from the home; Cohort Five treatment group children are three times as likely to be removed from the home. On a more positive note, the percentage of treatment group cases with a removal has declined from Cohort Three to Cohort Four, and from Cohort Four to Cohort Five. Twelve-Month Outcomes Twelve-month safety outcomes for Cohorts One, Two, Three, and Four are shown in Figure 6. Overall, families in the treatment group are significantly less likely to have children ages zero to five remaining safely in the home than comparison group families (p<0.05). Moreover, the percentage of young children in the treatment group families remaining safely in the home decreased in each consecutive cohort from Cohort One to Cohort Three; improvement was observed from Cohort Three to Cohort Four.

57%71%

78%63%

66%46%

64%48%

70%46%

63%64%

68%54%

14%

19%28%

19%24%

20%4%

13%12%

21%14%

18%19%

29%

3%6%

9%16%

16%39%

9%35%

13%14%

10%20%

14%14%

3%

6%14%

9%

9%8%

4%9%

4%9%

Comparison (N = 7)Treatment (N = 7)







No Removal and No Report in 6 Months No Removal and Report in 6 MonthsRemoval and No Report in 6 Months Removal and Report in 6 Months

Overall

Cohort One

Cohort Two

Cohort Three

Cohort Four

Cohort Five

Cohort Six


Looking at removals both with and without a new report, twice as many treatment group families had a child removed in Cohort Three than Cohort One. This difference is likely due to the policy change on safety plan removals in March 2018 which increased the number of children in state custody. Overall, there are significantly more treatment group children aged zero to five who are removed from their home (with or without a new report) than comparison group children (p<0.01). Furthermore, Cohort Three treatment group children are significantly more likely to have been removed from their home (p<0.05). Differences between treatment and comparison groups for Cohorts One, Two, and Four are not statistically significant. Figure 6. Percent of Cases with Children Age 0–5 Remaining in Home After Twelve Months

Q3. To what extent does MEPP increase rates of reunification? For children who were removed, it is hoped that their stay in substitute care is held to a minimum while parents receive services from MEPP. Reunification is measured among those families with a child removed from his or her parents’ custody before parents were enrolled in MEPP who continued to be in out-of-home placement at start of MEPP as well as for cases with a child removed within 12 months after the parent began MEPP. For this measure, reunification is defined as children zero to five for whom parental rights were reinstated or custody was dismissed to the parent (i.e., parents received custody of the child). At six months, only a small percentage of children between the ages of zero and five in either the treatment or comparison groups were reunified (five percent for treatment groups and eight percent for comparison groups). Overall, among children in all cohorts, the difference in the six-month rate of reunification between the treatment and comparison groups is not statistically significant (p>0.05).

44%32%

44%35%

57%35%

46%55%

47%38%

38%32%

36%13%

26%23%

38%23%

35%24%

9%22%

16%35%

4%23%

13%9%

11%22%

9%14%

4%17%

13%19%

4%14%

8%16%






No Removal and No Report in 12 Months No Removal and Report in 12 MonthsRemoval and No Report in 12 Months Removal and Report in 12 Months

Overall Cohort One

Cohort Two

Cohort Three Cohort Four


At twelve months, there is an increase in the percentage of children reunified across both groups. As shown in Figure 7, 19 percent of the children in the overall treatment group and 22 percent of children in the overall comparison group were reunified with their families within 12 months. Children removed from clients who entered MEPP in the first year of the Waiver were more likely to be reunified than comparison group children. However, this trend does not hold for the third or fourth Cohorts. No results are significant. Figure 7. Percent of Children Age 0–5 Reunified Within 12 Months

When parents are making progress toward reunification, one of the initial steps is a trial home placement. Comparing trial home placements among MEPP participants and the comparison groups, data show that roughly one in four children involved in a MEPP case were placed in a trial home visit within 12 months of the removal or enrollment date. Of those, all were reunified within six months of the trial home placement. Children in the overall comparison group were over twice as likely to be placed in a trial placement within twelve months of enrollment; these results are statistically significant (p<0.01). Figure 8. Percent of Children Age 0–5 Enrolled in Trial Placement Within 12 Months of Removal

26%19%

30%17%

14%18%

20%26%

22%19%






52%

24%

0% 10% 20% 30% 40% 50% 60%

Comparison (N=52)

Treatment (N=79)

Overall Cohort One

Cohort Two

Cohort Three

Cohort Four


Q4. To what extent does MEPP increase timeliness of reunification? To determine whether the time to reunification with their children has decreased among MEPP participants, PCG calculated the average number of days to reunification among those cases with a child ages zero to five who was reunified within one year of the enrollment date or the removal date if the child was removed within one year of enrollment (Figure 9). Overall, the average number of days to reunification was significantly longer (roughly six weeks, on average) for the treatment group than it was for the comparison group (p<0.01). In particular, the average time to reunification for MEPP children in Cohort Three was over four months longer than comparison group children. MEPP children in Cohorts One, Two, and Four were reunified slightly later than comparison children. Figure 9. Average Number of Days Until Reunification

Regression Analysis (Q9) A stepwise linear regression analysis was performed on the 12-month outcomes examined above to determine which combination of factors examined in the process component of this report, such as presenting client characteristics, service utilization, and fidelity measures (e.g., number of Matrix sessions), are correlated with positive or negative outcomes. Table 15 displays the factors that the regression analysis found to impact at least one outcome measure. Factors in green are positive results (e.g., less likely to have a new report, more likely to reunify) and conversely for red factors. Darker shaded factors are significantly correlated (p < 0.05).

204226

163287

190194

231269

197243






Overall Cohort One Cohort Two Cohort Three Cohort Four


The small sample sizes available for these outcomes prevent any definitive causation of factors on the outcomes. A power analysis reveals that a sample size of roughly 125 participants would result in an 80 percent confidence in the findings, which is larger than the populations displayed below for all three measures, i.e., new reports (102), new removals (69) and reunification (80). Table 15. Factors Impacting Child Welfare Outcomes at Twelve Months

Outcomes

Factor:

New Report Twelve Months

(n = 102)

New Removal Twelve Months

(n = 69) Reunification

(n = 80) Region 2 Region 3 Removal Prior to MEPP Previous Substantiated Report Previous Indicated Report Previous Unfounded Report Drug of Choice: Benzodiazepines Drug of Choice: Cocaine Drug of Choice: Heroin Drug of Choice: Marijuana Drug of Choice: Methamphetamine Drug of Choice: Opiates Drug of Choice: Other Employed Full Time Received Other Substance Abuse / IOP Services Received Education / Employment / Medical / Anger Services

Received Basic Need / Support / Informal Support Services

PAFAS Parenting Score PAFAS Family Score DASS Depression Score DASS Stress Score Youngest Child Age Oldest Child Age Weeks In MEPP Number of Matrix Sessions Number of Triple P Sessions Completed MEPP to Fidelity

Significant findings of the analysis are described below.


• MEPP Completion. Clients who completed MEPP to fidelity are slightly more likely to have their children reunified within 12 months than those who did not.

• Depression, Anxiety, and Stress. Clients with more depressive symptoms, as measured by the DASS, are slightly more likely to not have a new report of maltreatment within 12 months. Conversely, clients with more stress symptoms are less likely to not have new report of maltreatment.

• Parenting Skills. Clients with higher PAFAS family scores (meaning parents show fewer positive family practices) are significantly more likely to not have a new maltreatment reports within 12 months than clients with lower scores.

• Number of Matrix Sessions. Clients attending more Matrix IOP sessions are significantly more likely to have their child reunified within 12 months.

• Number of Triple P Sessions. Clients attending more Triple P sessions are more likely to have no new maltreatment report within 12 months than clients who attend fewer sessions, but they are less likely to have their child reunified within 12 months.

• Receipt of Services. Clients receiving basic need services, or supportive (e.g., mentor, AA) or informal (e.g., church, family) supports, were less likely to have a child removed within 12 months and more likely to have their child reunified within 12 months. Clients receiving education, employment, medical, or anger management services were more likely to have a child reunified within 12 months. Clients receiving an IOP other than Matrix were more likely to have a child removed from their home.

• Drug of Choice. Clients were less likely to have no new maltreatment report for all drugs of choice except marijuana. Conversely, clients whose drug of choice was benzodiazepines, heroin, or opioids were significantly more likely to reunify with their children than clients whose drug of choice was marijuana or methamphetamine.

• Previous Reports. Clients with a previous report (substantiated, indicated, or unfounded) are more likely to have a child removed than those without a previous report. Clients with a previous substantiated or unsubstantiated report are slightly more likely to reunify with their children within 12 months of the removal/MEPP enrollment date.

• Region. Clients from Region 2 are more likely to reunify with their children within 12 months of the removal/MEPP enrollment date.

Q5. To what extent does MEPP improve the well-being and functioning of children? Two data sources, CANS assessments and case record reviews, are used to assess improvements in child well-being over time. Data from the CANS assessment are used to assess both the needs and strengths of families in the treatment group. PCG collected and analyzed available CANS data from two assessment periods; the initial assessment


yielded 172 cases with a completed CANS and a follow-up assessment yielded 45 cases with a completed assessment. CANS and Case Record Reviews CANS assessments are broken into domains and items. There are seven domains, with a varying number of items within each domain: Caregiver (57 items); Child Strengths (7 items); Child Behavioral and Emotional Needs (11 items); Life Domain Functioning (13 items); Medical (8 items); Regulatory (3 items); and Risk Factors (9 items). CANS assessments are scored on a scale from zero to three, with zero meaning there is no need associated with the item and scores of two and three representing items with actionable needs. Figure 10 shows the percentage of items receiving a score of zero to three in each CANS domain for the initial and follow-up assessments. The Caregiver domain showed the greatest reduction of harm, rising eight percentage points in clients who scored a zero or one. Child Strengths scores of zero reduced by seven percentage points but scores of one increased by four percentage points. Figure 10. Comparison of CANS Item Score Percentages from Initial to Follow-Up CANS

64%

54%

71%

64%

77%

79%

80%

75%

71%

71%

23%

25%

21%

25%

21%

17%

14%

16%

16%

14%

10%

17%

6%

10%

3%

4%

6%

8%

10%

9%

3%

5%

1%

1%

1%

3%

5%

Follow Up (n=2261)

Initial (n=2252)

Follow Up (n=294)

Initial (n=295)

Follow Up (n=462)

Initial (n=463)

Follow Up (n=546)

Initial (n=547)

Follow Up (n=378)

Initial (n=379)

Care

give

rCh

ildSt

reng

ths

Child

Beh

avio

rLi

feFu

nctio

ning

Child

Risk

Fact

or

Score 0 Score 1 Score 2 Score 3


Of the seven domains, PCG investigated which areas overall showed growth (green) and which showed a decline (red) (Figure 1). PCG measured improvement by calculating the average percent of change of all actionable items within each domain from the initial CANS assessment to the follow-up assessment. The number of items reported for each domain in Figure 11 are those where at least one child contained an actionable need. Overall, domains that showed strong improvement and growth are Regulatory (70%), Child Behavioral Emotional Needs (73%), and Medical (75%). Other domains that showed improvement, but to a lesser extent, are Caregiver (17%), Life Domain Functioning (35%), and Child Strengths (50%). The only domain that exhibited a decline was Risk Factors (4%). Of the seven items in the Risk Factors Domain, four showed a decline and three showed growth; the item which showed the most improvement was Abuse and Neglect (n=23; 65 percent). Figure 11. CANS Domain Changes between First and Second Assessment

Figure 12 depicts the rate of positive change and negative change for each domain based on item changes within each domain. Of the seven domains, only three had items that experienced a decline in the follow-up CANS assessment. The Caregiver domain experienced decline in 12 of its 54 items (22%); Life Domain Functioning had four of 12 items experience decline (33%); and, the Risk Factors domain had four out of seven items experience a decline (57%). Child Behavioral Emotional Needs, Child Strengths, Regulatory, and Medical domains had no decline for their actionable items.

75%

73%

70%

50%

35%

17%

-4%

-10% 0% 10% 20% 30% 40% 50% 60% 70% 80%

Medical (n=2)

Child Behavioral Emotional Needs (n=9)

Regulatory (n=2)

Child Strengths (n=5)

Life Domain Functioning (n=12)

Caregiver (n=54)

Risk Factors (n=7)


Figure 12. CANS Domain Rates of Positive and Negative Change by Item

Overall, the items which have the highest rates of action needed on the follow-up CANS assessment are Substance Abuse (n=60), Collaboration with Other Parents/Caregivers (n=47), Financial Status (n=43), Caregiver Support for Permanency Plan Goal (n=42), and Involvement with Care of the Child (n=42). The percent change on actionability for each of these domain items varied (Figures 13 and 14). Substance Abuse improved by 40 percent; Collaboration with Other Parents/Caregivers improved by six percent; Financial Status improved by 19 percent; while Caregiver Support for Permanency Plan Goal worsened by seven percent and Involvement with Care of the Child worsened by five percent.

2

2

5

3

9

8

39

4

4

12

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Medical (n=2)

Regulatory (n=2)

Child Strengths (n=5)

Risk Factors (n=7)

Child Behavioral Emotional Needs (n=9)

Life Domain Functioning (n=12)

Caregiver (n=54)

Positive Change Negative Change


Figure 13. CANS Caregiver Domain Item Changes (n>24)

Figure 14. CANS All Other Domain Item Changes (n>10)

The domain items which displayed the largest relative improvement are Discipline (4 to 0), Nutrition Management (4 to 0), Curiosity (3 to 0), Failure to Thrive (3 to 0), and Oppositional (2 to 0). All five of these items had a rate change of 100 percent. However,

48%28%

-4%-24%

37%24%

28%24%

14%32%

13%32%

-5%-7%

19%-2%

6%40%

Community Involvement (n=25)History of Maltreatment of Children (n=25)

Condition of Home (n=25)Legal (n=25)

Mental Health (n=27)Resources (n=29)

Organizational Skills (n=32)Job Functioning (Employment/Education) (n=34)

Transportation (n=35)Understanding Impact of Own Behavior on Children (n=38)

Relationship with Abuser(s) (n=39)Responsibility in Maltreatment (n=41)

Involvement with Care of the Child (n=42)Caregiver Support for Permanency Plan Goal (n=42)

Financial Status (n=43)Inclusion of the Child in the Foster Family (n=43)

Collaboration with Other Parent/Caregivers (n=47)Substance Abuse (n=60)

46%

14%

79%

73%

39%

-11%

-10%

65%

10%

Family (n=13)

Extended Family (n=14)

Adjustment to Trauma (=19)

Family (n=11)

Relationship Permanence (n=23)

Maternal availability (n=19)

Prenatal Care (n=21)

Abuse/Neglect (n=23)

Substance exposure (n=39)Risk Factors

Life Domain Functioning

Child Behavior Emotional Needs

Child Strengths


the number of children that were characterized as having a need in those areas was too small to make any definitive claims. Domain items which displayed the largest lack of improvement are Labor/Delivery (5 to 10), Developmental (1 to 2), and Physical Health (1 to 10). Again, the number of children characterized with a need within these item is small. Data from the case record review (e.g., changes in physical health, mental health and education) were used to examine changes in child well-being and functioning among both treatment and comparison groups. The reviews provide a broader look at children’s health, mental health, and education over a full 12-month period of OCFS’ involvement in a family’s case and parents’ engagement in services; case review results are available for Cohorts One, Two, Three, and Four. Differences in children’s health and health care, mental/behavioral issues, or education in all cohorts were minimal for both the treatment and comparison groups. As shown in Figures 15, 16 and 17, the majority of children in both the treatment and comparison groups experienced improvement in all three areas, although overall, a greater proportion of treatment group cases had children with improvements in mental health and education. Conversely, a greater proportion of comparison group cases experienced improvements in health for Cohorts One, Two, and Three, but this trend is reversed for Cohort Four, where both groups are equal. There is little information reported on children’s educational performance due to their age. Overall, there are not significant differences between the treatment group and the comparison group regarding improvements in health, mental health, or education (p>0.05). Figure 15. Percent of Cases with Children with Improvement in Health by Domain Based on Case

Review

97%97%

97%96%

100%96%

100%93%

98%96%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Comparison (n = 29)Treatment (n = 32)





Overall Cohort One Cohort Two Cohort Three Cohort Four


Figure 16. Percent of Cases with Children with Improvement in Mental Health by Domain Based on Case Review

Figure 17. Percent of Cases with Children with Improvement in Education by Domain Based on Case Review

Less than 10 percent of cases in both treatment and comparison groups had children of school age and recorded information regarding their educational progress. Improvements in education included improvements in attendance and grades.

57%72%

79%61%

65%90%

52%70%

62%73%






60%57%

80%57%

86%100%

58%78%

69%74%






Overall

Cohort One Cohort Two

Cohort Three Cohort Four

Overall

Cohort One Cohort Two Cohort Three Cohort Four


Q6. To what extent does MEPP improve the functioning and well-being of family members?

Improved functioning and well-being of family members are assessed through three measures: parenting skills, using the PAFAS and parent survey; parental mental health using the DASS assessment; and overall well-being, assessed via the case record review. While the PAFAS, DASS, and parent survey are administered only to MEPP participants, the case record review enables evaluators to compare changes for both treatment and comparison groups in overall well-being. Parenting Skills The PAFAS asks parents about parenting behaviors associated with risk factors of child emotional and behavioral problems (e.g., coercive parenting, lack of parental adjustment). These behaviors are categorized into subdomains, and the changes in average subdomain scores are analyzed to determine if parents’ behaviors change as a result of participation in MEPP. A total of 120 clients completed both an initial and follow-up PAFAS assessment: 17 from Cohort One, 20 from Cohort Two, 15 from Cohort Three, 20 from Cohort Four, 35 from Cohort Five, and 13 from Cohort 6. Among all participants with initial and follow-up assessments, average domain scores were lower on follow-up assessments in all domains except Parental Teamwork, indicating some risk behaviors were reduced while the participant was enrolled (Table 16). In the Parenting Practice, Parent Adjustment, and Family Relationships domains, scores decreased significantly from initial to follow-up, indicating that surveyed MEPP participants reported increasing their use of positive parenting practices. Table 16. Average Initial and Follow-up Scores on Parent Practice Scales of MEPP Participants

Domain Range5

All Respondents (n=120)

Average initial score Average follow-up score Parenting practice 0–54 9.6 7.0*

Parent adjustment 0–15 6.6 4.6*

Family relationships 0–12 3.3 2.6*

Parental teamwork (N = 60) 0–9 2.9 2.8 *Indicates a statistically significant decrease in scores, a positive result (p<0.05).

5 Lower scores indicate better parenting skills and fewer high-risk parenting behaviors.


Within the Parenting Practice domain, Parenting Consistency followed by Parent Adjustment appeared to be the areas in which parents are rating themselves more highly following MEPP involvement. Questions on the assessment tool regarding consistency are related to following through on consequences. Questions regarding parent adjustment focus on how often parents feel stressed, happy, sad, and satisfied in their life, as well as how they cope with the demands of parenting. Case record reviews are used to collect information on parents’ use of punitive parenting practices; however, in both the treatment and comparison group cases such practices were not an issue for most parents. This is confirmed by the PAFAS, where scores on the Punitive Parenting subdomain have been low throughout the evaluation period. Parental Mental Health As with parenting skills, the evaluation collects information on the mental health and stress of parents from different sources: case record reviews and DASS assessments. Looking first at the case record review data available for the first four cohorts, information about changes in parental mental health and stress were not reported in a number of case records for both treatment and comparison group families. Among those parents for whom information was available, roughly the same number of treatment and comparison group cases experienced improved parental mental health and stress. Evidence used for this evaluation component included case record notes input by caseworkers on the parents’ well-being, family team meeting narratives (in which clinicians, family members, and other involved parties reported on a family’s progress), and parents’ self-reports. Building on the case review data, to gain more information about MEPP parents’ well-being specifically, the DASS is used to assess depression, anxiety, and stress among MEPP participants. DASS assesses the core symptoms of depression, anxiety, and stress experienced by clients in the past week. Based on the responses, a client’s symptoms can be categorized as Normal, Mild, Moderate, Severe, or Extremely Severe, and average scores are compared to assess changes in symptoms over time. Initial and follow-up DASS assessments were completed by 120 MEPP participants. Figure 18 displays the percentage of parents with normal, mild, moderate, severe, or extremely severe scores for each of the DASS domains on the initial and follow-up assessments. Initial scores show that anxiety symptoms were most common at the time of the initial assessment and nearly half of the participants had at least mild symptoms for Depression and Anxiety. The Anxiety domain showed the largest percentage point improvement, with 71 percent at follow-up reporting in the normal range, compared to 47 percent reporting in the normal range for the initial assessment. Additionally, in the Anxiety domain, a smaller percentage of parents reported extremely severe symptoms at follow-up, with more clients reporting in the normal range or at a lower severity level. Significant improvements were seen in the Depression and Stress domains as well, with


76 percent and 81 percent reporting in the normal range, comparatively (p<.05), at follow-up. Figure 18. Initial and Follow-up DASS Scores by Subdomain (n=120)

Comparison of the three domains showed that although average scores are fairly low, there was a significant decrease in scores in all three domains (Table 17). This indicates that based on client self-reports symptoms of depression, anxiety, and stress decreased during the course of MEPP involvement. In addition, 95 percent of parent survey respondents reported that they were better able to cope with their negative emotions, which will help with symptoms of stress, anxiety and depression when they do occur. Table 17. Average DASS Scores by Subdomain among MEPP Participants

DOMAIN Range6 All Respondents (n=81)

Average initial score Average follow-up score

Depression 0–42 10.7 5.6*

Anxiety 0–42 10.4 6.2*

Stress 0–42 13.9 9.0* *Statistically significant decrease in scores (p<0.05)

6 Lower scores indicate better parenting skills and fewer high-risk parenting behaviors.

81%

61%

71%

47%

76%

53%

11%

17%

4%

8%

13%

15%

4%

11%

13%

18%

6%

18%

4%

9%

4%

8%

4%

11%

4%

7%

18%

1%

4%

Follow up

Initial

Follow up

Initial

Follow up

Initial

Normal Mild Moderate Severe Extremely Severe

SUBDOMAIN Depression Anxiety Stress


Overall Functioning and Well-Being Parents in the treatment group appear to have experienced improvements in their overall well-being, based on the PAFAS and DASS scores and the case record reviews. Responses to the parent surveys support this conclusion, with an overwhelmingly positive response by parents about their confidence in handling their children’s needs, their ability to cope with their children’s behaviors, and their home life being more stable. Figure 19 displays the results of the survey administered to MEPP parents. Figure 19. MEPP Participant Agreement with Parenting Statements on Parenting Survey (n=24)

Q7. To what extent does MEPP lead to improved parental perceptions of child

welfare services? Parental perceptions of the child welfare system are measured using the parent survey and feedback collected during the interviews with MEPP provider and OCFS staff. The participants who completed the survey have very positive perceptions of their MEPP providers. As shown in Figure 20, nearly all reporting participants felt as though providers treat them with respect, and most felt that the staff members do not talk to them in ways that seem accusatory or blaming. Similarly, nearly all participants felt as though the staff both understand and try to help them address their needs. Of all the participants who completed the survey, all but one agreed or strongly agreed that providers are sensitive toward their cultural and religious beliefs. Participant perceptions of OCFS workers were also positive, with nearly all reporting that they felt the workers treat them with courtesy and respect, provide assistance with staying sober, and are sensitive to their cultural and religious beliefs. This is somewhat contrary to the focus group data that were collected. Focus groups generally reported that caseworker support was either hit or miss. Participants expressed that they experienced

4%

4%

4%

4%

4%

4%

4% 35%

30%

38%

35%

29%

26%

57%

65%

58%

61%

67%

70%

I feel more supported by my extended familyor community.

Our home life is more stable.

I learned what to expect and not to expectfrom my children at this age.

I am better able to cope with my children’s negative behavior.

I feel more confident about my ability to handle my children’s physical needs.

I feel more confident about my ability to handle my children’s emotional needs.

Strongly Disagree Disagree Agree Strongly Agree


higher levels of satisfaction with MEPP providers than caseworkers. Additionally, because very few parent surveys were completed, it may be that parents who did complete the survey were more likely to actively participate in the program or engage with services altogether, which may in turn have improved relations with caseworkers. Figure 20. Parent Perceptions of Staff Attitudes (n=24)

In addition to attitudes toward staff, the parent survey assessed MEPP’s responsiveness to parents’ needs by asking them whether they received the services that they wanted, as well as desired those services they received. Table 18 shows that of the four areas evaluated, three areas of want and receive match for at least half of the participants. That is, participants who wanted help knowing how to take better care of their children, getting off drugs and/or alcohol, and staying sober received those services. The area with the smallest proportion of success was helping participants to connect with extended family. Receiving informal supports like this from family were shown in the regression analysis to, in part, be correlated with improved outcomes. The overall findings show that most participants are receiving the assistance that MEPP is designed to provide, yet there are some participants who do not desire the support they are being offered.

4%

4%

4%

4%

4%

4%

4%

4%

8%

38%

38%

33%

25%

21%

30%

22%

17%

54%

58%

63%

71%

67%

70%

74%

79%

Treated me with courtousy and respect.

Provided the help that I needed to stay sober and carefor my children.

Was sensitive to my cultural and religious beliefs.

Talked to me in a way that was not blaming.

Were not confrontational while helping me.

Were sensitive to my cultural and religious beliefs.

Understood what I needed and tried to help me get it.

Treated me with respect.

Strongly Disagree Disagree Agree Strongly Agree

OCFS Caseworker Service Provider

The overall findings show that most participants are receiving the assistance

that MEPP is designed to provide, yet there are some participants who

do not desire the support they are being offered.


Table 18. MEPP Responsivity to Need (n=24)

Help knowing how to take care of my children better Received Not Received

Wanted 50% 4% Not wanted 38% 8%

Help getting off of drugs and/or alcohol Received Not Received


Help staying sober from drugs and/or alcohol Received Not Received


Help connecting with extended family who can assist me and my family Received Not Received


Outcome Evaluation Conclusions The intent of MEPP was twofold: to keep children safely in the home, if possible, and to increase reunification rates when removal was necessary. The analysis yielded inconclusive results regarding whether participation in MEPP reduced repeat maltreatment for children. Specifically, the treatment group has a slightly lower percentage of cases without a new appropriate report at six months than the comparison group. However, the differences between the treatment and comparison groups are not statistically significant (p>0.05). However, there were some specific positive outcomes at 12 months. Clients attending more Matrix IOP sessions were significantly more likely to have their child reunified and those attending more Triple P sessions were more likely to have no subsequent reports of maltreatment. Unfortunately, clients who attended more Triple P sessions were also less likely to have their child reunified in 12 months. Though not explored, this could be due to the complexity of cases and circumstances requiring parents to receive higher levels or extended Triple P treatment. Additionally, the analysis uncovered that significantly (p<0.05) more children in the treatment group were removed from their home within six and twelve months of enrollment into MEPP than comparison group children. Comparison group cases were nearly 20 percentage points more likely to keep a child in the home than MEPP cases. Furthermore, comparison group children were slightly more likely to reunify with their parents. MEPP participants also had significantly longer reunification times compared to the comparison group—six weeks longer on average (p < 0.01).


Treatment group participants showed improvement in well-being and functioning for both children and adult participants. Children showed improvement in most CANS domain items. Furthermore, improvements were evidenced for the items with a high number of children identifying actionable needs from the initial CANS to the follow up CANS. Specifically, substance abuse improved by 40 percent, collaboration with other caregivers improved by six percent, financial status improved by 19 percent, abuse/neglect improved by 65 percent, and substance exposure improved by 10 percent. Case record reviews showed a greater proportion of children in the treatment group with improvements in mental health and education than comparison group children, though results were not significant (p>0.05). Additionally, the average domain scores for the PAFAS parenting practice, parent adjustment, and family relationships domains were significantly improved on the follow-up assessments. The DASS assessment showed anxiety factors in parents displayed the largest percentage point improvement (47 percent in the normal range at the initial assessment vs. 71 percent in the follow-up assessment). Significant improvements were also reported for depression (23 percentage points) and stress (20 percentage points). A regression analysis was conducted to determine which system, practice, and child and family level factors were associated with the Waiver Demonstration’s effectiveness and improved outcomes. The analysis showed that MEPP participants who receive supports for basic needs or informal supports are more likely to have positive outcomes than clients not receiving those services. Conversely, those clients who reported their drug of choice was methamphetamine were less likely to have positive outcomes than clients with other drugs of choice. Other factors included in the analysis showed mixed results on their overall correlation with outcomes.


Cost Outcomes The cost evaluation is designed to address the following two research questions. Fiscal/Cost Questions

1. Are the costs of all services, including days spent in foster care, provided to a family through the child welfare system less than those of all services provided to similar families when the latter do not receive Waiver services (service use and cost)?

2. How cost effective is the Demonstration Project compared to the current method of delivering services?

The cost analysis compares maintenance costs for children placed out of the home and services provided to clients and children in MEPP cases to those in the comparison group. The analysis that follows breaks out costs for parents and children between the ages of zero and five. Table 19 shows the total costs incurred for members of the treatment and comparison groups. Average costs per case were higher among treatment groups in all cohorts than the comparison group cases, driven largely by costs for contracted MEPP services.

• Room and board costs include the costs paid to placement providers (including both licensed and unlicensed foster care) to provide a place for a child to live, including meals within one year of enrollment (or pseudo-enrollment). Data were taken from the payment table in MACWIS. In all cohorts except Cohort Four, the costs were lower among treatment cases than comparison cases.

• Other maintenance payments include diaper allowances, as well as allowances for clothing and personal spending, with the highest costs for both treatment and comparison groups incurred for clothing. Apart from Cohort Three, payments incurred for other maintenance items were lower for children in the treatment groups.

• Services include payment for a wide variety of fee-for-services. For children, they include childcare expenditures (the highest single expense for all groups), legal services, and education. For a small number of parents (one to two in each group), residential or halfway house services were also provided. Apart from Cohort Three, total costs for services tended to be lower for clients in the treatment groups than for those in the comparison groups.

• MaineCare payments include costs for contracted medical services and include payment for assessment and treatment services, including those to address a family member’s public health concerns. The amount reported is the total amount spent on the service including both federal and state shares.


Table 19. Total Costs of Services to Treatment and Comparison Groups (both Children and Parents)

Room and Board

Other Maintenance

Payments Services MaineCare Payments

MEPP Contracted

Services

Average Cost for

Case Overall

Treatment (n=205) $705,100.33 $116,782.73 $536,906.41 $3,765,118.18 $1,987,903.07 $34,691.76 Comparison (n=205) $815,106.44 $147,013.95 $527,467.53 $1,551,679.82 — $14.835.45

Cohort One Treatment (n=44) $131,071.53 $17,291.12 $94,694.39 $808,122.93 $268,167.09 $29,583.66

Comparison (n=44) $186,975.80 $25,477.73 $126,604.28 $333,043.47 — $15,015.18 Cohort Two

Treatment (n=46) $174,147.87 $23,789.18 $99,806.84 $844,855.79 $368,294.80 $32,845.53 Comparison (n=46) $191.644.04 $28,126.59 $147,830.68 $348,181.81 — $15,560.50

Cohort Three Treatment (n=45) $100,503.53 $29,930.19 $132,826.30 $826,489.36 $540,080.26 $36,218.44

Comparison (n=45) $159,687.64 $28,508.99 $97,404.10 $340,612.64 — $13,915.85 Cohort Four

Treatment (n=70) $299,377.40 $45,772.24 $227,245.03 $1,285,650.11 $811,360.92 $38,134.37 Comparison (n=70) $276,798.96 $64,900.64 $167,061.64 $529,841.89 — $14,837.19

Table 20 looks more closely at the types of contracted services provided to parents and children zero to five other than MEPP and excluding MaineCare services. Fewer dollars were spent on participants in the treatment group for parenting and substance abuse programs other than Matrix IOP and Triple P when matched to participants in the comparison group. However, MEPP participants spent more on drug and alcohol testing and medical care than those in the comparison group. Table 20. Costs for Services to Treatment and Comparison Groups

Service

Treatment Group Comparison Group Number of

Parents and Children

Number of Cases

Amount Spent

Number of Parents and

Children Number of Cases

Amount Spent

Basic Needs 107 76 $34,347.69 84 63 $42,815.26 Child Care 132 100 $195,740.28 107 86 $205,301.05 Drug & Alcohol Testing 174 123 $53,345.15 147 108 $28,943.61 Maintenance 96 71 $10,352.45 80 61 $7,107.88 Medical 58 54 $61,849.23 26 22 $22,716.04 Other 24 24 $8,535.66 42 35 $20,051.95 Other Parenting Programs 1 1 $2,914.16 15 13 $33,977.77 Other Substance Abuse Programs 16 16 $72,727.89 8 5 $82,389.91 Therapy 8 8 $6,758.45 5 5 $7,479.96 Travel 166 106 $90,335.45 162 99 $76,684.10

TOTAL 401 178 $536,906.41 347 160 $527,467.53


Payment for MaineCare services are broken out by service subcategory7 for children ages zero to five and parents in the treatment and comparison groups in Table 21. For both parents and children, the highest costs were incurred for the provision of mental health and counseling services followed by basic medical costs. The largest difference between the treatment and comparison groups is shown for the parents where MEPP adults have roughly five times more MaineCare service costs than comparison group adults. In particular, the costs for drug/alcohol treatment (either IOP or non-IOP) are roughly a factor of 10 times higher for MEPP clients. This further highlights the statements in the outcomes section that the comparison group may not be robust enough to make definitive claims about MEPP. Table 21. MaineCare Payments to Treatment and Comparison Group Participants

Children Ages 0 - 5 Parents

Service Treatment Group

Costs Comparison Group Costs

Treatment Group Costs

Comparison Group Costs

Basic medical $524,291.78 $318,383.26 $404,193.54 $40,011.26 Serious medical $165,932.22 $69,925.61 $206,524.41 $27,507.67 Medical devices / supplies $22,665.57 $17,610.57 $11,426.32 $754.41 Dental $33,806.38 $17,426.94 $8,295.00 $906.50 Physical testing / assessment $67,162.87 $48,861.34 $146,584.98 $19,863.10 Mental health testing / assessment $38,048.25 $28,600.83 $64,516.14 $31,888.24 Drug testing $23.40 $0.00 $199,716.79 $9,970.51 Drug/Alcohol treatment (IOP) $0.00 $0.00 $351,590.98 $16,100.00 Drug/Alcohol treatment (other/non-IOP) $0.00 $0.00 $338,513.38 $56,321.07 Mental health/counseling services $604,318.84 $569,884.83 $577,507.33 $277,663.68

TOTAL $1,456,249.31 $1,070,693.38 $2,308,868.87 $480,986.44 The cost evaluation shows higher costs were incurred overall for MEPP participants compared to cases involving delivery of traditional services. While room and board costs are generally lower for children of MEPP clients than comparison group children, this difference is not enough to make up for the amount paid for MaineCare payments. In total, MEPP has not resulted in a cost savings and cost roughly $4 million more to deliver to families when matched to comparison group families and children.

7 Subcategories were determined by PCG attributing MaineCare procedures and procedure groups to the most applicable subcategory.


Recommendations There is a striking difference between the success in achieving “system outcomes” in this study and the success in achieving “well-being outcomes.” The former are driven by the entire child welfare system. Decisions about outcomes such as reunification and safety are made by caseworkers, administrators and judges. During the course of the Waiver, new policies were promulgated, triggered by factors outside of MEPP that still impacted the outcomes. Perhaps due in part to such external factors, MEPP did not produce many positive changes in the system outcomes. On the other hand, the parental reports of improvements in their own stress, anxiety, depression and ability to manage their children generally indicate that the well-being results did improve. Those who engaged and completed the service made strong gains in their own well-being in addition to the rest of their families. These results prompt one to consider what about the overall MEPP approach is worth salvaging and what is not. The cost differences are large and to compensate for these added costs, the issues detracting from more positive system outcomes need to be addressed more directly. The need is underscored by the new federal Family First legislation which will incentivize states to provide drug treatment and parenting programs to child welfare families similar to MEPP. The issues outlined below were found in the evaluation to impede drug treatment programs from working effectively with child welfare clients. These issues will need to be addressed by human services and OCFS policymakers and other stakeholders including attorneys, judges, the treatment community and perhaps legislators. The first four are multi-system issues while the last two help explain why the child welfare system outcomes are not strong for MEPP clients.

1) Child welfare agencies, service providers, judges and attorneys do not agree on the philosophy of drug treatment; that is, whether it should lead to “harm reduction” or abstinence. What may be considered success by one party is not embraced by the others.

2) There is no consensus on the place of medication-assisted treatment and marijuana use, both medical and recreational. Does the inclusion of marijuana without specific child safety issues expand the reach of child welfare agencies beyond initial intention?

3) Judges may or may not value completion of a drug treatment program as a criterion for returning a child home, especially if other elements of the case plan are not met.

4) Parents’ attorneys often counsel against being evaluated for a level of care which is a requirement of intensive outpatient treatment since having results of a high treatment need makes for a stronger case against the parents. They also often do not approve of clients entering treatment programs.


5) Before implementing a child welfare “system change” such as reunification, caseworkers want to see a family’s compliance with all elements of a case plan, not just completion of drug treatment. This often includes having a job and stable housing. Parents, on the other hand, often see completing the program as the goal and are surprised when their children are not returned once the goal is met.

6) Intensive drug treatment programs can require three visits a week, generally during working hours, and the addition of parenting classes can make it difficult to find and keep a job. Therefore, it will take longer to complete the treatment plan and achieve an outcome such as reunification.

Despite different approaches and philosophies across providers, nearly all interviewees (caseworkers and providers) expressed sensitivity for participants and a desire to see this program succeed. Stakeholders were genuinely concerned about barriers and expressed commitment to finding solutions, and participants were able to verbalize several personal benefits they have received from the program. However, substance use recovery is a complicated issue. Analysis of the project outcomes, stakeholder interviews, focus groups, and staff surveys yielded several recommendations for future opportunities to influence adoption and spread of model enhancements. They are described below. Launch a Multi-disciplinary Process to Unify and Clarify Policy about Marijuana, Medication Assisted Treatment and Harm Reduction Strategies Use an existing mechanism such as the Court Improvement Program or Children’s Cabinet to address the issues raised here about the role of marijuana use and medication-assisted treatments in child welfare cases. These issues include the goals of drug treatment (i.e., harm reduction or abstinence) that will be acceptable to child welfare and the courts; what, if anything, is considered acceptable marijuana use in child protection; and what is the role of medication-assisted treatment. OCFS should translate the results into policy and practice guides for caseworkers. The guiding principle in modifying and clarifying policy should be the achievement of safety and permanency for the child. For instance, the selection of Matrix IOP as a key component of the project suggests that a policy or practice requiring total abstinence is not appropriate, because that is not what Matrix IOP seeks to achieve. In addition, case plan provisions that require parents to get and maintain jobs do not necessarily enhance the children’s safety and may be counterproductive, given the difficulties parents have in participating in the services and simultaneously holding down jobs. In designing new policies and practice, each proposal needs to be tested against the questions of the degree to which it is necessary for the safety and permanency of children and how it contributes to those outcomes.


Determine how Medicaid can Help Financially to Support Treatment and Parenting Programs for Child Welfare Families including Expansion of Treatment Options Medicaid has the potential to provide federal matching funds for evidence-based treatment and parenting programs. One of the major sticking points to the continuation of MEPP programming is funding. Parenting education in Maine is currently viewed as a supportive service; thus, it is not reimbursable via traditional Medicaid. However, parenting education is often recommended as part of permanency planning. It is understood that OCFS is already working closely with the Maine Legislature and MaineCare office to explore opportunities to expand Medicaid to cover Triple P. At the same time, other evidence-based parenting programs should be considered, particularly for parents of younger children or those whose children have been removed. Providers and OCFS staff recognized the importance of participants practicing parenting skills with their children. While a large percentage of clients already had their children removed from the home at the time of program enrollment, the idea of incorporating parenting education into visitation arose several times in stakeholder interviews. Generally, it was expressed that this method of parenting education could be more effective than traditional classroom style learning. Providers and caseworkers suggested that having hands-on practice to demonstrate skills and work through issues with a parenting clinician present would be useful not only to increase parent education but also to document concrete progress for permanency planning. This time could also be used to address specific issues OCFS staff identified during the investigation or when working with the families. Stakeholders suggested that this hands-on parenting education could be completed during supervised visitation or arranged playgroups. Though Matrix IOP is already a covered Medicaid service, providers report billing issues due to the length of the program. While many substance use programs are four weeks in duration, the fact that Matrix IOP is four times as long can sometimes cause issues with reimbursement, requiring extra justification and more frequent reauthorization of services. Therefore, OCFS should continue to work closely with MaineCare to remove obstacles for billing. One of the features of MEPP that made this program inherently more valuable than traditional substance use and parenting referrals was that Matrix and Triple P clinicians were reimbursed for time spent collaborating. Traditionally, behavioral health services are strictly fee-for-service; time spent discussing strategies with other clinicians, in attendance at client meetings, or returning phone calls or emails is not reimbursable. In one region with superior collaboration the clients experienced more success. Future programming should strategize a way to fund or reimburse provider collaboration as a mechanism to improve participant outcomes. Increase Communication Mechanisms Between Caseworkers and Providers Communication was a common theme throughout the interviews, focus groups, committee meetings, and staff surveys. Providers reported a desire to have faster delivery


of collateral information (including relevant history and case goals) from OCFS and more timely responses to phone calls and emails, whereas OCFS staff emphasized the importance of frequent, transparent updates from all providers about client progress and setbacks with recovery. All parties generally agreed that provider involvement in Family Team Meetings was helpful for permanency planning. Additionally, while collaboration between OCFS and providers for aftercare or discharge planning did not seem to be a common occurrence, both parties recognized that better coordination in this area would be helpful in supporting sustained recovery and setting clear expectations for clients. A mechanism for increasing communication was an effective MEPP champion. Each region designated MEPP liaisons and coordinators for providers and OCFS offices. However, the degree of their engagement ranged widely. Generally, regions with strong MEPP champions appeared to have better integration of programs, better communication between OCFS and providers, and more alignment with fidelity requirements. Assignment of a liaison or coordinator title was not enough. Future programming should recognize the importance of this role and take care to ensure that the person chosen has the time, effort, and energy to actively pursue cross-site collaboration for maximum effect. This includes regular provider presence in local OCFS offices to educate and answer caseworker questions about programming and referrals. Increase Training Across the Board Implementation of any new program requires training for OCFS staff and participating or impacted providers at the start and ongoing refreshers for veteran and newly hired staff. Interviewees reported that initial training prepared them for implementation, but follow up interviews revealed ongoing training needs, especially in light of staff turnover and growth of a less experienced workforce. Future programming should include ongoing training for OCFS staff about program fidelity as well as substance use, specifically around recovery, relapse, abstinence, and best practice. In turn, program providers should also receive training on the child protection system and how cases are moved through the system to give them a better understanding of OCFS requirements, limitations, and their role in helping participants navigate the system. Additionally, the Department should consider trainings for the community particularly with the court system, including drug court, the Bar Association, and defense attorneys for any new initiatives as court buy-in often impacts program success. Expand Eligibility When asked about eligibility criteria, both OCFS and providers frequently suggested that the age range of children be broadened to at least seven or eight. They cited two primary reasons; first, child age is the element of the eligibility criteria that most often disqualified a family from participation. Second, the Triple P curriculum does not clearly apply to children ages zero to three, thus the curriculum may not be as effective for parents who only have younger children. To the contrary, many families enrolled in MEPP also had older children to which new parenting skills could be applied. Interviewees suggested that


expanding the age criteria may be especially important for districts with smaller, more rural populations so that the program is more widely applicable to OCFS’ families. Family First legislation already allows any family with substance abuse issues that has a child who is a candidate for foster care to receive subsidized treatment when participating in a Title IV-E Clearinghouse-approved program. Therefore, we recommend expanding future programming efforts to include all families with children under age 18. Continue to Improve Transportation Transportation has repeatedly risen in conversation with participants, providers, and OCFS staff as a barrier to participant success. Focus group participants reported that transportation can be inconsistent and unreliable, which makes it difficult for individuals to get to group on time or arrive there at all. Some participants also reported travelling great distances, over an hour one way, to attend Matrix or Triple P groups. While OCFS can arrange transportation for clients in the MEPP program, caseworkers and supervisors acknowledge that the transportation system is flawed as it is often volunteer-operated and organizing rides for participants can be very time consuming. One supervisor reflected that a caseworker could spend an entire day arranging rides. Steps need to be taken to identify alternative, more reliable transportation sources and vendors.


Appendices Appendix A: Statistical Similarity of Treatment and Comparison Cases Table A1 shows the p-values of a chi-squared comparison of treatment and comparison group cases. Table A1. Significance Values of Comparison Group Matching Criteria

Measure Cohort 1 Cohort 2 Cohort 3 Cohort 4 Cohort 5 Cohort 6 District 0.76 0.97 0.73 0.90 0.95 1.00 Number of Kids 0.95 0.83 0.80 0.80 0.55 0.75 Number of Adults 0.92 0.79 0.75 0.93 0.93 0.65 Prior Involvement 0.49 0.37 1.00 0.86 0.64 0.42 Alcohol Risk 1.00 0.83 0.82 1.00 0.37 0.59 Drug Risk 0.73 1.00 0.73 0.51 0.81 0.64 Placed in Residential 1.00 1.00 1.00 1.00 1.00 1.00 Placed in Regular Foster Care 1.00 1.00 0.56 0.70 0.50 0.59 Placed in Relative Foster Care 0.65 1.00 1.00 1.00 0.79 1.00 Placed in Unlicensed Foster Care 0.79 0.82 1.00 1.00 0.73 0.60

The Department of Health and Human Services (DHHS) does not discriminate on the basis of disability, race, color, creed, gender, sexual orientation, age, or national origin, in admission to, access to, or operations of its programs, services, or activities, or its hiring or employment practices. This notice is provided as required by Title II of the Americans with Disabilities Act of 1990 and in accordance with the Civil Rights Act of 1964 as amended, Section 504 of the Rehabilitation Act of 1973, as amended, the Age Discrimination Act of 1975, Title IX of the Education Amendments of 1972, the Maine Human Rights Act and Executive Order Regarding State of Maine Contracts for Services. Questions, concerns, complaints or requests for additional information regarding the ADA may be forwarded to the DHHS ADA Compliance/EEO Coordinators, #11 State House Station, Augusta, Maine 04333, 207-287-4289 (V), or 287-3488 (V)1-888-577-6690 (TTY). Individuals who need auxiliary aids for effective communication in program and services of DHHS are invited to make their needs and preferences known to one of the ADA Compliance/EEO Coordinators. This notice is available in alternate formats, upon request.

Maine Title IV-E Demonstration Project Final Evaluation Report

Documents

Transcript of Maine Title IV-E Demonstration Project Final Evaluation Report