Children’s Intensive Services (CIS) Program Evaluation

Presented By:Elizabeth Earls – President/CEO

Rhode Island Council of Community Mental Health Organizationsand

Deborah O’Brien – Director, Quality Improvement The Providence Center

History and Overview Why Measure Outcome? In the 1990’s, as resource demands for health and

social programming increase, State and Federal leaders, managed care entities and consumers demand programs that are accountable and measure the effectiveness of their work

The Department of Children, Youth and Families (DCYF) begin to incorporate performance indicators into all of its service contracts. DCYF establishes “Outcome Committees” for its various service sectors with the goal of establishing measurement and accountability.

History and Overview Why Measure Outcome? The Community Mental Health Organizations

meet with DCYF staff for a number of months and determine that they will use the Child Behavior Checklist (CBCL) as the CIS performance measurement tool.

DCYF announced that as part of the planned revisions to the CIS contracts, it would mandate that the CBCL be completed on each child who received services from a CIS program.

History and Overview Why the Child & Adolescent Functional Assessment Scale (CAFAS)?

DCYF had never established the staff and the technical capacity to collect the data from the CBCL’s. Therefore, they have no ability to analyze or provide feedback to us on the success, or lack of success, in our programs.

Some clinicians had reported that their experience with the CBCL indicated that it was not a good measure of outcome.

History and Overview Why the Child & Adolescent Functional Assessment Scale (CAFAS)?

Through the Council’s Quality Improvement Committee, CMHCs compare their experience with DMHRH and its required Outcome Evaluation Instrument (OEI): DMHRH collects data and reports back to organizations quarterly. The OEI had been approved and utilized by several CMHO’s to fulfill their JCAHO – ORYX process.

History and Overview Why the Child & Adolescent Functional Assessment Scale (CAFAS)?

The RI Council determines that it will take the lead on this issue in the absence of action by DCYF. The Council sub-committee with both QI directors and Children’s Services Directors. The Committee identifies possible outcome measurement tools to be used in the CIS programs. The tools are reviewed across a determined set of criteria.

History and Overview Why the Child & Adolescent Functional Assessment Scale (CAFAS)?

Criteria used for the comparison include the following:

Target population Domains covered by the Tool Completer of the instrument (staff vs. client) Administration time Cost Norms/Breadth of use Reliability Ability to measure change over time Technical infrastructure requirements Clinical usefulness Face Validity

History and Overview Why the Child & Adolescent Functional Assessment Scale (CAFAS)?

In February 2000, the Council’s Member Organizations vote to use the CAFAS. It informs DCYF of its decision. DCYF agrees to assist with support of training and recognizes that the CAFAS will replace the CBCL.

History and Overview Why the Child & Adolescent Functional Assessment Scale (CAFAS)?

High degree of inter-rater reliability Sensitive to change over rating time periods Ability to identify high risk behaviors Sub-scales measure functioning in various

domains Identification of specific profile types with

children & adolescents Tested extensively and used in National studies:

Fort Bragg, Michigan CMHC, CMHS (REACH) Benchmark with other studies

History and Overview CAFAS Implementation

Council contracts with Dr. Kay Hodges, creator of the CAFAS, to train 40 clinicians as trainers on the CAFAS.

The Council has made an important decision to systematically collect data based on what the clinician knows and has complied into this single format. We will collect Statewide data and be able to measure our success and advocate for our programs.

We will provide the data to DCYF.

History and Overview CAFAS Implementation

Implementation Committee is created to establish a uniform protocol across the CIS Programs so that all staff are trained; are determined reliable raters; and are collecting data in a consistent manner.

History and Overview CAFAS Protocol

Issues being addressed by the Implementation Committee, include but are not limited to: Age of population to be assessed (7 to 17) Level of credentialed staff (masters and

bachelors) Administration intervals (1-month and 6-months) Train-the-trainer on CAFAS software Rater training, rater remediation, rater 18-

month boosting (345)

History and Overview CAFAS Protocol

Implementation Committee issues (continued): RI specific manual created for FAQ’s Data export software developed Identify and contract with database

consultant for aggregate reporting Satisfaction Measurement (To be

determined)

CAFAS – Self-Harmful Behaviors Subscale

Severe ImpairmentSevere disruption or incapacitation (30)

Moderate Impairment Major or persistent disruption (20)

Mild ImpairmentSignificant problems or distress (10)

Minimal or No ImpairmentNo disruption of functioning (0)

142 Non-accidental self-destructive behavior has resulted in or could result in in serious self-injury or self-harm (e.g., suicide attempt with intent to die, self-starvation).

146 Non-accidental self-harm, mutilation, or injury which is not life-threatening but not trivial (e.g., suicidal gestures or behavior without intent to die, superficial razor cuts).

149 Repeated non-accidental behavior suggesting self-harm, yet the behavior is very unlikely to cause any serious injury (e.g., repeatedly pinching self or scratching skin with a dull object).

151 Behavior is not indicative of tendencies toward self-harm

143 Seemingly non-intentional self-destructive behavior has resulted in or could likely result in serious self-injury (e.g., runs out in the path of a car, opens car door in moving vehicle), and youth is aware of the danger.

147 Talks or repeatedly thinks about harming self, killing self, or wanting to die.

144 has a clear plan to hurt self, or genuine desire to die.

CAFAS CAFAS Domains

School Home Community Behavior Towards Others Moods and Emotions Self-Harmful Behavior Substance Abuse Thinking Caregiver Scales

CAFAS -Severity of Impairment

14%

41%

34%

11%

1%

140 or higher –Intensive treatment

needed

100 to 130 – Moreintensive care thanoutpatient needed

50 to 90 – Servicesbeyond outpatient care

may be needed

20 to 40 – outpatientneeded

20 or less – someintervention may be

needed

CAFAS Outcome Results:CIS and Benchmark Programs

0

20

40

60

80

100

RI:CIS MCMHC FBEP

Average Admission Score Average Discharge Score

CIS Outcomesby Length of Treatment

0102030405060708090

100

0-3 Months 4-6 Months 7 Months orGreater

Admission CAFAS Discharge CAFAS

39%

26%

7%

5%

5%

5%

3%

2%

Physically DangerousToward Others

Self Harmful/ Suicidal

Substance Abuse

Legal Involvement2nd to Assault or

Weapon Use

Run Away

SexuallyInappropriate/ Sexual

Assault

Psychotic Behavior

Fire Setting

Areas of High Risk at Admission to CIS

Risk at Admission and Discharge

54%

31%

0%

10%

20%

30%

40%

50%

60%

Admission Discharge

Decrease in high risk behaviors

Next Steps Streamline the data import and

administration Identify and implement a satisfaction

tool Train and implement the Pre-School &

Early Childhood Functional Assessment Scale (PECFAS)

Analysis of aggregate data for all stake holders