Concurrent Validity of Alternative CANS Outcome

MetricsWilliam A. Shennum

Julian LeiroDelisa Young

Five AcresAltadena, California

What is test validity?

• The concept of validity was formulated by T.L. Kelley in 1927, who stated that a test is valid if it measures what it claims to measure.

• For example a test of intelligence should measure intelligence and not something else, such as memory.

Simplypsychology.org/validity

Concurrent validity

• Concurrent validity is a measure of how well a particular test correlates with a previously validated measure of the same construct.

• Concurrent validity relies upon measurements that take place at the same time.

Explorable.com

Who we are

• Non profit organization serving greater Los Angeles community

• Research & Evaluation Department– Outcome & performance measurement– Quality & business process improvement– Program evaluation– Project management

Five Acres’ service programs

Why CANS?

• Need for a common outcome measure across programs for use in planning & CQI.

CANS roll-out• Training• Inspiration – Dr. Lyons’ visit• Technology• Reporting

Individual Score Report (Excerpt)Client: “I” “I”

CANS Date: 9/24/2013 2/3/2014

Life Domain Functioning Intake Discharge1. Family 3 02. Living Situation 2 08. Medical 0 010. Sexuality 0 013. School Achievement 2 114. School Attendance 1 0Youth Strengths15. Family 3 116. Interpersonal 1 118. Educational 2 1

What About Program –Level CANS Outcomes?

• Several scoring options available• Which ones work best?–Easily understood by stakeholders–Closely associated with other

outcomes important to the organization (concurrent validity)

Study Design

Three service levels were included.

1. Community based mental health2. Wraparound3. Residential treatment

Study Design - continued

• Youth intake and discharge CANS were compared (N=233)

• Five CANS outcome scoring metrics were computed for each youth

• Statistical associations of each CANS metric with concurrently measured discharge outcomes were examined.

Concurrently Measured Outcomes

Service Intensity

Program Program Outcome/Success Measures

Lower

Higher

Community based mental health

• % Service completion (EBP’s)• Avg. YOQ points improved

(caregiver)

Wraparound

• % Program completion/ graduation

• % In community placement at discharge

Residential treatment (RBS)

• % Program completion/ graduation

• % In less restrictive placement at discharge

CANS Outcome Metrics Examined in This Study

Metric 1. Intake-discharge change scores

Method: Compute the average change from intake to discharge, for total score and for each domain

• Strength- easy to calculate• Weaknesses – unit of measurement not

common-sense, not tied to clients’ specific needs

Metric 1 – Intake/Discharge Change Scores, Example

Youth #CANS Total

Score IntakeCANS Total Score

Discharge ChangeRBS_2 15.5 10.4 5.1RBS_3 8.1 5.2 2.9RBS_4 1.6 6.3 -4.7RBS_5 10.7 3.9 6.7RBS_6 13.3 12.0 1.3RBS_7 6.7 4.6 2.1RBS_8 7.4 4.1 3.3

RBS_12 11.1 6.6 4.6RBS_13 3.0 3.7 -0.7RBS_14 11.3 3.6 7.7

Total Score 8.8 6.0 2.8

Program Results for Metric 1

Community Based

Services

Wraparound Residential Treatment

Avg. Change 2.0 0.7 1.9

Significance P<.001 P<.004 P<.02

Metric 2 - Reliable Change Index Analysis

Method: Determine whether each individual’s change score is statistically significant, by comparing it with a Reliable Change cutoff value

•RCI Formula: change in an individual’s score divided by the standard error of the difference for the test.•Parameters in formula include Intake and Discharge scores, test standard deviations, and test (rater) reliability.•Result is a yes-no score for each youth, for CANS total score, and each domain.

– Strength: statistically sound– Weaknesses: difficult to compute and explain to stakeholders

How to Calculate the RCI

M. Horsill, www.psy.uq.edu.au

1. Calculate standard error of measurement

2. Calculate standard error of the difference

3. Calculate RCI

If RCI is 1.96 or greater, the difference is statistically significant (95% confidence level, 2-tailed). 1.65 = one-tailed test cutoff

Metric 2 - Reliable Change Index Analysis, Example

Youth #CANS Total Score

Change RCI Score>=1.96?

(1=yes, 0-no)RBS_2 5.1 2.23 1RBS_3 2.9 1.29 0RBS_4 -4.7 -2.08 0RBS_5 6.7 2.98 1RBS_6 1.3 0.59 0RBS_7 2.1 0.95 0RBS_8 3.3 1.44 0

RBS_12 4.6 2.03 1RBS_13 -0.7 -0.29 0RBS_14 7.7 3.40 1

% Exceed RCI Cutoff 40.0%

Program Results for Metric 2

RCICommunity

Based Services

Wraparound Residential Treatment

>=1.96 14.9% 10.2% 24.2%

>=1.65 24.1% 12.6% 30.3%

Metric 3. Reduction in Number of Actionable Needs

Method: Compare the number of actionable needs at intake and discharge, total and by domain.

–Strength- addresses standout needs–Weakness- not tied to an individual’s specific needs identified at intake.

Metric 3. Reduction in Number of Actionable Needs, Example

Youth ## Actionable Needs Intake

# Actionable Needs Discharge Change

RBS_2 23 24 -1RBS_3 19 13 6RBS_4 3 10 -7RBS_5 23 7 16RBS_6 32 29 3RBS_7 12 6 6RBS_8 15 2 13

RBS_12 30 14 16RBS_13 5 6 -1RBS_14 24 6 18

Total Score 18.6 11.7 6.9

Program Results for Metric 3

Community Based

Services

Wraparound Residential Treatment

Avg. Change 4.5 1.4 4.4

Significance P<.001 P<.12 P<.03

Metric 4 - % Actionable Needs Resolved

Method: Track specific actionable items identified at intake, and calculate the % of these that are no longer actionable at discharge, total and by domain.

–Strengths - tailored to individual, easy for stakeholders to understand–Weakness – somewhat complicated scoring

Metric 4 - % Actionable Needs Resolved, Example

CANS Domain: Youth Strengths Intake

Discharge

Need Resolved? (1=yes, 0=no)

15. Family 3 1 116. Interpersonal 1 117. Optimism 2 1 118. Educational 2 1 119. Vocational U U20. Talents/Interests 2 1 121. Spiritual/Religious 2 2 022. Community Life 2 1 123. Relationship Permanence 2 2 024. Resiliency 1 125. Resourcefulness 2 1 1

% Needs Resolved 75%

Program Results for Metric 4

Community Based

Services

Wraparound Residential Treatment

% Needs Met 72.8% 49.2% 61.6%

Metric 5 – Actionable Needs Improved

Method: Track specific actionable items identified at intake, and calculate % of these that show any improvement, even if still actionable at discharge, total and by domain.

– Strength - tailored to individual, easy for stakeholders to understand

– Weaknesses – includes needs still remaining at discharge, somewhat complicated scoring

Metric 5 – Actionable Needs Improved, Example

Youth Behavioral/Emotional Needs Intake

Discharge

Any Improvement? (1=yes, 2=no)

41. Psychosis 0 0

42. Impulse/Hyper 2 0 1

43. Depression 1 0

44. Anxiety 1 1

45. Oppositional 3 2 1

46. Conduct 2 1 1

47. Adjustment to Trauma 3 2 1

48. Anger Control 2 2 0

49. Substance Use 0 0

% Any Improvement in Actionable Needs 80%

Program Results for Metric 5

Community Based

Services

Wraparound Residential Treatment

% Needs Improved

75.3% 59.4% 66.6%

Concurrent Validity of the 5 CANS Program Outcome Metrics

• Are the metrics statistically associated with concurrently collected measures of program success?

All CANS Metrics Significantly Associated with Concurrent Program

Outcomes  CANS Outcome Metrics

Association w/Concurrent Outcomes

1. Avg.

Change

2. RCI

Cutoff

3. Change

in # Needs

4. Needs

Resolved

5. Needs

Improved

Median Statistical Significance

p = .020

p = .043

p = .009

p = .013

p = .018

Specific CANS Domains were Positively Associated with

Concurrent Outcomes  CANS Outcome Metrics for

Selected DomainsFunctioning Youth

StrengthFamily

Needs & Strength

s

Behavior &

Emotional Needs

Median Correlation w/ Outcomes

r = .23

r = .33

r = .22

r =.28

Some Differences Across Program Were Evident

  CANS Outcome Metrics

Program / Outcome

Avg. Change

RCI Cutoff

Change in #

Needs

Needs Resolve

d

Needs Improve

dCommunity Based

Service completion

p<.001 p<.05 p<.001 p<.01 p<.05

YOQ Points p<.05 p<.05 p<.01 p<.01 p<.01

WraparoundProgram

completionp<.001 p<.001 p<.001 p<.001 p<.001

In community p<.01 ns p<.05 p<.10 ns

Residential (RBS)Program

completionp<.10 p<.05 p<.05 p<.05 p<.05

Less restrictive ns ns ns p<.05 p<.10

CANS outcomes for ‘successful’ youthCANS Outcome Metrics

Program

Avg. Change in Total Score

% Youth exceedin

g RCI1.65

Avg. Change in # of

Actionable Needs

Avg. % Actionable Needs

Met

Avg. % Actionable Needs

Improved

Community MH 2.3 29.0% 5.6 76.7% 78.9%

Wraparound 2.6 34.8% 7.0 68.9% 79.9%

Residential 3.1 36.8% 8.1 70.5% 74.6%

Average 2.7 33.5% 6.9 72.0% 77.8%

Trends• “Reduction in number of actionable needs”

and “% of actionable needs resolved” metrics tended to work best for most programs

• “Average change score” metrics worked well in program serving less severe youth

• Suggestive evidence favored youth strengths domain as a driver of overall improvement

• CANS outcome profiles for successful youth were similar across programs – benchmarks would assist in setting performance targets.

Take Aways• All CANS metrics examined showed evidence of

concurrent validity across a range of programs• All CANS domain improvements were related to

concurrent outcomes, with suggestive evidence that “youth strengths” had strongest relationship to success.

• Programs serving higher severity needs may benefit from measuring metrics related to actionable need resolution and improvement.

• Benchmark databases could assist programs in setting outcome targets for youth.

Contact Information

Bill ShennumDirector of Research & Evaluation

Five Acres(626)798-6793

bshennum@5acres.orgwww.5acres.org