DCFS Research and Evaluation Continuing Development of Wraparound as an Evidence-Based Practice...

DCFSResearch and Evaluation

Continuing Development of Wraparound as an Evidence-Based Practice

(Workshop A-11)

Sixth California Wraparound Institute

Tran M. Ly, Ph.D.Ming H. Lee, M.P.H.

Christopher J. Jarosz, Ph.D.

Department of Children and Family ServicesCounty of Los Angeles

June 7, 2010

2DCFS

Research and Evaluation

Workshop Outline

• Introduction• Overview of Wraparound’s evidence base• Los Angeles County’s placement and cost outcomes studies• Program evaluations• Logic model• Methods• Implementing an evaluation• Review and wrap-up• Resources

3DCFS


Introduction

4DCFS


Learning Objectives

• Be cognizant of the evidence base of Wraparound, including a recent example of a Wraparound evaluation in Los Angeles County.

• Learn what is a program evaluation, and why and when to conduct one.

• Recognize how the selection of study designs, measurement tools, and sampling strategies affect the types of questions asked and conclusions drawn in an evaluation.

• Practice building a basic logic model as an initial step in devel-oping an evaluation plan.

5DCFS


The Presenters

• Tran Ly has graduate training in psychology, education, and child development. She previously planned and conducted several studies involving families of children with various disa-bilities.

• Ming Lee has training in biology, and graduate training in public health and biostatistics. He previously planned and conducted studies in perinatal health indicators and their influences on birth outcomes.

• Christopher Jarosz has graduate training in the behavioral and biological sciences. He has planned and conducted behavioral, community health, and organizational research in the public and private sectors.

6DCFS


DCFS Research and Evaluation

• Conduct program evaluations and applied research for DCFS.• Provide technical consulting to other DCFS programs. • Plan and conduct departmental surveys.• Serve as the department’s liaison with outside researchers.

Website:http://dcfs.co.la.ca.us/rae/index.html

7DCFS


Overview of Wraparound’s Evidence Base

8DCFS


Residential Care

• “Residential care” and “group home care” are terms sometimes used interchangeably within the research literature, with no consensus on a single definition.

• Residential care is a term applied to a diverse array of services that are provided to youth living in such congregate care as apartments, emergency shelters, halfway homes, secure set-tings, and group homes.

9DCFS


Residential Care (continued)

• Within residential care, placement instability is a common occurrence associated with various negative outcomes such as child behavior problems and juvenile delinquency.

• Residential care placements in California were approximately 11% of the foster care population in 2000, but more than 50% of foster care expenditures.

• Group homes provide continuous staff supervision, which can make them an expensive placement option.

• Group home providers are categorized into Rate Classification Levels (RCLs) based on the level of care and services provided, with RCL 14 being the most restrictive.

10DCFS


Other Research Studies

• Bruns and Suter (2010) updated their summary of experimental and quasi-experimental research studies on outcomes of the Wraparound process.

• Nine controlled studies have been presented in peer-reviewed publications encompassing child welfare, juvenile justice, and mental health.

• Bruns and Suter note that each study has methodological weak-nesses.

• The studies considered in total indicate “superior outcomes for youth who receive wraparound compared to similar youth who receive some alternative service.”

Eric J. Bruns and Jesse C. Suter. Summary of the Wraparound Evidence Base: April 2010 Update (2010). Resource Guide to

Wraparound, Theory and Research, Chapter 3.5, National Wraparound Institute, 1-9.

11DCFS


Positive Outcomes

• Fewer placement changes, and moves to less restrictive environ-ments.

• Older youth more likely to be in a permanency plan at the time of follow-up.

• Improved ability to externalize problems, especially with boys in the study.

• Overall decline in Child and Adolescent Functional Assessment Scale (CFAS) scores.

• Increased school attendance, reduction in school disciplinary actions, and increased grade point averages.

12DCFS


Los Angeles County’s Placement and Cost Outcomes Studies

13DCFS


Los Angeles County Wraparound Program

• Wraparound provides Los Angeles County (LAC) with an alterna-tive to residential care for foster youth consistent with a national review panel report (Cole, 1998).

• LAC implemented Wraparound in 1998 as a 10-child pilot study as an alternative to residential care.

• The goal of the pilot was to explore the impact of Wraparound on reducing reliance on out-of-home care.

• An analysis of the outcomes suggested that Wraparound youth were more likely to return and stay at home compared to youth in residential care.

14DCFS


LAC Wraparound Program (continued)

• With closure of the County’s children’s shelter (McLaren) in 2003, Wraparound was identified as a primary mechanism for returning youth to their communities.

• Referrals were initially slow due to the lack of understanding of Wraparound.

• Another factor was the lack of research on its potential impact on youth, which at the time consisted of only a handful of controlled studies.

15DCFS


An Initial Study

• A LAC study was conducted in 2004 to compare the outcomes of Wraparound graduates with youth who were discharged from Rate Classification Level (RCL 12 or 14) and went into less re-strictive placements.

• RCL 12 and 14 were selected since youth must qualify at these levels for enrollment in Wraparound since they must be in or at imminent risk of placement in a residentially-based institutional setting.

• The two groups were followed for 2-1/2 years using administra-tive data.

16DCFS


An Initial Study (continued)

• The Wraparound graduates did substantially better than the youth who were discharged from RCL 12-14.

• The Wraparound graduates had fewer subsequent out-of-home placements and therefore less financial cost to the County of Los Angeles.

• A greater percentage of Wraparound graduates also exited foster care.

17DCFS


Wraparound Growth

• Residential care in Los Angeles County has been reduced by over 60 percent since the 2004 study.

• About 2,400 foster youth were in residential care in 2004, while the number declined to fewer than 1,000 youth in 2009.

• Wraparound enrollment was less than 200 youth in 2003, which increased to over 1,200 youth by 2009.

• The number of Wraparound providers increased from eight to 34 in 2006.

• In 2009, the Wraparound program was expanded to accommo-date up to 4,200 youth.

18DCFS


Further Study

• In 2009, we applied greater methodological rigor in attempting to replicate and expand upon the initial study.

• The areas we focused on including using entire populations in the study, providing a more detailed set of selection criteria for the cohorts, and use of a matched-comparison design.

• We found that the placement and cost outcomes of Wraparound versus RCL 12-14 were consistent with the findings of the initial study.

Michael Rauso, Tran M. Ly, Ming H. Lee, and Christopher J. Jarosz (2009), Improving outcomes for foster care youth with

complex emotional and behavioral needs: a comparison of outcomes for wraparound vs. residential care in Los Angeles

County. Emotional & Behavioral Disorders in Youth, 9, 53-76.

19DCFS


Further Study (continued)

• This year, we established new one-year cohorts of Wraparound graduates and youth discharged from RCL 12-14 to lower-level placements.

• Our principal goal was to determine how consistent the results were with the previous studies.

• We compared the placement outcomes and associated costs of the two cohorts as defined in the selection criteria shown on the next slide.

• We hypothesized that Wraparound would continue to result in better outcomes (less restrictive placements) and lower costs to the system.

20DCFS


Selection Criteria

Criteria Wraparound RCL 12-14

The case record is available in CWS/CMS X X

Graduated from Wraparound between 1 July 2007 and 30 June 2008

X

Discharged from RCL 12 or 14 to a lower placement level (< RCL 10) or home between 1 July 2007 and 30 June 2008

X

Had not previously been enrolled in the Wraparound program

X

Did not receive Wraparound services in the 12 months after discharge

X

Was in a Wraparound or RCL 12-14 placement for at least six months prior to graduation or discharge

X X

Was no older than 17 years, 0 months at the time of graduation or discharge

X X

21DCFS


Outcome Measures

• Case closures—did the case close during the 12 months after Wraparound graduation or discharge from RCL 12-14 to a lower placement level?

• Placements—how many placements, including placement types, did the youth have during the 12-month period after graduation or discharge?

• Costs—based on the placement types and time durations, what were the system costs during the 12-month period?

22DCFS


Demographics

Demographic Characteristic

Wraparound (N = 80)

RCL 12-14 (N = 98)

Age Ranges

5-to-11 years 16% 14%

12-to-17 years 83% 86%

Gender

Female 54% 59%

Male 46% 41%

Ethnicity

African American 23% 51%

Hispanic 51% 30%

White 15% 16%

Other 11% 3%

23DCFS


Case Closures

16.9

58.8

0.0

50.0

100.0

Wraparound RCL 12-14

Per

cent

age

of C

hild

ren

Percentages of Cases that Closedwithin 12 Months

24DCFS


Placement Outcomes

Figure 3. Average Number of Out-of-Home Placements

0.88

2.22

0

0.5

1

1.5

2

2.5


Ave

rage

Num

ber

of P

lace

men

ts

Average Number ofOut-of-Home Placements

Figure 4. Average Number of Days inOut-of-Home Placements

202

308

0

100

200

300


Ave

rage

Num

ber

of D

ays

Average Number of Days inOut-of-Home Placements

25DCFS


Placement Outcomes (continued)

41.3

51.3

7.66.1

57.1

36.7

0

20

40

60

No Placement 1-2 Placements 3 or More Placements

Pe

rce

nta

ge

of C

hild

ren

Wraparound (N=80) RCL 12-14 (N=98)

Children Who Had None Versus at Least One Placement

26DCFS


Figure 5. Distribution of Out-of-Home Placements

47.2

14.34.3

25.7

8.617.0

2.8

25.2

42.7

12.4

01020304050

Guardian/Relative Foster Family Court-Specif ied/ Small Family

FFA-Certif ied Group Home

Perc

enta

ge o

f Pl

acem

ent T

ypes


Distribution of Out-of HomePlacement Types

Placement Distribution

27DCFS


Cost Outcomes

Figure 6. Average Out-of-Home Placements Costs

$9,627*

$15,872*

$0

$10,000

$20,000


Ave

rage

Cos

t per

Chi

ld

Average Out-of-Home PlacementCosts

Statistically-significant, p < 0.01

28DCFS


Figure 7. Distribution of Out-of-Home Placements Costs

42.5 46.3

7.51.3

19.4

2.06.12.5

62.2

10.2

0

20

40

60

No Cost $1--$20,000 $20,001-$40,000 $40,001--$60,000 $60,001--$80,000

Placement Costs by Child

Perc

enta

ge o

f C

hild

ren

Wraparound

RCL 12-14

Distribution of Out-of-Home Placement Costs

Cost Distribution

29DCFS


Summary

• The findings support previous cost impact analyses that Wrap-around is more cost-effective and has better outcomes compared to traditional residential care.

• Relative to RCL 12-14 children, Wraparound children are more likely to have their cases closed within 12 months of graduation.

30DCFS


Summary (continued)

• Wraparound versus RCL 12-14 children in the 12-month period after graduation had:

– No or fewer out-of-home placements.– Placements, when they did occur, were often to less restric-

tive environments and require fewer number of days.– Financial costs associated with placements were significantly

less.

• Despite recent improvements in placement and cost outcomes for the RCL children, the outcomes of Wraparound graduates remain significantly better.

31DCFS


Summary (continued)

• By many indications, Wraparound has provided an effective mechanism for reducing the Los Angeles County’s reliance on out-of-home care.

• Wraparound has a positive impact in that youth are living in less restrictive and more stable living environments.

32DCFS


Policy and Practice Implications

• Wraparound provides an effective service delivery option for youth with behavioral and emotional problems.

• Wraparound, endorsed as a high-priority strategy in Los Angeles County, was recently expanded to accept youth who are not currently in or at imminent risk of placement into higher levels of residential care.

• A training model for service delivery is being developed for children’s social workers that shares many of the Wraparound values and principles.

• Wraparound principles and practices will soon influence the nature of residential care in Los Angeles County from long-term placements to planned, short-term, and individualized interven-tions.

33DCFS


Limitations

• The study design is a retrospective study, and therefore youth were not randomly assigned to groups.

• We examined placement and cost outcomes for a period of one year.

• In using administrative data, we do not have information on the behavioral characteristics or functioning of the comparison youth.

34DCFS


Future Research Directions

• We will be conducting a third, one-year cohort study to further assess the year-to-year stability of placement and cost outcomes for Wraparound and RCL 12-14.

• We are currently analyzing two-year cohort groups to determine longer-term outcomes for Wraparound and RCL 12-14.

• We are refining the selection criteria for equivalence of emotional, behavioral, and mental health difficulties of the Wraparound and RCL 12-14 groups.

35DCFS


Program Evaluations

36DCFS


What is a Program Evaluation?

An evaluation involves “carefully collecting information about a program or some aspect of a program… to make necessary

decisions about the program.”

“The type of evaluation… depends on what you want to learn about the program.”

“More recently… evaluation has focused on utility, relevance and practicality at least as much as scientific validity.”

(McNamara, 2006)

37DCFS


Possible Goals

• Measure the impact of services on clients and other stakeholders.• Improve the effectiveness and reduce the costs of service delivery.• Verify if the program is operating to plan and if the plans need to be

changed.• Facilitate meta-thinking about program goals, and if they are being

met.• Promote decision-making based on data in responding to organiza-

tional and community needs.

38DCFS


Possible Goals (continued)

• Compare programs to determine which best meet the intended needs.

• Establish a basis for duplicating or adapting programs for other venues and uses.

Adapted from:McNamara, Carter (2006). Field Guide to Nonprofit Program Design, Marketing and Evaluation (4th edition). 252 pages.

39DCFS


Types of Program Evaluations

• Program goals• Processes and process fidelity• Short- and long-term outcomes• Client satisfaction• Time, quality, and financial costs • Combination of two or more types

40DCFS


Stakeholders

• Who are the formal and informal decision-makers?• Who is likely to benefit from the evaluation?• Who can facilitate or possibly hinder the evaluation?• What resources can the stakeholders offer for the effort?• Is there a shared vision and commitment to mutual goals?

41DCFS


Shared Vision?

42DCFS


Group Exercise #1

Let’s say you are asked to conduct an evaluation of Wraparound service delivery in your community.

Identify the primary and secondary goals that you would want to achieve. What type of evaluation would you perform? Who

would be the three most influential stakeholders in supporting or benefiting from the evaluation?

43DCFS


Logic Model

44DCFS


What Is a Logic Model?

• A logic model is a useful tool for program development and evaluation and is usually part of an evaluation plan.

• A logic model:

– Describes the logical links between the program’s theories or assumptions, services, and outcomes desired.

– Illustrates a sequence of cause-and-effect relationships.

• While there many versions, a logic model is often a one-page graphical or schematic representation of what you do, why you do it, what you hope to achieve, and how you will measure that achievement.

45DCFS


Sample Template

Program vision: Population served: Population needs to be addressed by services:

Outcomes Indicators Measurement Services* Resources

* Include service assumptions

46DCFS


Building a Logic Model

• Establish your program vision and goals or objectives.• Describe your target population that will be served, including the

needs to be addressed.• Articulate the services and resources needed, including the

assumptions you are making.• Identify your program outcomes and indicators.• Determine how you will measure these outcomes and indica-

tors.

47DCFS


Vision Statement

• A vision statement is a brief mission statement of what you hope for and want for the families and community to achieve in the future.

• In general, vision statements are very broad and ambitious goals—the objective for the program is to be able to contribute towards that vision.

• Vision statements are not necessarily measurable—example, all children are “ready” for school by age 6.

48DCFS


Target Population and Needs

• In building a logic model, you will need to identify your target population.

• Describe its key characteristics which may include age, gender, ethnicity, educational status, economic status, and personal risk factors.

• Be as specific as possible in identifying the people who will or have received the services to ensure that intended outcomes are meaningful and appropriate.

• It is important to describe the population needs that your program intends to address—what will your services help them to do?

49DCFS


Service Needs and Assumptions

• In continuing the development of the model, briefly describe the services that your families will receive.

• Describe why you think the services will lead to the desired outcomes.

– What assumptions are you making?– Are the services informed by evidence-based practice?

• When possible, use theory and research to guide the develop-ment of your logic model.

50DCFS


Outcomes versus Indicators

• An outcome is a change that is likely to take place as a result of the target population’s participation in your program.

• Three types of outcomes are: long-term, intermediate, and short-term.

• Indicators tell you whether an outcome has been achieved.• Indicators are concrete specific descriptions of what you will

measure.• Also known as performance targets, indicators can specify a

targeted number or percentage of achievement.

51DCFS


Examples of Outcomes and Indicators

• An example of an outcome is “participants will make their home child-safe.”

• Some examples of indicators for this outcome might include:

– “Participants access and install supplies needed to have a child-safe home.”

– “85% of participants remove household hazards that jeopardize child safety.”

– “Participants have a list of emergency numbers posted where all family members can find them.”

52DCFS


Group Exercise #2

With this brief introduction to outcomes and indicators, are you able to distinguish between the two?

On your handout, please circle if you think the example is an outcome (O) or an indicator (I).

53DCFS


Methods

54DCFS


• Now, you are almost ready to implement your logic model.• How do you start?• The process of implementing your logic model should include:

– Selecting a study design.– Determining your available data collection options.– Analyzing your collected data and presenting them.

Next Steps

55DCFS


• Of the nine Wraparound research studies described by Bruns and Suter (2010), two types of research designs were used: randomized designs and quasi-experimental designs.

• In both design types, participants are grouped into treatment and comparison groups.

• In a quasi-experimental design, differences observed may be influenced by dissimilarities already present between compari-son groups.

Selecting a Study Design

56DCFS


Some Measurement Tools

• Questionnaires, surveys, and checklists• Structured and semi-structured interviews• Rating scales (e.g., CAFAS and CANS)• Administrative data• Direct observations• Focus groups• Documentation reviews• Case studies

57DCFS


• Will you be studying the entire population or sample from the population?

• When including the entire population is too costly or impractical, a sample from the population may be appropriate.

• When sampling, the objective is to draw a subset of the popula-tion that is representative of the characteristics of the population.

Populations vs. Samples

58DCFS


Sampling Methods

• Simple random sampling—everyone has an equal chance of being selected.

• Stratified sampling—used to ensure that enough participants with desired characteristics are selected.

• Nonrandom sampling—includes availability, purposive, quota, and snowball techniques.

59DCFS


Implementing an Evaluation

60DCFS


Implementation Plan

• Construct the logic model.• Select or develop measurement and data management tools.• Administer the evaluation tools.• Analyze and report on the results of the evaluation.

61DCFS


Group Exercise #3

Let’s say you are asked to conduct an evaluation of Wraparound service in your community. The initial step in implementing such an evaluation would be to develop a

logic model.

Please break into small groups and work on building a basic logic model using the template provided.

62DCFS


Review and Wrap-up

63DCFS


Learning Objectives Accomplished?

• Be cognizant of the evidence base of Wraparound, including a recent example of a Wraparound evaluation in Los Angeles County.

• Learn what is a program evaluation, and why and when to conduct one.

• Recognize how the selection of study designs, measurement tools, and sampling strategies affect the types of questions asked and conclusions drawn in an evaluation.

• Practice building a basic logic model as an initial step devel-oping an evaluation plan.

64DCFS


Conclusions

• We have given you a general overview of these topics:

– Program evaluation– Logic model– Methods– Evaluation plan

• There is enough information to teach a semester-long course for any of these topics.

• What we want you to take away from this workshop is that evaluating and planning a program is not a one-time or a one-step process.

65DCFS


Resources

66DCFS


Useful Web Links

Research Methods Knowledge Base

http://www.socialresearchmethods.net/kb/evaluation.php

Basic Guide to Program Evaluation

http://www.managementhelp.org/evaluatn/fnl_eval.htm

Logic Model and Toolkit

http://www.friendsnrc.org/outcome/toolkit/index.htm

DCFS Research and Evaluation Continuing Development of Wraparound as an Evidence-Based Practice...

Documents

Transcript of DCFS Research and Evaluation Continuing Development of Wraparound as an Evidence-Based Practice...