11

The Wraparound ProcessThe Wraparound ProcessAn in-depth look at fidelity patterns fromAn in-depth look at fidelity patterns from

a national dataseta national datasetKristen Kristen M. Leverentz-BradyM. Leverentz-Brady, M.A., M.A.

Jesse Jesse C.C. SuterSuter, M.A., M.A.University of Vermont Department of PsychologyUniversity of Vermont Department of Psychology

Eric J. Eric J. BrunsBruns, Ph.D., Ph.D.University of Washington Department of PsychiatryUniversity of Washington Department of Psychiatry

1818thth Annual Florida Mental Health Institute Annual Florida Mental Health InstituteResearch Conference on ChildrenResearch Conference on Children’’s Mental Healths Mental Health

March 7, 2005March 7, 2005

For more information: For more information: ebruns@u.washington.eduebruns@u.washington.eduwww.uvm.edu/~wrapvtwww.uvm.edu/~wrapvt or or depts.washington.edu/wrapevaldepts.washington.edu/wrapeval

22

IntroductionIntroductionJohn John BurchardBurchard’’ss Wraparound Fidelity Wraparound FidelityIndex (WFI-3) assesses adherence toIndex (WFI-3) assesses adherence toprinciples of the wraparound processprinciples of the wraparound processthrough interviews with caregivers, youths,through interviews with caregivers, youths,and providers.and providers. 11 Elements of Wraparound11 Elements of Wraparound 4 items per element4 items per element Responses scored by interviewer Responses scored by interviewer fromfrom 0 (low 0 (low

fidelity) to 2 (high fidelity) to 2 (high fidelity)fidelity)Requested by over Requested by over 250250 communitiescommunitiesUsed byUsed by at least at least 5050 communities nationally communities nationally

33

Prior Research on WFIPrior Research on WFI

WFI Total scores and most element scoresWFI Total scores and most element scoresfound to have good psychometric propertiesfound to have good psychometric properties(e.g., internal consistency, test-retest reliability)(e.g., internal consistency, test-retest reliability)WFI Total scores associated with externalWFI Total scores associated with externalexpert ratings of fidelity using more intensiveexpert ratings of fidelity using more intensivemethodmethodWFI Total scores discriminate WFI Total scores discriminate different types ofdifferent types ofprograms (e.g., wraparound vs. comparison)programs (e.g., wraparound vs. comparison)Emerging evidence for association betweenEmerging evidence for association betweenWFI scores and child and family outcomesWFI scores and child and family outcomes

44

Unanswered questionsUnanswered questionsFor communities using the WFI and researchersFor communities using the WFI and researchers

WhatWhat is the level of agreement between is the level of agreement betweenWFI respondents?WFI respondents?What is the underlying structure of theWhat is the underlying structure of theWFI? What does it tell us about the wayWFI? What does it tell us about the waywraparound occurs in the wraparound occurs in the ““real worldreal world””??What constitutes a What constitutes a ““goodgood”” (or (or ““acceptableacceptable””))fidelity score?fidelity score?

55

TodayToday’’s presentations presentationWe will present results of analyses of a nationalWe will present results of analyses of a nationalWFI-3 dataset for 667 youth from 10WFI-3 dataset for 667 youth from 10communities, focusing on:communities, focusing on:AgreementAgreement between youths, caregivers, and between youths, caregivers, and

facilitators,facilitators,How different hypothesized models for wraparoundHow different hypothesized models for wraparound

implementation fit with WFI-3 data (confirmatoryimplementation fit with WFI-3 data (confirmatoryfactor analysis), andfactor analysis), and

What the national dataset and previously publishedWhat the national dataset and previously publishedevaluation studies using the WFI-3 tell us aboutevaluation studies using the WFI-3 tell us aboutsetting standards for wraparound setting standards for wraparound fidelityfidelity

ImplicationsImplications for quality assurance, research,for quality assurance, research,and model developmentand model development

66

Extending understandingExtending understandingof the WFI:of the WFI:

Agreement between Agreement between WFIWFIrespondentsrespondents

Kristen M. Leverentz-Brady, M.A.Kristen M. Leverentz-Brady, M.A.University of Vermont Department of PsychologyUniversity of Vermont Department of Psychology

Wraparound Evaluation and Research TeamWraparound Evaluation and Research Teamwrapvt@uvm.eduwrapvt@uvm.edu

77

Background: WFI Properties andBackground: WFI Properties andPsychometricsPsychometrics

What has been assessed:What has been assessed: Test-retest reliabilityTest-retest reliability Internal consistencyInternal consistency Concurrent validityConcurrent validity Criterion-related validityCriterion-related validityWhat has not been assessed:What has not been assessed: Stability over time and typical trajectory ofStability over time and typical trajectory of

fidelity scoresfidelity scores Inter-respondent agreementInter-respondent agreement

88

National WFI SampleNational WFI SampleN=667 families from 10 collaborating sites in 9N=667 families from 10 collaborating sites in 9statesstates N=622 RF interviewsN=622 RF interviews N=490 caregiver interviewsN=490 caregiver interviews N=367 YouthsN=367 Youths

Data collection method:Data collection method: Sites received manual and training materialsSites received manual and training materials Each site completed Memorandum of Agreement toEach site completed Memorandum of Agreement to

administer WFI in full and to adhere to interviewadminister WFI in full and to adhere to interviewprotocolsprotocols

Majority of sites administered interviews by phoneMajority of sites administered interviews by phone18% of RF, 14% of Caregiver, and 8% of Youth interviews18% of RF, 14% of Caregiver, and 8% of Youth interviewsconducted face-to-faceconducted face-to-face

99

National WFI study sampleNational WFI study sample

367490622667Total NsTotal Ns8151921Minnesota 25172427Florida

23313131Nevada27323434Missouri16222222North Carolina12232626Minnesota 129717474Massachusetts12141718Pennsylvania

212226339366Nebraska23394449Indiana

N YouthN CGN RFN familiesSite

1010

Results: Agreement BetweenResults: Agreement BetweenRespondentsRespondents

For all three respondents, a moderateFor all three respondents, a moderatecorrelation was foundcorrelation was found ICC = .58ICC = .58For Individual respondents:For Individual respondents: RF-CG = .44RF-CG = .44 CG-Y = .49CG-Y = .49 RF-Y = .45RF-Y = .45

1111

Findings and implicationsFindings and implicationsModerate agreement found for WFI Total scoresModerate agreement found for WFI Total scoresacross all respondentsacross all respondents Higher than found in a meta-analysis of parent-mental healthHigher than found in a meta-analysis of parent-mental health

worker cross-informant scores for children (Achenbach,worker cross-informant scores for children (Achenbach,McConaughy, & Howell, 1987)McConaughy, & Howell, 1987)

ConsistentConsistent with with agreementagreement found in a meta-analysis found in a meta-analysis ofof ratings ofratings ofadults (Achenbachadults (Achenbach,, Krukowski Krukowski,, Dumenci Dumenci, &, & Ivanova Ivanova, in , in press)press)

LevelLevel of agreement provides evidence for of agreement provides evidence forreliability of WFI, but also suggests thatreliability of WFI, but also suggests thatindividual respondent scores may differindividual respondent scores may differmeaningfully across individual respondentsmeaningfully across individual respondents Considering all respondentsConsidering all respondents’’ scores individually will scores individually will

likely be important in considering fidelity levelslikely be important in considering fidelity levels

1212

Recipes for Wraparound:Recipes for Wraparound:Comparing wraparound models usingComparing wraparound models using

confirmatory factor analysisconfirmatory factor analysis

Jesse C. Suter, M.A.Jesse C. Suter, M.A.University of Vermont Department of PsychologyUniversity of Vermont Department of Psychology

Wraparound Evaluation and Research TeamWraparound Evaluation and Research Teamjesse.suter@uvm.edujesse.suter@uvm.edu

1313

OverviewOverview

Wraparound developed asWraparound developed as value-based modelvalue-based model(e.g., community-based, family-focused)(e.g., community-based, family-focused)Service Service providers have struggledproviders have struggled to translate to translatephilosophy intophilosophy into effective real-world practiceseffective real-world practicesRecent effortsRecent efforts provided more clearly specifiedprovided more clearly specifiedmodels of wraparoundmodels of wraparoundCurrent study provides an empirical test for howCurrent study provides an empirical test for howwell two wraparound well two wraparound modelsmodels fit with fit with ratings fromratings fromcaregivers on the WFIcaregivers on the WFI

1414

OutlineOutline

Models of wraparoundModels of wraparoundMethod for comparing modelsMethod for comparing modelsFindingsFindingsConclusionsConclusionsImplications and future directionsImplications and future directions

1515

Wraparound ElementsWraparound Elements

1.1. VoiceVoice and choiceand choice2.2. Team-basedTeam-based3.3. Community-basedCommunity-based4.4. Cultural competenceCultural competence5.5. IndividualizedIndividualized andand

strengths-basedstrengths-based

6.6. Natural supportsNatural supports7.7. Continuation of careContinuation of care8.8. CollaborationCollaboration9.9. Flexible resourcesFlexible resources

and fundingand funding10.10. Outcomes-basedOutcomes-based

(Burns, B. J., & Goldman, S. K., 1999).

1616

Necessary ConditionsNecessary Conditions

Team uses documentation forTeam uses documentation forcontinuous improvement andcontinuous improvement andaccountability.accountability.

5.5. AccountabilityAccountability

TeamTeam is aware of wide array ofis aware of wide array ofservices, identifies/develops naturalservices, identifies/develops naturalsupports, and designs/tailors servicessupports, and designs/tailors servicesto expressed needs.to expressed needs.

4.4. Acquiring services &Acquiring services &supportssupports

Team members capably perform theirTeam members capably perform theirroles on the team.roles on the team.

3.3. Capacity buildingCapacity building

Regular attendance and participationRegular attendance and participationby appropriate people.by appropriate people.

2.2. CollaborationCollaboration

Team adheres to practice modelTeam adheres to practice modelwhile promoting team cohesivenesswhile promoting team cohesivenessand high quality planning.and high quality planning.

1.1. Adherence toAdherence tophilosophical modelphilosophical model

(Walker, J., Koroloff, N. & Schutte, K., 2003).

1717

Steps forSteps for Model TestingModel Testing

WFI-3 caregiver interviews (n = 444)WFI-3 caregiver interviews (n = 444)Specify modelsSpecify models Each item assignedEach item assigned to only one factorto only one factor Factors allowed to correlateFactors allowed to correlateItem-level CFA using WLSMV Item-level CFA using WLSMV estimationestimation(Levent, 2004)(Levent, 2004)

RMSEA of .06 or lower indicates good fitRMSEA of .06 or lower indicates good fit CompareCompare to one-factor modelto one-factor model (parsimony test) (parsimony test)

1818

Findings: Model SelectionFindings: Model Selection

0.0590.057

0.067

0.053

0.063

0.04

0.05

0.06

0.07

10 Elements Necessary

Conditions

"g"

Model

RMSEA

items 44

items 40

Cut-off for good model fit <= .06

1919

ConclusionsConclusions

Take-home findingsTake-home findings Both models showed adequate fitBoth models showed adequate fit Both models better thanBoth models better than one-factorone-factor Removing a poorly fitting element improved fitRemoving a poorly fitting element improved fitPotential limitationsPotential limitations Selected sampleSelected sample WFI items designed for one modelWFI items designed for one model Caregiver onlyCaregiver only

2020

Implications for quality assuranceImplications for quality assuranceand program developmentand program development

Results support presenting WFI fidelityResults support presenting WFI fidelitydata for individual elementsdata for individual elements Such results can be used to support trainingSuch results can be used to support training,,

supervision, supervision, and program developmentand program development Results may be useful for assessing andResults may be useful for assessing and

addressing organizational and systemaddressing organizational and systemconditions in a communityconditions in a community

2121

Implications for researchImplications for research

Results provide evidence for validity of the WFIResults provide evidence for validity of the WFIand its underlying elementsand its underlying elements Future research may illuminate how best to combineFuture research may illuminate how best to combine

elements into subscales that possess adequateelements into subscales that possess adequatereliability (internal consistency)reliability (internal consistency)

Results provide evidence for validity of theResults provide evidence for validity of thenecessary supports modelnecessary supports model Future research may investigate relationshipFuture research may investigate relationship

between necessary conditions subscales constructedbetween necessary conditions subscales constructedfrom WFI data and Portland State RTC system andfrom WFI data and Portland State RTC system andorganizational assessmentsorganizational assessments

2222

Implications for NWIImplications for NWI

National Wraparound Initiative: supportsNational Wraparound Initiative: supportsconsideration of value-based principlesconsideration of value-based principlesduring process ofduring process of Further defining wraparoundFurther defining wraparound Developing specific phases & activitiesDeveloping specific phases & activities Creating implementation supportsCreating implementation supports Creating next generation of fidelity toolsCreating next generation of fidelity tools

2323

““Is it Wraparound Yet?Is it Wraparound Yet?””Bootstrapping Bootstrapping wraparoundwraparound

fidelityfidelity standardsstandards usingusing the WFI the WFIEric J. Eric J. BrunsBruns, Ph.D., Ph.D.

University of WashingtonUniversity of WashingtonDivision of Public Behavioral Health and Justice PolicyDivision of Public Behavioral Health and Justice Policy

Wraparound Evaluation and Research TeamWraparound Evaluation and Research Teamebruns@u.washington.eduebruns@u.washington.edu

206-685-2477206-685-2477

2424

BackgroundBackgroundFidelity measurement is the Fidelity measurement is the ““natural unionnatural unionof scientific and practical needsof scientific and practical needs”” ( (SalyersSalyerset al., 2003)et al., 2003) Assist inAssist in interpretation of study interpretation of study results,results,

determine effective components of modelsdetermine effective components of models Help programs and trainers apply QualityHelp programs and trainers apply Quality

Assurance activitiesAssurance activities Help agencies or jurisdictions with policy,Help agencies or jurisdictions with policy,

funding, certification decisionsfunding, certification decisions

2525

Fidelity benchmarks forFidelity benchmarks forwraparound?wraparound?

Wraparound model has been slow to Wraparound model has been slow to becomebecomestandardizedstandardizedNo single model can probably yet be consideredNo single model can probably yet be considered““evidence-basedevidence-based”” by traditional treatment by traditional treatmentresearch standardsresearch standardsNonetheless, WFI items are based on coreNonetheless, WFI items are based on coreprinciples that are widely accepted (and principles that are widely accepted (and recentlyrecentlyrefined)refined)Both researchers and collaboratingBoth researchers and collaboratingcommunities are interested in assistance communities are interested in assistance inininterpretinginterpreting wraparound fidelity scores wraparound fidelity scores

2626

What methods can be used toWhat methods can be used todetermine fidelity standards?determine fidelity standards?

Norm-referencedNorm-referenced Comparison Comparison toto a large, representative a large, representative samplesampleCriterion-referencedCriterion-referenced Self-referencedSelf-referenced = compare scores to your = compare scores to your

own scores over timeown scores over time Content-referenced Content-referenced = compare scores to an= compare scores to an

absolute criterion (e.g., absolute criterion (e.g., ““90% fidelity90% fidelity””)) Expectancy-referencedExpectancy-referenced = Prediction of = Prediction of

performance based on external criteriaperformance based on external criteriae.g., compare scores to a score shown to predicte.g., compare scores to a score shown to predictdesired client outcomesdesired client outcomes

2727

Current Current approachapproach

Our ability to apply the describedOur ability to apply the describedapproaches is variableapproaches is variableOurOur aim is to aim is to ““bootstrapbootstrap”” the process the processthrough a combination of methodsthrough a combination of methods Study 1:Study 1: Norm-referenced Norm-referenced study usingstudy using our our

national WFI-3 datasetnational WFI-3 dataset Study 2:Study 2: Criterion-referenced Criterion-referenced reviewreview of of

studies employing the WFIstudies employing the WFI

2828

National WFI study sampleNational WFI study sample

367490622667Total NsTotal Ns8151921Minnesota 25172427Florida

23313131Nevada27323434Missouri16222222North Carolina12232626Minnesota 129717474Massachusetts12141718Pennsylvania

212226339366Nebraska23394449Indiana

N YouthN CGN RFN familiesSite

2929

WFI Overall Fidelity ScoresWFI Overall Fidelity Scores

0.020.767Meanb0.090.80110b0.070.8009b0.110.7978ab0.100.7957ab0.090.7946ab0.120.7535ab0.090.7514a0.070.7353a0.090.7352a0.110.7221

Post-hocSDmean

Site(Rank)

0.6

0.65

0.7

0.75

0.8

0.85

0.9

Site

WF

I O

vera

ll F

idelity

sco

re

F (9,656) = 5.951, p<.0001

Mean

3030

Total WFI Results: FacilitatorsTotal WFI Results: Facilitators

0.030.805Meanb0.050.85510b0.100.8359b0.090.8278ab0.100.8207b0.080.8176ab0.170.8125ab0.090.7994ab0.070.7783a0.090.7632a0.110.7461

Post-hocRF SDRF mean

Site(Rank)

0.6

0.65

0.7

0.75

0.8

0.85

0.9

Site

WF

I-R

F T

ota

l F

idelity

sco

re

F (9,612) = 7.452, p<.0001

Mean

3131

Total WFI Results: CaregiversTotal WFI Results: Caregivers

0.040.737Meanb0.090.79210b0.140.7859b0.160.7788ab0.130.7697ab0.150.7656ab0.100.7305ab0.140.7134ab0.140.6943ab0.200.6932a0.190.6491

Post-hocCG SDCG mean

Site(Rank)

0.6

0.65

0.7

0.75

0.8

0.85

0.9

Site

WF

I-C

G T

ota

l F

idelity

sco

re

F (9,480) = 3.195, p<.001

Mean

3232

Total WFI Results: YouthTotal WFI Results: Youth

0.030.736Meanb0.040.84310b0.140.8219b0.120.7938ab0.090.7737ab0.140.7516ab0.130.7445ab0.140.7074ab0.140.6893ab0.150.6742a0.170.5651

Post-hocY SDY mean

Site(Rank)

0.6

0.65

0.7

0.75

0.8

0.85

0.9

Site

WF

I-Y

T

ota

l F

idelity

sco

re

F (9,357) = 3.659, p<.0001

Mean

3333

Summary: Mean WFI-3 site scoresSummary: Mean WFI-3 site scores

80.5

73.7 73.6

76.7

60

65

70

75

80

85

90

95

100

WFI-RF WFI-CG WFI-Y Overall Fidelity

WF

I F

idelity

3434

Study Study 2:2: Criterion-referenced reviewCriterion-referenced review

Published, in press, or formally presentedPublished, in press, or formally presentedstudies presenting WFI scores forstudies presenting WFI scores forindependent samples that either:independent samples that either:

Predict inclusion in a group orPredict inclusion in a group orAchievement of an external criterionAchievement of an external criterion

Five studies foundFive studies found Bruns, Leverentz-Brady et al., 2004Bruns, Leverentz-Brady et al., 2004 Ferguson, 2005Ferguson, 2005 Peterson et al., 2004Peterson et al., 2004 Rast et al., 2004Rast et al., 2004 Rast & VanDenBerg, 2004Rast & VanDenBerg, 2004 (Bruns, Rast, Walker, Peterson, & Bosworth, in press)(Bruns, Rast, Walker, Peterson, & Bosworth, in press)

3535

WFI scores predicting groupWFI scores predicting groupmembershipmembership

60

76

50

55

60

65

70

75

80

85

90

95

100

Trad CW Wraparound

64

75

50

55

60

65

70

75

80

85

90

95

100

Trad CW Wraparound

Ov

era

ll W

FI

sc

ore

Ferguson et al (2004): Randomizedtrial of Wraparound in California

Peterson et al (2004): Matchedcomparison study of WA in Nevada

3636

WFI scores associated withWFI scores associated withexternal criterionexternal criterion

72

84

50

55

60

65

70

75

80

85

90

95

100

Poorer supports for

WA

Greater supports for

WA

72

87

50

55

60

65

70

75

80

85

90

95

100

RF with Poorer

outcomes

RF with more

Positive outcomes

Ov

era

ll W

FI

sc

ore

Rast et al (2004): Facilitator-levelassociation between WFI and outcomes

Bruns et al (2004): Association between fidelity& program and organizational supports

3737

WFI scores for a program overWFI scores for a program overdevelopmental stagesdevelopmental stages

64

72

86

50

55

60

65

70

75

80

85

90

95

100

Pre-training Post-training Training with

coaching

Overa

ll W

FI sco

re

Rast & VanDenBerg (2004): Impact of coachingand certification on wraparound fidelity

3838

Summary: Is it Wraparound Yet?Summary: Is it Wraparound Yet?

60

7275

72

84

64

7276 77

86

64

72

80

87

50

55

60

65

70

75

80

85

90

95

100

Non-WA /

pre-training

WA

WA with

poorer

support/

outcomes

WA as usual

from comp.

studies

National

sample (min,

mean, max)

Well-

supported/

outcome-

based WA

Overa

ll W

FI sco

re

Borderline

Non-wrap

Acceptable

High-fidelity

3939

ConclusionsConclusionsThere is a discernable pattern of WFIThere is a discernable pattern of WFIFidelity scores from across studiesFidelity scores from across studies Wraparound vs. non-wraparound programsWraparound vs. non-wraparound programs Wraparound programs with different levels ofWraparound programs with different levels of

supportsupport Facilitators and/or programs that achievedFacilitators and/or programs that achieved

more positive outcomesmore positive outcomesSite-level scores from the national WFISite-level scores from the national WFIdataset show significant variability, but falldataset show significant variability, but falllogically within the patternlogically within the pattern

4040

ConclusionsConclusionsBy combining data fromBy combining data fromthese norm- and criteria-these norm- and criteria-referenced approaches, wereferenced approaches, wecan make provisionalcan make provisionalguessesguesses What is wraparound?What is wraparound? What is What is ““high-fidelityhigh-fidelity””

wraparound?wraparound?Also can be done forAlso can be done forindividual respondent WFIindividual respondent WFIscoresscoresLikely to be useful forLikely to be useful forcollaborating sites as well ascollaborating sites as well asin research studiesin research studies

4141

LimitationsLimitations

Approach truly is art as much as scienceApproach truly is art as much as science especially in the absence of outcomes dataespecially in the absence of outcomes dataWe donWe don’’t know much about thet know much about theprograms in the national sampleprograms in the national sampleInconsistent methods used acrossInconsistent methods used acrossstudies reviewed and programs instudies reviewed and programs innational samplenational sample

4242

ConcernsConcernsHow to set fidelityHow to set fidelitythresholds forthresholds forwraparound withwraparound withlocal variations?local variations?How to reconcile withHow to reconcile witha lack of modela lack of modelstandardization?standardization?Also, how to avoidAlso, how to avoidfidelity standardsfidelity standardsproviding a providing a ““ceilingceiling””as well as a as well as a ““floorfloor””??

4343

Wraparound EvaluationWraparound Evaluationand Research Teamand Research Team www.uvm.edu/~wrapvtwww.uvm.edu/~wrapvt www.depts.washington.edu/www.depts.washington.edu/

wrapevalwrapeval ebruns@u.washington.eduebruns@u.washington.edu wrapvt@uvm.eduwrapvt@uvm.edu

Tell us what you think!Tell us what you think!