Delaware's BEST Logic Model

Navigation Tutorial

• Click to return to the Main Menu

• Click to view Evaluation Outcomes

• Click to skip to the end of an animation

• Click blue text at the bottom of the screen to view additional information about selected outcomes

• Click buttons to view more detailed information about a particular measure, or to view graphs and data

Click Me

Acknowledgements

The development of this logic model and the corresponding data was a collaborative effort involving many partners and stakeholders including the BEST grant team and the evaluation committee. In particular the following individuals were instrumental: Marissa Smith Timothy Fowles Cathy Imburgia Mary Moor

Delaware’s B.E.S.T.

for Young Children and Their Families

Logic Model & Outcomes

Delaware Children’s DepartmentDivision of Prevention and Behavioral Health

Services

Main Menu:What would you like to do?

Soccer & Logic Model Menu

Evaluation Outcomes

Soccer and Logic Model

View Soccer Model Piece-by-Piece

Label Components of Soccer Model View Logic Model

Soccer Model Master

Delaware’s B.E.S.T. for Young Children and Their FamiliesDelaware Children’s Department | Division of Prevention and Behavioral Health Services

Home Opponent5 0

System of Care

Soccer Model Master


Home Opponent5 0

System of CareChildren and

Families

Soccer Model Master


Home Opponent5 0

System of Care

Providers

Soccer Model Master


Home Opponent5 0

System of Care

B.E.S.T. Team

Soccer Model Master


Home Opponent5 0

System of CareAgencies,

Organizations, Stakeholders

Soccer Model Master


Home Opponent5 0

System of CareBroader

Community

Soccer Model Master


Home Opponent5 0

System of CareLimits of System

(e.g., Geography, Resources)

Soccer Model Master


Home Opponent5 0

System of CareEvaluation

Soccer Model Master


Home Opponent5 0

System of Care

Evaluation Outcomes

Providers

Children and Families

Agencies, Organizations, Stakeholders

Broader Community

Satisfaction with Trainings | Adoption of Evidence-Based Practice| Fidelity to ModelBurnout

Evaluation Outcomes

Providers

Children and Families Broader Community


Strong & Representative Family Organization | Awareness of Child MH Issues & ResourcesStigma

Evaluation Outcomes

Providers



Broader Community

Funding & Resources for Child Mental Health | Publicity | Access to ServicesUnnecessary Duplication of Services | Systemic Barriers

Evaluation Outcomes

Providers



Broader Community

PreBERS: Family Involvement

Homework Compliance

DPICS

xx

Child and Family Outcomes


• Observational Coding Measure• Completed by therapist during session• Caregiver Interaction Skills• Positive “Do” Skills• Reflections, Labeled Praise,

Behavioral Descriptions• Negative “Avoid” Skills• Commands, Questions, Negative Talk



Homework Compliance

DPICS

xx




Homework Compliance

DPICS

xx

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 170123456789

10

PositiveNegative

Session Number

Ave

rage

DPI

CS S

core

s

Combined Home-based and Outpatient Sessions up to 8/1/2012; N Sessions = 2149


xx

• Parent report of completed homework collected at every session• Number of days completed HW• Possible number of days to

complete HW

Child and Family OutcomesDPICS

Homework Compliance



xx

(HW compliance graph here)


Homework Compliance



xx

• Parent-report measure• 42 items + 7 open-ended questions• Identifies a child’s:• Strengths• Emotion regulation• School readiness• Language and attention skills• Social confidence


Homework Compliance



xx

(PreBERS graph here)



Homework Compliance


DECA: Self Regulation

DECA: Self-Control

DECA: Attachment

DECA: Initiative

PreBERS:Emotion Regulation

PreBERS: Social Confidence



• Parent-report measure• Three versions:

• DECA 1–18M (33 items, 5 minutes)• DECA 18–36M (36 items, 5 minutes)• DECA 2–5Y (37 items, 5 minutes)

• Assesses:• Protective factors• Initiative• Self-regulation• Attachment/relationships• Self-control


DECA: Self-Control

DECA: Attachment

DECA: Initiative





20

25

30

35

40

45

36.76

41.06 40.83

Time

Att

achm

ent

*Graph shows t-scores

Clinical concern

p <.05 n.s.

N = 41 N = 31 N = 18


DECA: Self-Control

DECA: Attachment

DECA: Initiative









DECA: Self-Control

DECA: Attachment

DECA: Initiative





Baseline 6 Month Follow-up 12 Month Follow-up20

25

30

35

40

45

34.29

43.06 42

Time

Self-

Cont

rol

Clinical concern

p <.05 n.s.

*Graph shows t-scores


DECA: Self-Control

DECA: Attachment

DECA: Initiative









DECA: Self-Control

DECA: Attachment

DECA: Initiative





• Graph here


DECA: Self-Control

DECA: Attachment

DECA: Initiative









DECA: Self-Control

DECA: Attachment

DECA: Initiative





• Graph here


DECA: Self-Control

DECA: Attachment

DECA: Initiative





Child and Family Outcomes• Parent-report measure• 42 items + 7 open-ended questions• Identifies a child’s:• Strengths• Emotion regulation• School readiness• Language and attention skills• Social confidence

DECA: Self-Control

DECA: Attachment

DECA: Initiative





• (Graph here)


DECA: Self-Control

DECA: Attachment

DECA: Initiative






DECA: Self-Control

DECA: Attachment

DECA: Initiative





• (Graph here)


DECA: Self-Control

DECA: Attachment

DECA: Initiative





EQ-R2: Academic Performance

EQ-R2: Achievement

PreBERS: School Readiness

xx



xx



EQ-R2: Achievement



xx

• (Graph here)



EQ-R2: Achievement



xx

• Parent-report measure• 15 questions with subparts• Assesses child’s functioning in school• Topics covered:• School attendance/grade level• Special placements and plans• School achievement• Disciplinary actions



EQ-R2: Achievement



xx

• (Graph here)



EQ-R2: Achievement



xx

Child and Family Outcomes• Parent-report measure• 15 questions with subparts• Assesses child’s functioning in school• Topics covered:• School attendance/grade level• Special placements and plans• School achievement• Disciplinary actions


EQ-R2: Achievement



xx

• (Graph here)



EQ-R2: Achievement



CCSP-R: Cultural Competence

Multi-Sector Service Contacts

Youth Services Survey: Families

xx



• Parent-report measure• 27 items• Measures perceptions of service across

five domains:• Access• Participation in treatment• Cultural sensitivity• Satisfaction• Outcomes

Participation in Tx

Access to Services

Cultural Sensitivity

Satisfaction

Child and Family OutcomesYouth Services

Survey: Families




• Access to services graph here


Participation in Tx

Access to Services


Satisfaction





• Participation in Tx graph here


Participation in Tx

Access to Services


Satisfaction





• Cultural sensitivity graph here


Participation in Tx

Access to Services


Satisfaction





• Satisfaction w/ services graph here


Participation in Tx

Access to Services


Satisfaction





• Parent-report measure• 47 items with subparts• Report of child and family service use:• Services used• Service locations and convenience• Quantity of services received• Satisfaction with services• Payments for services

Convenient Location

Services Received

Quantity Received

Satisfaction

Length of Treatment

Continuity in Care






• Services received graph here


Convenient Location

Services Received

Quantity Received

Satisfaction

Length of Treatment

Continuity in Care





• Location graph here


Convenient Location

Services Received

Quantity Received

Satisfaction

Length of Treatment

Continuity in Care





• Quantity graph here


Convenient Location

Services Received

Quantity Received

Satisfaction

Length of Treatment

Continuity in Care





• Satisfaction graph here


Convenient Location

Services Received

Quantity Received

Satisfaction

Length of Treatment

Continuity in Care





• Continuity in care graph here


Convenient Location

Services Received

Quantity Received

Satisfaction

Length of Treatment

Continuity in Care





• Length of tx graph here


Convenient Location

Services Received

Quantity Received

Satisfaction

Length of Treatment

Continuity in Care





xx

• Parent-report measure (no staff as caregiver version)

• 13 items• Assesses:• Importance of culture in the lives of the

respondent• Caregiver’s perspective on provider’s

consideration of family’s beliefs, traditions, and practices when providing services






xx

• CCSP-R Graph here






x

National Evaluation

xx


x

xx


x

xx


x

xx

6 months

12 months

18 months

24 months

Grand total

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Average of DE Reten-tion (%)Average of National Retention (%)

National Evaluation



xx


Although this is considered a key outcome, no data currently are available.

x

EQ-R2

xx



xx

Expulsions

Suspensions

Child and Family Outcomes• Parent-report measure• 15 questions with subparts• Assesses child’s functioning in school• Topics covered:• School attendance/grade level• Special placements and plans• School achievement• Disciplinary actions

EQ-R2


xx

• Suspensions graph here

Child and Family OutcomesEQ-R2

Expulsions

Suspensions


xx

• Expulsions graph here

Child and Family OutcomesEQ-R2

Expulsions

Suspensions


x

Parenting Stress Index (PSI)

Caregiver Strain Questionnaire

xx



x

xx

• Parent-report measure (no staff version)

• 21 items• Assesses difficulties, strains, and

other negative effects related to caring for a child with emotional and behavioral problems





x

xx

• CGSQ graph here





• Parent-report measure (no staff version)• 36 items• Assesses:• Level of parental distress• Parent-child dysfunctional interaction• Difficult child characteristics

CG-Child Interax

Parental Distress

Difficult Child





• Graph: PSI Parental distress here


CG-Child Interax

Parental Distress

Difficult Child




• Graph: Dysfunctional Parent-Child Interactions here


CG-Child Interax

Parental Distress

Difficult Child




• Graph: PSI Difficult Child here


CG-Child Interax

Parental Distress

Difficult Child




x

Child Behavior Checklist (CBCL)

ECBI

xx



Problem

Intensity • Parent-report measure of child disruptive behavior• Intensity score• Problem score

• 36 items• Completed at every PCIT session



ECBI



1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 1790

100

110

120

130

140

150

Intensity

Session Number

Ave

rage

ECB

I Sco

res

Range of Significant Concern

Combined Home-based and Outpatient Sessions up to 8/1/2012; N Sessions = 2149

Problem

Intensity


ECBI



There was a significant decrease in ECBI Problem scores, β10 = -.56 (SE = 0.09), p < .001, and this change was not statistically different between treatment groups, β11 = -.04 (SE = 0.13), p = .76.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 205.00

7.00

9.00

11.00

13.00

15.00

17.00

19.00

21.00

23.00

25.00

Outpatient

Home Based

Clinically significant

Problem

Intensity


ECBI


• Standardized measure of children’s symptomatology• Caregivers’ report of their child’s problems,

disabilities, and strengths• Caregivers say how often statements describing

behaviors are true for their child• 17 social competence items, 113 behavior problem

items• Requires approximately 20 minutes to complete• Yields result scales in three areas:

• Syndrome Scales• DSM-Oriented Scales• Externalizing, Internalizing, and Total Problems

scales

Attention Problems

Aggressive Behavior

Externalizing

ADHD Problems

ODD Problems



ECBI


Baseline 6 Month Follow-up 12 Month Follow-up60

62

64

66

68

70

72

74

71.7

65.55

63.56

Time

Agg

ress

ive

Beha

vior

p <.05

n.s.

N = 54 N = 31 N = 18

Clinically significant

Child Behavior Checklist 1.5-5 (CBCL 1.5-5): Aggressive Behavior subscale (t-

scores)

Attention Problems

Aggressive Behavior

Externalizing

ADHD Problems

ODD Problems


ECBI


Child Behavior Checklist 1.5-5 (CBCL 1.5-5): Attention Problems subscale (t-scores)

• Graph: Attn Prblms here

Attention Problems

Aggressive Behavior

Externalizing

ADHD Problems

ODD Problems


ECBI


Child Behavior Checklist 1.5-5 (CBCL 1.5-5): Externalizing subscale (t-scores)

• Graph: Externalizing here

Attention Problems

Aggressive Behavior

Externalizing

ADHD Problems

ODD Problems


ECBI


Child Behavior Checklist 1.5-5 (CBCL 1.5-5): ADHD Problems subscale (t-scores)

• Graph: ADHD Problems here

Attention Problems

Aggressive Behavior

Externalizing

ADHD Problems

ODD Problems


ECBI


Child Behavior Checklist 1.5-5 (CBCL 1.5-5): Oppositional Defiant subscale (t-scores)

• Graph: Oppositional Defiant Problems here

Attention Problems

Aggressive Behavior

Externalizing

ADHD Problems

ODD Problems


ECBI


x

By Completion Status

xx

Child and Family Outcomes: Attrition


x

xx

Child and Family Outcomes: Attrition

8%

16%

10%

67%

Not Engaged Dropout CDI Dropout PDI Completer

28%

26%13%

33%

Outpatient CSM

c2 (3, N = 169) = 21.20, p < .001

By Completion Status


x

Comparison to Other Trainings

Overall Training Satisfaction

xx

Provider Outcomes


x

xx

• Therapists were surveyed following completion of a 40-hour PCIT training

Provider Outcomes




x

xx

Trainer Didactics Family Sessions

Videos Role-Play Manuals123456789

10

Post-Training

Dissatisfied

Satisfied

Provider Outcomes




x

xx

• Therapists were asked to rate how their experience with the 40-hour PCIT training compared to their experience with previous trainings

Provider Outcomes




x

xx

Much Better67%

Better22%

Neutral11%

Post-Training

None indicated “Worse” or “Much Worse”

Provider Outcomes




x

CSM Service Location

xx

Provider Outcomes


x

CSM Service Location

xx

Provider Outcomes

Interactive CSM Client Map


http://batchgeo.com/map/d46293290bc087ff7937666cb7263014

x

xx

Provider Outcomes


x

xx

Community Outcomes


xx

Community Outcomes

Curriculum Evaluation

Pre-Awareness Week Knowledge

Location of Awareness Program

• Families who attended Mental Health Awareness Week completed an online survey responding to the following items


xx

Community OutcomesLocation of

Awareness Program

Sussex County (3)

Kent County (6)

New Castle County (25)




Community OutcomesLocation of

Awareness Program

Awareness Week

Prior Participation


• Individuals responded to the following items keeping in mind how they felt before participating in Children’s Mental Health Awareness Week




Community Outcomes

Awareness Week


I knew that Awareness Week existed

Strongly Disagree (9)Disagree (10)Neither Agree nor Disagree (5)Agree (9)Strongly Agree (4)




Prior Participation


Community Outcomes

Awareness Week


I knew that Delaware's BEST for Young Children existed





Prior Participation


Community Outcomes

Awareness Week


I have participated in Awareness activities before





Prior Participation


Community Outcomes

Usefulness of Materials

Families’ Opinions

Children’s Opinions

Easy to Understand

• Individuals responded to the following items keeping in mind how they felt about the program curriculum





Community Outcomes

The curriculum materials were useful

Neither Agree nor Disagree (1)

Agree (18)

Strongly Agree (18)

*None responded ‘Strongly Disagree’ or ‘Disagree”





Easy to Understand




Community Outcomes

I understood how to use the curriculum ma-terials

Disagree (1)Neither Agree nor Disagree (0)Agree (19)Strongly Agree (19)

*None responded ‘Strongly Disagree’





Easy to Understand




Community Outcomes

The families liked the curriculum

Neither Agree nor Disagree (15)Agree (15)Strongly Agree (7)






Easy to Understand




Community Outcomes

The children liked the curriculum

Neither Agree nor Disagree (4)Agree (15)Strongly Agree (19)






Easy to Understand




x

xx

Community Outcomes


x

xx

Agency Outcomes


x

MSSC-R

xx

Agency Outcomes


x

xx

Agency Outcomes

• Parent-report measure• 47 items with subparts• Report of child and family service use:• Services used• Service locations and convenience• Quantity of services received• Satisfaction with services• Payments for services

**Listed in Child and Family Outcomes section

MSSC-R


x

xx

Agency Outcomes

• Graph hereMSSC-R


The “Logic”

Population/Context

OutcomesServices

Families

Providers

Logic Model

OutcomesServices

OutcomesServices

Agencies

OutcomesServices

One person’s service is another’s outcome…

Contact Us:

Tim FowlesDepartment of

PsychologyUniversity of

Delaware108 Wolf HallNewark, DE [email protected](302) 781-3213

Delaware’s B.E.S.T.261 Chapman Rd.Newark, DE [email protected]: (302) 781-

3210

mailto:[email protected]



Delaware's BEST Logic Model

Data & Analytics

Transcript of Delaware's BEST Logic Model