Evidence-Based Home Visiting Models

to Prevent Child Maltreatment – Assessing and Addressing Fidelity

February 12Anne Duggan, ScD

Research Supported by:NIH, CDC, HRSA MCHB, ACF

Annie E. Casey Foundation, Robert Wood Johnson Foundation, David & Lucile Packard Foundation

Hawaii Department of Health, Hawaii Family Support

Institute, Alaska State Health Department, Alaska Mental Health

Trust, Family League of Baltimore City; Safe & Sound Initiative;

United Way of Central Maryland

Today’s Talk

• Describe Evidence-based Models of Home Visiting

• Identify basic components of models and associated fidelity measures

• Introduce and demonstrate use of a conceptual framework for research to increase home fidelity and impact

Evidence-based Practices

An evidence-based practice, also

called EBP, refers to an approach to

prevention or treatment that is

validated by some form of

documented research evidence.

Rand Corporation Promising Practices Criteria for Research

Evidence Proven Promising

Types of Outcomes

Program directly impacts indicators of interest

Pgm impacts inter-mediary outcomes

Effect Size1 or more outcomes

changed by > 20% or > .25 sd

Change in outcome >1%

Statistical Significance

p<.05 p<.10

Comparison Groups

RCT or quasi-experimental Comparison group, but weaker

Sample Size

N > 30, each group N > 10, each group

Three Major HV Models

PAT

Pilot (1981) MI Statewide Scale Up (1985) National Scale Up (1986) RCT (1993)

NFPRCTs (1977, 1987, 1994) National Scale Up (1996) Nonprofit Replication Org. (2003)

HHS/ HFA

Demonstration Pilot of HHSP (1985) HFA National Scale Up (1992) HHSP RCT (1992, 1993) HI Statewide Scale Up (2001)

Designations of the Three Models

PAT NFP HFA

SAMHSA Pending PendingNot

Submitted

OJJDP PROMISING EXEMPLARY EFFECTIVE

Rand Corp. PROMISING PROVEN PROVEN

Red= All 3 programsBlue= 2 programsGreen= 1 program

Adoption of PAT, NFP and HFA Home Visiting Models

PAT: >3300 sites, 50 states

NFP: 118 sites, 25 statesHFA: 430 sites, 35 states

What are the Basic Components of a Home Visiting

Model?

• Families to be targeted

• Outcomes to be achieved

• Causal chain from inputs to outcomes

PAT, NFP, HFA Target Different Families

PAT All pregnant women and families with child <6 months old

NFP First time mothers, <26 wks gestation, and <19 years old or single or low SES

HHS/ HFA

CAN Risk per Family Stress Checklist >25Hx of abuse as child; SU, PMH, Criminal Hx; Low

self-esteem, poor coping ability; multiple life stressors; violence potential; unrealistic

expectations of child; harsh punishment of child; sees child as difficult/provocative

PAT, NFP, HFA Focus on Similar Outcomes

PAT

Increase parenting knowledge and behaviorIdentify developmental delays and health issuesPrevent CANIncrease school readiness and success

NFP

Improve pregnancy outcomes; Promote child health and development; Strengthen families’ economic self-sufficiency

HHS/ HFA

Improve family functioningPrevent CANPromote child health and development

Process

Screening & Assessment

Direct Home Visiting Services

Medical Home

Other Needed Community

Services

Family FunctioningParent Mental Health, SU, IPV

Social Support

Economic Sufficiency

Child OutcomesHealth and Development

Program Model

Outreach

Trust Building

Crisis Intervention

Case Management

Parenting Education, Role Modeling, Reinforcement

Framework for Influence of Home Visiting on Family and Child Outcomes

Parenting

Knowledge/Attitudes/Skill

Parent-Child Interaction

Environment for Learning

CAN

linkages

enrollment

Measures to Assess Fidelity to the Basic Components of an

EBHV Model

• How well families are targeted

• How well outcomes are achieved

• How well each element in the causal chain is carried out

What Do We Know about Home Visiting as a Preventive

Intervention?

1. Home visiting can improve outcomes, but effects tend to be modest and variable.

From Meta-Analytic Studies: HV can be effective, but effect sizes are

small.Domain ES

Cognitive Development (41 studies) .18*

Socio-emotional Development (24) .10*

CAN Prevention (7) .32

CAN – Prevention of Potential Abuse (13) .24*

Parenting Stress (4) .21

Parenting Behavior (37) .14*

Parenting Attitudes (15) .10*

Maternal Education (5) .13*

Maternal Employment (7) .02

Public Assistance (3) -.04

ES KeySmall .20

Medium .50

Large .80

*p<.05

Sweet & Applebaum, 2002

From single studies, we see that: Effects can vary over time, & across

subgroups.

Birth – 2 Yrs

All Families10% vs. 5%, NS

Poor, Unmarried Teens

19% vs. 4% p=.07

2-4 Yrs

All FamiliesNo group difference

Poor, Unmarried Teens

No group difference

Birth – 15 Yrs

All Families0.73 vs. 0.44

p<.05

Violent FamiliesNo group difference

What Do We Know about Home Visiting as a Preventive

Intervention?

1. Home visiting can improve outcomes, but effects tend to be modest and variable.

2. Programs like HSP/HFA target

– the right families and individuals,

– at the right time,

– focusing on the right outcomes

HHS/HFA Home Visiting Model - Who is Targeted, and When

• WHO IS TARGETED– Caregivers in Families with Multiple Malleable

Risks

– Kempe Family Stress Checklist (“at risk” > 25)

• WHEN ARE THEY TARGETED?– Prenatally, at Child’s Birth, Shortly Thereafter

HHS/HFA targets the right families – those with multiple, malleable risks for poor

parenting.

18%

34%41% 41%

19%24%

1%

13%

CAN Mother

Depressed

Child Language

Delay

Poor Classroom

Concentration

Not-at-Risk Families do better than At-Risk Controls

Outcomes in 1st Grade

HV Targets Right Individuals Caregivers = Primary Influence in EC

Child’s Developmental Trajectory

Brain Development

Caregivers

Exposure down with age

Community e.g. Socioeconomic and EC SERVICES

PeersExposure up

with age

Self-Regulation

Communicating and Learning

Making Friends & Getting Along

Environment

Risk & Protective Factors

Proximal

Distal

Genotype

Source: Adapted from Tremblay, R. E. 2006. www.excellence-earlychildhood.ca

HSP/HFA Targets Caregivers at the Right Time: Early Parenting Tracks into Grade School

AOR* p

Nonviolent Discipline 4.5 <.01

Psychological Aggression 5.9 <.01

Minor Assault 5.1 <.01

Severe Assault 7.6 <.01

Neglectful Behavior 3.7 <.01

Confirmed CPS Report 6.7 <.01

*AOR for Later Use if Tactic Used Birth–3 Years

Home Visiting Focuses on the Right Outcomes: Parenting and Its

Determinants

Parent’s Developmental

History

Partner Relations

Child Attributes

Social Network

PARENTINGPersonality / Relationship

Capacity

Child Outcomes

Adaptation of Belsky’s Framework

Stresses

The quality of parent-child interaction is most vulnerable to maternal relationship insecurity

under conditions of stress.

0

1

2

3

4

5

6

7

8

9

Low Stress High Stress

ContinuousSecure

EarnedSecure

Insecure

Mean and 95% CI

Phelps JL, Belsky J and Crnic K.

1998

HHSP/HFA Targets the Right Outcomes: Association of Maternal Depression & IPV with

Severe Physical Abuse of Child, Birth – 3 Years

Depression AOR 95% CI

None 1.0 Ref.

Possible (CES-D 16-23) 1.8 0.8, 3.8

Probable (CES-D >23) 3.7 1.9, 7.2

Intimate Partner Violence

Neither injured 1.0 Ref.

Only mother injured 0.8 0.1, 6.0

No partner 4.9 2.4, 10.0

Both partner & mother injured 6.4 2.9, 14.1

HHSP/HFA Targets the Right Outcomes: Depression and IPV Other Parenting

Behaviors

Parenting

Behaviors

Depressive Symptoms

Intimate Partner Violence

OR p OR p

Poor HOME Score 1.8 .05 1.2 .38

Poor NCAST 2.2 .02 1.1 .60

Neglect 1.7 .10 2.5 <.001

Assault on Esteem 2.4 <.01 2.8 <.001

Harsh Parenting 3.2 .01 2.3 <.01

What Do We Know about Home Visiting as a Preventive

Intervention? 1. Home visiting can improve outcomes, but

effects tend to be modest and variable.

2. Programs like HSP/HFA target– the right families and individuals,– at the right time,– focusing on the right outcomes

3. But even if the model seems right, desired outcomes might not be achieved.

Overall HSP/HFA Impact was Negligiblefor Most Outcomes

Hawaii Alaska

AOR p AOR p

Maternal Depression

0.97 .84 0.66 .16

Physical IPV 0.83 .19 0.82 .43

Poor HOME Score 0.87 .39 0.51 <.001

Poor NCAST Score 0.86 .29 0.79 .31

So what’s going on? Wrong model?

Type III Error Dobson and Cooke, Evaluation & Program Planning,

1980

• Unless fidelity of implementation is determined, it is not possible to determine whether negative impact is due to:

–Inadequacies in the model or

–Departures from the model

“Every system is perfectly designed to achieve exactly the results it gets.”

Donald M Berwick, M.D.Institute for Healthcare Improvement

• Let’s look at how home visiting services were provided – let’s look at coverage, duration and frequency of visits, visit content……

Percent of Families Screened, by Hospital

8984

88

64

82

67

H1 H2 H3 H4 H5 H6

% Screened

Overall(FY00-FY07) FY 05 FY06 FY07

Number of Births 153,403 18,023 18,364 19,069

Percent Screened 72% 77% 74% 76%

Percent Screen + 52% 50% 50% 52%

Percent Assessed 81% 85% 83% 83%

Percent Assess + 50% 47% 47% 47%

Percent Referred 70% 74% 82% 77%

HSP Screening & Assessment Rates

Attrition was higher than expected and there was substantial,

unintended variation across sites. (Hawaii)

0

20

40

60

80

100

120

0 4 8 12 16 20 24 28 32 36 40 44 48 52

Weeks of Age

Perc

ent

Agency BAgency AAgency C

Mean Number of Visits

Active AllAgencyFamiliesFamilies

B 22 16 A 19 11C 28 12

p <.01 <.01

Percent of Families Active

Visit Content also Differed from the Model: HVers Often Failed to Respond to

Parenting Risks

11%

21%14%14%

32%

8%

Poor Maternal MentalHealth

Domestic Violence Maternal SubstanceUse

All Families with Risk

All Families with Risk and High Dose

Programs Varied Substantially in Provision of Core Services

Hawaii Alaska

All Sites

Range All Sites

Range

Had 1st IFSP on Time 44% 28%-62% 24% 0% - 53%

Developmental Screenings 36% 23%-67% 48% 38% - 58%

Discussion of Risks

Poor Mental Health 15% 6%-30% 45% 9% - 88%

Domestic Violence 22% 13%-40% 24% 6% - 33%

Substance Use 15% 0%-33% 30% 11% - 42%

Factors forIntegrity

Model Complexity& Clarity

ImplementationSystem

ParticipantResponsiveness

Quality of Delivery

Service Integrity• Fidelity = Coverage

Duration, FrequencyVisit Content

• Competence

Component Analysis

To identify essential components of the model

Intervention

Model

Framework of Determinants of Integrity (Carroll et al., 2007)

Outcomes

Factors for Integrity•Implementation System

(Hiring, Training, Supervision, Curricula, Protocols, Monitoring, Linkage Agreements)•Family Attributes

Process

Screening & Assessment

Direct Home Visiting Services

Medical Home

Other Needed Community

Services

Family FunctioningParent Mental Health, SU, IPV

Social Support

Economic Sufficiency

Child OutcomesHealth and Development

Program Model

Outreach

Trust Building

Crisis Intervention

Case Management

Parenting Education, Role Modeling, Reinforcement

We need to understand how family attributes and the implementation system

moderate impact.

Parenting

Knowledge/Attitudes/Skill

Parent-Child Interaction

Environment for Learning

CAN

linkages

enrollment

Influence of Home Visiting Model

Complexity and Clarity (Hawaii)

Program impact was compromised …. by drift in the model itself that had arisen in taking the model to scale.

Original CAN Prevention Program

Designation as an EI Program for Children at Risk for CHCN due

to Environmental Factors

Case Plan focused on Risks that

Made Families Eligible

Parent-driven philosophy; IFSP with family as decision-makers in

setting goals & strategies

Influence of HV Model ClarityStaff and Recipient Understanding of

PAT – Similarity

• Belief that a strong relationship was important and beneficial to the parents.

– Differences

• Perception of home visitor’s expertise

• Perception of purpose of home visitor showing the child a new activity

- From Hebbeler & Gerlach-Downie, 2002

Influence of Implementation System - Training

SB6 HVers lacked basic knowledge of child development.

24

9

30

39

46

52

Social-Emotional

Language

Selected Age Too YoungSelected Correct Age RangeSelected Age Too Old

. Tandon et al. Success by 6 Evaluation, Baltimore, 2004.

Influence of Implementation System - Training

Training improved staff knowledge, at least short term.

19

24

5

9

49

30

65

39

32

46

30

52

SE-2006

SE-2004

Lang-2006

Lang-2004

Selected Age Too YoungSelected Correct Age RangeSelected Age Too Old

Comparison of Study Results with ICMQ Validation Study Results*

12 Months 24 Months• % Agreement Validation Studies 91% 89%

This study 88% 74%

• Sensitivity Validation Studies 72% 94%This study 0% 7%

• Specificity Validation Studies 92% 83%This study 90% 89%

*Bricker D and Squires J, the Effectiveness of Parental Screeningof At-Risk Infants: The Infant Monitoring Questionnaire.Overall sensitivity 4-36 months = 63%; overall specificity = 91%

The implementation system must include SKILLS TRAINING & FEEDBACK &

COACHING. Knowledge Shown

in Training Setting

Skill Shown in Training Setting

Skill Observed in Practice

Didactic & Discussion

10% 5% 0%

Skill Demonstrated in Training

30% 20% 0%

Practice/Feedback in Training

60% 60% 5%

Coaching in Practice Setting

95% 95% 95%

Joyce and Showers, 2002

RCT of Coaching to Promote Fidelity & Impact

• Statewide random assignment of home visiting teams

• Data sources: parent interviews; record review; observation of home visitors; surveys of and in-depth interviews with home visitors and supervisors.

15 Teams Stratified by

Caseloads and Retention Rates, then Randomly

Assigned to Three Study

Group

Usual HSP Services

Training in HFT + Usual Supervision

Training in HFT + Enhanced

Supervision

Influence of Participant ResponsivenessProgram outreach - most effective in a subset of

mothers.

0

20

40

60

80

100

16 32 48 64 80 96

Maternal Anxiety Score

Perc

ent

Acti

ve in P

rogra

m

Assertive OutreachRelaxed Outreach

An assertive outreach policy promoted retention of mothers with high relationship anxiety.

Home Visiting Impact Was Pronounced inMothers with High Relationship Anxiety

19%

57%

19%

31% 33%

58%

26%

68%

34% 32%

ALL AR Neither

High

Anxiety

High

Avoidance

High

Both High

Example: Depressive Symptoms

Where We Go from Here…

– Learn What Works, for Whom

– Improve Fidelity•Clarify existing models

•Build implementation system infrastructure

•Understand providers and recipients

•Build basic skills

– Enhance Home Visiting Models

Where We Are Going in Hawaii…

5-year ACF-funded project Staff training and supervision

CQI capacity

Targeting of families

Enhancements to the model

Reconciliation of funding incongruities