Adoption of Evidence-Based Primary Prevention Programs: Identifying

Characteristics of Ready, Willing, and Able Schools

Paul D. Flaspohler, Ph.D., Cricket Meehan, Ph.D.,

& Vanessa Watts, M.A.

Miami University

Kathryn E. Keller, MPA

Health Foundation of Greater Cincinnati

Learning Objectives

• To identify key features in determining a school's readiness and capacity to implement and sustain an evidence-based prevention program (EBP) targeting school violence

• To understand how to assess a school's readiness and capacity to implement and sustain an EBP

• To describe the promise of EBPs and related services and to describe the challenges inherent in dissemination of EBPs and related services

Working with the “Ready”

Matching Intervention to

Need

Mainstreaming Evaluation and Accountability

Building a Community of

Practice

6678 KEY STAKEHOLDERS Evidence of Need

Children (N=6678) in 15 different elementary and middle schools

0

200

400

600

800

1000

1200

1400

7 8 9 10 11 12 13 14

Age

0

200

400

600

800

1000

1200

1400

3 4 5 6 7 8

Grade

Ethnicity by School

0

100

200

300

400

500

600

A B C D E

Asian/Indian/PI

African American

Hispanic

White

SOME CHALLENGES Evidence of need

Barriers to Successful Development

I feel safe in my school 17 % strongly disagree/disagree

I worry a lot 59.7 % certainly/somewhat true

I have many fears 43.4 % certainly/somewhat true

I get mad easily 28.4 % most, all of the time

I fight a lot 28.4 % certainly/somewhat true

I like myself 22.7 % never/some of the time

I feel lonely 19.7 % most/all of the time

Other children pick on me 33.4 % somewhat/certainly true

SOME FORM OF UNIVERSAL PREVENTION COULD MEET NEEDS OF MORE THAN 3000 “AT RISK” CHILDREN IN SCHOOLS

WHAT DOES THE RESEARCH TELL US ABOUT MEETING THAT NEED?

The Research to Practice Gap Practice

• Community Systems

• Schools

• Health Agencies

• Community Coalitions

Science

• Basic research

• Efficacy

• Effectiveness

• Services Research

RESEARCH TO PRACTICE AND COMMUNITY CENTERED MODELS

Bridging the Gap

Research to Practice models

• Begin with basic research questions

• Based on generation of generalizable knowledge

• View “best practice” in terms of discreet programs and strategies to be adopted as intended in the original

Problems with IOM Model

• Mismatch between Innovation design and community capacity

• Mismatch between community/organization ideology and values that underlie a Innovation

• Pro-Innovation Bias

• Simplistic models of decision making

– (Miller & Shinn, 2008)

Community centered models

• Start with the community

• Focus attention on local needs (contextual knowledge)

• See “best practice” as a process rather than magic bullet programs

• Emphasize that practitioners, clients, and communities should have control as well as accountability

Problems with CC Models

• Rarely evaluated

• When evaluated, many prove not to work. Some prove worse than nothing at all.

• Often re-invent the wheel

CAPACITY In both research to practice models and community centered models

B. Distilling the Information—Prevention Research System

C. Supporting the Work—Prevention Support System

A. Putting It Into Practice—Prevention Delivery System

Synthesis

General Capacity

Building

Innovation-Specific

Capacity Building

General Capacity

Use

Innovation-Specific

Capacity Use

Policy Climate

Funding

Existing Research and Theory

Translation

Effective Dissemination/ Implementation requires attending to:

Dissemination/ Implementation

Characteristics of the

Innovation

Process of Transitioning Innovation to

Setting

Characteristics of the Setting

Practice

• Practice-Based Evidence

Best Practice Process

• Local control

• Self-evaluation

• Tailoring process and technology

Research

• Evidence –Based Practices

Bridging the gap

The SWEBP Process

Pre-planning

Planning

Implementation

Evaluation

Level 1 Grant

Level 2 Grant

Working with the “Ready”

Matching Intervention to

Need

Mainstreaming Evaluation and Accountability

Building a Community of

Practice

Working with the “Ready”

Core Team RFP Workshop

Capacity/ Readiness

Assessment

Planning Grant Proposal

Core Team – The Right People Administration

Principal

Superintendent

Education

Teacher

Health/ Mental Health

Counselor

School Psych

School Social Worker

Nurse

Other

Other

Parent

Community Member

RFP Workshop – Initial Commitment

• Core team must attend

• Access to Capacity/Readiness Assessment

• Access to Proposal

Capacity/Readiness Assessment

Can you use data?

• Readiness Assessment includes: – Collective Efficacy

– School Characteristics

– Strength-Based Practices

– School Readiness

– Life Satisfaction

Spreading the word…

• Staff complete assessment online

• Data is collected, aggregated and distributed by TA team

• Schools receive aggregate data to use in planning grant

Readiness Indicator: Staff Participation

0

10

20

30

40

50

60

70

80

90

100

Yr 2006 Yr 2007

Yr 2008

% Staff Response (funded)

% Staff Response (not funded)

Application Review Criteria • Eligibility (y/n)

• Commitment to EBP (20)

• Strength of Argument (10)

• Staff participation % (10)

• Survey Results (20)

• Behavioral Health Problems (10)

• SED Students (10)

• Poverty – F/R Lunch (5)

• Addresses Disparity (5)

• Budget (10)

Reducing applicant pool from 300+ Schools

0

2

4

6

8

10

12

14

16

18

20

2006 2007 2008

The SWEBP Process

Pre-planning

Planning

Implementation

Evaluation

Level 1 Grant

Level 2 Grant

Planning Phase

Needs and Resource

Assessment

Problem Identification

Program Selection/ Adaptation

Implementation Grant Proposal

Needs and Resource Assessment

• Informants – Staff Survey

– Student Survey • 3-5 grade; 6-8 grade

– Parent Survey

• Content (CASEL+) – Classroom behaviors

– Socio-emotional functioning

– School Climate

– Bullying and Violence

– Others

• Technical Assistance

– Consent/Parent Notification

– Packaging

– Administration (limited)

– Data entry • Remark

• Prezza

• TurningPoint

– Data analysis

Problem Identification

• Data informed

• Combine N/R data with other school data sources

NO MAGIC BULLETS

Students: Strengths and Difficulties

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

Rural Elementary

Rural Middle Urban Elementary

Urban Middle

emotion

conduct

Hyper

peer prob

Prosocial

Staff: Disruptive Behavior Problems

0

10

20

30

40

50

60

A B C D E

Staff: School Orientation to Innovation

0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

A B C D E

Program Selection/Adaptation

• Must be EBP

– Blueprints

– NREPP

– Whatworks.gov

• Consider

– Need

– Readiness

– Feasibility

• Plan for adaptation to local needs

Rea

dine

ss

Fe

asib

ility

Evidence of Need

EBPs

Olweus Bullying Prevention Program

PATHS

Life Skills Training

Caring School Communities

Project Achieve

Peacebuilders

The SWEBP Process

Pre-planning

Planning

Implementation

Evaluation

Level 1 Grant

Level 2 Grant

Implementation Phase

Program training

Ongoing technical

assistance

Monitoring process

Monitoring outcomes

Program Training

• “Innovation Specific” Capacity building – Training provided by certified program experts

• Coordinated where possible to reduce costs

• TA team serves as buffer between trainers and schools

Ongoing Technical Assistance

• Proactive and Reactive strategies to promote general capacity

– Using Data

– Parent Engagement

– Culturally Competent Adaptation

– Sustainability

• Quarterly meetings of programs in implementation phases to celebrate success and problem solve

Local evaluation

• Process monitoring

• Fidelity monitoring

• Outcomes monitoring As planned

With Quality

Outcomes

Outcome Evaluation

• Innovation Specific (program) Outcomes

• Common Outcomes

– Quality of Life

– Strengths and Difficulties

– Organizational Capacity

HOW DO YOU KNOW YOUR EFFORTS ARE WORKING, PAUL?

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

PATHS PATHS PATHS Olweus Olweus LifeSkills LifeSkills CSC CSC

First Year Fidelity

Two year fidelity

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

PATHS Olweus Olweus LifeSkills

Year 1

Year 2

The SWEBP Process

Pre-planning

Planning

Implementation

Evaluation

Level 1 Grant

Level 2 Grant

Evaluation Phase

Continuous Improvement

Sustainability

LESSONS LEARNED... Strengths:

• 100% process and outcomes monitoring!

• can help to raise awareness of the complex nature of planned change processes

• engagement in the process builds general capacity to participate in data-informed decision making

Challenges

• Are we being told what we want to hear?

• Working “the Cream”?

• Unsolvable systems problems?

• Sustainability of our efforts?

Next steps…

Agricultural Extension Model Professional Development Model

Best Practice Process involves

• Developing a cadre of skilled change agents to drive the work

– “the more I see, the less I know”

• Embracing the paradox – dynamic equilibrium

• Resisting “umbrage”

"Good judgment

comes from

experience

...and a lot of that

comes from bad

judgment.“ Will Rogers

THE HEALTH FOUNDATION OF GREATER CINCINNATI GRANTEES

Ready, Willing, and Able Schools

The Need Identified From Our Grantees’ Applications…

• “The socioeconomic status [of our families] often leaves students in a home environment that is deficient in stability. As a result, children are not taught social and emotional skills needed to succeed in school and social settings.”

• “Teachers are constantly correcting inappropriate behavior and modeling good social behaviors, which interferes with academic teaching time.”

• “Students come to school tired, hungry, and unkempt.”

The Need Identified From Our Grantees’ Applications…

• “[Our school] is making cuts in basic programs, staff and services in order to balance the budget deficit of $629,599.”

• “Mental health services in [our county] are few and fragmented in their accessibility and availability.”

• “[Our school] is located in a small isolated town. No public transportation exists in the town and resources are difficult to access and not within walking distance.”

WE HAVE THE NEED, BUT ARE WE READY?

The Health Foundation of Greater Cincinnati Grantee Schools

The Right People…

• “The staff at [our school] is dedicated to helping their students achieve both academic and social success. They are supportive of mental health programming within the school.”

• “The staff have expressed a desire to help students with issues such as social skills, anger management, peer socialization, respect for authority, peers, and themselves.”

• “The administration was very positive regarding implementing a school wide behavior program.”

Ability to Implement…

• “The planning team members are completely dedicated and invested in finding a program that will aid in preventing problems in the future as well as assisting student achievement in academics and with life.”

• “Once a program is identified as being a viable option, the planning team will visit schools that have implemented the program.”

• “The program, once selected, will have a special kick-off to jump start and encourage enthusiasm for the program.”

RESULTS FROM OUR GRANTEES Evidence of Ready, Willing, and Able Schools

Funded Schools, “Ready” for EBP

• Funding status (funded vs. not funded) was associated with a significant effect on the readiness variables, using Wilk’s Lambda, F(12, 984) = 1.974, p = .024

Believe EBP will be beneficial

Believe EBP will make their work easier

Believe now is good timing for EBP

Funded Schools, “Willing” to Work Together

• Funding status (funded vs. not funded) was associated with a significant effect on the individual teacher collective efficacy variables, using Wilk’s Lambda, F(12, 434) = 2.797, p = .001

Students come to school ready to learn

Teachers have the skills needed to produce meaningful learning and discipline effectively

Teachers have the ability to get through to difficult students

Home life contributes to students’ learning

Funded Schools, “Able” to Succeed

• Funding status (funded vs. not funded) was associated with a significant effect on the capacity variables, using Wilk’s Lambda, F(17, 424) = 2.501, p = .001

Openness

Friendliness

Stability

Safety

Valuing relationships

Caring about families

Valuing parents ideas

Well-regarded

Proactiveness

Coordination

Clearness of mission & goals

Focus on improvement

Helping students succeed

Performance

QUESTIONS?

Colleagues

• Abe Wandersman (USC) – Matt Chinman

– Pam Imm

– Dana Keener

– Kelly Lewis

– David Fetterman

• Funders – Kay Reitz (ODMH)

– Kate Keller (Health Foundation of Greater Cincinnati)

– Janet Saul (CDC)

– Kathy Lesesne (CDC)

• MHEDIC – Mark Weist

– Seth Bernstein

– Karen Weston

– Steve Evans

• OMHNSS-EPIC – Julie Owens (OU)

– Dave Julian (OSU)

– David Hussey (KSU)

– Betsy Tracey (CWRU)

– Eric Dubow (BGSU)

• CSBMHP • Carl Paternite

• Bob Burke

• Dawn Anderson-Butcher

• Amy Wilms

• Cricket Meehan

Graduate Students

• Melissa Maras

• Chris Reiger

• Rochelle Fritz

• Angie Ledgerwood

• Jennifer Elfstrom

• Raven Cuellar

• Holli Sink

• Dana E. Crawford

• Julie Platten

• Anna Hung

• Joel Gaffney

• Liz Brown

• Jim Mosher

• Kathy Conaway

• Gillian Finocan

• Karin Vanderzee

• D’Arcy Reynolds

• Tommy Koopman