IMPLEMENTING AND SUSTAINING EVIDENCE-BASED PROGRAMS

WITH INTEGRITY: GOAL ALIGNMENT BETWEEN PROGRAMS AND SETTINGS

Brian R. Flay, D.Phil.

Public Health, Oregon State University

Presented at Forum on Emphasizing Evidence-Based Programs for Children and Youth

Child Trends, Washington DC, April 27 2011

WORKING WITHIN AN ORGANIZATION RATHER THAN CHANGING THE WHOLE

ORGANIZATION • I focus on the idea of inserting a program into a

setting rather than reforming the setting – Many reading and math education programs– Many substance use prevention programs

• Life Skills Training, ALERT

– Many violence prevention programs• Second Step

– Programs to prevent multiple problem behaviors• Aban Aya

– Many social-emotional and character development (SACD or SECD) programs• Positive Action, PATHS

WHAT MAGNITUDE OF EFFECTS

CAN BE EXPECT? • Contrast efficacy and effectiveness trials (Flay, 1986)• Meta-analyses suggest that the effects of programs

evaluated under real-world conditions are generally smaller than the effects of the same program evaluated under controlled conditions– Lipsey (2011, personal communication):

• “We do regularly find smaller effect sizes for ‘routine practice’ programs than ‘research and demonstration’ programs, though have not always commented on that in our published papers.”

– Lipsey (1999): • ES for demonstration programs = .13; • ES for practical programs = .07 (though wide variability)

– Wilson, Lipsey & Derzon (2003): • ES for demonstration programs = .25; • ES for routine-practice programs = .10 (but very small N of routine practice

program evaluations)

WHY DOES THIS DIFFERENCE EXIST?

• Efficacy trials are often conducted by the same people who developed the program who also maintain a high level of control over the intervention delivery as well as the research design – Thus, they ensure a high level of implementation with integrity (but

rarely help to sustain a program!)

• Effectiveness trials may be conducted by third parties, who have less control over program implementation and, possibly, lower motivation to ensure program integrity – They also often have to train implementers who may be less

motivated to do the particular program well.

• Can this be changed?– Probably – by ensuring optimum implementation!

OPTIMIZING IMPLEMENTATION AND SUSTAINABILITY

• Most of the necessary conditions are the same as for comprehensive setting change (Slavin, this forum; Slavin & Madden, 2007; Wandersman, this forum; Wandersman, et al., 2008)– Support of the whole setting– Be true to the model– Investment and commitment– Ongoing professional development

• One condition is different– Alignment with overarching goals of the setting

• E.g., the goals of Success For All are automatically aligned with the goals of schools

SUPPORT OF THE WHOLE SETTING

• Readiness for change (Fixsen, this forum)• Leadership (e.g., principals)

– Need an ongoing champion as well as someone who monitors implementation amount and integrity

– But must avoid creating a “cult” – that might help short-term compliance but not long-term commitment

• Implementers (e.g., teachers) – [Though they are not always the best judges of what

programs to adopt!]– Needs to be a team effort – implementers need to

agree to the adoption of a program, be encouraged to work with each other, problem-solve together, etc.

SUPPORT OF THE WHOLE SETTING (CONTINUED)

• Recipients (e.g., students and their families)– For school programs, also need student and family

buy-in– Need for quick and easily observable

improvements/successes• Perceptions of ineffectiveness will bring it down

– Make sure program goals are relevant to receivers• E.g., students don’t always see the value of academic

achievement – they won’t be motivated to achieve if they see getting good scores as being mostly for the benefit of the teacher or the school

INVESTMENT IS ESSENTIAL

• Settings that pay for program materials, training and ongoing support usually implement better

• Continuous investment is necessary!– Refresh consumable materials– Ongoing tech support and training

• Continuous purchase of support and training from program developers is also helpful– Homegrown or third-part materials or training are

usually inferior and likely to be less effective

• Ongoing investment is a good indicator of ongoing commitment by decision-makes

BE “TRUE TO THE MODEL”

• Integrity with the original theory and key concepts of a program must be maintained if new settings/places are to obtain the same kinds of results as found in prior evaluations– Program elements cannot be re-invented for each setting

• Adaptation for the culture or context can be useful, but the key elements cannot be changed or dropped– Castro et al., 2010; Castro, this forum

• The intensity, breadth, comprehensiveness, etc. of the original program have to be maintained (Elliott & Mihalic, 2004; Mihalic, this forum)– No short-cuts– Unless there is evidence – Or implementers must evaluate the altered version

ONGOING PROFESSIONAL DEVELOPMENT

• Initial training is essential– Because of increased pressures/mandates, staff require extensive training,

even for well-packaged, easy to implement programs– Reviews of the effectiveness of the train-the-trainer (TTT) model are mixed

• “Little is known” (Herschell, et al., 2010)• TTT is less effective (ES = .09) than other approaches (ES = .20) (Conn et al., 2011)• Expert training is more cost-effective than TTT (Olmstead, et al., 2011)

• On-going technical support is also essential (Fixsen et al., 2005)– Staff seem to need a lot more hand-holding than they used to (again

because of increased demands)• Annual (re)training

– Refresher training for ongoing staff and new training for new staff (staff turnover is a big issue)

– Implementers are often starting from a low level of understanding and skill - they need ongoing training just like students need ongoing teaching

• You don’t learn most things from one learning session – you need practice, feedback, retraining for re-alignment, etc.

Alignment with the Overarching Goals of the Setting• This condition underlies all of the above – to the

extent that this is true, the others should follow.• Must become legitimate at all levels of the system

– E.g., link prevention program outcomes to academic achievement• Claims must be supported by high-quality evaluation evidence

– E.g., school-based or after-school social-emotional or prevention programs must be able to demonstrate that they also improve academics• Other research suggesting the link is not adequate to motivate

leadership or implementers – too big of a leap of faith– This was a big failure of substance use prevention researchers and

program developers and of the Safe and Drug Free Schools program

ALIGNMENT WITH THE OVERARCHING GOALS OF THE SETTING (CONTINUED)

• Programs need to quickly produce “easy to see” and desired effects quickly– E.g., improved classroom behavior, followed by improved

learning, followed by improved test scores

• Also need to meet the broader overarching needs/goals of the broader setting– Address the (whole) child, the family & the community

• Integrate developmental and prevention science, together with positive youth development to address the ‘‘whole child’’ (Bhattacharyya, et al., 2009)

• Align with cultural ideals – (Castro et al., 2010, this forum)– e,g, in Hawai’i, Pono Choices values of being right with yourself and

above reproach can help avoid unsafe sex and improve academics

CONCLUSIONS

• Inserting a program into a setting so that it is implemented with integrity and is sustained– Requires the same conditions as setting reform:

• Support of the whole setting• Investment and commitment• Being true to the model• Ongoing professional development

– Plus one additional condition:• Alignment with the overarching goals of the setting

– Behavior, Character and Achievement– Consistent with community values and norms

REFERENCES• Bhattacharyya, O., Reeves, S., & Zwarenstein, M. (2009). What Is Implementation Research? Research on Social Work

Practice, 19(5), 491-502. • Castro, F. G., Barrera Jr, M., & Holleran Steiker, L. K. (2010). Issues and challenges in the design of culturally adapted

evidence-based interventions. Annual Review of Clinical Psychology, 6, 213-239.• Conn, V. S., Hafdahl, A. R., & Mehr, D. R. (2011). Interventions to Increase Physical Activity Among Healthy Adults: Meta-

Analysis of Outcomes. American Journal of Public Health, 101(4), 751.• Elliott, D. S., & Mihalic, S. (2004). Issues in disseminating and replicating effective prevention programs. Prevention Science,

5(1), 47-53. • Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation Research: A Synthesis of the

Literature. University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network (FMHI Publication #231). Tampa, FL.

• Flay, B. R. (1986). Efficacy and effectiveness trials (and other phases of research) in the development of health promotion programs. Preventive Medicine, 15, 451-474.

• Herschell, A. D., Kolko, D. J., Baumann, B. L., & Davis, A. C. (2010). The role of therapist training in the implementation of psychosocial treatments: A review and critique with recommendations. Clinical Psychology Review, 30(4), 448-466.

• Lipsey, M. W. (1999). Can rehabilitative programs reduce the recidivism of juvenile offenders? Virginia Journal of Social Policy & the Law, 6(3), 611-641.

• Olmstead, T., Carroll, K. M., Canning-Ball, M., & Martino, S. (2011). Cost and cost-effectiveness of three strategies for training clinicians in motivational interviewing. Drug and Alcohol Dependence. doi: 10.1016/j.drugalcdep.2010.12.015

• Slavin, R. E., & Madden, N. A. (2007). Scaling up Success For All: The first 16 years. In S. B & S. McDonald (Eds.), Scale-up in education (pp. 201-228). Lanham, MD: Rowman & Littlefield

• Wandersman, A., Duffy, J., Flaspohler, P., Noonan, R., Lubell, K., Stillman, L., . . . Saul, J. (2008). Bridging the gap between prevention research and practice: The interactive systems framework for dissemination and implementation. American Journal of Community Psychology, 41(3), 171-181.

• Wilson, J. W., Lipsey, M. W., & Derzon, J. H. (2003). The effects of school-based intervention programs on aggressive behavior: A meta-analysis. Journal of Consulting and Clinical Psychology, 71(1), 136-149. doi: 10.1037/0022-006X.71.1.136

THANK YOU

• I thank the Office of the Assistant Secretary for Planning and Evaluation for sponsoring this Forum

• Thanks to Child Trends for organizing and hosting it.

• I thank my wife, Carol G. Allred, very much for her help with this! – As the developer, seller and trainer for the

Positive Action program (www.positiveaction.net), she has far more real-world experience with these issue than I do!