The Feasibility of the Realist Approach withCanadian Evaluations

Donna Smith-Moncrieffe

Canadian Evaluation SocietyOttawa Convention Centre

June 17,2014

The Feasibility of the Realist Approach withCanadian Evaluations

Donna Smith-Moncrieffe

Canadian Evaluation SocietyOttawa Convention Centre

June 17,2014

2

Summary of the PresentationSummary of the Presentation

1. Public Safety: Crime Prevention Evaluation Research

2. Realist Approach Using Multi-site evaluations

1. Using Multi-site theories to assess the feasibility of the Realist Approach

3. Case Study 1● Assessing the Critical Elements in the Theory of Change● Assessing between-site differences● What Works for Whom and in What Circumstances?

4. Case Study 2● Understanding the Context● Outcome Assessment● Utilizing New Theory to Strengthen the Intervention

5. Summary/Recommendations/Discussion

3

Realist Approach using multi-site evaluationsRealist Approach using multi-site evaluations

4

Case Study 1: Stop Now and Plan®

Assessing the critical elements in the Theory of ChangeCase Study 1: Stop Now and Plan®

Assessing the critical elements in the Theory of Change

5

Case Study 1 Cont’d: Assessing Between-site Differences

Case Study 1 Cont’d: Assessing Between-site Differences

Key elements Site 1 Toronto Site 2 Edmonton Site 2Cree, Quebec

Effect sizes related to emotional regulation (aggression)

Moderate to High(0.54-1.17 effect size)

Method: Quasi-experiment with comparison group

Moderate to High(0.4-0.61 effect size)

Method: Quasi-experiment with comparison group

Statistical Assumptions not met

Method: Single site longitudinal analysis

Fidelity rating High High Low

Contextual differences

• Project implemented as per the model

• Project implemented as per the model

• Literacy in English

• Parental engagement is low

• Logistical Challenges

• Cultural differences

6

Case Study 1 cont’d : What Works for Whom and it What Circumstances?

Case Study 1 cont’d : What Works for Whom and it What Circumstances?

● In this case we identified Site 3 as not reaching the typical reductions in aggression;

● Questions about what worked for whom and in what circumstances were answered via assessing the between-site differences

Aboriginal youth who reside in rural communities will have more difficulty completing the required number of cognitive therapy sessions due to distance between the home and the program location;

These Aboriginal youth will have more difficulty understanding the content if their first language (Cree) and cultural norms are not adapted in their curriculum;

The parents of Aboriginal youth at risk who reside in rural communities have a higher rate of concern related to stigmas and therefore their involvement in parental groups are significantly reduced;

These three key variables in the Cree site affected the achievement of typical outcomes. These confounding variables were not present in the other two sites.

7

Case Study 2: Alternative Suspension ProgramStep 1: Understanding the Context

Case Study 2: Alternative Suspension ProgramStep 1: Understanding the Context

Examining the Context Description

What is the aim of the program? To reduce crime by :•Decreasing school disciplinary action•Increasing school attachment•Decreasing antisocial behavior

What is the nature of the target population?

• Youth between 12-17 years who are at risk of being suspended or have been suspended

• Persistent behavior challenges• Difficulties in academic and social life

In what context and settings does it operate?

• Urban setting• High deprivation, high dropout rates,

juvenile delinquency and criminality• At-risk communities (low education rate,

high unemployment rate, high proportions of single-parent families and of low-income household

What are the prevailing theories about how school suspensions reduce crime?

• Drop out of school is a pathway to delinquency and criminality

• Anti-social behavior in school has been found to be related to adult criminal behavior and violent crime

8

Case Study 2 cont’d: Step 2: Outcome Assessment

Case Study 2 cont’d: Step 2: Outcome Assessment

● Interim school data results (2012- 2013 ) indicate that although the three sites have delivered the same program, Edmonton has had greater impact on key variables;

● The original theory works best for Edmonton and not the other sites;

● Examine what works for whom in Step 3

Variable Site 1Edmonton

Site 2Moncton

Site 3 Chilliwack

Disciplinary actions

71.4% decrease 45.3% decrease

60.5% decrease

Improvement in School Behaviour(end of year)

73.5 % improvement

45.6 % improvement

68.3%Improvement

9

Case Study 2 cont’dStep 3: Utilizing New Theory to Strengthen the Intervention

Case Study 2 cont’dStep 3: Utilizing New Theory to Strengthen the Intervention

● What are the elements that may be contributing to efficacy in one site and not another?

● At the interim stage of the evaluation, the ANOVA results indicate that the following variables may be contributing to between-site differences:

Previous suspension history varies by site with Edmonton having relatively lower rates of prior school suspensions than Moncton. Further analysis is required to determine if this variable is contributing to outcome differences

School act legislation enforced in one site

● The Edmonton site was limited to YMCA activities only during the suspension period. Participants in the two other sites had access to a variety of other complementary activities outside of the YMCA.

● During the final two years of this longitudinal evaluation, these variables will need to be addressed and then tested via multivariate analyses to determine what elements enhance efficacy for the participants in their unique contexts

10

SummarySummary

● Realist Theory is feasible and useful in a Canadian context:

Prioritize evaluation questions that focus on the context;

Examine the elements in the theory of change;

Use a multi-site framework where possible to examine between-site differences;

Modify elements of the program or develop a new theory of change to enhance favorable change for program participants operating in unique contexts; and

● The approaches used in Case Study 1 and 2 can be used retrospectively or can be used at the interim evaluation stage to apply modifications and enhance outcomes

11

Recommendations Recommendations

● Use program models with standardized content to facilitate the identification of what works for whom and in what circumstances;

● Use mixed model techniques (multivariate analysis and qualitative methods to isolate the between-site differences and the elements that are contributing to variation in the original theory of change;

● Develop detailed theories of change and consider modifying theories during long term evaluations to ensure that the appropriate context is considered in the “evidence chain”;

● Ensure that stakeholders understand that the context, and theory of change in each of the sites are as important as the quantitative outcomes;

● Develop and share new theories of change for each of the sites to clarify what works, for whom in each site.

12

DiscussionDiscussion

● How will we advance the Realist Approach in Canada?

● How will we ensure that evaluators use innovation and rigor to build and rebuild theories of change that take into consideration the context of each intervention?

● Who will facilitate and support the development and exchange of information that will ensure this approach is integrated into evaluation plans?

13

14

Thank YouThank You

References

Astwood Strategy (2014) Impact Evaluation Final Report, Public Safety Canada

Laliberté, D. (2014). Alternative Suspension Impact Evaluation Summary, 2013-2014 Updates, Public Safety, to be published.

Pawson R., & Tilley, N. (2006) Realist Evaluation. Development Policy Review Network Thematic Meeting, Re Report on Evaluation , Amsterdam, Netherlands

R.A. Malatest & Associates Ltd (2014). Multi-site Evaluation of the Alternative Suspension Program. Second Annual Report, Submitted by Glenda Shaw-Garlock, G. to Danièle Laliberté, Senior Evaluation Analyst.

Smith-Moncrieffe, D (2013) Stop Now and Plan: Impact Evaluation Summary 2013-2014, Public Safety Canada

Treasury Board of Canada, Theory-Based Approaches to Evaluation: Concepts and Practices , Published by the Government of Canada

Special thanks to evaluation team members: Donna Smith-Moncrieffe, Danièle Laliberté and Giselle Rosario for compiling evaluation data for analysis and use for this presentation