1

The Role of Policy Evaluation in Violence Prevention

Jennifer L. Matjasko, PhD Behavioral Scientist

Division of Violence Prevention National Center for Injury Prevention and Control

Centers for Disease Control and Prevention

National Center for Injury Prevention and Control Division of Violence Prevention

2

Overview

� Overview of the policy process

� The role of policy evaluation in the policy process

� Types of policy evaluation

� Policy evaluation and violence prevention (examples and illustrations weaved throughout)

� Conclusions

2

3

Overview

� Overview of the policy process

� The role of policy evaluation in the policy process

� Types of policy evaluation

� Policy evaluation and violence prevention

� Conclusions

3

4

The Policy Process

5

Overview

� Overview of the policy process

� The role of policy evaluation in the policy process

� Types of policy evaluation

� Policy evaluation and violence prevention

� Conclusions

5

6

The Role of Policy Evaluation in the Policy Process

“Application of evaluation principles and methods to examine content, implementation, or impact of policy.”

Purpose: • Choosing among policy options • Informing implementation • Demonstrating impacts and value of

policy • Informing evidence base and future

policies • Accountability for resources invested

7

The Role of Policy Evaluation in the Policy Process

Problem Identification

Describe Policy Options

Assess and Select Policy

Options

Develop Strategy to Implement

Policy Implementation & Compliance

Evaluate Effects and

Impact

Implementation Evaluation

Impact Evaluation

Problem Identification Policy Analysis

Strategy and Policy

Development Policy

Enactment Policy

Implementation

8

Overview

� Overview of the policy process

� The role of policy evaluation in the policy process

� Types of policy evaluation

� Policy evaluation and violence prevention

� Conclusions

8

9

The Role of Policy Evaluation in the Policy Process

Problem Identification

Describe Policy Options

Assess and Select Policy

Options

Develop Strategy to Implement

Policy Implementation & Compliance

Evaluate Effects and

Impact

Implementation Evaluation

Impact Evaluation

10

Content Evaluation

� Does policy content clearly articulate goals, implementation, and intended change?

� What was the context of policy development? ��Essential in interpreting implementation and impact evaluation results

� Tools/Resources

� Legislative Records/Databases (Lexis Nexus, THOMAS, govtrack, state and local government websites)

� Logic Models

11

Content Evaluation Example: State-Level Anti-Bullying Laws

� The US Department of Education’s Analysis of State Anti-Bullying Laws � http://www2.ed.gov/rschstat/eval/bullying/state-bullying-

laws/state-bullying-laws.pdf

� To what extent do states’ bullying laws cover key legislative and policy components?

�Content analysis of anti-bullying laws according to Dept of Ed key components �Phase of the policy process?

US Dept of Ed (2011)

12

Content Evaluation Example: State-Level Anti-Bullying Laws

� Some findings

� Forty-six states have bullying laws and 45 of those laws direct school districts to adopt bullying policies.

� Thirty-six states include provisions in their education codes prohibiting cyberbullying or bullying using electronic media.

US Dept of Ed (2011)

13

Implementation Evaluation

� Was policy implemented as intended? � Understanding how a policy was implemented including critical differences between

planned and actual implementation ��Can be critical in interpreting impact evaluation results � Tools/Resources

� Constituent interviews � Case studies � Observations

14

Implementation Evaluation Example: State-Level Anti-Bullying Laws

� Another component of US Department of Education’s Analysis of State Anti-Bullying Laws � How are state laws translated into practice at the district and

school levels?

�Key constituent interviews in select states and districts

�Phase of the policy process?

US Dept of Ed (2011)

15

Implementation Evaluation Example: State-Level Anti-Bullying Laws

� Some findings

� Districts located in states with more expansive legislation

produced the most expansive school district policies. � However, there were some exceptions. Several school districts

in states with less expansive laws also substantially expanded the scope and content of their policies beyond the minimum legal expectations.

US Dept of ed (2011)

16

Impact Evaluation

� Did policy produce intended outcomes/impact? Can those changes be attributed to the policy?

� Short-term or intermediate outcomes as well as long-

term changes or impacts � Compare costs of a policy to the resulting benefits

� Tools/Resources

� Administrative data � Cost data from key constituents � Methods for constructing comparison groups � Policy evaluation experts

17

Impact Evaluation Example: Business Improvement Districts (BIDS) and Violent Crime

� Business Improvement Districts � Community economic development model � Special assessments on commercial properties to augment

public services

�Pre/post among 30 BID neighborhoods in Los Angeles

�Phase of the policy process?

MacDonald et al. (2010)

Problem Identification Policy Analysis

Strategy and Policy

Development Policy

Enactment Policy

Implementation Evaluate Effects

and Impact

18

Impact Evaluation Example: Business Improvement Districts (BIDS) and Violent Crime

� Some findings

� BID implementation was associated with a 12% reduction in the incidence of robbery and an 8% reduction in the total incidence of violent crimes.

� Implementation data informed these results: private security example.

MacDonald et al. (2010).

19

Overview

� Overview of the policy process

� The role of policy evaluation in the policy process

� Types of policy evaluation

� Policy evaluation and violence prevention

� Conclusions

19

20

Policy Evaluation and Violence Prevention

� A broad policy landscape

� Few policies have been rigorously evaluated

� Evaluations are needed at all stages of the policy process � To identify promising policy options � To specify the characteristics and active ingredients of a policy

intervention � To ascertain fidelity of implementation � To identify impact and potential unintended consequences of

policies

21

Violence Prevention Policy Evaluation Things to consider

� What is the policy? � Who is impacted? � What violence-related knowledge or behaviors do you

want to change? Or, what are the violence-related knowledge or behaviors that might be changed by a particular policy intervention?

� What is a realistic time frame to see change? � What are the potential unintended consequences? � What data sources are available to evaluate policy? � What other policies might confound findings?

22

Data Sources to Inform Evaluation

� Crime data � Public surveys � Medicaid and other claims data � Behavioral health data � ED and hospitalization data � Mortality data (e.g., NVDRS) � Census data

� Others? � New data sources to conduct evaluation?

24

Policy Evaluation Challenges/Solutions

Challenges � Evaluation is often an

afterthought � Data sources may not

be available � Bringing the right

players together may be difficult

� Statistical challenges with multiple interventions at the same time

Solutions � Include evaluation

planning early on � Identify data needs

prior to determine feasibility of the eval

� Identify key stakeholders as early as possible in the eval planning process

� Partner with academic institutions, other experts to conduct evaluations

25

Conclusions

� Policy evaluation is critical

� Plan evaluation before the implementation of policy (when possible)

� Demonstrating impact can be challenging when multiple interventions have been implemented

� CDC and others are available to provide technical assistance

Jennifer L. Matjasko jmatjasko@cdc.gov

Thank You

The findings and conclusions in this presentation are those of the author and do not necessarily represent the views of the Centers for Disease Control and Prevention.