Going Beyond Evaluation Buzzwords

Lauren Decker, Ph.D. Edvance Research

Presented at Texas ACE Annual Conference, Austin, TX.

August 7, 2013

One word or phrase expressing how you feel about evaluation

“If you don’t know where you are going, how are you gonna’ know when you get there?”

 Yogi Berra

Where are we going and how will we get there?

  What’s a logic model and how does it help?   Defining seemingly inter-changeable terms   Time and tool to begin a logic model (or refine one)

  What is actually happening and how does it help to know?   Program and evaluation importance of understanding

implementation fidelity

  Bringing thoughtful program lessons into evaluation plan and use   What makes a great evaluation?   Planning and asking the right question(s)

Logic Models and Theories of Change

Logic Model vs. Theory of Change

  Logic Model = graphically depicts what your program is intended to do   Key program components

  Theory of Change = why your program operates as intended and how components and activities are hypothesized to move outcomes   Assumptions underlying

expected change

Simple Logic Model

Inputs Outputs Outcomes

Resources Activities Participation Short

Medium

Long

What we do

Who we reach

What results we expect

What we invest

Logic Model Example

How does having a logic model help my program?

Program phase

  New program

  Existing program

  Redesigning existing program

Logic model use

 Creation & planning

 Documentation & communication

 Reinvention, improvement, & expansion

 Use the paper provided to begin a logic model for your program

 Remember to include resources, intended activities, participants, short-, mid-, and long-term outcomes

  If you can also represent the theory of change (arrows)

What does our program logic model look like? 10 minutes

What should we include in our program Logic Model?

  Identify:   All key components of program

 Resources, intended activities, participation, short-, mid-, and long-term outcomes

  Components and pathways(mediators) through which program is expected to produce intended outcomes (theory)

  Student outcome(s) program is designed to improve  Short-, mid-, long-term outcomes  Other outcomes as well if relevant

I have a Logic Model, now what?

– What is fidelity of implementation?

Fidelity of Implementation

 The extent to which key components of the program are delivered as originally intended by the developer   Key components = strategies, practices,

activities and behaviors that are critical to defining your program

Logic Model Example

Fidelity of Implementation   the extent to which key components of

the program are delivered as originally intended by the developer   Key components = strategies, practices,

activities and behaviors that are critical to defining your program

  When is your program not your program?

How do I measure fidelity of my program?

Fidelity

Structural key

components

Interactional key

components

Structure Process

How does measuring fidelity help my program?

  Understanding fidelity helps you know about:   Participation   Activity delivery   Content

  Example questions you can answer:   Were activities implemented by program staff

according to design?   Did staff receive program content as planned?   What were the adaptations that were made to the

program?   What were the factors that may have affected program

fidelity?

“I think you should be more explicit here in Step Two.”

Evaluations without measuring fidelity…

 Visit someone else’s logic model in process

 For the activities listed suggest:   What type of

fidelity could be measured

  Ways of measuring that type of fidelity for the activity

How to measure fidelity in your program? 10 minutes

What do we need to do to understand fidelity of our program?

 Periodically measure implementation fidelity   Measured separately for each key

component  Thresholds specified for determining

whether key components of program were implemented with fidelity

 Assess and Report whether each key component was implemented with fidelity

What makes a great evaluation?

High-quality evaluations include:

 Evaluators who clearly use sound evaluation principles and practices (transparency, consideration of context, etc.)   Complex programs require thoughtful, directed

evaluations  Focus from the start on use of evaluation

processes and products to provide information to, and about, the program being evaluated   Alignment to the program’s logic model and

theory of change   Include provision for some level of

implementation fidelity

High-quality evaluations include:

  Preparation and planning for variation within multi-site evaluations

  Consideration of clear deadlines for completion and deliverables set by program

  Upfront acknowledgement of the limitations of the proposed evaluation

  Minimizing jargon to increase clarity of communication of results

  Striking a balance between consultancy/ recommendation support and maintaining unbiased distance as an independent evaluator

Before meeting with an evaluator:

 Complete your program logic model and theory of change   Or have discussions and ideas of key

components for development assistance from evaluator

 Consider what you really want to learn   What outcomes do you want an evaluation to

focus on for your program?

What do you really want to know? Make sure you ask the right question!

Before meeting with an evaluator:

 Complete your program logic model and theory of change

 Consider what you really want to learn  What data is already available or will need to

be collected to answer your right question(s)?   If you believe data will need to be collected do

you want to collect them or have the evaluator conduct the data collection?

  Think about whether you have identified SMART outcomes!

SMART Outcomes are:

 Specific   Reflect simple information that is communicable and

easily understood  Measurable

  Can changes by measured in reliable and valid ways?  Achievable

  Able to be collected and sensitive to change during the allotted time

 Relevant   Reflect information that is important and likely to be used

 Time bound   Progress can be tracked at desired frequency within

allotted time

Source: Adapted from World Bank Group.

Questions for an evaluator   How does your proposed evaluation align with our

program logic model and theory of change?   What data do you need from us to conduct the

evaluation? What data to you propose to collect, when, and in what format?

  How intrusive will the data collection be for our program?

  Will your evaluation be able to tell us if the program was in place as intended?

  What we really want to know from an evaluation is _________. How will your proposed evaluation answer this question?

Questions to ask yourself about an evaluation proposed to you

  Does the evaluation design reflect my logic model and theory of change?

  Do the proposed outcomes follow my logic model and theory of change?

  Is it feasible to collect outcome data at the time I expect to see changes (short- medium- and long-term outcomes) in those selected?

  Are the selected outcomes and measures relevant for my program and my stakeholders?

  Does the evaluation include SMART outcomes that will be evaluated?

Logic Model Example

Questions? Thank you.

Lauren Decker, Ph.D. ldecker@edvanceresearch.com