v.2.2.1

The Guide to

The SYSTEMS EVALUATION PROTOCOL

ii

iii

TABLE OF CONTENTS

7 7

é

PREFACE

ACKNOWLEDGEMENTS

Figure 1.

Phases of

Program

Evaluation

Throughout the Protocol there are sidebars that will be of interest to systems

These are added for their supplementary value and will enhance understanding of the

reader should be able to use the Protocol even

Figure 2.

Protocol

Terminology

for System

Hierarchy

Throughout the Protocol

want to use as a group

Universe Multiverse

Atom

Solar SystemPlanet

Galaxy

Figure 3.

Phases of Evaluation Planning

7

of the human thrower and use mechanical devices like a catapult we can predict even

This discussion format

workgroup members should be broken up into small groups (2-5 people in each

is presented and at each

given 2-3 minutes to think about and take personal

5-7 minutes to share and

process can be repeated

prompts generally allow

share the notes created at

mind when you think about

Figure 4.

Hypothetical map

of stakeholders

especially colleagues

of stakeholders can be

program member list the stakeholders individually onto separate pieces of

it notes should also work

place the stakeholders on

stakeholders near their

taking turns in a formal

own criteria for similarity

When the diagram is complete (when everyone

discuss the process and

have developed - this may

stakeholders on their own

stakeholders they had not

VII.

for stakeholders because they lead to discussions about the meaning of what they are

Figure 5.

Phases in

Program

Lifecycle

iNITITATION dEVELOPMENT sTABILITY Dissemination

PHASE I pHASE II PHASE III PHASE IV

State of the Program

cHARACTERIZING A PROGRAM'S EVOLUTION

discuss how and why the program has changed

The following prompt is designed to help the

the group to best communicate the story of the

are several working groups engaged at the

groups take turns sharing and describing their

Figure 6.

Phases in

evaluation

Lifecycle

Process &

Response

PHASE I pHASE II PHASE III PHASE IV

State of the Evaluation

cHARACTERIZING An Evaluation'S EVOLUTION

changeComparison

& ControlGeneralizability

Figure 7.

Lifecycle

Alignment

is concerned with the origin and the

Generalizability

Phase I Phase II Phase III Phase IVPhase I Phase II Phase III Phase IV

Prog

ram Lif

ecycle

Evaluatio

n Lif

ecycle

Phase IA

Phase IB

Phase Iv

A

Phase Iv

B

Phase iIIB

Phase iIIa

Phase IIa

Phase IIB

Prog

ram and Evaluatio

n Lif

ecycle Def in

itio

ns and alig

nment

the theory of natural

all of the life sciences

Figure 9

Sample Logic

Model

Figure 10

Sample Path

from activity to

outcome

Figure 11

Pathway from

activity to

outcome

Figure 12.

pROGRAM pathway

model

feedback can be especially valuable to the logic and pathway model development process when programs reviewing each

(

useful insights about the answers to

in a manner that is

makes strategic

contributes to our

Figure 13.

Stakeholders

located on

pROGRAM pathway

model

Review

Figure 14.

Evaluation Scope

within it as shown in

Figure 15.

lITERATURE

MAPPING (Golden

Spike)

to delve deeper into understanding stakeholder

key stakeholders about

stakeholder whom they would be interested in

agree on a plan for how to capture stakeholder

workgroup member should

that the stakeholder input

interview the stakeholder

and if agreeable establish

group should then create

interview based on the list provided in

and pathway model during the interview can provide

the program before sharing

to a peer

brainstorm as many strategies as possible and then narrow them down based on your

stakeholders outside of the working group can provide a

program implementers and/or stakeholders who are

most column with each of their

peer or stakeholder is then

may also include columns

working group can also use the brainstorm guide to develop

for an imagined or well-known

neutral program will help

of their choices and the challenges facing them as they

and

do not

TimeRules out

other possible

Shows Program lifecycle phase it may be appropriate for

No No No

post pre No No No

No No

and

Analysis the process of deriving order and meaning from data

that may be outside the program

must screen the data for accuracy - allowing you to go back and clarify

the programan abstract or general idea inferred or derived from empirical evidence

not receive the treatment

on one scale are associated with low values on another

WikipediaWikipedia

building

person in the leadership role who will facilitate the protocol and guide

the larger systems within which the program is embedded

statement

in the future

Lifecycle analysis

Long-term outcomes

outcomes to long-term outcomes

Pathway model

Post-only

Pre-post

Program

Program boundary

a step-by-step guidethe process of deriving order and meaning from data using non-numerical methodsthe process of deriving order and meaning from data using numerical

they already possess

Wikipedia

Random assignmentin the study

Reliability

Sample

Wikipedia

SEP

for that programShort-term outcomes

Stakeholder analysis

Stakeholders

Subscale a smaller set of items on a measure that have shared validity and reliability independent of the larger measure

SurveySystem

take place

which the program is embedded

between each

Timeline

completedTreatment group

program change

that they were designed to measure

[

and [

[

[ORGANIZATION

ORGANIZATION

[

The [

of the [

and the [

[

PERSONNEL

TRAINING

HARDWARE

SOFTWARE

THANK YOU!

DESCRIPTION

FUNDING

INPUTS

PROGRAM STAGE

PARTICIPANTS

LOGIC MODEL

DESCRIPTION

EVALUATION TRAINING

PROGRAM

IN OUT

INITIATION DEVELOPMENT STABILITY DISSEMINATION

PHASE I PHASE II PHASE III PHASE IV

STATE OF THE PROGRAM

CHARACTERIZING A PROGRAM'S EVOLUTION

INITIATION DEVELOPMENT STABILITY DISSEMINATION

STATE OF THE EVALUATION

PROCESS &RESPONSE

PHASE I PHASE II PHASE III PHASE IV

CHANGE COMPARISON &CONTROL GENERALIZABILITY

CHARACTERIZING AN EVALUATION’S EVOLUTION

and the

dist

ribut

edth

is

Phase I Phase II Phase III Phase IVPhase I Phase II Phase III Phase IV

Phase IA

Phase IB

Phase Iv

A

Phase Iv

B

Phase iIIB

Phase iIIa

Phase IIa

Phase IIB

ev

alu

ae

le

e e

B

A

C

ev

alu

ae

le

e e

checklistchecklist

checklistchecklist

checklistchecklist

outputs

with a

on the

Use:

or an

weaker