Program Evaluation

George Vukotich, Ph.D. Copyright © 2004

Intergovernmental Executive Development Program

Program EvaluationMeasuring Success

It all begins with setting goals and knowing where you are going.

SMART GOALS

Specific

Measurable

Agreed On

Realistic

Timed

Relevant Measurement

Activity vs. Productivity

Effectiveness vs. Efficiency

Direct vs. Indirect

Are we measuring the right thing.

Price / Quality / Innovation

What is your organization known for.

What is your approach to providing services.

What to Evaluatee.g. Evaluating IEDP

The course material

The instructors

The facility

Quality of interaction with others

What was learned

Evaluation ProcessADDIE Model

Analyze

Design

Develop

Implement

Evaluate

Evaluate to What Standard

Benchmarking

Based on pre-set goals

Compare to other similar projects Similar industries, organizations,

products

Past / Historical standards

How to evaluate

Forms Interviews Focus Groups Self Analysis Grades Supervisors Review Work Productivity

Degree of Success

To what extent does something

need to be effective to.

Formative vs. Summative

Conducting the evaluation during the process and making adjustment along the way.

Waiting until the process is complete and evaluating the results.

Test – Retest Methodology

Sampling

Data – Not just getting it, but using it.

Reliability and Validity

VALIDITY:

Is what we are evaluating the right thing to evaluate to help us be successful.

RELIABILITY:Will we get consistent measures over time.

Kirkpatrick’s 4 - Levels

Level 1 – Reaction

Level 2 – Learning

Level 3 – Behavior

Level 4 -- Results

Balanced Scorecard

Cost / Benefit Analysis

Employee job satisfaction

Students completing a program

Reduction in processing errors

Customer service ratings

Unintended Consequences

What is the cost of making an error in our evaluation.

♦ Turnover

♦ Time to Hire, Cost to Hire

♦ Reduced Processing Costs

♦ Improved Safety

♦ Reduced Insurance Costs

CORRELATIONS

Return on Investment (ROI)

Theory into practice

How can your team use “program evaluation” to help you be more successful.