February 2009

Measuring Value: Using Program Evaluation to Understand What’s

Working -- Or Isn’t

Juliana M. Blome, Ph.D. , MPHOffice of Program Analysis and Evaluation

National Institute of General Medical Sciences

MORE Program Directors MeetingColorado Springs, Colorado

June 12, 2009

NIGMS

February 2009

“Program Evaluation…!?”

NIGMS

February 2009

Program Evaluation: What is it?

Program evaluations are individual, systematic studies that use objective measurement and analysis to answer specific questions about how well a program is working.

- GAO/GGD-00-204 Program Evaluation

NIGMS

February 2009

Evaluation Answers Questions Such As….

• Does it work?• How well does it work?• Does it do what we want it to? • Does it work for the reasons we think it does? • Is it cost effective? • Are the benefits worth it? • What are the unintended consequences?

NIGMS

February 2009

Research vs. Program Evaluation

Evaluation

Judges merit or worth Policy & program interests of

stakeholders paramount Provides information for

decision-making on specific program

Conducted within setting of changing actors, priorities, resources, & timelines

Research

Produces generalizable knowledge

Scientific inquiry based on intellectual curiosity

Advances broad knowledge and theory

Controlled setting

NIGMS

February 2009

Why bother?

To gain insight about a program and its operations

To improve practice - to modify or adapt practices to enhance the likelihood of success

To assess effects – to determine if we’re meeting our goals and provide evidence of effectiveness

NIGMS

February 2009

Guidelines for Conducting Successful Evaluations

Invest heavily in planning early on Integrate evaluation into ongoing program activities Use knowledgeable, experienced evaluators

NIGMS

February 2009

Evaluator Skills

Evaluation theory and methods Research methods (design, planning, statistics,

qualitative and quantitative methods) Data collection, analysis and interpretation Communication and Interpersonal skills Content area skills Project management Ethics

At universities and colleges, this type of expertise is found in the social and behavioral sciences departments!

NIGMS

February 2009

Evaluation Costs

The National Science Foundation's “rule of thumb” about evaluation budgets is 10%

of the total grant amount.

NIGMS

February 2009

Types of Evaluations

Needs Assessment What is nature & extent of the issues program should address? Planning phase

Feasibility Study Is evaluation appropriate and/or affordable? Maturity/timeliness issue? Process or outcome evaluation produced

Process Evaluation Is program is being conducted & producing output as planned? How can process can be improved?

Outcome Evaluation To what extent have a program’s goals have been met?

NIGMS

February 2009

StandardsUtility

Feasibility

Propriety

Accuracy

Engage Stakeholders

Focus Evaluation

Design

Describe the program

Gather credible evidence

Justify conclusions

Use and share lessons learned

CDC Framework for Program Evaluation

Steps

Milstein et al, Health Promotion Practice, July 2000, Vol 1(3): 221-228

For more info:

http://www.cdc.gov/EVAL/

NIGMS

February 2009

Evaluation Standards

UtilityEvaluations should serve the practical information needs of a given audience

FeasibilityEvaluations take place in the field and should be realistic, prudent, diplomatic and frugal

ProprietyThe rights of individuals affected by evaluations should be protected

Accuracy Evaluations should produce and convey accurate information about a program’s merit and/or worth

Guiding Principles for Evaluators, American Evaluation Association, www.eval.org

NIGMS

February 2009

CDC Framework: Key Steps in Evaluation

1. Engage stakeholders

2. Describe the program

3. Focus the evaluation design

4. Gather credible evidence

5. Justify conclusions

6. Ensure use and share lessons

NIGMS

February 2009

Step 1- Engage Stakeholders

Who are the stakeholders?

Those involved in program operations, those affected

by the program operations, and users

of evaluation results

NIGMS

February 2009

What are the goals and specific aims of the program?

What problem or need is it designed to address?

What are the measurable objectives? What are the strategies to achieve the objectives?

What are the expected effects?

What are the resources and activities?

How is the program supposed to work?

Step 2 - Describe the Program

NIGMS

February 2009

“I think you should be more explicit here in Step Two.”By Sidney Harris, Copyright 2007, The New Yorker

NIGMS

February 2009

Step 3 - Focus the evaluation design

• What do you want to know?

• Consider the purpose, uses, questions, methods, roles, budgets, deliverables etc.

An evaluation cannot answer all questions for all stakeholders.

NIGMS

February 2009

Step 4 - Gather credible evidence

Evidence must be believable, trustworthy, and relevant

• Information scope, sources, quality, logistics • Methodology & data collection • Who is studied and when

NIGMS

February 2009

Step 5 - “Justify” Conclusions

Consider data:

• Analysis and synthesis - determine findings

• Interpretation - what do findings mean?

• Judgments - what is the value of findings based on accepted standards?

• Recommendations – - what claims can be made?

- what are the limitations of your design?

NIGMS

February 2009

Step 6 - Use and share results

Share lessons learned with stakeholders!

Provide feedback, offer briefings. disseminate findings

NIGMS

February 2009

Are you overwhelmed?

Next Session – Moving from the abstract to the concrete