Evaluation Designs and MethodsProgram Evaluation Basics Webinar Series

Mary E. Arnold, Ph.D.Professor and Youth Development Specialist

Oregon State University

4-H Professional Development WebinarFebruary 14, 2013

Webinar Agenda

Building on previous month’s topic of focusing and planning your evaluation we will:

• Explore the concept of rigor in evaluation, and its particular place in Extension and 4-H evaluation

• Learn the role that evaluation questions play in determining evaluation design and data collection methods

• Explore common evaluation designs• Explore common and innovative evaluation methods• Learn of resources available to support evaluation efforts in 4-H

and Extension

Elements of Rigor

• Evaluation design

• Conceptualization of program constructs &

outcomes

• Measurement strategies

• Timeframe of the evaluation study

• Program integrity

• Program participation and attrition

• Statistical analysesBraverman, M. T., & Arnold, M. E. (2008). An evaluator’s balancing act: Maintaining rigor while being responsive to multiple

stakeholders. In M. T. Braverman, M. Engel, R. A. Rennekamp, & M. E. Arnold (Eds.) Program evaluation in a complex

organizational system: Lessons from Cooperative Extension. New Directions for Evaluation, 120, 71-86.

Rigor and the 4-H Organization

• Who determines standards of rigor?

• How do decisions about evaluation methods get made?

• How, and to what extent, is the quality of a completed evaluation determined?

(X O)

O = “Observation” (data collection) X = “intervention” (program)

Post Only Design

Evaluation Question Example: What skills do campers report developing at science camp?

TIME

Never

Sometimes

Usually Always

I can use scientific knowledge to form a question1 2 3 4

I can ask a question that can be answered by collecting data1 2 3 4

I can design a scientific procedure to answer a question1 2 3 4

Post Only Control Group Design

E (X O)C (X O)

Evaluation Question Example: Do youth who attend 4-H summer science camp have better science skills than youth who do not attend?

O = “Observation” (data collection) X = “intervention” (program)

E = Experimental group (program participants) C = Control group (non-participants)

TIME

What is the level of rigor? What cannot be said?

Never

Sometimes

Usually Always

I can use scientific knowledge to form a question 1 2 3 4

I can ask a question that can be answered by collecting data 1 2 3 4

I can design a scientific procedure to answer a question 1 2 3 4

One Group Pre-Test/Post-Test

(O X O)

Evaluation Question Example: Do youth have higher levels of positive youth development at the end of the program than they did at the beginning?

TIME

Strongly Disagree Disagree Agree

Strongly Agree

I feel good about my scholastic ability O O O O

I feel accepted by my friends O O O O

I can figure out right from wrong O O O O

I can do things that make a difference O O O O

What is the level of rigor?What cannot be said?

Retrospective Pre-Test

( O X O)

Evaluation Question Example: Do youth have higher levels of positive youth development at the end of the program than they did at the beginning?

TIME

TIME

For each of the following items, please indicate how you felt before participating in this program, and how you feel now after participating in this program.

1 = Strongly disagree2 = Disagree3 = Agree4 Strongly Agree

Before After1 2 3 4 1 2 3 4

I feel accepted by my friendsO O O O O O O O

I can figure out right from wrongO O O O O O O O

I can do things that make a differenceO O O O O O O O

Control Group Pre-Test/Post Test

Evaluation Question Example: Do youth in program develop higher levels of PYD than youth who do not participate?

E (O X O)C (O --- O)

Strongly Disagree Disagree Agree

Strongly Agree

I feel good about my scholastic ability O O O O

I feel accepted by my friends O O O O

I can figure out right from wrong O O O O

I can do things that make a difference O O O O

TIME

LOW

-

Leve

l of

PY

D -

HIG

H

E

C

Time Series Design with Control Group

O O O O X O O OLOW

-

L

evel

of

PY

D -

HIG

H

E

C

Choosing an Evaluation Data Collection Method

Some Common Methods

• Archival data (records and documents)

• Surveys (mailed, electronic, phone)

• Interviews (phone, face to face, group)

• Focus group interviews

• Observation

• Tests (scenarios or skill/knowledge tests)

Important StepsWhat is the key concept that must be measured in each evaluation question?

Did youth participants in the YA4-H! Teens as Teachers program pilot increase their own consumption of fruits and vegetables?

Who has knowledge of this potential change?Several sources may emerge: Youth, parents, leaders, teachers, friends, records, observations.

What sources of data will be acceptable to stakeholders?

What expertise and funding is available to make a particular method practical?

Existing Documents and RecordsHave you ever considered meeting notes, minutes, videos, registrations, test scores, forms, records, reports as possible sources

of data?

• Considerably more cost effective than original data collection

• Data are not affected by the act of collecting it

• Programs collect lots of information that is never used, and too often we forget to look for existing data that can answer the question

What existing information could answer the question of increased fruits and vegetable consumption?

Surveys (Mailed, Electronic, Phone)

How could surveys help us assess increase in youth consumption of fruits and vegetables?

Designing Surveys: A Guide to Decisions and Procedures. Ronald Czaja and Johnny Blair (2005)

Internet, Mail, and Mixed-Mode Surveys: Tailored Design Method (3rd Ed.)John Dillman, Jolene Smyth, & Leah Melani Christian (2008)

How to Conduct Surveys: A Step by Step Guide Arlene Fink (2013)

Survey Research Methods (4th Ed.)Floyd Fowler (2009)

Interviews (Phone, Face to Face, Group)

Designing and Conducting Your First Interview (text book)Bruce Friesen (2010)

Qualitative Interviewing: The Art of Hearing Data (3rd Ed.)Hebert and Irene Rubin (2011)

Focus Groups; A Practical Guide for Applied Research (2nd Ed.)Richard Krueger (1994)

Interviewing as Qualitative Research: A Guide for Researchers in Education and the Social Sciences (4th Ed.)Irving Seidman (2012)

Why might interviews be a good method for collecting data about youth consumption of fruits and vegetables?

Direct Observation

Situations where you want direct information• May be more reliable than asking people if they are using new practices

When you are trying understand an ongoing behavior, process or unfolding situation or event

• Observing camp counselors before, during and after a training program

When there is physical evidence, products or outcomes that can be readily seen• Inspecting project records, newsletters, signs

When written or other data collection procedures seem appropriate• Programs to vulnerable or underserved audiences, when language or

literacy is a problem

Key ResourceCollecting Evaluation Data: Direction ObservationEllen Taylor-Powell and Sara Steele- Booklet available on the State 4-H Website

Could we use direct observations to measure youth consumption of fruits and vegetable?

Tests and Scenarios

Useful when assessing learning that requires specific knowledge that must be turned into action for the program to be considered successful.

Requires the ability to do a real pre –post test process

Can be very creative! Think skits and role plays• Camp counselor training• Youth leadership programs• Risk management training

Can you think of how a scenario evaluation might be a useful method for assessing youth consumption of fruits and vegetables?

Summary

Stay tuned on March 6th for our next webinar, which will focus more in depth on creating high quality questionnaires!

Your Evaluation Question Determines Your Design Determines Your Methods

That’s all for now!Join in next month for:

Creating High Quality Questionnaires

Don’t forget to complete an evaluation of today’s webinar at: http://www.surveymonkey.com/s/4HEvaluationwebinar