How’s it Working? Evaluating Your Program
description
Transcript of How’s it Working? Evaluating Your Program
How’s it Working? Evaluating Your Program
MAAPS Conference, 7 May 2010Debra Smith & Judah Leblang
Program Evaluation & Research GroupSchool of Education, Lesley University
PERG Founded 1976
Over 600 program evaluation and research studies in various educational settings
Also offers professional development and consultation
Session participants will:
Be introduced to the basics of program evaluation through an example
Define a question or questions about their own program
Identify methods for collecting data that would help to answer their question/s
Discuss next steps
What is program evaluation?
A type of applied research focused on systematically collecting and analyzing data to help answer questions about a program, or some aspect of a program, in order to make decisions about it.
Purposes
Accountability
Program development
Generating knowledge
Formative vs Summative
Formative evaluation offers feedback along the way to improve programs
Summative evaluations “sum up” the results of a program at the end of a period of development or implementation.
Audiences
Funders Program leaders Program participants Organizational partners Others
Evaluation process
1. Goals/ logic model
2. Questions
3. Evaluation plan4. Data
collection
5. Data analysis
6. Reporting
PROGRAM
An example: Evolutions
• After school program begun in 2005, connected with Peabody Museum of Natural History at Yale University—initially involved approximately 40 low SES/ minority students
Evolutions program goalsTo provide opportunities for students to:• Prepare for post-secondary (college)
education; • Learn about scientific—and other
careers; • Expand their knowledge of and interest
in science (science literacy); • Develop transferable skills for the
future; and • learn about the Peabody
Museum/museum careers.
Logic models
Map a coherent chain of connections between goals, resources, activities and what you expect (short term), want (over an intermediate period) and hope (in the long term) to happen.
They also reflect your assumptions and theory of action or change.
Logic Model
Key Concepts
Category Resources or Inputs
Activities—
Outputs Short-term outcomes
Long-term outcomes
General information
Staff, funds, materials, space, etc
What we plan to
do/who we will do it
for
The results of
our program—direct outputs
Outcomes (changes) at completion
of the project year or soon after
Outcomes (changes)
several years
beyond completion
of the project
And EVO exampleCategory Resources
or InputsActivities— Outputs Short-term
outcomesLong-term outcomes
General information
Staff, funds, materials, space, etc
What we plan to do/who we will do it for
The results of our program
—direct outputs
Outcomes (changes) at
completion of the project
year or soon after
Outcomes (changes)
several years beyond
completion of the project
EVO examples
Full time project
director funds from
Peabody Museum and other funders, classroom space, etc.
In-depth exploration of:
science topics
tours of Peabody
collections Yale scientist
labs
Students will meet:
at least 6 scientists students
will visit no less than 2
natural history museums
Students will: learn skills
associated with producing
a museum exhibition
Understand key science
themes
Students will: understand
different types of careers
within disciplines
understand the college application
process
be inspired to pursue a
career in the sciences
Goal
Rationale
Assumptions
Resources Activities Outputs
Short-term outcomes
Mid-term outcomes
Long-term outcomes
Logic models may look different..
Develop a logic model for your own program/ project
Evaluation process
1. Goals/ logic model
2. Questions
3. Evaluation plan4. Data
collection
5. Data analysis
6. Reporting
PROGRAM
Questions: Think Goldilocks
Specific but not too detailed
Important but not too broad in scope
Key Questions: Part One
How does EVO prepare students for college or high school?
How are EVO students involved in developing an exhibit at the museum?
Do students develop increased “science literacy,” as defined by EVO staff?
Key Questions: Part Two
How (if at all) do students express more confidence about and interest in doing science?
Are students more aware of careers in science?
How (if at all) do students demonstrate increased knowledge of the college application process, and develop criteria for choosing a college that meets their needs?
What questions do you want to answer about your program?
Evaluation process
1. Goals/ logic model
2. Questions
3. Evaluation plan4. Data
collection
5. Data analysis
6. Reporting
PROGRAM
Data collection methods
Observation
Interviews/ focus groups
Surveys
Document/artifact review
PERG Evaluation MatrixEvolutions 2005-06 Data collection activities>>
EVALUATION QUESTIONS:
Observe Evo students
Student focus groups
Interview project director
Review project docs and artifacts
Examine pre-post survey
Student prep for college/academic planning
√ √ √ √ √
Student involvement in museum exhibit
√ √ √
Students' development of science literacy
√ √ √ √
Student learning √ √ √ √ √
Students' interest in science/environment
√ √ √ √ √
Students' confidence in doing science
√ √ √
Students' interest in/knowledge of science careers
√ √ √ √
Technical considerations: Validity
Will the data answer the questions?
Are we asking the right questions?
Triangulation
Is there adequate triangulation (use of multiple methods and/or data sources) to ensure validity?
Drafting your own matrix: What data will help you answer your questions?
Evaluation process
1. Goals/ logic model
2. Questions
3. Evaluation plan4. Data
collection
5. Data analysis
6. Reporting
PROGRAM
Collecting data
Make sure your plan is doable given time and resources available.
Design instruments to focus your data collection, ensure consistency and avoid bias.
Be organized: take notes, develop a system for tracking/ filing your data.
Collecting data
Communicate clearly about what you are doing, why and how the findings will be shared and used.
Be mindful of human subjects protections. Does your organization have an institutional review board (IRB)?
The First Year: site visit
On-site data collection• Focus groups with students• Interviews with director, project staff• Observation of end of year event• Parent interviews
Evaluation process
1. Goals/ logic model
2. Questions
3. Evaluation plan4. Data
collection
5. Data analysis
6. Reporting
PROGRAM
Analyzing data
What stands out?
What are the patterns?
What are the similarities?
What are the differences?
Is more information needed?
Reliability
Are the patterns in the data, or judgments about the data, consistent?
Validity, again
Is the data helping you answer the questions?
Is the data credible?
Evaluation process
1. Goals/ logic model
2. Questions
3. Evaluation plan4. Data
collection
5. Data analysis
6. Reporting
PROGRAM
Reporting
Consider purpose and audience/s
Reporting relevant findings, questions/ recommendations
Engaging stakeholders in discussion
Using findings to inform next steps
Results of the first-year evaluation
• The impact of the evaluation on EVO—more focused program, clearer objectives, suggestions for sustainability.
• Evidence of program success: Retention, student engagement, positive changes in students’ view of doing science and scientists.
The Ongoing Evaluation--shaping the program:
• Implementation of evaluator suggestions—examples: informational interviewing, developing a smaller exhibit, refining requirements for students
EVO: 2006-Today
• Continued development and expansion of EVO—2006 until today: Expansion of the program from approximately 40 to more than 80 students, introduction of internships and Sci Corps.
– Different areas of science focus—environmental awareness, geoscience, depending on funding sources.
Evaluation resources
W.K. Kellogg Foundation Evaluation Handbook www.wkkf.org/Pubs/Tools/Evaluation/Pub770.pdf
Kellogg Logic Model Development Guide www.wkkf.org/Pubs/Tools/Evaluation/Pub3669.pdf
Basic Guide to Program Evaluationwww.managementhelp.org/evaluatn/fnl_eval.htm
Evaluation resources
Program Evaluation & Research GroupLesley University29 Everett St.Cambridge, MA 02138www.lesley.edu/[email protected]