Evaluation Framework, Methods, Tools and Data Analysis

32
UNLEASH the POWER of the Evaluation Framework, Methods, Tools and Data Analysis

description

Evaluation Framework, Methods, Tools and Data Analysis. Agenda. Why Evaluate? Evaluation Framework Purpose of the Evaluation Evaluation Plan (Methods, Sample) Evaluation Tools Analyse and Summarize Results Presenting Information Wrap Up and Questions. Discussion Questions:. - PowerPoint PPT Presentation

Transcript of Evaluation Framework, Methods, Tools and Data Analysis

Page 1: Evaluation Framework, Methods, Tools and Data Analysis

UNLEASH the POWER of the

Evaluation Framework, Methods, Tools and Data Analysis

Page 2: Evaluation Framework, Methods, Tools and Data Analysis

Agenda1. Why Evaluate?2. Evaluation Framework3. Purpose of the Evaluation4. Evaluation Plan (Methods, Sample)5. Evaluation Tools6. Analyse and Summarize Results7. Presenting Information8. Wrap Up and Questions

Page 3: Evaluation Framework, Methods, Tools and Data Analysis

Discussion Questions: What do you currently evaluate in the literacy service

planning process?

Why do you conduct evaluations?

Page 4: Evaluation Framework, Methods, Tools and Data Analysis

Why Evaluate? Provide an opportunity to assess and correct your

course on an ongoing basis

Evaluation is important for accountability purposes

Evaluation is important for learning – which leads to better quality practice when widely shared

Page 5: Evaluation Framework, Methods, Tools and Data Analysis

Determine how the Literacy Service Plan has fared in action

Assess the effectiveness of the Literacy Service Plan in terms of its perceived intentions and results

Assess the effectiveness of the literacy service planning process (to be able to make improvements in the process)

Page 6: Evaluation Framework, Methods, Tools and Data Analysis

Evaluation Framework1. Identify the purpose of, and audience for, the evaluation

results2. Identify what information needs to be collected and from

whom3. Select from different research methods4. Select sample/population5. Design evaluation tools6. Administer evaluation tools7. Analyze and summarize results8. Present information

Page 7: Evaluation Framework, Methods, Tools and Data Analysis

Identify the Purpose and Audience of the Evaluation Results

Need to be clear about the purpose of the evaluation. What are you evaluating?

Five Forms of Evaluation Effort evaluation – inputs Performance evaluation – outputs Effectiveness evaluation – outcomes Efficiency evaluation – costs Process evaluation – what worked well and how could the

process be improved

Page 8: Evaluation Framework, Methods, Tools and Data Analysis

Formative or summative evaluation?

Formative Evaluation To collect information that can be used primarily for

development and improvement

Summative Evaluation To make an overall judgement about the

effectiveness of a program/service/process

Page 9: Evaluation Framework, Methods, Tools and Data Analysis

When you construct your evaluation questions, you will refer back to the purpose of the evaluation to make sure the questions you ask will produce the information you want

Page 10: Evaluation Framework, Methods, Tools and Data Analysis

Discussion Question: Which form(s) of evaluation would you use in the

literacy service planning process and why?

Effort evaluation – inputs Performance evaluation – outputs Effectiveness evaluation – outcomes Efficiency evaluation – costs Process evaluation – what worked well and how

could the process be improved

Page 11: Evaluation Framework, Methods, Tools and Data Analysis

Develop an Evaluation Plan

What information will be collected

(focus)

How the information will be

collected (methods)

Who or what sources will provide the information

(sample, sources)

Purpose of the Evaluation

Page 12: Evaluation Framework, Methods, Tools and Data Analysis

Research Methods

Asking people for information

SurveysInterviewsFocus groups

Observing and analyzing existing materials

Look at data that already exists

Two basic ways to collect information:

Page 13: Evaluation Framework, Methods, Tools and Data Analysis

Samples

Questions to Ask:

How many people or pieces of print material should be included in your evaluation?

Which people or print material should be included in your evaluation?

How should they be selected?

Page 14: Evaluation Framework, Methods, Tools and Data Analysis

Evaluation Tools

Questions are the key!

Questionnaires/Surveys

Key Informant Interviews

Focus Groups

Page 15: Evaluation Framework, Methods, Tools and Data Analysis

Designing Questions

Closed and open ended questions

For written surveys, list all possible options for closed questions, including “no opinion” or “don’t know”

Page 16: Evaluation Framework, Methods, Tools and Data Analysis

Open EndedWhat are your hobbies? (please list below)

_______________________________

ClosedWhat are your hobbies? (please select all responses that apply to you) Reading Running Knitting Eating Other (please list): ___________________

Do you like ice cream? (please select the most appropriate response) Yes No Don’t Know

Page 17: Evaluation Framework, Methods, Tools and Data Analysis

Be sure that questions are neutral - watch out for the halo effect

Ask questions so that people can admit to none of the responses (what, if any,…)

Avoid agree or disagree statements

Avoid double-barrelled questions

Page 18: Evaluation Framework, Methods, Tools and Data Analysis

Double-Barrelled Questions:

What feedback do you have about our parenting groups and our youth groups?

How would you rate your level of satisfaction with our volunteer training program and ongoing support for volunteers?

Page 19: Evaluation Framework, Methods, Tools and Data Analysis

Avoid:

Sensitive or embarrassing questions

Double negatives (Do you agree or disagree that nonprofits should not receive government funding?)

Too many alternatives

Unimportant questions

Complex language (jargon)

Page 20: Evaluation Framework, Methods, Tools and Data Analysis

Analyze and Summarize Results

What do you do with all of the information you’ve gathered?

Page 21: Evaluation Framework, Methods, Tools and Data Analysis

Data Analysis

Quantitative Data Data which describes reality using numbers

Qualitative Data Data which describes events, persons, etc.

without the use of numbers (words)

Page 22: Evaluation Framework, Methods, Tools and Data Analysis

Quantitative Data Analysis

Tabulate the # of responses for each category in the questions

Average, range, most frequent response

62% of clients are satisfied with the services provided by our organization

Youth programs received an average rating of 3.5 out of 4

All services were rated between 2.82 and 3.43 out of 4

Page 23: Evaluation Framework, Methods, Tools and Data Analysis

Qualitative Data Analysis

Read the responses and identify general themes

Sort each individual response into themes

Count # of responses in each theme to determine priority

Page 24: Evaluation Framework, Methods, Tools and Data Analysis

Considering Findings

Four processes to make sense of evaluation findings:

1. Description and Analysis2. Interpretation3. Judgement4. Recommendations

Page 25: Evaluation Framework, Methods, Tools and Data Analysis

Description and Analysis: Organizing the raw data into a form that reveals the basic results. Presents the facts of the case and the actual data

Interpretation: This goes beyond the data to meaning and significance. What do the results mean? What’s the significance of the findings? Why did the findings turn out this way? What are the possible explanations?

Page 26: Evaluation Framework, Methods, Tools and Data Analysis

Judgement: Bringing values to bear on the analysis and interpretations. To what extent and in what ways are the results positive or negative? What is good, bad, desirable or undesirable?

Recommendations: What should be done? What are the action implications of the findings?

Page 27: Evaluation Framework, Methods, Tools and Data Analysis

Presenting Information Data needs to be arranged, ordered and presented in a

reasonable format that allows readers to quickly detect patterns in the data

Visual presentations of information: Bar graph Scatterplot Figures (graphs and other diagrams)

Pie chart Flow charts Line graphs Tables

Page 31: Evaluation Framework, Methods, Tools and Data Analysis

Wrap Up and Questions

Page 32: Evaluation Framework, Methods, Tools and Data Analysis

References

Ministry of Tourism and Recreation. (1982) Enjoying Research. Government of Ontario.

Patton, Michael Quinn. (1986) Utilization-Focused Evaluation. California: SAGE Publications.