Designing an Evaluation Plan

Get it rolling…

To generate a good plan means logically working through a series of issues

stakeholders and their concerns

constraints

translate concerns into key evaluation questions

selection of data gathering methods to address key questions that are to be the focus

When do you make a plan?

Planning for evaluation should occur as part of the other planning activities associated with project start up.

Form a team

An evaluation group should be established and basic management issues need to be addressed

Identifying stakeholders

Understanding the stakeholders and the audience of the evaluation report(s) will shape:

the goals/objectives of the evaluation

the questions to be asked and when

the methods of data collection, analysis and reporting

Identifying concerns

Stakeholders will likely differ in their concerns and what they want to find out, but these are not necessary mutually exclusive.

Concerns will vary from project to project.

Concerns will change over the life of the project.

Stages

Pre-implementation Which concerns need to be addressed during

the design and development of the project?

Post-implementation Short term Medium term Long term

Constraints?

These factors will determine the size and scale of the evaluation and what the evaluation team can do practically.

Budget and resources Time Availability of competent staff Pre-specified evaluation objectives,

methodologies and/or reporting procedures Legal or ethical issues Availability of data ‘Political’ considerations

The questions…

You must spend time on getting the evaluation questions right. OR, you may get the wrong answers, or

answers to questions you didn't ask or want to know about.

Action QuestionsHigh Value Questions

Data Gathering

It is best to use a number of data gathering techniques and/or sources of data to substantiate findings. This is known as a process of triangulation

— the use of multiple investigative methods or information sources to get the answer to the question at hand.

Data Sources

Students — prospective, current, past, withdrawn

Colleagues — teaching partners, tutors, teachers external to the project

Discipline/instructional design experts

Professional development staff

Graduates and employers

Documents and records — teaching materials, assessment records, past SETLs, assessment statements and tasks

Selecting a methods…

Paradigm for the study (empirical, interpretive, critical theory-based, pragmatic)

Time involved in preparing to use the particular method/tool (e.g. preparation of a bank of questions for a questionnaire)

Time involved in gathering or recording the data — on the part of the data collector; on the part of the 'evaluee/s'

the time needed to analyse and report the data

the scale involved — the number of students, staff required for valid/authentic data.

Method – Now what?

The skill/expertise required to use the method

The expertise, personnel and/or resources required to analyze and/or report the data.

Storing the data

Making sure that data is safe and not lost

Thinking through filing categories; e.g. by question type; data source; data method

Considering confidentiality requirements & other safeguards arrangements to access data

Ethical Considerations

In any evaluation, the rights and welfare of 'subjects' need to be respected and protected.

Privacy: Some data gathering techniques may be perceived as an invasion of privacy if prior consent on the part of the subject(s) has not been gained.

Confidentiality: Much information that subjects provide is given in confidence unless specific permission to use 'private' information.

Websites to explore

http://www.managementhelp.org/evaluatn/fnl_eval.htm

http://www.utas.edu.au/pet/sections/developing.html