C.E.L.T Developing a Comprehensive Faculty Evaluation System Jeff Schlicht, Ph.D. Associate...

C.E.L.T

Developing a Comprehensive Faculty Evaluation System

Jeff Schlicht, Ph.D.Associate Professor

Department of Health Promotion and Exercise [email protected]

Measurement vs. Evaluation

Measurement… is the process of systematically assigning numbers to the individual members of a set of objects or persons for the purpose of indicating differences among them in the degree to which they possess the characteristics being measured.

Robert L. Ebel, “Measuring Educational Achievement”, 1965

Measurement vs. Evaluation

Evaluation is the process of interpreting a measurement by comparing it to a value (or set of values) that represent an agreed upon standard. This process involves judgment.

A Comprehensive System = Measurement and Evaluation

In order to develop a comprehensive faculty evaluation system, we must:

1) Develop technically good (valid & reliable) measurement instruments

2) Agree upon a university standard for desirable faculty performance

How Do We Currently Measure Faculty Teaching Performance?

In the HPX department we use:

1) Student rating forms

2) Peer observation (in classroom)

Are either of these forms of measurement valid and/or reliable?

The answer is 1) we don’t know and 2) no

Student Ratings of Faculty Performance

The development of a valid and reliable questionnaire, i.e. a student rating form, is a technical process that has a body of science behind it.

Our department does not know who created the current rating form, and whether it has ever been tested for validity and reliability.

Student Ratings of Faculty PerformanceDos and Don’ts

DO include a balanced response scale• equal number of positive and negative responses• mirror-image opposites

– Strongly agree agree disagree strongly disagreeDO develop items (questions) important to faculty AND

studentsDO conduct field tests of the items ( = collect data) to do

reliability and validity analysesDO develop norms (a year’s worth of data) to allow fair

and appropriate interpretation (i.e. comparison)DO consider using a professionally designed student

rating form

Student Ratings of Faculty PerformanceDos and Don’ts

DON’T• Have two items in one question (double-

barreled response item)• Simply hand back a summary sheet of student

ratings to instructor – include personal consultation to foster instructional improvement

• Consider individual student responses in faculty evaluation

Peer Observation

The short answer: DON’T do it for personnel decisions

(can be useful for faculty enrichment)

The addition of another faculty member or administrator into the classroom ALWAYS alters the classroom dynamic in such a way as to provide an abnormal view of what usually happens there

THOSE DATA ARE NEVER VALID

Peer Observation

If you must do it for faculty evaluation consider:

• multiple visits by multiple (4) peers– 8 visits = adequate sampling– develop valid, reliable checklist

• train observer team in use

– prepare students, instructor– schedule post-observation conference

• videography

8 Steps to a Comprehensive Faculty Evaluation System

Step 1: Define the faculty role model

• What should be evaluated?– Start at the department level

• Have faculty write down everything they do at WCSU to create a master list of responsibilities

• Determine faculty roles– What are the things we are accountable for?

• Assign the items from the master list of faculty responsibilities to the various faculty roles


Step 2: Determine faculty role model parameters on a University (system?) level

• How important should each role be?– Static vs. dynamic role model

• Static = fixed weights

• Dynamic = range of weights

This is missing from our current system, and therefore we do not control subjectivity as well as we could


Step 2: Determine faculty role model parameters on a University (system?) level

• Static Role Model– Load Credit = 50%– Creative Activity = 25%– Productive Service = 15%– Professional Service = 10%


Step 2: Determine faculty role model parameters on a school, then University (system?), level

• Dynamic Role ModelMinimum

Maximum

– Load Credit 35% 75%– Creative Activity 20% 50%– Productive Service 5% 45%– Professional Service 0% 40%


Step 2: Determine faculty role model parameters on University (system?) level

• Dynamic Role Model– The advantage of a dynamic role model is that it

allows various levels of the University to play to their strengths

• Allow each department to set dynamic range values, then select extreme max. and min. as University standards


Step 3: Use observable or otherwise documentable achievements/products/performances to define all the roles in the faculty role model

• Teaching (student interactions that promote learning)– Content appropriateness/currency

– Instructional design skills

– Instructional delivery skills

– Instructional assessment skills

– Course management skills


Step 4: Define weights for each component of a single role – departments should be allowed to have their own weights, as long as they fit into the University structure. This embeds department values into the evaluation design:

• Teaching min max

– Content appropriateness/currency 10% 40%

– Instructional design skills 10% 30%

– Instructional delivery skills 20% 60%

– Instructional assessment skills 20% 50%

– Course management skills 0% 30%


Step 5: Identify where you will collect the data to evaluate the roles• Use sources that have first hand experience with the performance in

question – preferably multiple sources of input• Teaching

– Content appropriateness/currency• syllabus review by peer• Self-review

– Instructional delivery skills• Student evaluation• Peer evaluation???• Self-review

– Course management skills• Chair or administrative assistant evaluation• Self-review


Step 6: Define weighting for information provided by different sources

• Teaching min max– Students 15% 50%– Self 15% 40%– Peer 15% 30%– Chair 15% 30%


Step 7: Determine how to gather information

• Questionnaire

• Interview

• Checklist


Step 8: Design data collection instruments

Evaluation for Promotion and Tenure

Evaluation is the process of interpreting a measurement by comparing it to a value (or set of values) that represent an agreed upon standard.

This process involves judgment.

In a system like the one described today, evaluation is an on-going process involving multiple sources and evaluators. We are constantly comparing ourselves against the WCSU Faculty Role Model, so we have immediate feedback about our performance.

Evaluation for Promotion and Tenure

Promotion and tenure decisions can be based on comparing our performance against a

pre-determined, minimum required composite Faculty Role Model score. In

other words, prior to becoming eligible for advancement, we would already know the

outcome.

Comprehensive Faculty Evaluation

Practical steps we can take to improve the system:• Initiate a discussion about what it means to be a

faculty member at WCSU in the 21st century• Discover new ways to measure faculty performance,

tests those ways to make sure they are valid and reliable, and create a feedback loop that uses that information to enhance faculty performance

• Lobby the union and the University to more actively support faculty enrichment

Final Thoughts About Faculty Evaluation

Why should we do it?

• Promotion and tenure is important, but…

the true value of assessment lies in its ability to direct faculty enrichment. If evaluation is not tied to enhancing faculty performance, it will be perceived as punitive, a hurdle, rather than as a path toward self-improvement.

Acknowledgements

These materials are summarized from the ideas of Raoul A. Arreola, University of Tennessee

Health Science Center, presented at the Center for Educational Development and Assessment

professional enrichment seminar in Orlando, FL. March 10-11, 2008, and available in the text:

“Developing a Comprehensive Faculty Evaluation System,”

3rd edition, Arreola RA, Anker Publishing Co., Bolton, MA 2007

C.E.L.T

Developing a Comprehensive Faculty Evaluation System

Jeff Schlicht, Ph.D.Associate Professor

Department of Health Promotion and Exercise [email protected]

C.E.L.T Developing a Comprehensive Faculty Evaluation System Jeff Schlicht, Ph.D. Associate...

Documents

Transcript of C.E.L.T Developing a Comprehensive Faculty Evaluation System Jeff Schlicht, Ph.D. Associate...