“if student ratings are part of the data used in personnel decisions,

one must have convincing evidence that they add valid

evidence of teaching effectiveness” (Wilbert McKeachie,

1996, American Council of Learned Societies p. 3).

UNCW Ad Hoc Committee“Use of SPOTs for RTP”

Report to Faculty Senate, January 18, 2011

Charge to Committee

To investigate and report as to what empirical research has found regarding the validity of SETEs (Student Evaluations of Teaching Effectiveness) in measuring teaching effectiveness (if they are not valid, or of low validity should UNCW be using SETEs in RTP, etc?).

What do UNCW and UNC documents actually say about using SETEs for purposes of RTP?

If SETEs are not valid measures of teaching effectiveness, what should be used to measure teaching effectiveness?

Provide information about how other universities have attempted to resolve these issues.

Develop recommendations for the UNCW Faculty Senate to discuss and possibly vote.

Committee Membership and ProcessMembers (Past and Present RTP chairs, etc.)

Craig Galbraith, (Management, past UNCW RTP Committee member and chairperson, 2005-2010);

John Fischetti (Education; current UNCW RTP Committee member and chairperson, 2010-present);

Barry Wray (Information Systems and Operations Management);

John Taggart (Environmental Studies), Regina Felix (Foreign Language and Literatures), Susan Roberts (Clinical Research, Nursing), Yaw Chang (Mathematics and Statistics; former member

of UNCW RTP Committee)Bruce McKinney (Communications; President, UNCW

Faculty Senate) – ex officio, non voting memberProcess

Bi-weekly meetings, rely on “scholarly” research (not opinions), meeting with student government representatives

Charge 1Are SPOTs a Valid Measure of Teaching Effectiveness? –

NO (particularly Global Measure)Studies indicate between 6% and 20% of teaching

effectiveness captured by SETEsAlmost all early studies were TAs in Freshman courses, with little

control over course – can not be generalized to most university courses

Newer studies indicate SETEs can be easily manipulated by instructors (such as giving chocolates, grades, etc.) or negatively biased (by a single comment, etc.)

Newer studies indicate SETEs really measure “likability” and “attractiveness”

Newer studies indicate high SETEs actually associated with lower student learning

Source: 1,800 students in 116 classes (87 instructors) using standardized learning outcome tests; Galbraith, Merrill & Kline, Research in Higher Education (forthcoming), 2011

Charge 2UNC Policy (section

400.3.1.1) - 1993Some student evaluation

MUST be usedDoesn’t specify weightingWarns against comparing

SETEs with other faculty, departments, university

States that that SETEs are most effectively used to compare changes over time

(Specific sections of UNC Policy are provided in report)

UNCW Policy“SPOT results are appropriately used

for annual merit evaluation summaries, consideration for salary raises, RTP, and post-tenure-review decisions”

“it is strongly suggested that peer and student evaluations be given similar emphasis in personnel recommendations.“

UNCW RealitySPOTs, Q16 are used extensively, and

sometimes almost exclusively, for RTP

SPOTS are typically compared across departments and university (quintiles) in reports

Charges 3 and 4

3: What can be used?Faculty mixed about peer

evaluations, but generally positive about validity

As SETEs became more dominant, peer evaluations became fewer

Few “validity” studies published on peer evaluation

4: Universities are all over the board with SETEsU. of Minnesota (no global

question, direct peer evaluations are weighted more)

Southeast Missouri StateUses IDEA instrumentEach department drives

questionsSETEs can not count more than

33% of measure of teaching effectiveness (based on IDEA Center’s recommendations)

Charge 5: Recommendations Recommendation 1: Eliminate the global question (Q16 from UNCW

SPOTs)  Recommendation 2: For RTP, Q1 to Q15 are not reported with any

department, school, or university averages, quintiles, or categorical statements such as above average, average, etc.

Recommendation 3: Q1 to Q15 be reported for RTP only as a comparison over time for that particular instructor.

Recommendation 4: That some questions on the SETEs be tailored to specific departmental missions and expectations.

Recommendation 5: Quantified SETEs (UNCW) cannot be weighted more than 25% to 33% as an assessment for teaching effectiveness for purposes of RTP.

Recommendation 6: That the qualitative/written comments from student evaluations be more systematically administered and reported in order for them to be used more effectively in RTP decision.

Charge 5: Recommendations Recommendation 7: That the individual faculty member has a right to formally

comment, explain, or respond to student evaluations (either quantitative or qualitative/written comments) for purposes of RTP, and that these comments, explanations, or responses be formally included as part of the SETE reporting process.

Recommendation 8: That the process of peer evaluation of teaching be more uniform between departments and Schools within UNCW in order for them to be used more effectively in RTP decision.

Recommendation 9: That peer evaluations be weighted at least equal to student evaluation of teaching (quantified question and student written comment section) for purposes of RTP

Recommendation 10: That UNCW investigate using the IDEA Center’s evaluation system for student evaluation (http://www.theideacenter.org/node/5) . However, no global question should be used, and any quantified SETE process should never be weighted more than 25% to 35% as an indicator of faculty teaching effectiveness