Slide 1

Ritual or reality: do student evaluations have any effect on teacher thinking and practices? Presentation at the Australasian Higher Education Evaluation Forum (AHEEF), Christchurch, November, 2010 Sarah Stein, Jo Kennedy, Dorothy Spiller, Trudy Harris, Stuart Terry & Lynley Deaker

Slide 2

The Project Unlocking the Impact of Tertiary Teachers Perceptions of Student Evaluations of Teaching Ako Aotearoa (National Centre for Tertiary Teaching Excellence) National Project Fund grant ($150k plus $20k extension) 2010 - 2011

Slide 3

Project Partners From Left: Lynley, Sarah, Trudy, Dorothy, Jo & Stuart Otago Polytechnic Stuart Terry University of Waikato Dorothy Spiller Trudy Harris University of Otago Sarah Stein Jo Kennedy Lynley Deaker

Slide 4

Research question How do the current evaluation processes and practices influence teachers thinking and behaviours in relation to student learning at all stages of the teaching and learning cycle?

Slide 5

Objectives 1.Explore tertiary teachers perceptions about student evaluations and identify how those perceptions impact on their thinking and practices. 2.Explore how tertiary teachers make use of information obtained from student evaluations at all stages of the teaching and learning cycle. 3.Compare teacher thinking and behaviours around evaluations across three participating institutions. 4.Make recommendations about evaluation processes so that evaluations can be optimally used for teaching development and enhancing student learning.

Slide 6

Where we are up to... Literature Review complete Questionnaire - complete, data analysis continues Interviews - underway

Slide 7

Literature Review: Questions gap between acceptance and use to inform practice using multiple sources of evaluation-how could it help teacher emotions and receiving student feedback relationship between teaching beliefs and using evaluations to inform teaching

Slide 8

Literature Review: Questions What interpretation guidance and PD support would help in the use of evaluations to inform teaching? What institutional support and incentives would help to integrate student evaluations and professional development? What are the perceptions about the ownership of the student evaluation process?

Slide 9

Progress to date: the questionnaire Developed a questionnaire for teaching staff with some variation in language to contextualise to each institution Piloted with a small sample from Otago and Waikato Universities Online for 3 weeks from 22 April 2010 and sent to 2445 teaching staff. Received 1065 responses or 45%

Slide 10

Progress to date: interviews 20 interviews are being carried out at each institution due for completion in early 2011 Interviewees selected from the questionnaire respondents who volunteered a range selected to explore stories behind the questionnaire responses

Slide 11

Next Steps data analysis analysing questionnaire data using a range of processes looking at comparisons and variances between and within the groups (ongoing) coding of the qualitative data into themes links with questions that emerged from the Literature Review this analysis has informed the selection of interviewees and the questions to be asked during the interview

Slide 12

Early Data Analysis Questionnaire data

Slide 13

Teacher perceptions of student evaluation data & influences on practice Q9 To what extent do your reasons for using student evaluations influence your teaching decisions? ( E.g., decisions about learning outcomes, teaching strategies, assessment, etc. 1=A great deal, 5=Not at all) Q11 To what extent does your institutions use of student evaluation data influence your teaching decisions? (E.g., decisions about learning outcomes, teaching strategies, assessment, etc. 1=A great deal, 5=Not at all) Q13 My course design refinements are influenced by student evaluation results: ( 1=A great deal, 5=Not at all) Q15 My willingness to try new teaching approaches is constrained by the possible negative effects on my student evaluations. ( 1=A great deal, 5=Not at all )

Slide 14

Q9 To what extent do your reasons for using student evaluations influence your teaching decisions? For example: decisions about learning outcomes, teaching strategies, assessment, etc. 1=A great deal, 5=Not at all Otago Polytechnic differs significantly from the two Universities for Q9 (P-value=0.044) Q9 Teaching decisions influenced by reasons for doing student evaluations

Slide 15

q10 Comment Data - Theme Coding all Institutions

Slide 16

Q 10 comments (reasons for influencing decisions) 1a I believe we must value how students interpret our teaching methods; process and especially how they value feedback with regard to their progress on a particular course or module. Students evolve intellectually and have a global perspective of where they fit in the world and change with it. We must be aware of those changes and expectations of our students globally; academically and culturally.

Slide 17

Q 10 comments (reasons for influencing decisions) 1b Not everything is popular! Being confident (but not arrogant) as a teacher is important as well as being reflective. I continually reflect on my teaching.

Slide 18

Q 10 comments (reasons against influencing decisions) 2a It is my perception that the evaluations are largely a measure of popularity rather than of learning/teaching. I believe that the students are generally not in a position at the time the evaluation is set to comment on their learning outcomes or my teaching.

Slide 19

Q 10 comments (reasons against influencing decisions) 2b After 15 years I can tell if a course has worked or not. I am currently revising a whole course because I or it was not relevant to the students. I can see that in their faces; attentiveness and attendence/ participation.I don't need a survey to tell me that. Although it did confirm it.

Slide 20

Q9 & Years Teaching in the Tertiary Sector Q9 To what extent do your reasons for using student evaluations influence your teaching decisions? For example: decisions about learning outcomes, teaching strategies, assessment, etc. 1=A great deal, 5=Not at all There was a significant difference within the groups based on the number of years experience in tertiary teaching (P-value=0.028)

Slide 21

Q11 Institutional use of student evaluations as an influence on teaching decisions All 3 institutions are significantly different from each other (P-value=0.000), with Otago Polytechnic staff the most influenced. Q11 To what extent does your institutions use of student evaluation data influence your teaching decisions? E.g., : decisions about learning outcomes, teaching strategies, assessment, etc. 1=A great deal, 5=Not at all

Slide 22

Q13 - Course/paper design refinements influenced by student evaluations Both Universities are significantly different to Otago Polytechnic staff (P-value=0.004), but not from each other. Polytechnic staff have more of a perception that student evaluation results influence course/paper design. Q13 My course design refinements are influenced by student evaluation results: 1=A great deal, 5=Not at all

Slide 23

Q15 - Teachers willingness to try new teaching methods constrained by evaluations There is a significant difference between Otago University and the other two institutions regarding teacher perceptions that student evaluation results constrain their willingness to try new teaching methods (P-value=0.000) Q15 My willingness to try new teaching approaches is constrained by the possible negative effects on my student evaluations. 1=A great deal, 5=Not at all

Slide 24

Q4 teachers activity after they receive results The categories that were most frequently used were: Q4(b) Read the open question comments made by the students - 86% rated 1 or 2 Q4(a) Spend time going over the data and responses 80% rated 1 or 2 Q4(g) Actively look for feedback about teaching and assessment 72% rated 1 or 2 Q4(d) Compare the data with previous evaluations 57% rated 1 or 2 Q4(e) Discuss the results with colleagues/teaching team 55% rated 1 or 2 Least used were : Q4(f) Seek assistance with interpreting the results from others 19% rated 1 or 2 with 55% rating 4 or 5 Q4(c) Provide students with feedback on the results 15% rated 1&2 with 62% rating 4 or 5

Slide 25

Q4 teachers activity after they receive results Q4 When you receive the results from your appraisals/evaluations do you 1=always, 5=never There is a significant difference between institutions (F=21.210, p=0.000). The difference is between the two universities and the Otago Polytechnic.

Slide 26

And now. Watch this space.