Australasian Higher Education Evaluation Forum (AHEEF) 2008"Evidence based Decision Making: Scholarship and Practice"

2-3 October, 2008Australasian Higher Education Evaluation Forum (AHEEF)

2008"Evidence based Decision Making: Scholarship and Practice"

2-3 October, 2008

eVALUAte: development and validation of a

teaching survey

Beatrice Tucker, Beverley Oliver and Julie-Ann Pegden

eVALUate surveys for student feedback

eVALUate unit survey (for all current students enrolled in

course work units) eVALUate teaching survey(enables students to give feedback to

individual teachers on their request)

Literature Review• Systems for Measuring Teaching Quality –

controversial, validity is questionable(Pounder 2007, Kulik 2001, Kember & Leung 2008, Remedios &

Lieverman 2008, Burdsal & Harrison 2008)

• Used for different purposes – teacher development, promotion, awards– for improving quality of courses, quality audits,

benchmarking, funding allocation – student choice of study (Tucker, Jones, Straker, & Cole, 2003, Spencer & Schmelkin, 2002,

Ballantyne, 2003; Ory, 2000, Department of Education Science and Training, 2004, Carey & Gregory, 2003)

• Students and teachers have differing opinions about what they perceive to be good teaching (McLean 2001)

eVALUate surveys at Curtin

eVALUate unit survey– for unit review, annual and

comprehensive course review, student choice of units, benchmarking, promotions

eVALUate teaching survey - for teacher reflection and development,

promotions, awards

Development of survey

• 2003 – committee – students – teaching academics– Deans of teaching and learning– academics with expertise in survey design

and evaluation systems (statisticians)

• Literature was reviewed – excellent teacher characteristics, current

instruments used• Unit and teaching items included in

one survey Oliver, Tucker, Gupta, & Yeo (2008)

Method - focus group

StudentsManagers

Teaching academics

de Bono's Six Thinking Hats

Method - item development

Characteristics of a good teacher

Focus Group Literature Other Surveys

Item

eVALUate teaching survey1. Appears knowledgeable in this subject area

The teacher seems to have a good understanding of the subject.

2. Is enthusiastic in teaching this unitThe teacher makes the subject interesting and conveys his or her enthusiasm for the subject.

3. Is well organisedThe teacher has material prepared on time, is punctual and structures activities in ways that help learning.

4. Communicates clearlyThe teacher is easy to understand in face-to-face, online, written and other formats and explains concepts clearly.

5. Is approachableThe teacher encourages students to ask questions and seek help.

6. Provides useful feedbackThe teacher provides timely and helpful feedback so you can learn.

7. Is an effective teacherOverall, this teacher helps you to learn.

eVALUate teaching survey

8. Please comment on [the teacher]'s teaching strengths.

9. Please comment on how you think [the teacher] might improve the teaching and learning in this unit.

Testing and statistical analysis

• Piloted (x4) to student groups in 2003-4 tested for face validity

• Pilot 4 - Feedback from teaching staff and students

Surveys separated• Trialled to all Faculties in 2006 • Factor analysis and Rasch analysis –

tested the categorical scale and psychometric properties of the items (instrument measures multiple teacher attributes and not just one dominant trait)

How students give feedback

Uptake by teachers

0

200

400

600

800

1000

1200

1400

1600

Sem 1 06(trial)

Sem 2 06 Sem 1 07 Sem 2 07 Sem 1 08

No

. of

su

rve

ys

/ te

ac

he

rs

Teaching surveys Teachers

Teaching surveys drive unit survey response rates

0

10

20

30

40

50

60

70

80

90

100

Sem 2 06 Sem 1 07 Sem 2 07 Sem 1 08

Re

sp

on

se

ra

te (

%)

Without teaching evaluations With teaching evaluations

** * *

*significant differencep<0.001

Conclusion• All teaching staff are required to conduct a

teaching survey annually

• Teaching surveys help drive unit response rates

• Teaching eVALUate reports are used to reward staff: for promotion and awards