Interrogating Course Evaluation

UWC Induction 20 March 2015

Rita Kizito - kizitorita@gmail.com

Aim of this part of the session

• Interrogate ‘evaluation’ from own

perspective

• Come up with an evaluation

strategy

Common Purpose of evaluation

• Required by universities near end of

each course *

• Can be used by

administration/departments as an

important element in making

decisions about promotion

• Could be a source of great pride or

trepidation

(Cashin, 1999; Clayson, 2009)

Purposes of evaluation

auditdevelopment

“quality assurance”

(Biggs, 2003; Edström, 2008; Patton, 1997)

appraising teachers

developing/ improving courses and teaching effectiveness

“quality enhancement”

What should the focus of the

evaluation be?

What should the focus of the

evaluation be?

• To check that something is working

What is that something ?(e.g. course,

degree programme, activity)

What do we mean by ‘working’ ? (what goal

should be achieved?)

• To figure out how ‘it’ can be improved

What should the focus of the

evaluation be?

Teaching

( teacher)

Learning (outcomes)

Teaching (process)

Learning

(process)

“Rather than ratings, teachers

should be asked to include

their course analyses in their

teaching portfolio in order to

show their ability to both

analyze the student learning

experience and the quality of

the student learning outcomes ,

and to improve these with

adequate course development

measures”

(Edström, 2008, p.104).

“Evaluation is often viewed as a test of effectiveness – of materials, teaching methods or whatnot - but this is the least important aspect of it. The most important is to provide intelligence on how to improve these things”

What should the purpose of

evaluation be ?

( Bruner, 1966, as cited in Ramsden, 2003, p. 233).

Typical questions

• Have student attitudes changed?

• Has the lecturer approaches changed?

• Have student learnt something in class?

• What is happening in class?

• Have classroom practices changed?

• Are students engaged?

• Am I meeting students’ needs?

Evaluation strategies – student

attitudes & perceptions

Have student attitudes changed?

• Likert-scale questions & surveys

• Could be used to measure the effect of a

course, a degree or a long term change

over a number of years

Evaluation strategies – Lecturer

approaches

Have lecturer approaches to teaching

changed?

• Teaching Practices Inventories

• Can help departments reflect on teaching,

by allowing comparisons between different

courses/departments

Evaluation strategies – Student

learning

Have students learned anything?

• Measure student performance

• Important to standardise testing (using

locally developed or already developed

tools)

• Important to conduct pre and post-testing,

before and after a learning intervention

Is there evidence that the Graduate Attributes

have been adopted by the students?

Evaluation strategies – classroom

practices

What is going on in the class?

• Video recordings

• Peer observations (fellow lecturers) using

observation protocols

• This data can be used in your Teaching

Portfolio

Have classroom practices changed?

• Use of innovative teaching & learning

practices

• Providing useful feedback etc.

Evaluation strategies – student

engagement

Are students engaged?

• Observation of about 10 students to see if

they are engaged by recording student

activity

• What teaching & learning activities result in

high levels of student engagement?

Evaluation strategies – student

learning needs

Are students learning needs being met?

• Focus groups or Interviews

• Anonymous surveys/midterm surveys

(online, or on paper) – letter to the

facilitator

• Assignment by assignment survey (time it

took, approach, resources used etc. )

• Keep, Start, Stop

Kirkpatrick’s (1994) four level model

Level What is measured Examples

1 Reaction - changes in perception , satisfaction levels

How students feel about the learning experience

Feedback forms

2 Learning – changes in knowledge, skills, attitudes

Increase in student knowledge and skills

Informal/informal assessment before and after learning interventions

3 Changes in behavior/practices

How far learning is applied in practice resulting in personal changes

Observations and interviews of students over time

4 Results - noticeable changes in results or conditions.

How far the module/course impacts on program or institutional factors ( student performance, retention, throughput).

Use institutional data to identify whether the program/module shifts the nature of student participation/ performance/engagement.Student surveys.

Some Drawbacks

• Students are not always good at evaluating

teaching effectiveness. Popularity is usually

mistaken for good teaching

• Response rate is always less than 100%

• Susceptible to bias - voluntary

participation/polarization

• Questions are usually general –missing the

finer details of practice

The UWC Evaluation Guideline document

Instruments

GENERIC

EVALUATION DISCIPLINE

SPECIFIC

Question Bank

A standard set of questions that

students are invited to answer

anonymously at the end of each

course. ( often, the results are given

weight in promotion and tenure

decisions)

Run centrally and completed by all

students to provide the institution with

data on how students are performing

with regards to institutional strategies

and plans

Your evaluation strategy

Which evaluation methods could you consider

implementing in your teaching?

References

• Biggs, J. (2003). Teaching for quality learning at university: What the

student does, Buckingham, UK: SRHE and Open University Press.

• Cashin, W. E. (1999). Student ratings of teaching: uses and misuses. Ch 2,

In P. Seldin and Associates (Eds). Changing practices in Evaluating

Teaching: A practical guide to improved faculty performance and

promotion/tenure decisions. Pp 25-44. Bolton, MA. Anker Publishing.

• Clayson, D. E. (2009). Student evaluations of teaching: Are they related to

what students learn? A meta-analysis and review of the literature. Journal

of Marketing Education. 31 (1), 16-30.

• Edström, K (2008), Doing course evaluation as if it matters most. Higher

Education Research and Development, 27 :2: 95-106

• Kirkpatrick, D.L. (1994). Evaluating Training Programs. The Four Levels.

San Francisco: Berrett- Koehler.

• Ramsden, P (2003). Learning to teach in higher education ( 2nd edition).

London: Routledge Falmer.

• Schimpf, N (2015) Evidence-based Active Learning:

Evaluating Classroom Practices. Presentation made at the University of the

Western Cape

Thank you