Test development cycle Evaluation, item analysis and feedback KAYSERI, 29-31 JANUARY 2014

Aims

!  Test development cycle !  Test evaluation !  Constructive feedback !  Item banking

Test development cycle

Plan

Write

Use Evaluate

Revise

Test evaluation

Test evaluation

!  How we view statistical data

!  Other, less quantifiable factors?? ! commitment, aptitude, attitude to learning,

rapport with teacher and other learners, creativity, intelligence, etc.

objective numerical fact

Assessing a test

Statistical evaluation !  objective results

!  based on calculations

!  expressed in numerical data ! fact?

!  doesn’t recognise external factors

!  can only be black-white – or other factors need to be calculated

Holistic evaluation !  subjective impressions

!  based on own experience

!  expressed as opinion

!  takes into account unmeasurable factors

!  can be shaded – depending on circumstances

!  finding out how difficult our test is !  Facility Value

!  finding out how our students performed in comparison to one another (a ‘norm’) !  Discrimination Index

Statistical evaluation: item analysis

see handout

Constructive feedback

Constructive feedback

!  clear and direct

!  improvements that can be made ! courses of action

!  ‘sandwich’ your feedback !  something positive about the subject: „Your test is really comprehensive.” !  your main general criticism: „However, I think it’s too long – maybe you’d like

to cut a few items?” !  specific critical points: „In Task 6, I don’t think you’ll need twelve items – six

would do just as well. Some of them are quite problematic anyway, as there are several possible answers. Maybe you could just keep the unambiguous ones?”

!  conclude with something constructive: „I think with the revisions we talked about you’ll cover everything that needs testing at this point.”

Test checklist (adapted from Alderson et al., 1995) 1 Don’t just look! Approach each item as a test taker.

2 To do your test tasks, get someone who doesn’t know them.

3 Nobody writes good tests alone. Ask for others’ feedback.

4 Be prepared to change things. We all write bad items.

5 Revise any item where you get an unexpected response.

6 Check your tests against the syllabus. Anything missing?

7 Assess the test method – is it the best way to get results?

8 Is there a context? Does it feel real?

9 What are your marking criteria? Do students know them?

10 What if your own judgement is different from the test score?

Item banking

Item banking

What? !  test items !  identifying data

!  content

!  level

!  course

!  format

!  key and scoring !  item analysis: FV and DI !  notes of previous use

Why? !  workload saving

!  pre-trialled items

!  informed selection of items for a particular use

!  more balanced tests

!  consistent standards

Thank you!

zoltan.rezmuves@digimedia.hu

consonantvoiced.blogspot.com