RELIABILITY BY DESIGNPrepared by Marina Gvozdeva, Elena Onoprienko, Yulia Polshina, Nadezhda Shablikova

Outline

• Factors affecting reliability• Ways to increase reliability• Classroom test reliability• Tips for classroom tests

Two Components of Test Reliability

Candidates’ PerformanceReliability of the scoring

Ways to increase reliability

Factors affecting reliability:

• test factors;• administrative factors;• affective factors.

(Coombe et al., 2007:xxiii)

• fluctuations in the learner;• fluctuations in scoring;• fluctuations in test administration.

(Coombe et al. 2010)

Factors affecting reliability:

Reliability vs validity

E.g. 100 MCQ translation items: may be reliable, but will not be valid as a writing

test.

Increasing reliability contributes to validity: an unreliable test cannot be valid.

Reliable, not valid Not reliable, not valid

Reliable and valid

Reliability vs validity

Ways to increase reliability

Get enough samples of behaviour:• the more items there are in a test, the more

reliable it is;• additional items should be independent of

each other and of existing items;• a test should be neither too long nor too

short.

• do not allow test takers too much choice;• write clear items with simple instructions;• include items with a range of difficulty;• ensure that test materials are easy to read

and have a clear layout;• make sure test takers know what to expect

in the test.(Hughes, 1989:39)

Ways to increase reliability:

Compare four tasks

a) Write about tourism.b) Write a composition on tourism in this country.c) Write a composition on how we might develop the tourist

industry in this country.d) Discuss the following measures intended to increase the

number of foreign tourists coming to this country:i. more/better advertising and/or information (where? what

form should it take?);ii. improve facilities (hotels, transportation, communication

etc.);iii. training of personnel (guides, hotel managers etc.).

(Hughes, 1989:38)

Ambiguous items

Complete the sentences using the words and word combinations from Essential Vocabulary:

1. He has been promoted three times this year. No doubt he is … 2. Yes, that party was a disaster but it was not me who organized it, so

you …3. I am thinking of buying this laptop but I am not sure if I can afford it.

Could you …4. I think politics is really boring – it ….5. The movie was very touching but I managed to … II. Name: 6. three types of waste;7. three ways to dispose of garbage.

Scoring reliably:

• use items that permit scoring which is as objective as possible;

• make comparisons between candidates as direct as possible;

• provide a detailed scoring key;• train scorers.

Reliability for speaking assessment:

• designing the rating process – rating scale;• reporting scores and giving feedback –

setting cut scores;• assuring intrarater / interrater reliability.

Reliability for classroom speaking assessment

Intrarater / interrater reliability – training raters:

• understanding criteria for assessment;• agreement with other raters;• consistency of performance.

Reliability of teacher made tests:

• increasing number of items increases reliability;

• moderate difficulty level increases reliability;• having items measuring similar content

increases reliability.

Tips for writing short answer/completion items

There should be only one short,concise answer.

Allow for partial credit.

(Coombe C. et al, 2010:30-45)

Tips for cloze/gap-fill items:

• ensure that answers are concise;• provide enough context;• develop and allow for a list of acceptable

responses;• don’t put a gap in the first sentence of a

paragraph or text. (Coombe C. et al, 2010:34)

Tips for true/false questions:

• write items that test meaning rather than trivial detail;

• questions should be written at a lower level of language difficulty than the text;

• consider the effects of background knowledge;

• questions should appear in the same order as the answers appear in the text.

(Coombe C. et al., 2010:30)

• make sure you paraphrase questions in simple clear language;

• avoid ‘absoluteness’ clues;• add another option where plausible to

decrease the guessing factor. (Coombe C. et al., 2010:30)

Tips for true/false questions:

Tips for writing matching items:

• give more options than premises;• number the premises and letter the options;• make options shorter than premises;• avoid widows;• make it clear to test takers if they can use

options more than once. Coombe C. et al., 2010:33

Tips for writing good MCQ:

• write MCQ that test only one concept;• provide as much context as possible;• keep sensitivity and fairness issues in mind;• standardize the number of response options;• one response option should be an

unambiguous correct or best answer.(Coombe C. et al, 2010:25-7)

• all response options should be similar in length and level of difficulty;

• avoid writing absurd or giveaway distracters;• avoid extraneous clues;• make the stem positive.

(Coombe C. et al, 2010:25-7)

Tips for writing good MCQ: