STANDARD SETTING Prepared by Ludmila Kozhevnikova and Viktoria Levchenko Based on material by...

STANDARD SETTING

Prepared by Ludmila Kozhevnikova and Viktoria Levchenko Based on material by Anthony Green

Aims of the unit:

• to familiarize students with the main concept of standards in language setting;

• to explain the main principles and terms of standard setting;

• to train participants in a common interpretation of the CEF levels;

• to give practice in aligning examinations to CEF.

Lecture outline

• Standards in language testing.• Issues involved in setting linguistic standards.• Standards for tests/testers/test-takers. • CEFR: principles and issues.• Standard-setting process. The notion of a cut score. • Standard setting and the CEFR:

– stages– methods– concerns.

STANDARDS IN LANGUAGE TESTING

Standard(s)

…refer(s) to:•A level of performance required or experienced.•A set of principles which can be used as a basis for evaluating what language testers do. Standards in this sense may lead to codification in an agreed set of guidelines or Code of Practice. Such codification indicates a concern to establish professional ethics.

(Davies, A. et al. (1999, p. 185))

Standards involve issues of:•uniformity•authority•evaluation.

(Crowley 1989)

What are standards?

Languages are plural, fluid, creative, unpredictable: ‘in a real and defensible sense every individual speaks a language different from every other’ (Whitney, 1875)Tests (like teaching) privilege certain forms of a language as standard(s)Tests impose uniformity – they reward ‘correct’ languageTests are authoritative – they create and spread language learning targetsTests involve evaluation –they police language learningWhich (and whose) languages should be tested? Who should decide?

Linguistic standardsWhat we test: a standard

language?

Standards for tests and testers - how tests are conducted and

usedStandards for participants in language tests…APA Standards for Educational and Psychological TestingALTE Quality Management SystemETS Standards for Quality and FairnessCodes of conduct: ALTE, EALTA, ILTA

•What levels of validity, reliability, impact etc. are minimally sufficient for a language test?•Who determines and who polices standards? Minimum requirements or aspirational guidelines?•How similar would two tests need to be in order for us to claim ‘equivalence’ or ‘comparability’ between them?

System of rules or procedures resulting in the assignment of a number to differentiate between two or more states or degrees of performance.Setting performance targets for language learners.In order to establish common levels of proficiency, tests must be comparable in terms of quality as well as level, and common standards need, therefore, to be applied in their production.

(ALTE Code of Practice)

Standards for test takers - how good is good enough?

Appropriate cut scores

THE COMMON EUROPEAN FRAMEWORK OF REFERENCE FOR LANGUAGESAn emerging global standard for interpreting the results of language tests?

The Common European Framework of Reference defines communicative

proficiency:• at six/ nine levels, arranged in three (broader) bands:

A1/ A2(+) B1(+)/ B2(+) C1/ C2 • in relation to:

written/ spokeninteraction/production/ mediation

• in the form of functional “can do” statements presented in 54 separate illustrative scales

• ‘The establishment of common standards [between systems of qualifications and between learning, teaching and testing] is a main justification for the introduction of CEF’ (Trim 2002, p.18).

Defining L2 proficiency in functional terms means that the same descriptors can be used as a basic reference in:

• defining learning goals;– selecting/developing learning materials and activities;– judging the success or otherwise of learning.

In what sense(s) does the CEFR represent standards for language tests?• aspirational or minimum? authoritative? comprehensive?• ‘focus is largely on matters of process rather than those of product’

(Milanovic et al. 2002 p.5)

The Common European Framework of Reference

English Profile - introducing the CEFR for English. Information

Booklet, p.3

http://www.englishuk.com/uploads/assets/training/English_Profile_Information_Booklet.pdf

Shared understanding? What levels do the following descriptors belong to?

University of Bedfordshire 14

Can find and understand relevant information in everyday material, such as letters, brochures and short official documents.

Can follow extended speech even when it is not clearly structured and when relationships are only implied and not signalled explicitly.

Can recognise significant points in straightforward newspaper articles on familiar subjects.

[Common European Framework of Reference]

Can find and understand relevant information in everyday material, such as letters, brochures and short official documents. B1Can follow extended speech even when it is not clearly structured and when relationships are only implied and not signalled explicitly.C1Can recognise significant points in straightforward newspaper articles on familiar subjects.

B1


Shared understanding? What levels do the following descriptors belong to?

Terminology issues

B1What is meant or implied by ‘understand’,

‘follow’ or ‘recognise’What is meant by ‘a wide range of lengthy,

complex texts’ or ‘familiar subjects’? What are ‘short official documents’ or ‘straightforward newspaper articles’?

The assessment pyramid

Can do statements

Elaborated can do statement with guidance on conditions and criteria, explanation of terms

A1/ B2 etc.

Examples of types of activities and features of performance

Tasks and performance samples to illustrate levels McNamara and Roever (2006: 217)

STANDARD SETTING AND THE CEFR

18

Validity evidence of linkage of examination/test results to the

CEFR

Linking methodologies in the Council of Europe Manual ‘Relating Language Examinations to CEFR’

[2009:15]

1) Familiarisation:

• training activities to ensure participants in linking process have detailed knowledge of CEFR;

• necessary before Specification and Standardisation procedures;

• section 3.6 in the CEFR [English pp. 3336] describes salient features of levels.


2) Specification:

• self-audit of coverage of the examination (content and tasks types) against categories in CEFR Chapter 4 “Language use and the language learner” and CEFR Chapter 5 “The user/learner’s competences”;

• as well as serving a reporting function, these procedures also have a certain awareness-raising function that may assist in further improving the quality of the examination concerned;

• see Manual Chapter 4: Forms A2 and A8A20 focus on content analysis and the relationship of content to the CEFR.


Specification

Use Form A10 to assign a CEFR level to a sample text [Manual,pp.133-134]


• What do test takers have to do to arrive at an answer?– Read slowly and carefully?– Read quickly and selectively?– Understand a single word? A phrase? A

sentence? Several sentences? An entire text?– Is the necessary information explicitly provided

for them?• How does this vary by CEFR level?


Cognitive processes

3) Standardisation training and benchmarking:

• using CEFR illustrative samples for spoken and written performance to deepen familiarity with the CEFR levels (based on familiarisation) and assure that ratings of performances reflect the constructs described in the CEFR;

• training to standardise interpretation of levels in this way before:– benchmarking local performance samples and

tasks/items;– standard setting.

• corroborates the claim based on specification.


Benchmarking local samples of performance:

• rating local samples, using the CEF rating instruments;

• discussion of spread and iteration until suitable agreement is achieved.

4) Standard setting:

• deciding how to allocate students to a CEFR level on the basis of test performance – borderline performances and cut scores;

• familiarisation, specification and standardisation can be seen as preparatory activities to lead to valid and rational decisions;

• see Manual Chapter 6 and Further Material.



Standard-setting judgment

How will the minimally competent candidate – e.g. the marginally B1 reader/ listener – perform on the tasks in the test?Requires qualified, well trained panelists:

– often experts in relevant subject (e.g. doctors for a test of medical English);

– need to modify experts’ perception to focus on minimum acceptable performance;

– feedback provided to panelists.[Manual, 2009:59]


Procedural evidence

Who are the panelists?– representative of key stakeholder groups;– qualifications - experts in relevant subject;– confidentiality.

TEST-BASED METHODS


Test-based methods

• Angoff method;• yes/no extension of Angoff;• bookmark method;• basket method.


Angoff “method”:

• experts estimate probability that hypothetical, randomly selected minimally competent candidate will be able to answer each question correctly;

• addition of expert’s estimates = passing score;• average across experts = panel’s recommended

passing score.


Criticisms of Angoff methods:

• difficult judgment for panelists to make – people don’t usually think in terms of probabilities in this way;

• the method is subject to decision bias, e.g. some experts, whose ratings do not coincide with those of the group, may experience pressure to change their ratings, therefore, questioning their own judgment.

34

Yes/No variation:

• experts estimate whether or not the minimally competent candidate will be able to respond correctly to each item (Yes/No);

• add the Ys to calculate expert’s passing score;• average across panel = panel’s recommended

passing score;• cut point = panel recommendation +/-

standard error (x 1 or x 2).

35

• more popular with experts;• feedback not necessarily needed;• quicker to implement.

Yes/No variation:

36

Bookmark method:

• test questions are given in a ‘booklet’ (Ordered Item Booklet- OIB) from easy to hard;

• panelists each insert bookmark between pages (items) when the probability of a correct response by the Minimally Competent Candidate (hereafter - MCC) falls below some predetermined Response Probability (RP) such as 50%;

• for CEFR levels this may be done several times – there is a bookmark placed at the A2 cut point, the B1 cut point etc.

37

• total number of items preceding bookmark is experts’ passing score;

• multiple small groups arrive at decisions;• discussion may be allowed between rounds;• often involves multiple rounds; data usually

isn’t shared until the second or third rounds.

Bookmark method:

38

• results often shown graphically across rounds;

• frequently convergence occurs after 1st round;

• average across experts = recommended cut point;

• cut point can be set at +/- standard error x 1 or standard error x 2.

Bookmark method:

LEARNER BASED METHODS

40

Basket method:

• similar to Yes/No method;• panelists place each item in a basket at A1 or A2

level etc. according to the question:“At what level can a test taker already answer the following item correctly?”

• cut score = sum of items placed in the relevant basket or at levels below;

• if A1 has 2 items, A2 7 and B1 12, a test taker needs 7 + 2 + 12 correct responses (i.e. 21) to be placed at B1 or higher.

Learner based approaches to standard setting

Contrasting Groups method and the Borderline Group:•alternative judgments of learner level;

– by learners – self-assessment;– by teachers – judgments of learner level;– alternative tests or assessments.

•compare test results with the alternative judgment.

41

42

Main Point

Many methods, all aimed at providing a structured and reasoned approach to identifying:• cut points;• ranges of probable values;• procedural validity evidence.

5) Validation

• Validation is a continuous process of quality monitoring – of the test itself and of linking procedures

• Linking of a test to the CEFR cannot be valid unless the examination or test that is the subject of the linking can demonstrate validity in its own right.

43

Quality concerns

An examination that has no procedures for ensuring that standards applied by markers are equivalent in severity, or that successive forms of tests administered in different sessions are equivalent, cannot make credible claims of any linkage of its standard(s) to the CEFR because it cannot demonstrate internal consistency in the operationalisation of its own standard(s). (Manual p.9)

It is important to check that each step in linking is justified and successful.

STANDARD SETTING Prepared by Ludmila Kozhevnikova and Viktoria Levchenko Based on material by...

Documents

Transcript of STANDARD SETTING Prepared by Ludmila Kozhevnikova and Viktoria Levchenko Based on material by...