www.britishcouncil.org 1

Challenges in Developing and Delivering a Valid Test

Michael King and Mabel LiNAFLE, July 2013

www.britishcouncil.org 2

Contents

1. Integrated Language Education System

2. Basic Concepts in Language Assessment

3. Considerations in Language Assessment – IELTS Approach

4. Q&A

www.britishcouncil.org 3

1. Integrated Language Education System

LearnersLearners

Curriculum Delivery

Curriculum Delivery

CurriculumCurriculum

Language Assessment

Language Assessment

www.britishcouncil.org 4

2.

Language Assessment: Basic Concepts

www.britishcouncil.org 5

What is assessment?

Noun

The evaluation or estimation of the nature, quality, or

ability of someone or something: "the assessment of

educational needs".

Synonyms

appraisal - rating - valuation - estimation - evaluation

www.britishcouncil.org 6

Why do we need to do assessment?

Evaluate learning

Measure achievement

Inform future teaching

Rank stu

den

ts

Benchmarking

Motivate students

Evaluate teaching

Sele

ction

Find gaps in learning

www.britishcouncil.org 7

What kinds of assessment are there?

Assignment

Conversation

PresentationSelf a

ssessm

ent

Observation

TestEssay

Portfo

lio

Peer assessment

Qu

estio

nn

air

e

Checklist

QuizGroup task

www.britishcouncil.org 8

3. Considerations in Language Assessment

What are the qualities of a good test?

www.britishcouncil.org 9

IELTS Approach to Test Qualities

• Validity • Reliability • Impact• Practicality

R

PI

V

Validity: test content

Test content matched to needs of candidates

Task-based approach to testing

Testing of integrated skills

Variety of Listening and Reading item types used

Face to face Speaking Test and direct marking

Accepting all standard varieties of native-speaker English

Validity: continual research based development

1980 ELTS introduced (English Language Testing Service)

Original 4 skills test

1989 tri-partite international partnership created, 1st delivery of IELTS

1995 major systems and content revision

2001 revised Speaking Test

2005 revised Writing assessment criteria

2007 introduction of half band for Speaking and Writing

Constants up to date: 9-band scale, skill profile & overall score, f2f

interview

Reliability of test materials

Reliability of results

Scoring system

Marker/Examiner selection and training

Double marking and EOR

Research and validation analysis

Monitoring: sample and targeted

Results benchmarked with international standards

How do you create effective test tasks and rating scales for Writing and Speaking?

Theoretical relevance

The test’s criteria and tasks should be defensible in terms of valid theories of oral performance

Discriminating power

The test criteria and tasks should enable relevant distinctions to be made between candidates

Assessability

The features selected for assessor attention should be salient enough for a single assessor to notice them

IELTS W/S Band Descriptors

Writing Speaking

Task Achievement/Response

Fluency and Coherence

Coherence and Cohesion Lexical Resource

Lexical Resource Grammatical Range and Accuracy

Grammatical Range and Accuracy

Pronunciation

www.britishcouncil.org 17

Results benchmarked with int’l standards

http://www.cambridgeenglish.org/about-us/what-we-do/international-language-standards/

Impact

Long-standing concern for the relationship between

IELTS and its contexts of use

Test development partly shaped by this concern

Early 1990s growing interest in test impact/washback

Systematic investigation of IELTS impact began in 1995

Consultation processes with stakeholders to ensure- IELTS is fit for purpose - cut scores are set appropriately

Use of GT module for immigration and employment

Standards setting events

General Medical Council, UK (June 04)

National Council for State Boards of Nursing, USA (Oct 04)

Canadian Immigration and Citizenship (May 05)

Main outcomes of IELTS Impact Studies

IELTS is a reasonable direct communicative performance test of the

four macro skills

IELTS is challenging but fair

IELTS uses authentic and appropriate content

IELTS is motivating for learners and teachers

IELTS engenders a positive effect on classroom activities and

learning processes

Practicality 可行性 48 test administrations per year

Global network of 900 test venues in over 130 countries/regions

Emphasis on standards, consistency and security

global standards

test centre selection, training, auditing

secure stock management

on-line results verification

Provision of information through a variety of channels

On-line registration

Results available 13 days after the test

Post test services

4. Q&A

www.britishcouncil.org 23

Acknowledgement

References:

Presentation for teachers event, Nick Charge

www.ielts.org