Canadian English Language Benchmark Assessment for Nurses · Test Impact and Washback. TESOL...

Canadian English Language Benchmark Assessment for Nurses

Test Impact and Washback

TESOL National Conference Boston, MA

March 26, 2010

Presented by: Catherine Lewis and Blanche Kingdon

What is CELBAN?

Canadian English Language Benchmark Assessment for Nurses (CELBAN)

- an occupation-specific English language assessment

tool used to assess the threshold English language proficiency of internationally-educated (IE) nurses seeking entry into nursing employment in Canada.

Why was CELBAN developed?

Internationally-educated (IE) nurses require proof of language proficiency when obtaining licensure to practice in Canada.

Traditionally the English proficiency tests used have

been: TOEFL/TSE, TOEIC, IELTS, MELAB and other similar tests


English proficiency tests often used are inadequate because… • not based on a Target Language Use (TLU)analysis of

the workplace (nursing) (content and context does not represent the nursing workplace);

• not validated with the target population (IE nurses); • test-takers do not receive feedback on strengths and

weaknesses in the productive skills (speaking & writing).


Some IE nurses who can pass these academic language tests, cannot function in the workplace – and others who take these language tests and fail, may have the potential to function successfully on the job.

There was an need identified by key stakeholders for a more appropriate language proficiency assessment tool for nursing.

How was CELBAN developed?

In 2002, the Centre for Canadian Language Benchmarks (CCLB) facilitated the development of an occupation-specific English language assessment tool for nurses, CELBAN. CELBAN was seen as a model for other professions and occupations. Funding for CELBAN was provided by:

• Provincial Governments: Alberta – British Columbia – Manitoba – Ontario • Federal Government: Citizenship and Immigration Canada

Why nursing?

• Canada is experiencing an increasing shortage of nurses.

• IE nurses could help ease the shortage.

• Nursing stakeholders expressed enthusiastic support for such a project.

CELBAN is a project of the Centre for Canadian Language Benchmarks (CCLB)

Canadian Language Benchmarks (CLB)

Canadian Language Benchmarks (CLB) Descriptors:

• The CLB is a set of national standards for English as an additional Language.

• They describe what a person can do in English in the 4 skill areas of speaking, listening, reading, and writing using real life tasks.

What is a “Benchmark”?

A benchmark (CLB) is a description of the English language abilities one should demonstrate at one of the 12 levels. • Range: CLB Level 1 (describes a literate student with

no English) to CLB Level 12 (describes a graduate student in university program or a highly-proficient practicing professional)

• Four skills: Speaking, Listening, Reading and Writing. • Performance and Situational Conditions: Social

Interaction, Information, Instruction, Suasion, Business/Service Texts

CLB Sample Descriptors Speaking CLB 5: • can participate with some effort in routine social

conversation

• grammar and pronunciation errors are frequent and may sometimes impede communication

CLB 8: • can communicate effectively in most daily social and

familiar work situations

• grammar and pronunciation errors rarely impede communication

CELBAN

CELBAN - developed by researchers at Red River College in Winnipeg, Manitoba, Canada.

Prior to Phase II, the development of CELBAN: • Feasibility study

• Phase I – An Analysis of the language demands of Nursing across Canada – a TLU Analysis (using the CLB).

Based on analysis of the data gathered in Phase I, the English language demands of the nursing profession were determined to be:

Skill CLB Level

Speaking 8

Listening 9

Reading 8

Writing 7

Data obtained in Phase I, the TLU

analysis of the nursing profession was used in the development of CELBAN.

Situational Uses of Language

Family10%

Professional34%

Clients56%

Observations of Nurses (Chart 1)

Individuals with whom Nurses Interact

Pie_chart_obs

ComfortsComfortsComforts1

ClarifiesClarifiesClarifies

DiscussesDiscussesDiscusses

ExplainsExplainsExplains

DescribesDescribesDescribes

InformsInformsInforms

Ask_InformationAsk_InformationAsk_Information

Ask_HelpAsk_HelpAsk_Help

Offers_HelpOffers_HelpOffers_Help

Gives_InstructionGives_InstructionGives_Instruction

ApologizesApologizesApologizes

SuggestsSuggestsSuggests

Small_TalkSmall_TalkSmall_Talk

Responds_QuesResponds_QuesResponds_Ques

PhonePhonePhone

Freq

Mean

ST_DV

Situation

Language Tasks

57

1.19

2.09

20

0.42

0.91

73

1.52

2.07

322

6.71

5

97

2.02

2.25

115

2.4

3.88

340

7.08

6.78

35

0.73

1.06

35

0.73

1.06

142

2.96

3.1

19

0.4

0.81

99

2.06

2.88

87

1.81

2.86

99

2.06

2.38

51

1.06

1.66

demo_obs

Clients

Family

Professional

Situational Uses of Language

899

154

538

Sheet1

TasksFreqMeanST_DVSituationFreq_S

Comforts571.192.09Clients899

Clarifies200.420.91Family154

Discusses731.522.07Professional538

Explains3226.715

Describes972.022.25

Informs1152.43.88

Ask_Information3407.086.78

Ask_Help350.731.06

Offers_Help350.731.06

Gives_Instruction1422.963.1

Apologizes190.40.81

Suggests992.062.88

Small_Talk871.812.86

Responds_Ques992.062.38

Phone511.061.66

Sheet2

Sheet3

Comforts4%Clarifies

1%

Discusses5%

Explains21%

Describes6%Informs

7%

Asks for Help2%

Offers Help2%

Gives Instruction9%

Apologizes1%

Suggests6%

Small Talk5%

Responds to Questions6%

Asks for Information22%

Phone3%

Observations of Nurses (Chart 2)

Language Tasks

Pie_chart_obs







Asks For InformationAsks For InformationAsks For Information

Asks for HelpAsks for HelpAsks for Help

Offers HelpOffers HelpOffers Help

Gives InstructionGives InstructionGives Instruction



Small TalkSmall TalkSmall Talk

Responds to QuestionsResponds to QuestionsResponds to Questions

PhonePhonePhone

Freq

Mean

ST_DV

Situation

Asks for Information22%

57

1.19

2.09

20

0.42

0.91

73

1.52

2.07

322

6.71

5

97

2.02

2.25

115

2.4

3.88

340

7.08

6.78

35

0.73

1.06

35

0.73

1.06

142

2.96

3.1

19

0.4

0.81

99

2.06

2.88

87

1.81

2.86

99

2.06

2.38

51

1.06

1.66

demo_obs

Clients

Family

Professional

Situational Use of Language

899

154

538

Sheet1

TasksFreqMeanST_DVSituationFreq_S

Comforts571.192.09Clients899

Clarifies200.420.91Family154

Discusses731.522.07Professional538

Explains3226.715

Describes972.022.25

Informs1152.43.88

Asks For Information3407.086.78

Asks for Help350.731.06

Offers Help350.731.06

Gives Instruction1422.963.1

Apologizes190.40.81

Suggests992.062.88

Small Talk871.812.86

Responds to Questions992.062.38

Phone511.061.66

Sheet2

Sheet3

Sheet3







Asks For InformationAsks For InformationAsks For Information

Asks for HelpAsks for HelpAsks for Help

Offers HelpOffers HelpOffers Help

Gives InstructionGives InstructionGives Instruction



Small TalkSmall TalkSmall Talk

Responds to QuestionsResponds to QuestionsResponds to Questions

PhonePhonePhone

Freq

Mean

ST_DV

Situation

Language Tasks

57

1.19

2.09

20

0.42

0.91

73

1.52

2.07

322

6.71

5

97

2.02

2.25

115

2.4

3.88

340

7.08

6.78

35

0.73

1.06

35

0.73

1.06

142

2.96

3.1

19

0.4

0.81

99

2.06

2.88

87

1.81

2.86

99

2.06

2.38

51

1.06

1.66

Description of CELBAN

Content of all CELBAN tasks:

• is based on data collected in Phase I, the TLU analysis of the nursing profession.

• was created with input from nursing consultants and experts to ensure authenticity.

Speaking Assessment

The speaking assessment consists of: • an oral interview • two role plays involving two assessors (with one

candidate).

• Scores are assigned as CLB Levels. • Clear criteria for evaluation are used. • Feedback on strengths and weaknesses is

provided on candidate’s report.

Listening Assessment

• The listening assessment is composed of: video scenarios - various settings including hospital,

home, clinic, and medical office. audio scenarios - phone calls and one shift-to-shift

report. • Scenarios – variety of interactions, i.e., between nurses and

patients, family members and other professionals. • Question format - standard multiple choice questions

(MCQs); some in chart format. Note: Listening, Writing & Reading tests are administered as a group.

Reading Assessment

• Reading assessment includes two sections: Skimming and Scanning Reading Comprehension

• Text - formats such as charts, patient notes, manuals, and information texts related to health issues.

• Question format - short answer (word or phrase), multiple choice, and cloze (fill-in-the-blank) format.

Writing Assessment

• Writing assessment includes two sections:

i) Form-filling

ii) Report Writing

• Scores are assigned as CLB Levels.

• Clear criteria for evaluation

• Feedback on strengths and weaknesses is provided.

Facts about CELBAN • Nursing licensing bodies across Canada have

accepted CELBAN as an option for demonstrating English language proficiency.

• A national administrative centre (CELAS Centre)

has been set up at Red River College to provide centralized national administration of CELBAN.

• As of September 2009, official CELBAN

Administration Sites have been set up in five of the ten provinces and two territories.

The Development of CELBAN Rigorous, comprehensive, and collaborative: Stage One: Planning Stage Two: Developing test specifications and

first draft of tool (speaking, listening, reading and writing) Stage Three: Piloting of draft versions Stage Four: Statistical analysis and revisions Stage Five: Completion of final version of test

materials (including guidebooks for administrators & invigilators, and speaking assessors) Stage Six: Implementation of CELBAN (Version

One)

CELBAN Products and Services

• Official CELBAN (Version One) (2003) • CELBAN website: www.CELBAN.org (2003) • CELBAN Readiness Self-Assessment(CRSA) (2004) - on-line: via website - off-line: a) for candidates b) for institutions • Official CELBAN (Versions Two and Three)(2005) • Institutional CELBAN (Versions One and Two)(2007)

http://www.celban.org/

Institutional CELBAN • Institutional CELBAN - developed for educational

institutions which provide nursing language bridging programs to internationally-educated nurses.

• Institutional CELBAN (modeled after the official

CELBAN), with same content/context and language components; but there are a few differences:

– Purpose – Length – Administration/training – Quality control

Institutional CELBAN vs. Official CELBAN

Features Institutional CELBAN Official CELBAN

1. Content and Context

Based on Phase I, the analysis of the language demands of nursing

Based on Phase I, the analysis of the language demands of nursing

2. Language Components

•Listening, Writing, Reading (Group) •Speaking (Individual Assessment

•Listening, Writing, Reading (Group) •Speaking (Individual Assessment

3. Purpose •Entry or exit into nursing language bridging programs •Diagnostic purpose

•Means of demonstrating English language proficiency as one requirement for gaining licensure to enter nursing profession in Canada

Institutional CELBAN vs. Official CELBAN cont’d


4. Length •L/W/R: 2 hours (+ scoring) •Speaking: 20-25 minutes (+scoring)

•L/W/R: 3 hours (+ scoring) •Speaking: 30-35 minutes (+scoring)

5. Administration and scoring and training

•Training is not required but is recommended and available from CELAS Centre staff if requested • Administration, scoring and reporting is done on-site by institution

• Training and Certification by CELAS Centre staff (CELBAN Trainers) is required • Invigilators and Speaking assessors must be approved and trained by CELBAN trainers • All components of test materials are couriered to CELAS for scoring and reporting

Institutional CELBAN vs. Official CELBAN cont’d


6. Quality Control • Letter of Agreement signed by purchasing Institution commits users of the tool to follow appropriate test protocols for secure materials storage, administration, usage, scoring, etc.

• Formal Licensing agreements are made between official CELBAN site and the CELAS Centre • CELAS Centre protocols must be followed to ensure standardized administration of CELBAN occurs and security is maintained. • Any breaches in procedure or security are immediately reported by CELBAN sites to CELAS. • Speaking assessment tapes are periodically reviewed by CELAS staff to maintain quality control.

Institutional CELBAN - available for purchase by educational institutions since October, 2007 - has training option available for teams at purchasing institutions - used in nursing language bridging programs in many institutions

Test Impact and

Washback

Test Impact and Washback Since the implementation of CELBAN, the impact and washback from this unique occupation-specific English language assessment tool has been experienced by a variety of stakeholders* in many different contexts.

* Stakeholders include test-takers, educators, researchers, test developers, employers , policy analysts, regulators, etc.

Test Impact and Washback 1. Test Impact - how stakeholders are impacted by a test

over time, positively or negatively.

Impact refers to the wider influence of tests on:

a) educational systems (e.g., test results used to made decisions about curriculum planning, funding allocation, admission, etc.)

b) society (e.g., immigration policy, licensing for professionals, etc.)

Language testing has consequences beyond the immediate

teaching/learning context.


Use of test scores can impact significantly on career and life changes for individual test-takers. (Social consequences of testing = “consequential validity”)

Widespread acceptance of a test often impacts

- publishers - institutions

(Impact can be positive or negative.)

Note: A baseline study is often conducted to describe an educational context before the introduction of an innovation intended to cause change.


2. Washback is limited to influences on

teaching and learning (including curriculum materials).

Washback can be positive or negative.


• Positive washback occurs when the implementation of a test results in – enhanced curriculum – improved communicative language teaching – better communicative language learning

(and ultimately better employees, once learners leave the educational environment to enter the workforce)


• Negative washback occurs when the implementation of a test – becomes the entire focus of a program (i.e.

‘teaching to the test’) – results in de-contextualized linguistic forms

and not authentic language being taught – results in a mismatch between what is taught

and tested, and what is needed thereafter.


Test Impact related to CELBAN

• CELBAN aligned - with other English language proficiency tests

• CELBAN scrutinized - in standard-setting exercises by

educators and regulators • CELBAN explored - for adaptation for use by

other health professions - for administration in other countries

No formal impact study has yet been undertaken since CELBAN is still in its infancy (less than 5 years since implementation).


Washback related to CELBAN

Positive Washback

• Institutional CELBAN - instructor is informed about gaps to fill throughout program from specific diagnostic feedback obtained

• Program planners can determine appropriate

entrance CLB levels for nursing language bridging programs


Negative Washback • “CELBAN Prep” courses and programs are being developed and

delivered in various locations; – many have not had outcomes and curriculum reviewed by CELBAN

developers or other qualified stakeholders; - IENs may be paying for courses and participating in tasks which have

no clear correlation to tasks and demands of CELBAN, and/or of the nursing profession

- Opportunists may be financially benefitting from IENs paying for ‘prep’ materials with the false believe that the materials are valid and beneficial.

• Some candidates who take official CELBAN may be inadequately

prepared and may not successfully complete official CELBAN.

Test Impact and Washback Examples of positive washback from teams using the Institutional CELBAN in educational programs: Examples: • helps program planners to set standards for

entrance to programs • provides instructors with means for conducting

interim assessment within programs • provides instructors with diagnostic data in

speaking/writing which was consolidated and used to design gap filling tasks to enhance curriculum

• increases instructor confidence: – in accuracy and reliability of results (answer keys are provided ,

exemplars are provided for scoring writing, and two assessors are recommended for the speaking component)

– in developing new teacher-generated tasks


More examples of positive washback… • Feedback to candidates on strengths and weaknesses

in speaking and writing extremely useful in preparation for clinical practice

• Identification of individual strengths and weaknesses provides information for instructors designing individualized learning plans for a variety of learning groups

• New courses have been developed by instructors ,e.g. Enhanced listening for Health Care; Enhanced Speaking for Health Care; on-line in-house practice site on Moodle for students to practice tasks related to CELBAN


Goal is to make the impact and washback from CELBAN as positive as possible… ……..all stakeholders will benefit.

References Alderson, J.C. and Wall, D. (1993). Does washback exist? Applied Linguistics 14,

116-29. Bachman, L. and Palmer, A. (1996). Language Testing in Practice, Cambridge:

Cambridge University Press Cheng, L. , Watanabe, Y., and Curtis, A. (Eds.): Washback in Language Testing:

Research Contexts and Methods. Lawrence Erlbaum and Associates, 2004. Applied Linguistics 26/3: 459-474. Oxford University Press 2005.

Hawkey, Roger. Consultant in language testing to University of Cambridge ESOL

Examinations. A Study of the Impacts of IELTS, especially on candidates and teachers. ESOL.org

Epp, L. and Lewis, C. (2009). Innovation in Language Proficiency Assessment:

CELBAN. Bosher, S., & Pharris, M. (Eds.) In Transforming Nursing Education. The Culturally Inclusive Environment. New York: Springer Publishing Company.

Taylor, Lynda. Key Concepts in ELT. Washback and Impact. ELT Journal Volume

59/2 April 2005. Oxford University Press. doi:101093/eltj/ccio30

Publications Available

• Phase I: Analyzing the English Language Demands of the Nursing Profession Across Canada (2002)

• Phase II: The Development of CELBAN: The Canadian Language

Benchmark Assessment for Nurses (CELBAN) (2003) • Phase III: The Implementation of CELBAN (2003) • Developing an Occupation-Specific Language Assessment Tool (Using

the Canadian Language Benchmarks) A Guide for Trades and Professional Organizations (2004)

Reports are available from the Centre from Canadian Language

Benchmarks (CCLB); see website: www.celban.org (Note: click Resources; then History, Background and Research)

http://www.celban.org/

Contact Information

Language Training Centre, Red River College 300 – 123 Main Street Winnipeg, Manitoba

Canada R3C 1A3

Catherine Lewis Email: [email protected] Phone: (204) 945-0085 Blanche Kingdon Email: [email protected] Phone: (204) 945-8933

mailto:[email protected]:[email protected]

Slide Number 1What is CELBAN? Why was CELBAN developed? Why was CELBAN developed?Why was CELBAN developed?�How was CELBAN developed?�Why nursing?�CELBAN is a project of the Centre for Canadian Language Benchmarks (CCLB)Canadian Language Benchmarks (CLB)�Canadian Language Benchmarks (CLB) Descriptors:��What is a “Benchmark”?�CLB Sample DescriptorsCELBANSlide Number 14Slide Number 15Slide Number 16Slide Number 17Description of CELBAN�Speaking AssessmentListening Assessment�Reading AssessmentWriting AssessmentFacts about CELBANThe Development of CELBANCELBAN Products and Services Institutional CELBANInstitutional CELBAN vs. Official CELBANInstitutional CELBAN vs. Official CELBAN cont’dInstitutional CELBAN vs. Official CELBAN cont’d�� Institutional CELBAN��- available for purchase by educational institutions since October, 2007��- has training option available for teams at purchasing institutions ��- used in nursing language bridging programs in many institutions � ��Test Impact �and �Washback�� Test Impact and Washback��Since the implementation of CELBAN,�the impact and washback from this unique �occupation-specific English language�assessment tool has been experienced �by a variety of stakeholders* in many different contexts.�� Test Impact and WashbackTest Impact and WashbackTest Impact and WashbackTest Impact and WashbackTest Impact and WashbackTest Impact and WashbackTest Impact and WashbackTest Impact and WashbackTest Impact and WashbackTest Impact and WashbackTest Impact and WashbackReferencesPublications AvailableContact Information

Canadian English Language Benchmark Assessment for Nurses · Test Impact and Washback. TESOL...

Documents

Transcript of Canadian English Language Benchmark Assessment for Nurses · Test Impact and Washback. TESOL...