RATERS TRAINING AS A SAFETY CONCERN

Sergey MELNICHENKOCompLang Aviation Training Centre

(Moscow), Deputy Director

Associate Member

ICAO LPR Implementation WorkshopRussia, St-Petersburg, May 24-26, 2011

RATERS TRAINING AS A SAFETY CONCERN

Associate Member

- RATER AS A DECISION MAKER - SELECTION OF PERSONNEL - AB INITIO TRAINING - RATERS CERTIFICATION - INTER-RATER RELIABILITY - INTRA-RATER RELIABILITY - RASCH ANALYSIS - RECURRENT TRAINING - EVALUATION OF RATER’S ADEQUACY

Associate Member

Raters Training as a Safety Concern

• RATER AS A DECISION MAKER• SELECTION OF PERSONNEL• AB INITIO TRAINING• RATERS CERTIFICATION• INTER-RATER RELIABILITY• INTRA-RATER RELIABILITY• RASCH ANALYSIS• RECURRENT TRAINING• EVALUATION OF RATER’S ADEQUACY

Associate Member

Raters Training as a Safety Concern

Aviation English test results are felt around the globe

Raters Training as a Safety Concern

Associate Member

Inadequacy in Level Evaluation

Level 3 instead of Level 4:

Loss of job and/or occupationFinancial losses for AO or ANSPMarket lossesRedistribution of air transportation

servicesLack of confidence for test-takers

Associate Member

Inadequacy in Level Evaluation

Level 4 instead of Level 3:

Aviation safety is compromisedDegradation to Level 2Overconfidence for test-takers

Raters Training as a Safety Concern

Associate Member

Raters Training as a Safety Concern

• RATER AS A DECISION MAKER

• SELECTION OF PERSONNEL• AB INITIO TRAINING• RATERS CERTIFICATION• INTER-RATER RELIABILITY• INTRA-RATER RELIABILITY• RASCH ANALYSIS• RECURRENT TRAINING• EVALUATION OF RATER’S ADEQUACY

Associate Member

Raters Training as a Safety Concern

ICAO recommendations on interlocutors and raters qualifications

Raters Training as a Safety Concern

Associate Member

ICAO recommendations on interlocutors and raters qualifications (Doc 9835)

Familiarity with ICAO documentationAnnex 1;Doc 9835;ICAO Rated Speech Samples CD;ICAO Circ 318

Raters Training as a Safety Concern

ICAO recommendations on interlocutors and raters qualifications (Doc 9835)

Familiarity with ICAO documentationOperational and/or language specialist

expertiseLevel 5 or Level 6 proficiencyFamiliarity with Aviation EnglishRATERS CODE OF ETHICS

Associate Member

Associate Member

Raters Training as a Safety Concern

• RATER AS A DECISION MAKER• SELECTION OF PERSONNEL

• AB INITIO TRAINING• RATERS CERTIFICATION• INTER-RATER RELIABILITY• INTRA-RATER RELIABILITY• RASCH ANALYSIS• RECURRENT TRAINING• EVALUATION OF RATER’S ADEQUACY

Associate Member

Raters Training as a Safety Concern

• Theory of language testing • ICAO documents: Annex 1, Annex 10, Doc 9432, Doс 9835, Circ 318• The ICAO Proficiency Rating Scale and holistic descriptors• Correlating speech samples with the ICAO LPR Scale

Associate Member

Raters Training as a Safety Concern

Raters training course should be aimed at experienced non-native and native English language teaching professionals and subject-matter experts who are currently teaching Aviation English and who want to be able to rate according to the ICAO Language Proficiency Scale

Associate Member

Raters Training as a Safety Concern

• RATER AS A DECISION MAKER• SELECTION OF PERSONNEL• AB INITIO TRAINING

• RATERS CERTIFICATION• INTER-RATER RELIABILITY• INTRA-RATER RELIABILITY• RASCH ANALYSIS• RECURRENT TRAINING• EVALUATION OF RATER’S ADEQUACY

Associate Member

Raters Training as a Safety Concern

• Certification by national Civil Aviation Authority or by international aviation organization is mandatory

Associate Member

Raters Training as a Safety Concern

• Is geographical principle necessary?

Raters Training as a Safety Concern

Associate Member

• Are quotas necessary?

Associate Member

Raters Training as a Safety Concern

• RATER AS A DECISION MAKER• SELECTION OF PERSONNEL• AB INITIO TRAINING• RATERS CERTIFICATION

• INTER-RATER RELIABILITY• INTRA-RATER RELIABILITY• RASCH ANALYSIS• RECURRENT TRAINING• EVALUATION OF RATER’S ADEQUACY

Associate Member

Raters Training as a Safety Concern

Different raters but same results

Associate Member

Raters Training as a Safety Concern

• RATER AS A DECISION MAKER• SELECTION OF PERSONNEL• AB INITIO TRAINING• RATERS CERTIFICATION• INTER-RATER RELIABILITY

• INTRA-RATER RELIABILITY• RASCH ANALYSIS• RECURRENT TRAINING• EVALUATION OF RATER’S ADEQUACY

Raters Training as a Safety Concern

Associate Member

Different times but same results

Associate Member

Raters Training as a Safety Concern

• RATER AS A DECISION MAKER• SELECTION OF PERSONNEL• AB INITIO TRAINING• RATERS CERTIFICATION• INTER-RATER RELIABILITY• INTRA-RATER RELIABILITY

• RASCH ANALYSIS• RECURRENT TRAINING• EVALUATION OF RATER’S ADEQUACY

Associate Member

Raters Training as a Safety Concern

Evidence of the validity of the responses

Associate Member

Raters Training as a Safety Concern

Evidence of the validity of the responses

Correctional refresher course

Raters Training as a Safety Concern

Associate Member

Evidence of the validity of the responses

Correctional refresher course

Administrative measures to marginal raters

Associate Member

Raters Training as a Safety Concern

• RATER AS A DECISION MAKER• SELECTION OF PERSONNEL• AB INITIO TRAINING• RATERS CERTIFICATION• INTER-RATER RELIABILITY• INTRA-RATER RELIABILITY• RASCH ANALYSIS

• RECURRENT TRAINING• EVALUATION OF RATER’S ADEQUACY

Associate Member

Raters Training as a Safety Concern

Annual refresher course for raters

Associate Member

Raters Training as a Safety Concern

Annual refresher course for raters

New materials

Raters Training as a Safety Concern

Associate Member

Annual refresher course for raters

New materials

Practical training

Associate Member

Raters Training as a Safety Concern

• RATER AS A DECISION MAKER• SELECTION OF PERSONNEL• AB INITIO TRAINING• RATERS CERTIFICATION• INTER-RATER RELIABILITY• INTRA-RATER RELIABILITY• RASCH ANALYSIS• RECURRENT TRAINING

• EVALUATION OF RATER’S ADEQUACY

Associate Member

Raters Training as a Safety Concern

Rater is not a lifetime profession

RATER TRAINING IS A SAFETY CONCERN

Associate Member

Raters Training as a Safety Concern

Thank you