NOTECHS - MEDESIM€¦ · Web viewThis study resulted in a proposal for a new non-technical skill...

NOTECHS

The evaluation of non-technical skills of multi-pilot aircrewin relation to the JAR-FCL requirements

J.A.G. van Avermaete and E.A.C. Kruijsen, editors

FINAL REPORT

-- CONCEPT --

Report on research carried out in the periodmarch 1997 - march 1998

The NOTECHS consortium:

National Aerospace Laboratory NLR (NL)Deutsches Zentrum für Luft- und Raumfahrt (DLR)Laboratoire IMASSA (F)University of Aberdeen (GB)

Sponsors:

RLD (NL)LBA (D)DGAC (F)CAA (GB)DG-VII (EU)

FINAL REPORT (VERSIE 0.3) NOTECHS

- ii - 5/18/2023

NOTECHS FINAL REPORT

SummaryThis report is the result of the NOTECHS project jointly carried out by DLR, IMASSA, NLR, and the University of Aberdeen. Upon a request of the JAA Project Advisory Group on Human Factors, the project group studied potential methods to assess multi-pilot aircrew on their NOn-TECHnical Skill proficiency (hence the project name). This study resulted in a proposal for a new non-technical skill evaluation method, based on training methods currently in use with several European airlines and on results of past research. The NOTECHS method divides non-technical skills into four categories comprising a total of 15 elements and behavioural markers for each of these elements.Due to the sensitivity within the pilot community associated with the assessment of pilot skills in general and of non-technical skills in particular (a problem normally not encountered with training), several issues required additional attention. The NOTECHS system has the potential to overcome the cultural differences that implicitly affect non-technical skills ratings given by pilot-examiners. This however requires more research, which will be conducted by the JAR TEL project.

This NOTECHS method is formulated as a proposed Acceptable Means of Compliance (AMC) with the new JAR-FCL regulations concerning the assessment of non-technical skills. Further Interpretative and Explanatory Material (IEM) text has also been suggested in this report.

The project was supported with a financial contribution by LBA, DGAC, RLD, CAA, and the Directorate General VII of the European Union.

Keywords: JAA requirements, JAR-FCL, JAR-OPS, non-technical skills, CRM, pilot training, pilot license, pilot examinations, Human Factors, NOTECHS, JAR-TEL.

5/18/2023 - iii -

Document control page

Document title The evaluation of non-technical skills of multi-pilot aircrewin relation to the JAR-FCL requirements

Authors:(in random order)

NameJ.A.G. van AvermaeteE.A.C. KruijsenH. NijhuisK.M. GoetersJ. HörmannR. FlinL. MartinR. AmalbertiC. Valot

CompanyNLR

DLR

University of Aberdeen

Lab. IMASSA

Document Revision log:Issue Date Pages Remarks0.1 16-01-1998 82 This version distributed among NOTECHS and JAR-

TEL partners. General Conclusions still missing.0.2 30-09-1998 Review within NLR0.3 12-10-1998 FINAL Draft

Acknowledgement:The authors wish to thank Cees van Gelderen and Hans Sypkens for their valuable inputs in the NOTECHS discussions. Undoubtedly, the underlying document would have been different without knowledge of the experiences of KLM and the Dutch CAA examiners’ commission (ACE-P) respectively.

This report is the result of the NOTECHS project jointly carried out by DLR, IMASSA, NLR, and the University of Aberdeen. The project was conducted upon a request of the JAA Project Advisory Group on Human Factors to study potential methods to assess multi-pilot aircrew on their non-technical skill proficiency. The project was supported with a financial contribution by LBA, DGAC, RLD, CAA, and the Directorate General VII of the European Union.

Distribution list

J. Meijer RLD/LI/VZC. Schaap RLDS. Baker CAA Human Factor UnitK. Mohamed DGACA. Samel DLRMr. Zimmermann LBAM. Rosati R.A.I.J.P. Henrotte CEC DG-VII

K.M. Goeters DLRJ. Hörmann DLRR. Amalberti IMASSAC. Valot IMASSAE. Perreaut IMASSAJ. van Avermaete NLR/VEE. Kruijsen NLR/VVH. Nijhuis NLR/VER. Flin University of Aberdeen

E. Andlauer Sofréavia

Table of contents

Introduction 1

1. Workpackage 1 : General philosophy and guiding principles

1.1 Objectives of Work Package 11.2 Scope and goals of NOTECHS: A definition of Non-Technical Skills

1.2.1 What the NOTECHS project is aiming at:1.2.2 What the NOTECHS project is not aiming at:

1.3 NTS requirements within present regulations1.3.1 ICAO1.3.2 JAR-FCL1.3.3 JAR-OPS

1.4 Outline of deliverables1.5 Philosophy of evaluation and guiding principles

1.5.1 Principle no. 1: “The need for technical consequences”1.5.2 Principle no. 2 “Use of a two-point rating scale”1.5.3 Principle no. 3: “Explanation of unacceptable rating required”1.5.4 Principle no. 4: “Repetition required”1.5.5 Principle no. 5: “No evaluation of personality traits”

2. Workpackage 2 : Review of existing NTS training and evaluation systems

2.1 Abbreviations2.2 Introduction2.3 Description of current NTS training systems

2.3.1 Air France2.3.2 Lufthansa German Airlines2.3.3 British Airways2.3.4 KLM Royal Dutch Airlines2.3.5 KLM Royal Dutch Airlines - SHAPE

2.4 NASA/UT/FAA2.5 RLD ACE-P2.6 Discussion of commonalities and differences2.7 Conclusions2.8 References

3. Workpackage 3 : Components of Non-technical skills

3.1 Method3.1.1 Structure of the NOTECHS system3.1.2 Design principles

3.2 Co-operation3.2.1 Definition3.2.2 Elements of co-operation3.2.3 Concepts of co-operation3.2.4 Debates over definition of co-operation3.2.5 Research findings3.2.6 Consequences for measurement

3.2.7 References - Co-operation3.3 Leadership and managerial skills

3.3.1 Definition3.3.2 Controversies about definition3.3.3 Concepts in the related literature3.3.4 Elements in existing systems3.3.5 Consequences for measurement within the NOTECHS system3.3.6 List of selected elements and behaviour examples3.3.7 References - Leadership and managerial skills

3.4 Situation awareness3.4.1 Definition3.4.2 Debate over definition3.4.3 Measuring cognitive processes3.4.4 Model/ Theory3.4.5 Framework3.4.6 Behavioural examples3.4.7 References - Situation awareness

3.5 Decision making3.5.1 Definition3.5.2 Types of decision3.5.3 Selected Research Findings3.5.4 Models of pilot decision making3.5.5 Framework3.5.6 Elements3.5.7 Behavioural Examples3.5.8 References - Decision making

4. Workpackage 4 : Using the NOTECHS evaluation system

4.1 Introduction4.2 Rationale behind the proposed AMC and IEM4.3 Proposal for AMC and IEM

4.3.1 The AMC4.3.2 The proposal for IEMs

5. Conclusions

Introduction

If there is one topic in the field of civil aviation that has got more and more attention over the past two decades than it must be that of Human Factors on the flight deck or CRM. The abbreviation CRM is deliberately used here instead of the full name, as the meaning of the capital "C" has changed over the years. Originally, CRM meant Cockpit Resource Management, but nowadays it is normally refered to as Crew Resource Management to stress the relation between cockpit and cabin crew. Recently, in several forums it has been suggested that the "C" should stand for Company to reflect the role of maintenance department, cargo and passenger departments and the like. There is a risk that the meaning of the concept devaluates with this: everyone knows the word CRM, but interprets its meaning in their own manner. It seems very common to associate CRM above anything else with authority differences on the flight deck (or between cockpit and cabin crew depending on the capital C), while in fact the topic should be interpreted much broader. It is probably wiser to refer to the topic under discussion with the term non-technical aspects, as is done in the new JAR-FCL (Joint Aviation Authorities-Flight Crew Licensing) requirements. The wording non-technical distinguishes the associated pilot skills from their technical counterpart which refers primarily to the traditional stick and rudder skills.. Unfortunately, the exact contents included under non-technical aspects, and the distinction between technical and non-technical aspects, is not defined by JAR-FCL. In fact, the underlying document will argue that it is not possible to draw a natural border between the two.

In addition, the JAR-FCL does ask for the assessment of non-technical skills, but it does not state how this evaluation should be done. In a 1996 survey conducted by the JAA-Project Advisory Group on Human Factors (JAA-PAG) on existing research needs in the different JAA member states several times an urgent need was expressed to start research into the way(s) the non-technical skills of flight crew members should be evaluated. This in the light of the (then) forthcoming new JAA rules. Next to the already mentioned JAR-FCL the evaluation of non-technical skills is also mentioned in the JAA Operations documents (JAR/OPS subpart N).

The underlying objective to include non-technical aspects in the JAA-requirements is to reach a harmonised application. Without the availability of an Acceptable Means of Compliance (AMC) providing clear guidelines on how the requirements can be met, it is not very likely that such a harmonisation will be reached for the assessment of non-technical skill proficiency. Knowing this, the JAA-PAG contacted four institutes with the request to set-up a research project which could provide background material for those responsible to write the AMC and Interpretatory and Explanatory Material (IEM). These four institutes, the Dutch NLR, German DLR, French IMASSA, and the University of Aberdeen, set up a project proposal which was presented to the JAA-PAG. The national authorities of the Netherlands, Germany, France and the United Kingdom, and the Directorate General VII (DG-VII) of the European Union supported the idea for the project and together they provided the budgets for the project. The project was named NOTECHS and started March, 1997 to run for twelve months.

Given the limited time available the project team realised that it would not be feasible to come up with a completely new evaluation system built from scratch and validated to ensure its proper functioning. Therefore it was made clear in the project plan that NOTECHS would limit itself to come up with background material for the development of AMC and IEM material for JAR-FCL, Part 1,

Subpart F, paragraph 240. However, the project results would clearly have relevance to other related JAR-documents such as JAR-OPS. When reading this NOTECHS report it is good to note though that as far as NOTECHS is concerned it is the check situation that matters: JAR-FCL does noet deal with training, but with the issue of granting or reneweing a license. The primary users of the NOTECHS system will be the examiners. It is not unlikely that small modifications or "add-ons" are desired when a company starts to use the NOTECHS system in their training.

NOTECHS was set up to build on earlier research, which included the results of earlier initiatives by:CAA/University of Aberdeen,DLR/LBA,RLD/ACE-P,NASA,KLMs Feedback and Appraisal System (FAS).

In the NOTECHS project one of NLRs test pilots, who works part-time as a line pilot on the Boeing 737 for KLM, participated. In addition the NOTECHS team was supported in their work by two operational pilots who had been involved in two of the earlier initiatives mentioned above. Both persons, one from KLMs working group on CRM, the other an examiner for the RLD and an IFALPA member active in its working groups in this field, participated in the project in a private capacity. Their inputs proved invaluable for the project.

The NOTECHS results should be used in the establishment of harmonised and unambiguous ways to comply with the basic rules. A prerequisite to reach this goal was to come to a clear definition of non-technical skills, and the formulation of a method to test the proficiency on these skills. It was realised from the start of the project that, without the possibility to test and validate the proposed method, it would be difficult to guarantee a robust and sensitive assessment tool. Fortunately there appeared to be a possibility for a larger consortium led by Sofréavia to continue work on this validation after the NOTECHS project. In the DG-VII JAR-TEL (Joint Aviation Requirements Translation and Evaluation of Legislation) project the NOTECHS method will be tested in a large scale study involving examiners in a number of different countries. The operational involvement in JAR-TEL will even be larger than in NOTECHS with the participation of British Airways and AlItalia.

It should be kept in mind when reading the underlying document that it was beyond the scope of the NOTECHS team to debate the fact that non-technical skills are included in the JAR-requirements in the first place. The JAR-FCL requirements as laid down in the draft text of October 1996 were taken as a given fact. It was clear that the JAR-FCL text would have some further revisions before reaching a final status, but it would have proven impossible to anticipate or include these amendments in the course of the NOTECHS project.

The remainder of this document is split into five different parts which report on the activities conducted in NOTECHS. Part A describes the general principles for the evaluation method as described by the NOTECHS team. The next part presents a summary of the inventarisation of existing initiatives to do the evaluation of non-technical skills. Based on the existing systems, and after a thorough literature review the skeleton for the evaluation system was set up: part C describes the decomposition of non-technical skills in behavioural categories. Part D presents the proposed

formulation for AMC and IEM, and the final part (E) contains the general conclusions for the NOTECHS project.

With the conclusion of the NOTECHS project a reference framework has been set up which can function as a basis for a harmonised non-technical skill assessment method. The partners involved in the project truly believe that this system will act as a helpful tool to the examiner for use in the check situations referred to in the JAR-requirements. This belief has to be substantiated in a validation study in which one of the main variables to take into account should be the influence of (national and company) culture. JAR-TEL will fulfil this important role while extending the application of the NOTECHS system to JAR-OPS as well.

PART A.Workpackage 1 : General philosophy and guiding principles

edited by C. Valot & R. Amalberti (IMASSA)

1Workpackage 1 : General philosophy and guiding principles

1.1Objectives of Work Package 1Work Package 1 describes the general philosophy followed by the consortium to propose material for inclusion into JARs concerning the evaluation of non-technical skills of flight crew members.

The report on this work package is divided into four parts. In the first part, the scope and goals of the evaluation of NTS are described. This part includes a review of related regulations (ICAO recommendations, JAR-FCL and JAR-OPS). The second part treats the fundamental principles of evaluation. The third part describes the work strategy used by the consortium to reach the project goals. The fourth and last part contains the proposed text for inclusion in AMCs and IEM related to existing JAR codes.

1.2Scope and goals of NOTECHS: A definition of Non-Technical Skills Non-technical skills (NTS) usually refer to pilots' attitudes and behaviours in the cockpit not directly related to aircraft control, system management and Standard Operating Procedures (SOPs) – the latter are referred to as technical skills (TS).Classic examples of non-technical skills are cockpit authority, crew co-ordination and co-operation, communication and collective decision making, human errors and conflict management, stress and workload management, attention, vigilance and confidence.It is explicitly assumed that a poor exercise of non-technical skills may facilitate errors in technical areas and has the potential to endanger the flight. This is the reason why training NTS is expected to improve flight safety.

Non-technical skills training and checking requirements have recently become part of the aviation regulations. This is the first step of a significant effort to increase crew members’ awareness of these “new” skills. The goal of this effort is to reduce the risk (ultimately: to ensure) that no individual behaviour in flight could lead to an incident or accident.

The requirement for the evaluation of NTS is already present within ICAO documents and provisional JAR-FCL and JAR-OPS. However, these requirements are subject to evolution prior to the actual implementation of JAR-FCL and JAR-OPS. Moreover, although the requirement for evaluation of NTS is already stated in the JARs, the way implementation method is still under discussion. The JAR regulations incorporate a possibility to define one or more Acceptable Means of Compliance (AMC) with these regulations; one of the goals of the NOTECHS project is to provide background material for such an AMC.

1.2.1What the NOTECHS project is aiming at: The central goal of the NOTECHS project is to develop a feasible and efficient method for assessing pilots’ non-technical skills during flight and simulator checks. Additionally, the consortium will formulate a proposed training curriculum for the examiners who will apply the NTS evaluations.

This method is designed to be applicable to the initial pilot licensing process (qualification for a commercial or airline pilot’s licence) as well as to proficiency checks and conversion courses distributed over the professional life of any pilot (e.g., annual recurrent training, command course, type qualification course).

The NOTECHS project effort is also expected to serve a more global harmonisation of human factors requirements between JAR-FCL and JAR-OPS. Non-technical skills or equivalent terms – as mentioned in JAR-FCL and JAR-OPS – are not entirely analogous. The main difference is that JAR-FCL uses a more flight-oriented definition of NTS (based on the Multi Crew Co-ordination concept) when JAR-OPS uses a more psychologically oriented definition (based on the definition and concept of CRM). The definition and clarification of human factors concepts required to establish an evaluation grid for non-technical skills should help the ad-hoc JAA committees (Human Factors Steering Group, Ops Committee and FCL Committee) to improve the harmonisation of all regulations in this respect.

1.2.2What the NOTECHS project is not aiming at:The goal of the NOTECHS project is not to debate on the need for regulations concerning the evaluation of Non-Technical Skills within the JARs. This is the responsibility of the ad-hoc JAA committees.

Therefore, the draft for an AMC and IEM related to NTS evaluation as presented in the last part of this report is generic and should be applicable with little modification to many regulations (JAR-FCL 1.030, 1.240, 1.250, or JAR-OPS 1.943, 1.945, 1.955, 1.965). The goal of this draft is restricted to concisely formulating the material and methods suggested within the report and to serve as input material for the regulatory process within the JAA committees.

The NOTECHS project is neither intending to focus on, nor to propose changes to the human factors tools, programs and courses required by the JARs to teach human factors to pilots, i.e. the ab initio human factors course (JAR-FCL) and the Multi-Crew Co-ordination course (JAR-FCL).

The NOTECHS project does not intend to develop any tool for training pilots in the use of non-technical skills. The scope of the NOTECHS project is restricted to NTS evaluation, not training.

1.3NTS requirements within present regulationsThis presentation includes all paragraphs and sections related to NTS, including those sections referring to NTS training.

1.3.1ICAOIn annex 1 on Personnel Licensing, requirements for ATPL Licensing (2.5) involve "Knowledge of Human Performance and Limitations". Among various skills, the applicant shall have demonstrated the ability to perform "procedures for crew incapacitation and crew co-ordination, including allocation of pilot tasks, crew co-operation and use of checklists"…

with a degree of competency appropriate to:

“ c) exercise good judgement and airmanship […]

f) understand and apply crew co-ordination and incapacitation procedures;”

In annex 6, Operation of Aircraft, the training programme for flight crew members includes:

“...training in knowledge and skills related to human performance and limitations... ”“...the training programme shall be given on a recurrent basis as determined by the State of the Operator and shall include an examination to determine competence ”.

1.3.2JAR-FCL The state of the JAR-FCL when WP1 was conducted (mid-1997)was that the evaluation of non-technical skills is mandatory during the proficiency check for aeroplane type/class ratings and ATPL holders (appendix 1.240 subpart F).

The stepping stone of NTS assessment is the JAR-FCL 1.240.This article defines requirements for class and type ratings as follows:

“The skill test contents and sections for a rating for multi-engine multi-pilot aeroplanes are set out in Appendices 1 and 2 to JAR-FCL 1.240”.

1.3.2.1JAR-FCL - Training for Non-technical skillsNon-technical skills are based upon the Multi Crew Co-ordination (MCC) concept. A brief definition of MCC is given in JAR-FCL 1.001:

“Multi-crew co-operation: The functioning of the flight crew as a team of co-operating members led by the pilot-in-command”.

1.3.2.1.1AMC FCLAMC FCL 1.261(d) offers a more comprehensive survey of the aims of MCC:

“-3- The objectives of MCC training are optimum decision making, communication, division of tasks, use of checklists, mutual supervision, teamwork, and support throughout all phases of flight under normal, abnormal and emergency conditions. The training emphasises the development of non-technical skills applicable to working in a multi-crew environment.”

“-4- The training should focus on teaching students the basics on the functioning of crew members as teams in a multi-crew environment, not simply as a collection of technically competent individuals. Furthermore, the course should provide students with opportunities to practice the skills that are necessary to be effective team leaders and members. This requires training exercises which include students as crew members in the PF and PNF roles.”[…]

“-7- In principle, the purpose of crew co-ordination procedures is to achieve the following aims:a. The pilot-in-command fulfils his managing and decision-making functions irrespective whether

he is PF or PNF.b. The tasks of PF and PNF are clearly specified and distributed in such a manner that the PF can

direct his full attention to the handling and control of the aircraft.c. Co-operation is effected in an orderly manner appropriate to the normal, abnormal or emergency

situations encountered.d. Mutual supervision, information and support is ensured at all times.”

Specific attention is devoted by the AMC to the reinforcement process for non-technical skills acquisition: “REINFORCEMENT-13- No matter how effective the classroom curriculum, interpersonal drills, LOFT exercises, and feedback techniques are, a single exposure during the multi-crew co-operation course for the initial issue of a multi-pilot aeroplane type rating will be insufficient. The attitudes and influences which contribute to ineffective crew co-ordination are ubiquitous and may develop over a pilot’s lifetime. Thus it will be necessary that the training of non-technical skills will be an integral part of all recurrent training for revalidation of a multi-pilot aeroplane type rating as well as of the training for the issue of further type ratings.”

This training for MCC is designed with a specific course (AMC FCL 1.261(d)):

“a. interfaces- examples of software, hardware, environment and liveware mismatches in practice

b. leadership / “followership” and authority- managerial and supervisory skills- assertiveness- barriers- cultural influence- PF and PNF roles- professionalism- team responsibility

c. personality, attitude and motivation- listening- conflict resolution- mediating- critique (pre-flight analyses and planning, ongoing review, post-flight)- team building

d. effective and clear communication during flight- listening- feedback- standard phraseologies- assertiveness- participation

e. crew co-ordination procedures- flight techniques and cockpit procedures- standard phraseologies- discipline”

1.3.2.1.2IEM FCLThe IEM FCL 1.261 (d) gives a survey of conditions and contexts to build situational exercises for MCC training:

“1. Requirements for multi-crew co-operation (MCC) training are set out in AMC FCL 1.261 (d). This IEM gives guidelines for the exercises that may be used in MCC training.2. The exercises should be accomplished as far as possible in a simulated commercial air transport environment. The instruction should cover the following areas:

a. pre-flight preparation including documentation, and computation of take-off performance data;

b. pre-flight checks including radio and navigation equipment checks and settings;c. before take-off checks including powerplant checks, and take-off briefing by PF;d. normal take-offs with different flap settings, tasks of PF and PNF, call-outs;e. rejected take-offs, crosswind take-offs, take-offs at maximum take-off mass, engine failure after V1;f. normal and abnormal operation of aircraft systems, use of checklists;g. selected emergency procedures to include engine failure and fire, smoke control and removal, windshear during take-off and landing, emergency descent, incapacitation of a flight crew member;

h. early recognition of and reaction on approaching stall in differing aircraft configurations;i. instrument flight procedures including holding procedures, precision approaches using raw navigation data, flight director and automatic pilot, one engine simulated inoperative approaches, non-precision and circling approaches, approach briefing by PF, setting of navigation equipment, call-out procedures during approaches, computation of approach and landing data;j. go-arounds, normal and with one engine simulated inoperative, rejected landing, support of the PF by the PNF;k. landings, normal, crosswind and with one engine simulated inoperative, transition from instrument to visual flight on reaching decision height or minimum descent height/altitude.”

1.3.2.2JAR-FCL: Training the instructors to train for MCCSome guidelines are given in the form of a curriculum (in AMC-FCL 1.365; “Course for the type rating instructor rating for multi-pilot aeroplanes TRI(MPA)”) for the training of instructors to teach, train and test applicants with MCC:

“ 1. The learning process2. The teaching process3. Training philosophies4. Techniques of applied instruction

a. Theoretical knowledge - Classroom instruction techniquesb. Flight - Airborne instruction techniques

5. Student evaluation and testinga. Assessment of student performanceb. Analysis of student errors

6. Training programme development7. Human performance and limitations relevant to flight instruction8. Hazards involved in simulating systems failures and malfunctions in the aeroplane during

flight.”

1.3.2.3JAR-FCL: Checking and testing Non-technical skillsThe field of checking is defined in Appendix 1 to JAR-FCL 1.240:

“ Appendix 1 to JAR-FCL 1.240 &1.295: Skill test and proficiency check for aeroplane type rating and ATPL:11. The following matters shall be specifically checked when testing applicants for the ATPL

(A) or a rating for multi-pilot aeroplanes extending to the duties of a PIC…/…(a) management of crew co-operation[…]

13. The applicant shall demonstrate the ability to:(c) exercise good judgement and airmanship(f) understand and apply crew co-ordination and incapacitation procedures, if

applicable; and(g) communicate effectively with the other crew members if applicable.”

A specific article, JAR-FCL 1.262, points out the “Skill”:

“ b) Multi-pilot skill test. An applicant for a type rating for a multi-pilot aeroplane shallhave demonstrated the skill required for the safe operation of the applicabletype of aeroplane in a multi-crew environment as pilot in command or copilot asapplicable…

c) Multicrew co-operation. On completion of the MCC training the applicant shall either demonstrate the ability to perform the duties of a pilot on a multi-pilot aeroplane by passing the type rating skill test on multi-pilot aeroplanes as set out in Appendices 1 and 2 to JAR-FCL 1.240, or shall be given a certificate of completion of MCC as shown in Appendix 1 to AMC FCL 1.261(d).”

1.3.3JAR-OPSIn the AMC OPS 1.945/1.955/1.965, Crew Resource Management (CRM) is defined as:

“ 1. CRM is the effective utilisation of all available resources (e.g. crew members, aeroplanesystems, and supporting facilities) to achieve safe and efficient operation.

2. The objective of CRM is to enhance the communication and management skills of the flightcrew members concerned. The emphasis is placed on the non-technical aspects of the flightcrew performance.

3. CRM training should include the following elements:- Statistics and examples of Human Factor related accidents

- Human perception, the learning process- Situational awareness- Management of workload, tiredness or fatigue, and vigilance - management of stress- Operator’s Standard Operating Procedures- Personality type, delegation, leadership, effective communication skills- The CRM loop with the notion of synergy- Effective communication and co-ordination within the flight crew and between the

crew and other personnel- Error chain and taking decision to break the error chain- Implication of automation on CRM”

1.3.3.1Conversion courseIn JAR-OPS, non-technical skills are involved in the conversion course:

“CRM is incorporated in the conversion course” JAR.OPS 1.945 -9- and related AMCs & IEMs

1.3.3.2Recurrent trainingIn JAR.OPS 1.965 and Appendix 1 are defined the content and modalities of recurrent training:

“ CRM, an operator shall ensure that each flight crew member undergoes a CRM training as a part of a recurrent training” JAR.OPS 1.965 -e- and related AMCs & IEMs.“ CRM training is conducted by suitably qualified personnel” JAR-OPS 1.965 -a-“ An operator shall ensure that each flight crew member undergoes recurrent training and checking […] relevant to the type of aeroplane on which the crew member is certificated to operate.”“ A recurrent training and checking program is established […]” JAR-OPS 1.965 -a 1&2- and related AMCs & IEMs.

1.3.3.3Recurrent checkingIn Appendix 1 to JAR.OPS 1.965 are defined the requirements with regard to recurrent checking:

“ Operator proficiency check must be conducted by a Type Rating Examiner”“ When assessing CRM skills, the examiner required by JAR-OPS 1.965(a)(3)(ii) should occupy an observer’s seat.”

For line checks, in the same Appendix 1:

“ (ii) The flight crew must be assessed on their Crew Resource Management skills.”

The AMC of 1.945/1.955/1.965 states that, for CRM training:

“ the flight crew member should not be assessed either during or upon completion of a course before a full-length syllabus course is completed.”“ The flight crew member should not be assessed either during or upon completion of a recurrent training.”

“ When a flight crew member undergoes an operator proficiency check, line check or command course, then CRM skills should be included in the overall assessment.”

The IEM OPS 1.965 on recurrent training and checking states that:

“1. Line checks, route and aerodrome competency and recent experience requirements are intended to ensure the crew member’s ability to operate efficiently under normal conditions.2. The line check is performed in the aeroplane.3. Line Checks

[…]3.2 In addition to the above duties, flight crew members should be assessed on their crew

resource management skills. The pilot-in-command, or co-pilot acting as pilot-in-command, should also demonstrate his ability to “manage” the operation and take appropriate command decisions. When assessing crew resource management skills, the examiner required by JAR-OPS 1.965(a)(3)(ii) should occupy an observer’s seat.

4. Proficiency Training and Checkinga. When a flight simulator is used, the opportunity should be taken, where possible,

to use Line Oriented Flying Training (LOFT).”

1.3.3.4A summary of present OPS and FCL regulations concerning NTSThe following tables (Table 1) present a summary of the various references to NTS in the JARs (definition, training, checking, etc.):

Table 1JAR-FCL JAR-OPSMCC concept: CRM concept:The functioning of the flight crew as a team of co-operating members led by the pilot-in-command

The effective utilisation of all available resources (e.g. crew members, aeroplane systems, and supporting facilities) to achieve safe and efficient operation

Training for MCC Training for CRMDevelopment of non-technical skills applicable to working in a multi-crew environment

To enhance the communication and management skills of the flight crew members concerned. The emphasis is placed on the non-technical aspects of the flight crew performance

Content of MCC training Content of CRM trainingExamples of software, hardware, environment and liveware mismatches in practice

Statistics and examples of Human Factor related accidents

Human perception, learning processSituational awareness

Management of workload, tiredness or fatigue, and vigilance - management of stress

Crew co-ordination procedures Standard phraseologies

Operator's Standard Operating Procedures

Leadership / “followership” and authority; managerial and supervisory skills, team building

Personality type, delegation, leadership, effective communication skills

Personality, attitude and motivationDecision making

The CRM loop with notion of synergy

Effective and clear communication during flightMutual supervisionTeamwork, and support

Effective communication and co-ordination within flight crew and between crew and other personnel

Error chain and taking decision to break the error chainImplication of automation on CRM

Realisation of MCC training Realisation of CRM trainingA single exposure during the multi-crew co-operation course for the initial issue of a multi-pilot aeroplane type rating will be insufficient. The attitudes and influences which contribute to ineffective crew co-ordination are ubiquitous and may develop over a pilot’s lifetime. Thus it will be necessary that the training of non-technical skills will be an integral part of all recurrent training…

An operator shall ensure that each flight crew member undergoes a CRM training as a part of a recurrent training.An operator should ensure that flight crew member complete the major elements of the full length CRM course over a four year recurrent training cycle.

Checking of Non-technical skillsMCC

Checking of Non-technical skillsCRM

Skill test and proficiency check Operator proficiency check conducted by type rating examiner

An applicant shall demonstrate the skill required in a multi crew environment in a multi-crew environment as a pilot-in-command or a copilot as applicable…On completion of the MCC training the applicant shall either demonstrate the ability to perform the duties of a pilot on multi-pilot aeroplanes

The flight crew members must be assessed on their crew resource management skills. The flight crew member should not be assessed either during or upon completion of a course before a full length syllabus course is completed.

1.4Outline of deliverablesThe deliverables of the NOTECHS project include:∑ the material for establishing a generic AMC for the evaluation of NTS. An Acceptable Means of

Compliance (AMC) constitutes a means, or several alternative means, but not necessarily the only possible means by which a requirement can be met. It should however be noted that when a new AMC is developed, any such AMC (which may be additional to existing AMCs) will be amended into the document following consultation under the NPA procedure. This AMC will include the definition, philosophy and guidelines of evaluation.

∑ the material for establishing a generic IEM for the evaluation of NTS. Interpretative/ Explanatory Material (IEM) helps to illustrate the meaning of a requirement. This material will include a method for evaluation and the minimum tools and requirements for training instructors to use the method.

1.5Philosophy of evaluation and guiding principlesThe proposed philosophy relies upon five principles. Three of them are dedicated to non-technical skills assessment and the last two define general principles for examiner qualification and content.

1.5.1Principle no. 1: “The need for technical consequences”Because of the emotional nature of human behaviours and attitudes, the evaluation of non-technical skills creates multiple opportunities for subjective ratings. Such a subjective evaluation is not desirable, or even acceptable.

The “need for objective consequences” principle is a protection against this risk. It is termed as follows:

Non-technical skills cannot be rated and cannot provoke a FAILED condition out of the context of a related objective technical consequence leading to compromised flight safety in the short or long term.

This principle also means that the observation of markers of a non-technical skill deficiency, i.e. insufficient communication or bad tempered pilot, is not a sufficient motive to declare that the pilot has failed the test. To declare an applicant failed for the reason of non-technical skills, the instructor will need firstly to have observed one or more unsafe flight conditions and secondly to have established a clear causal relation between the unsafe conditions and a deficiency in one or more non-technical skills.

The new evaluation will not add new reasons to fail any test. The new evaluation system goes beyond the present system by pointing to the human factors causes behind a pilot’s deficiency. The pilot should however fail the same test with or without the evaluation of Non-technical Skills because of the need for an objective adverse effect on flight safety, which is already a cause of failure in any present check.

However, one can note that it depends on the authority of each company and its Flight Standards department to decide what is the running definition of “safety engaged”. For example, a check policy may decide that the combination of poor non-technical skills with repetitive and related technical

flaws, even if these technical shortcomings are individually not enough to provoke a failure, may justify at the very end a failed exam.

Table 2 summarises the central idea of “going beyond the present explanation of a failure” with the introduction of NTS. The future grid will enable the instructors to have access to in-depth causes of pilot errors.

Table 2 NOTECHS "going beyond present explanation level" principle.Pilot-induced flight safety problems

Present grid of perception of pilot error and related evaluation (descriptive analysis)

Future grid of evaluation inclusive of NTS (in-depth analysis)

Any unacceptable observable flight consequence due to pilots actions. Flight safety compromised.

? Pilot deviating from procedures or expected standards.? Poor system handling? Poor decision making? Poor co-operation

? Communication? Situation Awareness? Decision making? Leadership

Note that this evaluation of NTS is part of the more global attempt to identify the source of pilot errors, and therefore to reinforce selectively the right area of pilot training, if required.

1.5.2Principle no. 2 “Use of a two-point rating scale” From the evaluation point of view, whatever the type of test (pilot’s licence, command course, type qualification, recurrent), the form used by examiners offers only two possibilities: ACCEPTABLE or UNACCEPTABLE.This logic applies to the evaluation of non-technical skills only.

At the very end, the goal of a NTS evaluation is to separate acceptable behaviour (Passed) from unacceptable behaviour (Failed). An applicant is declared as failing on non-technical skills only if technical safety is compromised (Principle 1).

It will not come as a surprise that there is no such thing as a clear-cut border between the FAILED and PASSED areas. Depending on such things as company culture and examiner personal standards, the exact border will lie somewhere within the area 2 depicted in Figure 1.

The FAILED rating (area 1) corresponds to compromised safety and, with no doubt, all instructors from all companies should agree.

The PASSED rating (area 3) means that concepts and attitudes are well assimilated and exercised; no doubt, all instructors from all companies should agree on PASSED.

In area 2, opinions may vary among companies depending on culture. Even if companies agreed on principle 1, one could assume that the effective adjustment level will be a question of discussion. It will be the scope of the JAR TEL studies to define the final adjustment of this boundary. However,

note that the final adjustment of the rating scale content (performance markers for each rating) will only be available after the field testing of the rating scale as planned in JAR TEL, 1999. Until this time, each operator (and indeed perhaps even each examiner) will have to define its own pass-fail border within the boundaries of area 2.It should be noticed that the intermediate rating (area 2) may also correspond to non-technical skills which still ask for more training, even though they have not caused safety problems.

FAILED PASSED

various opinions

area 1 area 2 area 3

10080604020

common agreement

common agreement

Perc

enta

ge o

f agr

eem

ent

amon

g in

stru

ctor

s

Figure 1 No clear-cut border between FAILED and PASSED.

1.5.3Principle no. 3: “Explanation of unacceptable rating required”For each negative rating on a non-technical skill category, the examiner should give an explanation in free text on the exam form, using standard phraseology, e.g. as defined within the NOTECHS system. Special attention should be given to indication of the applicable NOTECHS element and the connection with the resulting technical flight safety endangerment.

1.5.4Principle no. 4: “Repetition required”Most human behaviours are part of a general style. The goal of the NTS evaluation is not to focus on an isolated behaviour but merely to identify more permanent weaknesses which could be susceptible to training.This is to say that a certain leitmotiv of similar behaviours during the check would have to be recognisable to conclude that the pilot has problems in this area.

Aside from this general idea of a leitmotiv, the instructor should give a second chance to the applicant anytime this applicant has compromised flight safety due to the exhibition of a non-technical skill deficiency, and therefore is supposed to FAIL to the test. This second chance should consist of one repetition of the exercise, the situation or the procedure where the deficiency was observed. It has to be stressed that this second chance is only proposed to prevent that pilots are being failed on non-technical behaviour close to the border between PASSED and FAILED. It is normal to give a candidate a second chance on certain technical group items, but for some serious "technical failures" the candidate is never granted a second chance. The proposed NOTECHS repêchage should have the same spirit. If the nature of the technical failure is such that a second chance is not granted according to the relevant JAR-requirements, than the fact that there are non-technical reasons for the failure can never lead to this second chance.

1.5.5Principle no. 5: “No evaluation of personality traits”Any evaluation should only be based on observable behaviours.

When inferences (interpretation of facts) require access to social or cognitive skills, they should be limited and based on obvious and observable facts and behaviours. The grid must exclude any reference to a pilots’ personality or to emotional attitudes. That is the reason why a set of behavioural markers is described in the following part of the NOTECHS report. These behavioural markers are designed to support an objective judgement of the trainee.

PART B.Workpackage 2 : Review of existing systems

edited by E. Kruijsen and H. Nijhuis (NLR)

2Workpackage 2 : Review of existing NTS training and evaluation systems

2.1AbbreviationsACRP Aerospace Crew Research ProjectCRM Crew (Cockpit) Resource ManagementDLR Deutsche Forschungsanstalt für Luft- und Raumfahrt (German Aerospace Research

Establishment)DTD Department of Training Development (British Airways)FAA Federal Aviation AuthorityJAA Joint Aviation AuthoritiesJAR Joint Aviation RegulationsLLCV4 Line/LOS Checklist, version 4LOFT Line-Oriented Flight TrainingLOS Line-Oriented SimulationNASA National Aeronautics and Space AdministrationNLR Nationaal Lucht- en Ruimtevaartlaboratorium (National Aerospace Laboratory)NTA Non-Technical AspectsNTS Non-Technical SkillsRLD Rijksluchtvaartdienst (Dutch Civil Aviation Authorities)UT University of Texas

2.2IntroductionThe NOTECHS strategy is to base – if possible – its new NTS evaluation system on NTS training and/or evaluation systems presently in use with European airlines [van Avermaete & van Blanken 1997]. This section describes an inventory of a number of current NTS systems, including a short overview of the experiences of users (i.e. instructor pilots, examiners and line pilots) with these systems. Then follows a discussion on commonalities and differences between the systems, and some concluding remarks which may serve as an input for the next workpackage.

Four existing NTS training systems will be discussed, namely those in use at Air France, British Airways, Lufthansa German Airlines and KLM Royal Dutch Airlines. A new system, now under development at KLM, that will be specifically tailored to integrate TS and NTS assessment, has been added. Additionally, two research systems have been included in the discussion: a system used for diagnostic and data-collection purposes developed by the NASA/University of Texas/FAA Aerospace Crew Research project, and a system developed under auspices of the Dutch Civil Aviation Authorities for future use during flight exams. The information on the airline systems has been provided by training specialists employed by these airlines, and the research systems are described from literature: [Taggart 1995; Helmreich, Butler, Taggart & Wilhelm 1994] and [Rijksluchtvaartdienst 1997], respectively. Although not in use with an airline in Europe, the NASA/UT/FAA system has been added since many training systems in use with airlines (e.g. British Airways) are based on adaptations of this system. As far as possible, the systems have been described in a common way, making use of the same terminology, to facilitate comparisons. It must be said however that the different terminology used in the various systems for the description of (approximately) the same behavioural markers poses a problem in itself; where translation to common terms was not possible or desired, the original terms have been maintained.

2.3Description of current NTS training systems

2.3.1Air France

1.1 Name of the program Air France has given no specific name to its NTS training and evaluation system.

1.2 Developed by Air France, internally.1.3 Developed for Air France.1.4 Year A number of NTS-related items were included in the evaluation

form for annual simulator checks in 1984. Development of the NTS training system, intended for use in LOFT, began in 1994.

1.5 System foundation The LOFT training system is based on items stemming from the basic CRM courses that were already in use at Air France.

Status2.1 Role within the

companyThe NTS items on the annual check are true examination items; the LOFT system is used for training only.

2.2 Year of introduction Introduction of both the NTS items on the annual check forms and the LOFT system shortly followed the start of development (item 1.4).

2.3 Implementation phase

The LOFT system was gradually introduced to all pilots and instructors in each flight division. Concurrently, a system enabling events reporting by pilots was developed.

2.4 Changes in the system

No significant changes have been incorporated since the introduction, and Air France presently does not see any need for changes.

Appearance3.1 Implementation For the annual check evaluations, the instructor fills out a check

form with standard notations.During the LOFT de-briefs, both pilots fill out a sheet with remarks on NTS behaviour of the other crew member. The sheet helps them with “human factors awareness”. The instructor fills out a sheet describing crew functioning during the flight. These sheets, and a videotape of the crew’s performance during the simulator session, are used to aid the discussion during the de-brief. After the de-brief, the instructor’s sheet is destroyed.

3.2 Relation TS-NTS On the annual check form, the TS and NTS items are separated.The LOFT debriefing is mainly (about 80%) about NTS issues; the de-brief is usually referred to as the “CRM de-brief”.

3.3.1 Number & labels of categories

The annual check evaluation form contains four categories:1. Aircraft handling2. Flight management3. Knowledge

4. Professional behaviourOnly the last category is regarded as related to non-technical skills.The LOFT de-brief sheets have no categories, only elements (according to how the terms are used within the NOTECHS project)

3.3.2 Number & labels of elements

The NTS category on the annual check evaluation form has the following elements:4. Professional behaviour

4.1 Exactness4.2 Attention management4.3 Checking and decision making

4.3.1 Planning4.3.2 Task prioritisation4.3.3 Decision making

4.4 Intellectual honesty [sic.]4.5 Good captain’s/copilot’s attitude

The LOFT de-brief elements are the following:- Synergy- Communication- Action projects (sic.)- Stress- Fatigue and work load- Use of automation- Decision making- Errors- Organisational problems

3.3.3 Application of rating scales

The annual check elements are rated according a three-point scale: (–), (+), (++).No rating scales are applied for the LOFT de-brief.

3.4 Possible consequences of rating

Any minus (–) in the Professional Behaviour category on the annual check form implies that the pilot has failed the test.The LOFT sessions are not rated and thus have no consequences.

3.5 System output filed The annual check forms are filed.The LOFT de-brief sheets are not filed.

Operators4.1 Used by The annual check forms are filled out by instructor pilots.

During the LOFT de-briefs, pilots use the system for mutual feedback.

4.2 Evaluator training Instructors are trained in a seminar on CRM de-briefs. The training consists of an explanation of NTS elements and video case studies that the instructor must de-brief.

Use5.1 Description of

system useBoth the check flights and the LOFT sessions are repeated annually. On the annual check, each pilot is rated individually. The CRM de-brief following the LOFT session focuses on both the individuals and the crew as a whole.

5.2 Feedback to pilots The CRM de-brief following the LOFT session involves a discussion on NTS issues in which both the instructor and the pilots participate. They are aided by information sheets.

5.3 Known problems/difficulties

A reported problem concerns difficulties encountered in building and maintaining a good balance between flight safety and training on one side and assessment and possible consequences for pilots on the other.

5.4 Evaluator experiences

No information is available on this topic.

5.5 Acceptance No information is available on this topic.

MiscellaneousNone.

2.3.2Lufthansa German Airlines

1.1 Name of the program Lufthansa CRM Quick Reference List (CRM-QR)1.2 Developed by Lufthansa German Airlines, in co-operation with the German

Aerospace Research Establishment DLR1.3 Developed for Lufthansa (later also used within Hapag Lloyd, a German

Charter operator)1.4 Year 19941.5 System foundation The CRM-QR is based on the contents of a CRM course that

was developed by DLR together with Lufthansa in 1993. The course contents were derived from the FAA National Plan on CRM training and on a survey of training needs within Lufthansa. The topic “stress management” has been dropped.


companyThe CRM-QR is used for training only, not for evaluations. It serves as a guideline for the instructor to give systematic feedback on CRM-related behaviour in refresher courses, LOFT and Video-LOFT. It is also used for the same purpose in initial CRM seminars.

2.2 Year of introduction 19942.3 Implementation

phaseThe CRM-QR was introduced for all pilots at the same time; at first only for the initial CRM seminars, later it was also included in courses for instructor pilots and in Video-LOFT.


No significant changes have been incorporated since the introduction in 1994. The training arm of Lufthansa, Lufthansa Flight Training GmbH (LFT) has regrouped the items into four instead of the initial three categories, since this seemed to make the scope of the behaviour covered more complete.

Appearance3.1 Implementation The instructor uses a form with the items grouped into four

categories plus a list of definitions for the items. Observations are qualitative in free text format.

3.2 Relation TS-NTS CRM-QR covers CRM behaviour in CRM terminology. However, by its nature it is clearly overlapping with any technical aspects.


There are four categories:1. Management2. Teamwork3. Assessing and Deciding4. Communication


22 items are defined by their desired behavioural markers. They are divided among the four categories as follows:1. Management:

1.1 planned ahead1.2 avoided time pressure1.3 fought distraction1.4 set goals1.5 structured decision making

2. Teamwork:2.1 strived for positive first impression2.2 cared for human redundancy2.3 brought in knowledge in time2.4 supported others2.5 accepted criticism and reservations

3. Judgement and decision-making:3.1 used all sources of information3.2 searched for options3.3 evaluated advantages and disadvantages3.4 announced intentions3.5 delegated sensibly3.6 analysed decisions

4. Communication:4.1 asked for opinion of others4.2 listened to others carefully4.3 expressed own point of view adequately4.4 gave feedback4.5 stated also unclear reservations4.6 talked about discrepancies


No rating scales are applied.


Observation of not-required behaviour can lead to recommendations for specific training. As a feedback tool, CRM-QR has behaviour reinforcing and correcting functions.

3.5 System output filed No comment on this subject by the users.

Operators4.1 Used by Used during CRM seminars by pilots for mutual feedback.

Instructors and check pilots use the system also during refresher and (Video-)LOFT.

4.2 Evaluator training Instructors and check pilots are trained in a four-day seminar on instructing and moderating skills. Additionally they are trained in using the system and giving feedback in training course on Video-LOFT (one day classroom, one day simulator).


system useThe form is completed during and after the training session. Observations are made on the individual pilot. However, since crewmembers are interacting, the individual observations cannot be seen independently.

5.2 Feedback to pilots The preferred feedback method is self-debriefing of the crew.5.3 Known

problems/difficultiesNot the full scope of CRM-related behaviours is observable. Even for the observable aspects of communication and interaction it is very difficult to define the markers independently of the situation. Situation-specific items can be checked more easily but generally are not representative for the behaviour category under consideration.


The users are glad that they now have a systematic tool for giving feedback on the non-technical aspects of a flight.

5.5 Acceptance If the system is used in the correct manner, the pilots being observed are very positive towards the feedback. However, the instructor pilot should not exaggerate this area by trying to analyse the pilots’ personality.

MiscellaneousNone.

2.3.3British Airways

1.1 Name of the program CRM Foundation Course. Other training modules are: the Core Course for Trainers and Advanced Trainer Skills. For convenience, the basic course will be referred to as “Reflections” throughout although this is not its official title.

1.2 Developed by British Airways — Flight Operations Development Training (FODT) department ; and in the future the fleet training departments.

1.3 Developed for British Airways.1.4 Year 1989 - 1991 for the initial course and ongoing for all the

subsequent courses — recurrent training and courses for trainers etc.

1.5 System foundation The training modules are based on a combination of the ICAO (Canada) guidelines and first principles i.e. the skills of the FODT. The FODT took the basic model from ICAO. However, the Core Course for Trainers is derived from Helmreich’s course and only modified slightly to fit BA’s procedures.


companyCRM courses are used for training only, although BA is preparing for the day when CRM will be examined. The Team Skills (behavioural markers) list is used in training and LOFT sessions.

2.2 Year of introduction The basic CRM course first ran in 1991, the Core Course for Trainers had its debut in 1995 and the Advanced Trainer Skills in 1997.


The CRM course was introduced for the whole company but specific groups were targeted in the beginning. So, the first pilots to go through the course were mainly instructors and check pilots and trainers. FODT also put together a group of 50 of these instructors/examiners to act as advisors on course development and progress. The content has shifted however: BA’s original focus was on the awareness of the need for CRM, now it is on the need to put CRM into practice.


The original CRM course (Reflections) did not contain the Team Skills list. The system has not been changed; rather, it has taken shape. DTD is building up a structure of a CRM programme from the basis of the Reflections course. It has built on the Core Trainers course and the Advanced Team Skills and are now working on a recurrent training course which is interactive with the safety reports. The material from a much earlier flight skills course (1980s) has been reworked into these 3 courses.

Appearance3.1 Implementation The CRM course material consists of a variety of presentations

and exercises which are summarised in a booklet. The booklet contains key points and examples. The FODT philosophy for training is to make it action-orientated, so all courses include exercises.

3.2 Relation TS-NTS This is a CRM-only course but thereafter the emphasis is on integrating technical and non-technical skills. This is why recurrent training will be based around findings from safety reports — teaching the CRM with the technical skills. In a future phase BA is developing a routine check report which does include CRM assessment. Here technical and NTS are combined in one report.


The Team Skills list contains 8 categories:1. briefing2. inquiry/advocacy/assertiveness3. feedback4. communications/decisions5. leadership/followership6. interpersonal relations/climate7. preparations/plans/vigilance8. workload/avoiding distractions

The future phase will have 6 categories.


The 8 categories (called “basic team skills”) have a number of markers addressing each to give 40 items in total. This list of markers is being expanded gradually as the programme develops; a future phase plans 53 items plus 2 summary items.No labels are available.


There is no rating scale. The future phase has one rating scale for the overall assessment. It has 5 categories, but the unsatisfactory category is kept distinct from the 1-4 scale of standard of performance.


LOFT is non-jeopardy, therefore it is not rated officially. If a pilot performs unacceptably during a LOFT check then s/he will be put in for further training, so problems are not ignored. Pilots also do a 6-monthly check in which there is one box for CRM skills: comments where free-text comments and remarks can be placed.Future phase consequences would initially be non-jeopardy but could be used for jeopardy ratings if demanded (by the JAA).

3.5 System output filed All records of checks go onto a pilot’s training file (all remarks and comments by his/her instructors). BA does not hold a specific CRM database.

Operators4.1 Used by The Team Skills list is used by pilots and instructors in training.

For checks there is only the comments box.4.2 Evaluator training Instructors and examiners have all received the basic course and

most have been through the Core Course for Trainers. The Advanced Trainer Skills course concentrates on debriefing, facilitation and communication skills.


system useThe BA system is action-oriented; it focuses now (after the initial emphasis on awareness) on making pilots put the CRM skills they have learned into practice. The BA system is also being designed and structured so that in the future it forms a closed loop: safety reports feed into the training and recurrent CRM training will be put together focusing on the key topic(s) arising from the analysis of safety reports. The safety report system BASIS codes up the reports using the Team Skills list as a reference so NTS problems are coded in BA CRM terms. FODT feels that only basic CRM-training can be useful company-wide, thereafter recurrent training needs to be more aircraft-type specific and will therefore be put together by the fleet training departments.

5.2 Feedback to pilots LOFT sessions have debriefs built into the session.In future phases the routine checklist has to be signed off by the pilot to say that s/he understands all the comments and marks made on the list. Therefore, debriefing is built into this list officially, the examiner will have to go through the list with the pilot after the session and account for their marking.


None: FODT are solving problems as they go; the CRM training system at BA is in a continuous state of development.


Instructors are positive to varying degrees. The issue of recurrent checking and training lies in large part with the fleets and the instructors who run fleet training. Some of them are proactive and developing future checklists whilst others wait for directives from FODT, but everyone is working towards it at some level.

5.5 Acceptance No one has ever rejected the Team Skills training, however there has been no official survey of the courses.

MiscellaneousNone.

2.3.4KLM Royal Dutch Airlines

2.3.4.1KLM Feedback and Appraisal System (FAS)Apart from interviews with KLM training personnel and the author’s personal experiences with the system, additional information was obtained from a paper written by specialists from the training development department of KLM [Antersijn & Verhoef 1994].

1.1 Name of the program Feedback and Appraisal System (FAS). The NTS part of this training system is called WILSC, after the first letters of the behavioural categories in the system.

1.2 Developed by KLM Royal Dutch Airlines1.3 Developed for KLM1.4 Year 1990 (start of programme definition)

1992 (start of operational try-out phase)1994 (WILSC fully operational)

1.5 System foundation The raison d’être for FAS is the notion that TS and NTS training should be structured. The NTS part of FAS (called WILSC) was developed first. Much NTS-related training was already conducted at KLM, but there was no relation between different trainings. The WILSC behavioural categories were derived from interviews with KLM instructors, pilots and flight engineers. They brought in on-the-job situations where NTS would play a part, and the training development department integrated these into a behavioural marker list.


companyThe FAS is used for training only, not for evaluations; however, during a proficiency check debriefing the examiner may use WILSC terminology.

2.2 Year of introduction 19942.3 Implementation

phaseThe system was introduced first in the A310 and DC-10 fleets by means of a special type recurrent session with FAS-related study items. Presently all fleets are using the same system.


No significant changes have been implemented since the initial introduction.However, presently a work group is devising a new system that can be used for evaluation, as prescribed by the new JAR rules. Although this system is based on the FAS, it will probably differ substantially from its predecessor.

Appearance3.1 Implementation WILSC consists of glossary of NTS terms (a ‘dictionary’ which

is to ensure that the instructor and the pilots speak the same language) and a reference part, containing descriptions per behaviour item of effective, or desired, behaviour (as defined by KLM).

3.2 Relation TS-NTS FAS is defined as having a technical and a non-technical (WILSC) part, so that both types of behaviour can be dealt with during training and debriefs within a common framework. The technical part of FAS has not been developed yet, however.


There are five categories, denoted WILSC by the acronym of their names:1. Work Attitude2. Information Management3. Leadership4. Stress Management5. Co-operation


There are 14 items, divided among the categories as follows:

1. Work attitude:1.1 procedure orientation1.2 assertiveness1.3 exercise of self-criticism1.4 sense of responsibility1.5 service orientation1.6 image

2. Information management:2.1 information analysis2.2 planning and anticipation2.3 decisiveness

3. Leadership:3.1 task-oriented leadership3.2 people-oriented leadership

4. Stress management:4.1 stress management

5. Co-operation:5.1 working with others5.2 attentiveness


There is no rating scale within FAS. However, descriptions of “effective” and “ineffective” behaviour (as defined by KLM) are given as examples of the top and bottom ends of a scale.


KLM has no NTS rating system. The instructor can give free-text comments on the test sheets.

3.5 System output filed Any free-text comments made by the instructor are stored in the pilot’s training file, together with all other training results.

Operators4.1 Used by KLM instructors and pilots for CRM training courses (2 x 3

days), type qualification courses, type recurrent sessions (one or more FAS elements are part of the homework study items).

4.2 Evaluator training FAS training is an integral part of KLM instructor training.


system useAll pilots receive specific CRM training during two 3-day courses. For every type recurrent session, a part of the WILSC list of behaviours is specified as homework study item. It is up to the instructor to decide how he wants to use the FAS during the debrief, but the use of WILSC terminology prevents miscommunication etc.

5.2 Feedback to pilots There is no standard method to provide feedback to pilot: the instructor decides if and how s/he wants to make use of WILSC terminology during the debrief.


None. Initial effectivity studies pointed out that pilot acceptance of the system is good and that the list of behavioural markers is complete, i.e. covers those behaviours necessary for crew operations. The study also pointed out that certain markers pertained more to simulator sessions, and some others more to e.g. route training and normal line operations. The users of the system are taught to use only those markers that are necessary during their debriefs.


See previous point.

5.5 Acceptance See previous point.Miscellaneous

None.

2.3.5KLM Royal Dutch Airlines - SHAPE

1.1 Name of the program SHAPE (the acronym for the first letters of the behavioural categories in this system)

1.2 Developed by KLM Royal Dutch Airlines1.3 Developed for KLM1.4 Year Development began in 1996. Presently, the system is still under

construction; the first limited evaluations within KLM are taking place.

1.5 System foundation One of the recommendations of the working group that developed the FAS system (see KLM-FAS) was that an NTS assessment system should be derived from FAS. The SHAPE items are therefore based on the WILSC items from FAS (all WILSC items are included in SHAPE, though they are grouped differently). One of the first steps of the working group was to combine TS and NTS to form an integrated description of the functioning of a pilot or flight engineer. Hence, SHAPE succeeds both the TS and the NTS parts of FAS.KLM studied comparable systems from other airlines (Swissair,

Aer Lingus, Air France, Northwest Airlines, Qantas and United Airlines) and made extensive use of the work of dr. Hawkins [Hawkins 1984] and of the ACRP [Helmreich, Butler, Taggart & Wilhelm 1995].


companySHAPE will be used within KLM for quality assurance and performance improvement of cockpit personnel both in TS and NTS through measurement and comparison with a KLM standard (which still has to be determined).

2.2 Year of introduction the system has not been introduced yet.2.3 Implementation

phasePresently the system is being tested on a limited scale by instructor pilots during simulator prof checks.


not applicable.

Appearance3.1 Implementation KLM still has to decide which items from SHAPE can be rated

during exams. These items will appear on (computerised) score sheets and can be checked as “good”, “acceptable” or “not acceptable”. Good and acceptable marks will lead to passing the test.KLM has not yet decided on other implementation aspects, such free-text comments and debrief aspects.KLM is also developing a new assessment system, in which a structured approach of performance in a training/exam exercise must lead to consistent ratings.

3.2 Relation TS-NTS Technical and non-technical aspects are integrated in SHAPE. A new assessment system is being set up in which technical and non-technical SHAPE items are used in combination to arrive at a score for an exercise.


There are five categories, denoted SHAPE by the acronym of their names:1. Self2. Human Interaction3. Aircraft4. Procedures5. Environment and Situation


There are 29 items, divided among the categories as follows:

1. Self:S.1 assertivenessS.2 workload managementS.3 imageS.4 exercise of self-criticismS.5 sense of responsibilityS.6 self confidence

2. Human Interaction:H.1 working with othersH.2 task-oriented leadershipH.3 people-oriented leadershipH.4 communicationH.5 service orientationH.6 attentiveness

3. Aircraft:A.1 system knowledgeA.2 automation handlingA.3 manual handlingA.4 instrument scanA.5 outside scanA.6 panel scan

4. Procedures:P.1 knowledge of aircraft proceduresP.2 knowledge of company regulations and proceduresP.3 procedure orientationP.4 normal/conditional procedure handlingP.5 emergency and abnormal procedure handlingP.6 administration

5. Environment and situation:E.1 position awarenessE.2 situational awarenessE.3 information analysisE.4 planning and anticipationE.5 decisiveness


KLM intends to rate exercises (flight phases encountered during an exam, where an abnormal situation may or may not occur) according a three-point scale: “good”, “acceptable” and “not acceptable”. A “training recommended” remark may accompany the acceptable rating, and “training required” must accompany the not acceptable rating. The overall rating for the exam will be “proficient” or “not proficient”, depending on whether a “not acceptable” has been scored for any exercise.


The new rating system will be used for line and proficiency checks, so failing such a test will mean a grounding of the applicant.

3.5 System output filed No details are known on this subject yet.Operators

4.1 Used by The system will be put in use by KLM in the near future. Present planning indicates an in-use date of July 1998.

4.2 Evaluator training All examiners using this system are KLM instructor pilots. They will have received extensive training with the system during their instructor’s course.


system useThe system will be used for all exams at KLM: type ratings, line checks and proficiency checks.

5.2 Feedback to pilots Presently, applicants are thoroughly debriefed after each exam. This will remain so when the new system has been put into place.


There is no information on this topic yet: the first, limited evaluation is presently taking place. Early discussions with pilot instructors indicated that the definitions of the terms used in the SHAPE system will have to be very accurate and unambiguous for the system to work properly.


See previous point.

5.5 Acceptance See previous point.Miscellaneous

None.

2.4NASA/UT/FAA

1.1 Name of the program The NASA/UT/FAA Line/LOS checklist (version 4) (LLCV4)1.2 Developed by NASA/University of Texas/FAA Aerospace Crew Research

Project (ACRP)1.3 Developed for Airlines operating multi-crew aeroplanes.1.4 Year 1987 (original version)

1994 (version 4)1.5 System foundation The project started out with an experimental data form to collect

information on individual CRM performance of pilots receiving training at a large airline. The behavioural markers have their origin in the analysis of accidents and incidents specifically where human factors issues were causal. Based on these observations, the data form was modified in a continuous process to the current version, number 4.


companyThe LLC itself is used as a diagnostic and data collection tool. Several airlines have however implemented NTS training systems, based on adaptations of the LLC.

2.2 Year of introduction 1994 (version 4)2.3 Implementation

phasenot applicable.


The LLC is being updated as part of an ongoing process. A short version of the LLCv4 has been developed in 1996 for the evaluation of crew human factors skills in line flight settings.

Appearance3.1 Implementation A group of airlines within the ACRP have set up groups of

observers (typically 18-25 persons) who use the LLC to record observations of line flights and LOFT. The data are collected in a non-jeopardy setting, usually on fleets other than the observer’s own. These observations form an audit of the airline’s NTS system performance.The LLC in itself is not used for training or evaluation.

3.2 Relation TS-NTS Although the form contains an item “overall technical proficiency”, the LLC is designed specifically for the collection of data regarding non-technical issues. Therefore, any invasion of technical issues would contaminate the observations and are left out as much as possible.


There are 6 categories or “target areas”:1. Team Management & Crew Communications2. Situational Awareness & Decision Making3. Automation Management4. Special Situations5. Overall Observations6. Operational Considerations


There are 31 specific assessments, divided among the categories and described as desired behaviours. Below, they are listed in short:

1. Team Management & Crew Communications1.1 team concept and environment for open communications1.2 quality of briefings 1.3 cabin crew is included as part of team1.4 group climate is appropriate to operational situation1.5 crew members ask questions regarding crew actions and

decisions1.6 crew members state their information with appropriate persistence1.7 operational decisions are clearly stated to other crew

members1.8 captain establishes proper balance between authority and

crew1.9 participation

2. Situational Awareness & Decision Making2.1 time and workload management2.2 prioritisation of secondary operational tasks2.3 demonstration of high vigilance during high and low

workload2.4 preparation for expected or contingency situations

3. Automation Management3.1 automation operation guidelines established and

followed

3.2 PF/PNF tasks established and followed3.3 periodical review and verification of automated systems

status3.4 verbalisation and acknowledgement of system entries

and changes3.5 time management (for system programming)3.6 automation used at appropriate levels

4. Special Situations4.1 quality and timing of positive and negative feedback4.2 acceptation of performance feedback4.3 handling of conflicts4.4 recognition of fatigue and alertness management4.5 (avoidance of) creation of self-imposed workload and

stress4.6 sharing of operational knowledge and experience

5. Overall Observations5.1 overall technical proficiency5.2 overall crew effectiveness

6. Operational Considerations6.1 severity of abnormals and other systems events6.2 complexity of operating environment


A four-point scale is used:1. Poor: observed performance is significantly below expectations2. Minimum Expectations: observed performance meets minimum requirements but there is ample room for improvement.3. Standard: demonstrated behaviour promotes and maintains crew effectiveness. This is the level of performance that should be normally occurring in flight operations.4. Outstanding: performance represents exceptional skill in the application of specific behaviours, and serves as a model for teamwork.

These ratings must be given for four distinct phases of flight:a. pre-departure (including flight preparation)b. take-off and climbc. cruised. descent/approach and landing

In addition, ample room for free-text comments is provided.3.4 Possible

consequences of rating

None: the LLC data collection is always non-jeopardy.

3.5 System output filed see previous point. The data collected is used for assessing NTS training system performance and for e.g. the design of next year’s recurrent courses.

Operators(not applicable)


system useSee point 3.1

5.2 Feedback to pilots No feedback is provided directly.5.3 Known

problems/difficultiesSince the LLC is mainly a research tool, the difficulties documented are not relevant for airline use. However, a shorter version of the LLCV4 has been developed specifically for use during line checks [Wilhelm & Helmreich 1996].


No evaluator experiences with the “shortened” LLCV4 have been documented yet.

5.5 Acceptance Not applicable.Miscellaneous

none.

2.5RLD ACE-P

The Dutch CAA (Rijksluchtvaartdienst RLD) examiners’ commission (ACE-P) developed in 1995 an NTA assessment system for use in examinations of Dutch pilot’s licence applicants. The system was validated in a test involving a “shadow-assessment” by examiners during actual examinations. RLD has added a chapter in its examiner’s handbook on NTA, but the actual implementation of NTA assessments during licence and type rating exams will be implemented when the new JAR-FCL becomes operational.

1.1 Name of the program WMCL (the acronym for the behavioural components in the system)

1.2 Developed by the Dutch CAA (Rijksluchtvaartdienst RLD) examiners’ commission (ACE-P)

1.3 Developed for the Dutch CAA

1.4 Year 1993: concept1994: evaluation1997: version III in the RLD examiner’s handbook

1.5 System foundation The ACE-P based its system on the JAR-FCL text and literature available at the time (e.g. the NASA behavioural indicator system, ICAO documents and the KLM experiences, see paragraph 2.3).


companyThe system will be used for examinations by RLD examiners.

2.2 Year of introduction The system will be implemented concurrently with the JAR-FCL or earlier as decided by the RLD.


Text on the system has been added to the RLD examiner’s handbook. The assessment form will be changed when the system becomes active.


None; the evaluation revealed some problems (the examiner should not be part of the crew, remarks on the education of examiners and the definition of “not acceptable” behaviour) which were immediately implemented in the system.

Appearance3.1 Implementation see § 2.3

3.2 Relation TS-NTS The description in the examiner’s handbook is very specific about this point: when an applicant’s behaviour is deemed “not acceptable” when:1. an obvious relation with one or more technical aspects

exists2. the observed behaviour has led to (potential) danger to the

flight3. the “not-acceptable observed behaviour” recurs throughout the exam


1. Work Attitude2. Management3. Co-operation4. Leadership


1. Work Attitude1.1 keeping to procedures and regulations1.2 assertiveness1.3 responsibility/self-criticism

2. Management2.1 information analysis2.2 planning and anticipation2.3 decisiveness

3. Co-operationco-operation/team spirit

4. Leadership4.1 task-oriented leadership4.2 people-oriented leadership


The score form contains one NTA box that can be filled with a (+) or a (-) for acceptable or unacceptable behaviour. A (-) in the NTA box must always be accompanied by an unsatisfactory score on one of the technical categories.


A (-) in the NTA box accompanies a score below 6 on a technical category, which in itself is a reason for failing the test.

3.5 System output filed All exam forms are filed by the RLD.

Operators4.1 Used by The system will be used by RLD examiners for marking

applicants for CPLs, ATPLs and type ratings of multi-pilot aeroplanes.

4.2 Evaluator training The evaluators are not trained, but the examiner’s handbook contains a chapter on NTA assessment.


system useThe test score sheet contains 5 boxes for overall scores on technical groups and one box for the NTA score. A (-) on NTA must always be accompanied by a score below 6 on a technical group. When a technical group is re-examined, the NTA group will also be re-examined.The score form also contains boxes for the 4 categories, but scores (+) or (-) are used for clarification purposes only.

5.2 Feedback to pilots All applicants are debriefed after their exam.


See § 2.4 for some results of the evaluation.


None, since the system is not active yet.

5.5 Acceptance not applicable

MiscellaneousNone

2.6Discussion of commonalities and differences

1All airlines that use NTS systems were at least involved in the development of their system. Sometimes they were assisted by research establishments. A number of airlines have adapted their NTS system from the NASA/UT LLC system.

2. All investigated NTS systems are used primarily or even exclusively for training. When NTS are assessed during exams, insufficient non-technical performance is never a direct reason for failing an applicant (although Air France will fail an applicant when s/he displays insufficient

“professional behaviour”). This has been formalised in the RLD system, where it is stated that an insufficient score on NTS will only be given when an “obvious relation” exists with one or more technical aspects. The new KLM system has completely integrated NTA and technical aspects.

3. Most airlines keep non-technical aspects separate from technical skills. British Airways teaches NTS separately from TS, but tries to integrate them thereafter. See also the remark on the RLD and new KLM systems in the previous point.

4. Some behavioural markers appear in several systems, such as e.g. “working with others”, “information management” and “decision making”. However, all systems use different names and definitions for their markers, which prevents accurate comparisons.

5. None of the systems seems to effectively translate its NTS categories and elements into unambiguous and objectively measurable behaviours. All systems use connections with day-to-day operational situations to facilitate use by instructors and acceptation by the pilots; some systems contain a “dictionary” which provides a working definition of the terms used, and some have a description of positive and negative examples of behaviour associated with an element (as defined by the company). These clarifications are not accurate enough however to enable a pilot-examiner to provide a reliable rating per element.

6. The airlines that have a rating scale for NTS assessment do not differentiate at an objective behaviour level, and all have different scales. But all do differentiate between sufficient and insufficient performance. Unfortunately, the threshold between sufficient and insufficient remains rather obscured. Airlines worked around this problem by providing space for free-text comments on NTS performance, and RLD and KLM call for an obvious relation with insufficient performance in one or more technical categories (where the line between pass and fail is very clear).

7. The NASA/UT and new KLM systems provide ratings for different flight phases, all other systems provide one rating for the whole flight (or check).

8. Data on the relation between NTS assessments and actual pilot NTS performance would be very helpful, but are hardly available. British Airways is in the process of setting up a system where safety reports provide input for NTS training.The limited number of user experiences which are described, are favourable, though. Some remarks address incomplete coverage of behaviours by categories and elements, and the notion that behaviours used for simulator training may be different from those to be used during normal line operations. The latter implies that behavioural markers used for simulator checks may have to differ from those used for line checks.

9. BA and the ACRP speak of continuous NTS system improvements, Lufthansa and KLM say they did not change their systems (a lot). However, KLM’s decision to start development of a whole new training and evaluation system may be interpreted as a system improvement.

10. All airlines train their instructor pilots in the use of and training in NTS.

2.7Conclusions

1. The fact that no airline has decided to use an off-the-shelf NTS system for training or assessment indicates that cultural differences may play an important role in NTS system design. Especially when an NTS system will be used for assessment (i.e. pilots may fail exams due to insufficient NTS performance), the whole subject of implementation becomes extremely sensitive with the pilot community.

2. The differences in rating scales notwithstanding, all airlines try to define a distinction between sufficient (pass) and insufficient (fail) NTS performance. A further gradation in sufficient performance may depend on existing rating conventions within the airline. The boundary between pass and fail is not sharply defined, which may lead to inconsistent pass-fail decisions by examiners, which in turn may lead to pilot acceptation problems.

3. Clear and unambiguous definitions of all terms, categories and elements in an NTS system are necessary for proper, consistent fail-pass decisions and clear pilot debriefings, especially when the system will be used by several airlines. Pilots will have to be taught the exact meaning of the terms used in the NTS assessment system.

4. It would be advisable to set up a system of pilot NTS performance feedback (e.g. comparable to the NASA/UT system), so that the NOTECHS NTS training and evaluation system could be adapted to changing operational procedures and expanding knowledge of the subject.

2.8References Antersijn, P. & Verhoef, M. (1994). Assessment of non-technical skills: is it possible? In N. McDonald, N. Johnston & R. Fuller (Eds.), Applications of Psychology to the Aviation System.: Avebury, Aldershot.

Avermaete, J.A.G. van & Blanken, M.J. van (1997). NOTECHS: The Evaluation of Non-Technical Skills of Multi-Pilot Aircrew, Project Plan (Issue 1.0). NLR Contract Report CR 97037 L. National Aerospace Laboratory NLR, Amsterdam.

Flin, R. & Martin, L. (1997, in press). Behavioural Markers for CRM. In: Proceedings of the Ninth International Symposium on Aviation Psychology. Ohio State University, Columbus, Ohio.

Hawkins, F.H. (1984). Human factors in flight. Gower, Aldershot.

Helmreich, R.L., Butler, R.E., Taggart, W.R. & Wilhelm, J.A. (1995). The NASA/University of Texas/FAA Line/LOS Checklist: A behavioral marker-based checklist for CRM skills assessment. Instructions for using the LLCV4. Technical Paper 94-02 (Revised 12/8/95). Aerospace Crew Research Project, Austin, Texas.

Rijksluchtvaartdienst RLD, Examencommissie voor Beroepsvliegbewijzen (1997). Handboek Examinator deel III: Standaardisatie Niet Technische Aspecten (NTA). Rijksluchtvaartdienst, Hoofddorp. Dutch Directorate General of Civil Aviation RLD, Examination Commission for

Commercial Pilot’s Licences (1997). Examiner’s Handbook part III: Standardisation Non Technical Aspects (NTA). Dutch Directorate General of Civil Aviation RLD, Hoofddorp, the Netherlands.

Taggart, W.R. (1995). The Aerospace Crew Research Project Line/LOS Checklist: Assessing System Safety and Crew Performance. In: Proceedings of the Eighth International Symposium on Aviation Psychology. Ohio State University, Columbus, Ohio.

Wilhelm, J.A. & Helmreich, R.L. (1996). The Line/LOS Checklist for Check Pilots: A short form for evaluation of crew human factors skills in line flight settings. Technical Report 96-6, Aerospace Crew Research Project. University of Texas, Austin, Texas.

PART C.Workpackage 3 : Components of Non-Technical Skills

edited by R. Flin, L. Martin (Aberdeen), K.M. Goeters and H. Hörmann (DLR)

3Workpackage 3 : Components of Non-technical skills

3.1MethodThe method for the development of the NOTECHS system consisted of three interleaved phases of work. (i) Review of existing systems to evaluate proficiency in non-technical skills (Workpackage 2, plus additional material from Aberdeen University CAA project on Behavioural Markers for CRM, Flin & Martin, 1997), (ii) Literature search for relevant research findings relating to key categories of non technical skills identified in (i) and discussed in (iii), (iii) Extended discussions at NOTECHS meetings (April; June) and additional meeting of Workpackage 3 group (August). The review of existing systems is presented in the report of Workpackage 2 and the review of the background literature is given below.

As stated in the conclusion of Workpackage 2, it was decided that none of the existing systems could be adopted in their original form, nor did any single system provide a suitable basis for simple amendment, although particular attention was paid to two of the principal frameworks, namely the KLM WILSC/SHAPE systems and the NASA UT system (Helmreich et al, 1996). Therefore it was agreed that an original system should be devised in Workpackage 3.

Following the two discussion meetings with whole group, the work-package leaders had a two day meeting to finalise the draft NOTECHS system, which is shown below in Table 3 and figures 2 to 4 with further details provided on the development of each category.

3.1.1Structure of the NOTECHS systemThe four primary categories effectively subdivide into two social skills categories (Co-operation; Leadership and Management) and two cognitive skills (Situation Awareness; Decision Making); it was judged to be unnecessary to add another level to the system by introducing this distinction explicitly. It should be noted that for the social skills, behaviours are generally emitted in the form of communication (verbal and non-verbal) which can be directly observed. The cognitive skills are only partially observable since much happens ‘in the head of the pilot’. Hence for evaluation purposes, these cognitive processes must be inferred from observable behaviours, (e.g. actions or communication - thus when a pilot states a decision, the observer can infer that some option selection has taken place; discussion of alternative divert airports reveals that option generation and comparison processes are being employed.) Not all non-technical rating systems include cognitive skills explicitly, due to the indirect methods which must be used for their evaluation. Nevertheless, a basic tenet of CRM is that pilots should communicate in a manner which reveals their mental models and thinking processes to the other crew members. Thus it seemed appropriate to evaluate these critical cognitive skills which have been shown to contribute to flight safety, and which are taught to pilots as fundamental components of CRM.

The category ‘Communication’ is featured in a number of systems but is not included here as a separate category. This is because communication skills are inherent in all four categories and the listed behaviours all involve communication.

A category of ‘Personal Awareness’ skills (eg. stress, fatigue) was considered but rejected due to

difficulties in observing, or inferring except in the most extreme cases.

In relation to the four categories, a number of derived elements were examined and for each element a series of indicative behaviours were identified. At the elemental level, this set was compared against the KLM SHAPE system and the NASA UT system to confirm that essential elements had been encompassed. In the final version 3 to 4 elements for each of the four categories were selected, giving a total of 15 elements. For each element a number of exemplar behaviours were included. The exemplar behaviours were phrased as generic (eg. closes loop for communications), rather than specific (eg. reads back to ATC), to give an indication of type, and to avoid specifying particular behaviours which should be observed. This should also ensure that the system is as widely applicable as possible. Accompanying guidance notes to the finished system will provide further details.

3.1.2Design principlesIn addition to those criteria stipulated in Workpackage 1, the following principles were used to guide the final choice of components and descriptor terms.

The basic categories and elements should be formulated with the maximum mutual exclusivity. This is only achievable to a certain degree, given the interdependence of the various non-technical skills in flight deck operations.

A rule of parsimony was applied, in that the system should contain the minimum number of categories and elements in order to encompass the critical behaviours.

The terminology used should reflect everyday language for behaviour, rather than psychological jargon.

The skills listed at the behaviour level should be directly observable in the case of social skills or could be inferred from communication, in the case of the cognitive skills.

Table 3 Non technical Skills (NOTECHS) Framework: Categories and Elements

CATEGORIES ELEMENTS

Team building and maintainingCo-operation Considering others

Supporting othersConflict solving

Use of authority/assertivenessLeadership and Maintaining standardsManagerial skills Planning and co-ordinating

Workload management

System awarenessSituation awareness Environmental awareness

Anticipation

Problem definition/diagnosisDecision Option generationMaking Risk assessment/option choice

Outcome review

Figure 1 Non-Technical Skills (NOTECHS) Framework: Elements and Behaviours for Category - Co-operation

Figure 2 Non-Technical Skills (NOTECHS) Framework: Elements and Behaviours for Category - Leadership and Managerial Skills

Figure 3 Non-Technical Skills (NOTECHS) Framework: Elements and Behaviours for Category - Situation Awareness

Figure 4 Non-Technical Skills (NOTECHS) Framework: Elements and Behaviours for Category - Decision Making

3.2Co-operation

Klaus-Martin Goeters

3.2.1Definition

Co-operation is the ability to work effectively in a team/crew.

Co-operation requires team building and maintaining, so that co-operative actions are based on a mutual agreement of team-mates/crewmembers in a positive group climate. Such a climate is also obtained by factors like consideration/ support of other team-mates/ crewmembers and conflict solving skills. Co-operation deals with the question of how people function as a working group. It does not refer to the work itself, such as the quality/quantity of output.

Good co-operation is largely dependent on an active and open communication between team-mates/crewmembers. However, communication is not a basic element of co-operation alone. According to the concept of the NOTECHS group communication is a general mediator of all four non-technical skills categories, but some aspects of communication may fall more into one or the other category. Thus these particular aspects become secondary descriptors of the categories. All those aspects of communication belong to co-operation which deal with building a climate for open communication, sensitivity for different aspects of messages (e.g. also the emotional component) and the awareness of the difference of verbal vs. non-verbal communication.

The two interpersonal categories of the NOTECHS system, namely Co-operation and Leadership/ Managerial Skills are interrelated to a degree, since both refer to group processes. The solution for reaching a clear distinction is that in the NOTECHS system co-operation involves team building and maintaining which is in some other systems subsumed in leadership/ managerial skills. On the other hand the aspects of co-ordination and responsibility although often discussed as parts of co-operation became elements of leadership/ managerial skills in the NOTECHS terminology. The conceptual difference is that Co-operation is concerned with mutual assistance and team atmosphere during work, while Leadership/ Managerial Skills covers all aspects of initiative and co-ordination and goal setting.

3.2.2Elements of co-operationThe summary definition of co-operation comprises different elements:I. Team building and maintaining: Establishing positive interpersonal relations and active

participation of the team-mates/crewmembers in fulfilling the tasksII. Consideration of others: Acceptance of others and understanding their personal conditionIII Support of others: Giving help to other team-mates/crewmembers in case they need assistanceIV. Solving interpersonal conflicts: Articulation of different interpersonal positions with suggestions

for solutionsConsideration and support of others may be grouped together as one element since in practice both aspects are very closely interrelated, but for the sake of clear concepts both elements will be separately described.

Communication as a mediator element of co-operation: Building a climate for open communication and the awareness of the emotional parts of messages and of non-verbal communication. These aspects of communication largely fall into Element I, but partially also under the other elements.

3.2.3Concepts of co-operationCo-operation as a professional or scientific term is not a unique, clearly defined concept. Depending on the frame of reference it can mean different things. Often it is only indicating a field of application or interest like in the following examples: “training of co-operation” or “ research on co-operation”. Under such conditions it is usually not necessary to have a stringent definition for co-operation. A loose collection of describing terms may be sufficient. When one looks into source material of this kind one will find a set of keywords as in the following references:- Working in groups for problem-solving (Guirdham, 1990): Co-operative rather than competitive;

role behaviour; norms and conformity; handling conflicts; openness; not defending weak positions; considering new ideas on their merits, not on the basis of who originated them; trying to improve working relationships.

- Co-operation and competition (Baron & Paulus 1991): Reciprocity and co-operation: reacting on the behaviour of others, communication.

- In the WILSC categories of the KLM Feedback and Appraisal System co-operation is decomposed into two aspects:

C1. Working with others- initiating and maintaining an active role in the team.- being composed in reactions towards others.- getting along with different types of people.- being open to the suggestions of others.- participating in consultations and decision-making.- being loyal to other members of the team, backing people up.

C2. Attentiveness- recognising signals from others.- listening effectively.- being aware of the effect of one's own behaviour on others.- taking other's feeling into consideration.- showing respect for others.

- The Lufthansa CRM Quick Reference lists good CRM practice (=behaviour markers) in 4 categories: Management, Communication, Teamwork and Judgement & Decision Making.

The aspect Teamwork highly corresponds with the NOTECHS term Co-operation. It is defined by the following items:

- Trying to give a positive first impression - Taking care of human redundancy- Bringing in knowledge in time- Supporting others- Acceptance of critique and suggestions

- In the ICAO Human Factors Digest Nr. 2 (1989) six major areas are mentioned which should be taught in human factors training of flight crews. The first area in the list is Communication/Interpersonal skills" which covers the following aspects:

1. cultural influence2. barriers, e.g. rank, age, crew position3. polite assertiveness4. participation5. listening6. feedback7. legitimate avenues of dissent

From the presented information it can be seen that several sources combine co-operation and communication more or less completely. The reason for this is that well-known scientific models of communication are concerned a lot with interpersonal aspects. Two of them which enable deeper insights into on-going communication processes are to be mentioned:- A model based on information theory by Schulz v. Thun (1981). It pinpoints the 4 components of each message, namely the factual content, the information regarding the relation between sender/receiver, the disclosure of the sender and the appeal to the receiver.

- Transaction analysis TA: (see Berne, 1967). Guirdham (1990) writes: " An outline of the personality theory built into TA is needed before the communications aspects can be discussed. The fundamental proposition of this theory is that each adult personality is composed of, and at any time may be operating óut of` one of three égo states`. Theses three ego states are known, in terminology deliberately chosen by Berne to be easily understood by non-psychologists, as ´Parent`, Ćhild` and Ádult`". A CM may fall into the behaviour pattern of one of these roles due to the ongoing interaction. Certain transactions provoke the corresponding reactions. Example: If the captain "sends a message" to the F/O out of his parents-ego (not from the more rational adult-ego) like "You could show more engagement!" this will result in a reaction of the F/O dependent on his behavioural disposition in his childhood ( e.g. more adaptive "Next time I try my best!" or more rebellious "Do it yourself!").

These models make aware that the on-going communication contains besides the pure factual content of messages always indications for the interpersonal relations between sender and receiver. The later aspect falls still under co-operation in the NOTECHS system, but not the first one. As it is mentioned in the definition of co-operation (see above) the NOTECHS group disentangled Co-operation and Communication to a certain degree and follows the concept that communication is a general mediator which is a vehicle for all four NOTECHS categories, but uses different facets in different areas.

If one has to think about scales and assessment tools for the evaluation of observed co-operation, one needs a more precise definition of this category than just a list of items. It must then be clearly stated which factors are included and which are not. Particularly, the boundaries and differences with other categories in the total assessment system have to be described. Typically the NOTECHS definition of co-operation is the result of a discussion regarding a scale system which should distinctly and parsimoniously cover the non-technical skills components of cockpit work. This assessment system includes two aspects of interpersonal skills (i.e. Co-operation and Leadership/ Managerial Skill) and of cognitive skills (i.e. Situational Awareness and Decision Making). The chosen definition of co-operation is such that an overlap with the other categories is avoided.

3.2.4Debates over definition of co-operationThe definition of co-operation is the result of a thorough discussion in the NOTECHS group. The definition went at least through three major metamorphoses. The starting point was to try a synthesis of a variety of definitions/descriptions coming from different sources (alphabetical order): Baron and Paulus (1991); DLR AC for pilot applicants 1995; Eißfeldt, Goeters, Hörmann, Maschke und Schiewe (1994); Guirdham (1990); ICAO Circular 216 and 217(1989); JAR-FCL 3 (Flight Crew Medical Requirements, Manual p. 176/177, 1996); JAR-FCL 1, Subpart F (AMC FCL 1.261 (d): Multi-crew co-operation course); KLM Internal Papers: Documentation of the Feedback and Appraisal System (1996); Lufthansa Internal Papers: Documentation of the CRM Quick Reference (1997); NASA/UT/FAA Line/LOS checklist (1996).

Although a lot of literature is already available, a clear model by which the relevant set of non-technical skills can be distinctly ordered and described is missing. Partially, this situation results from the fact that the different non-technical skills show significant degrees of overlapping and relevant factors could be aggregated in different ways. The NOTECHS solution is not just another approach to this problem, but claims to be distinct and exhaustive and simultaneously offering a parsimonious scale system, which is easily applicable in check situations.

Originally, the NOTECHS group assumed a close link between co-operation and communication and thought of communication as an element of co-operation. One recognised that there are also interrelations to the other main factors: Particularly, good leadership needs a good co-operation of crewmembers and an effective decision making is based on the exchange of relevant information. Even situational awareness requires the flow of necessary information. Following these ideas it became clear that communication is a central mediator for all relevant non-technical skills and thus could not be solely subsumed in one category. The solution adopted was to split it up into different elements which can be integrated in the fourfold categorical system (e.g. establishing an atmosphere of open communication is a typical behaviour which should be observed in the category "co-operation"). In general industrial psychology co-operation is often contrasted to competition. Both aspects are seen as “natural” counterparts in normal business. Due to safety reasons the cockpit work should be free of competition between crewmembers. They should not work against each other. Mutual support is of utmost importance. This is also seen by the airlines which often contract pilots based on the principle of seniority, thus avoiding the competition between the cockpit crew.

What makes the distinction between co-operation and leadership? Although there is some overlap in basic elements there exist some clear differences: While co-operation is possible between equal partners, leadership is often based on given or incidental differences in the positional level or the degree of responsibility. Comparisons between Captain vs. First Officer or PF vs. PNF are well-known distinctions in the cockpit crew. Key words of leadership not to be found in the definition of co-operation are: management of hard and life ware, taking initiative, planning and allocation of individual tasks.

3.2.5Research findingsAccident research emphasises the importance of proper co-operation in the cockpit. Thus, exercises in teamwork have been established in all programs of Crew Resource Management (CRM) training. An analysis by Sears (1995) listed the causal factors in 93 well documented flight accidents with casualties or hull loss in commercial jet operation from 1977 to 1984. For each causal factors the percentage of cases was determined taking all 93 accidents as 100%. The following causal factors indicate deficits in relevant non-technical skills, probably also including a lack of effective co-operation: 33% Pilot deviated from basic operational procedures26% Inadequate cross-check by second crewmember10% Captain ignored crew inputs

3.2.6Consequences for measurementSince co-operation is referring to the interaction of crewmembers fulfilling their task it can be expected that this aspect is to be observed and assessed with sufficient reliability. Most of the actions of the crewmembers are visible and often based on the ongoing communication which can be followed (and if necessary even recorded). It is important to let the crew work under typical circumstances. Line Oriented Check (LOC) scenarios seem to be the best frame for the observation and rating of crew co-operation. The checker needs to be aware what has to be assessed in co-operation. Therefore he needs a list of behavioural markers by which the concept is defined.

List of potential markers for co-operation (as collected from available documents; markers differ in degree of relevance)

The markers for the different aspects of co-operation are:

Teambuilding and maintaining- Establishes atmosphere for open communication and participation.- Encourages inputs and feedback from others (lowers the barriers).- Does not compete with others.(- Shows verbal and non-verbal expressivity.)(- Eye contact with others.)

Consideration of others- Takes notice of the suggestions of other CM - even if s/he does not agree.- Takes condition of other CM into account.- Gives personal feedback.(- Expresses personal acceptance of other CM.)(- Mentions strong and weak points of other CM tactfully.)

Support of others- Helps other crew member in demanding situation- Offers assistance

Conflict solving- Keeps calm in conflicts.- Suggests conflict solutions.- Concentrates on what is right rather than who is right.

From this list those behaviour markers are chosen, which are generic enough and which also have a chance to be observed in check events of aircrews. Note that the markers listed above between brackets are not part of the NOTECHS system. (page 52) visualises the composition of co-operation by its most relevant elements and behaviours. The following table gives examples of good and poor co-operation.

Table 4 Examples of good and poor co-operation

Element Good practice Poor practice

Team building and maintaining

Establishes atmosphere for open communication

Blocks open communication

Encourages inputs and feedback from others

Keeps barriers between CM

Does not compete with others Competes with others

Consideration of others

Takes notice of the suggestions of other CM even if s/he does not agree

Ignores suggestions of other CM

Takes condition of other CM into account

Does not take account of the condition of other CM

Gives personal feedback Shows no reaction to other CM

Support of others Helps other CM in demanding situations

Hesitates to help other CM in demanding situations

Offers assistance Does not offer assistance

Conflict solving Keeps calm in interpersonal conflicts

Overreacts in interpersonal conflicts

Suggests conflict solutions Sticks to own position without considering a compromise

Concentrates on what is right rather than who is wrong

Accuses other CM of making errors

3.2.7References - Co-operation

Baron, A. & Paulus, B. (1991). Understanding Human Relations. Boston: Allyn and Bacon.

Berne, E. (1967). Spiegel der Erwachsenen. Hamburg: Rowohlt. [The Games People Play].

Eißfeldt, H., Goeters, K.-M., Hörmann, H.-J., Maschke, P., Schiewe, A. (1994). Effektives Arbeiten

im Team: Crew-Resource-Management-Training für Piloten und Fluglotsen. DLR - Mitteilung 94-09.

Guirdham, M. (1990). Interpersonal Skills at Work. New York: Prentice Hall.

ICAO (1989). Human Factors Digest No. 2: Flight crew training: Cockpit resource management (CRM) and line-oriented flight training (LOFT). ICAO Circular, 217.

KLM (1996). Feedback and Appraisal System: KLM Internal Paper.

Lufthansa Flight Training - Human Factors Department (1997). The CRM Quick Reference: Lufthansa Internal Document.

Schulz v. Thun, F. (1981). Miteinander Reden: Störungen und Klärungen. Reinbek: Rowohlt.

Sears , R.L. (1995). A new look at accident contributors and the implications of operational and training procedures. Proceedings of the 38th International Flight Safety Foundation Symposium, Boston.

3.3Leadership and managerial skills

Hans-Jürgen Hörmann

3.3.1DefinitionEffective leadership and managerial skills mean to achieve the joint task completion within a motivated, fully functioning team through co-ordination and persuasiveness.

The core of effective leadership is to set highest priority on the joint completion of a given task. Leadership responsibilities include the active and goal-directed co-ordination of the working activities within the crew. This is always a reciprocal process. Without complementary behaviour of the crew, leadership behaviour is less effective. All crewmembers are expected to dedicate their efforts and initiative to the safe and efficient achievement of the flight goals. However the final and legal responsibility for the operation on the whole is undivided, with the pilot in command (PIC). Crew responsibilities include monitoring and challenging each other whenever differences in concepts or actions are perceived. Within the scope of delegated tasks crewmembers have the same responsibilities as the PIC.

ICAO has defined a leader as “a person whose ideas and actions influence the thought and the behaviour of others. Through the use of example and persuasion, and an understanding of the goals and desires of the group, the leader becomes a means of change and influence” (ICAO, Human Factors Digest, 1, 1989a). It is important to distinguish between leadership, which is acquired, and authority, which is assigned. Leadership is one aspect of teamwork, and the success of a leader depends on the quality of his/her relationship in the team. The crewmembers should feel that they are an integral part of a well-run, well-organised operation in which their inputs are essential to reach commonly valued goals and overall success of the operation.

In distinction to the category co-operation, leadership and managerial skills focus more on the goal-directed initiative the crewmember under consideration is investing into management and co-ordination functions. This includes also positive influences on the motivation and morale of the crew. Whereas co-operation covers more the interactive processes without explicit appointed roles and independently of authority differences of the individuals.

The leader has a clear concept for the operation and provides general standards and directions for the completion of the different tasks. The tasks are allocated according to defined roles, specific experience, as well as to the present level of workload of the crewmembers. This concept for the operation is interactively briefed and always open for the contributions from other crewmembers. In order to ensure proper support and the participation from all parts of the crew, active care is taken to establish and maintain closed loop communication. A leader motivates, activates, and monitors others and encourages the crew to monitor and challenge her/himself and each other in a non-degrading way.

3.3.2Controversies about definitionThere is a debate among CRM experts whether to define leadership and followership skills directly in relation to the role of the captain (leader) and other crewmembers (followers) or whether these skills must be conceived as two sides of the same coin. Whereas it is clearly recognised that a specific responsibility is connected to the command role, and that the captain is the leader of the entire crew, each crewmember has a leadership responsibility that is important for example to effective decision-making. Depending on the allocation and delegation of tasks first officers or flight engineers can have equal opportunity for leadership behaviour in their position. Acquired leadership skills can enhance any position regardless of the assigned authority as every crewmember has to subscribe to their duties and functions.

As part of an FAA project, Pettitt and Dunlap (1997) state that it is the implementation of the skills by leaders and followers that is different, not the skills themselves. For example, the manner in which a captain influences a first officer is considerably different than the manner a first officer chooses to influence a captain -- yet both influence. Other authors like Guy Smith from Northwest Airlines have the feeling that the notion of followership skills are a neglected subject (see Industry CRM Developers forum in the Internet, 5/97). In a case example Smith attributes the behaviour of a first officer who acts in non-routine events (e.g. request for direct vectors, unscheduled PA announcements) without consulting with the captain, to a lack of followership skills. However, communicating one’s intention is one of the core elements of effective teamwork. How can one expect to be backed up by someone else who does not know the intentions of the actions?

Robert Jenney (Industry CRM Developers forum in the Internet, 5/97) from the company Aviation Safety Connection made a plea for omitting the term followership in describing teamwork within a flightcrew, since followership implies “subservience” (rather than service) and submission (rather than loyalty). It is based on an authoritarian rule, and the transition from pure doers to critical, active thinkers becomes difficult.

Instead of combining leadership and followership a different concept is suggested in some of the recent US CRM trainings: Leadership with participation / Assertiveness with respect. Each of the functions have less effect without their complement. Leadership does not work without participation

of the crew. Participation without leadership can create chaos and confusion. Similarly assertiveness without respect can have the same negative impact on crew effectiveness as respect without assertiveness.

Based on these assumptions it was concluded by NOTECHS to leave out the term “followership” and to suggest one definition for leadership and managerial skills that is applicable independently of the crew position.

3.3.3Concepts in the related literatureIn the academic literature there are discussions about the question whether effective leadership can be defined independently from the characteristics of the situation and of the crewmembers. Some approaches focus on the personality of a leader. For example Fiedler’s (1978) Contingency Model distinguishes between task- and person-orientated leadership style which is basically anchored in the leader’s personality. However, this approach does not leave much room for training and is not suitable for behaviour assessment within the framework of NOTECHS.

Blake and Mouton (1978) postulate in their famous Grid-model two independent dimensions of leadership styles: task-orientated and people-orientated. However, this model is very global. It does neither take into account characteristics of the crew nor of the situation. A leadership style with maximum task- and maximum people-orientation should always be optimal. Trollip and Jensen (1991) have added some helpful advice on how to adjust the basic styles according to the tasks and the crewmembers.

Hersey and Blanchard’s (1977) model of situational leadership adds as a third dimension different maturity levels of the crew varying from low motivation, knowledge, abilities to high motivation, knowledge, abilities. Though this seems to be a valuable additional variable for leadership styles, the disadvantages of these models are that they are too general for a description of the group dynamics that determine the effectivity of leadership behaviour. They only focus on the style of the leader, leaving the crew more or less as passive objects.

For CRM training a situational approach to leadership behaviour is indicated. Situational leadership recognises the need for leaders to adapt to the desires, attributes, and readiness (both job readiness and psychological readiness) of the other team members to whom the team leader is linked. Ginnett (1993) describes crew behaviour as a shell in which the captain as a leader breathes life. The power of the shell can be enhanced or diminished depending on the quality of leadership behaviour. Hackman (1993) emphasises this team-as-a-whole issue in stating that understanding the behaviour and performance of cockpit crews requires careful attention of the dynamics in the whole crew, not just the behaviours of the individual team members. He considers three facts about cockpit crews: First, they are teams, second the captain is the team leader, and third cockpit crews are richly entwined with the organisational, technological and regulators contexts in which they operate.

However, when assessing individual behaviour during check-rides, NOTECHS should consider these environmental determinants of the assessed skills. NOTECHS is assuming that leadership and managerial skills can be acquired during training and that individual behaviour can be evaluated by taking the social and environmental context into account.

3.3.4Elements in existing systemsLeadership is closely interconnected with areas like communication, co-operation, decision making, and workload management. It was decided to combine leadership and workload management into one category and name it “Leadership and Managerial Skills”. In order to avoid further overlapping with other categories, the NOTECHS group attempted to identify typical core elements for each category. The following list compiles elements that are connected to leadership and managerial skills in existing systems.

ICAO:ICAO (1989b) lists different elements in the area of leadership/”followership” without further definitions

- team building activities- managerial and supervisory skills: plan, organise, direct, control- authority- assertiveness- barriers- cultural influence- roles- professionalism- credibility- team responsibility- time/workload management

Other approaches separate the tasks for captains and other crew members: Balanced authority gradient (captain), assistance and assertiveness (other crew).

KLM:KLM carefully defines

- task-oriented leadership as planning and organising the work of the crew byo dividing tasks, setting prioritieso checking, correcting as requiredo using authority if required

- people oriented leadership as stimulating and motivating the crew byo delegating responsibilitieso maintaining open and communication and mutual understandingo motivating, showing appreciation, and giving support

and

- stress management as recognising and dealing with stress and the ability to recover byo remaining open to new information and suggestions of otherso keeping presence of mindo not becoming rigid under stress

This understanding of stress management by KLM comes closer to what NOTECHS refers to as situation awareness.

NASA/UT Line/LOS Checklist:The NASA/UT/FAA Line/LOS checklist (1996) does not explicitly contain the term leadership. Instead it is subsumed under the category “team management & crew communication”, including:

- co-ordination of flight deck activities and crew member participation- team concept and open communication established- effective advocacy and assertion- proper balance between crew member participation and command authority

Workload management is subsumed under situational awareness & decision making.

Team building and maintenance, setting the tone for the crew co-operation seem to be one of the main functions of a leader according to the NASA/UT and KLM scales. However, it was decided by NOTECHS to access the corresponding behaviour under the category co-operation.

3.3.5Consequences for measurement within the NOTECHS systemAs a complementary process, leadership and managerial skills should be measurable with the one set of markers. The skills themselves should be the same for leaders and followers. The way they are implemented in behaviour may differ according to position.

According to the concept of the four NOTECHS categories it was decided to extract four major elements for leadership and managerial skills from the existing systems:

Use of authority and assertivenessCreate a proper challenge and response atmosphere. The given command authority of the PIC should be adequately balanced to assertiveness and crew member participation. If situation requires, decisive actions are expected.

Providing and maintaining standardsThe compliance with essential standards (SOPs and others) for the task completion should be ensured. Therefore the crew should mutually supervise and intervene in case of deviations from standards. If situation requires, non-standard procedures might be necessary to apply. Such deviations shall be announced and consulted in the crew.

Planning and co-ordinationIn order to achieve top performance and to prevent workload peaks or dips an appropriate concept for organised task sharing and delegation has to be established. Plans and intentions have to be communicated so that the whole crew understands the goals and that the activities conducted by the crew do reflect proper co-ordination.

Workload managementClear prioritisation of primary and secondary operational tasks. Based on a sound planning, tasks are distributed appropriately among the crew. Signs of stress and fatigue are communicated and taken into account as performance affecting factors. Available external and internal resources (including automation) are used to accomplish task completion in time.

3.3.6List of selected elements and behaviour examples

Use of authority and assertivenessProper balance between command authority and crew member participation Proper authority/assertiveness balanceUses authority if necessaryMonitors and challenges others whenever differences in concepts occurAlways open for and interested in suggestions from crewReflects on suggestions of othersConvinces others with arguments instead of pure power (i.e. stripes)Consults crew members before decidingActs decisively if situation requires Motivates other crewmembers

Providing and maintaining standardsDemonstrates will to achieve top performance Notices criticality of sticking to the SOPs Intervenes wherever necessary, if norms and boundaries are not metWith crew being consulted deviates from standards if situation requires

Planning and co-ordinationEncourages crew participation in planning and task completionIntentions and goals are clearly stated and confirmedCo-ordinates crew activities on ground Delegates tasks and responsibilities

Workload managementUses all available (external and internal) resourcesRealises that stress and fatigue are affecting performance Takes care that crew climate is adequate to present level of workload Makes appropriate use of automation in order to avoid workload peaks or dips Recognises demands posed by automation, disengaged as necessaryManages task accomplishment within available time

From this list of behaviour examples thirteen behaviours (see figure 3, page 53) were selected as part of the NOTECHS system. Next, for each selected behaviour examples of good (desired) and poor (undesired) practice are given.

Table 5 Examples of good and poor leadership and management


Use of authority and assertiveness

takes initiative to ensure crew involvement and task completion

hinders or withholds crew involvement

takes command if situation requires, advocates own position

passive, does not show initiative for decisions, own position not recognisable

reflects on suggestions of others ignores suggestions of others

motivates crew by appreciation and coaches when necessary

does not show appreciation for the crew, coaches very little or too much

Providing and maintaining standards

subscribes to SOPs, makes sure SOP compliance in crew

does not comply to SOPs, does not monitor crew for SOP compliance

intervenes if task completion deviates from standards

does not intervene in case of deviations

with crew being consulted, deviates from standards if necessary

deviation from standards are neither announced nor consulted

demonstrates will to achieve top performance

does not care for performance effectivity

Planning and co-ordination

encourages crew participation in planning and task completion

plans only for him/herself, crew not involved

plan is clearly stated and confirmed

intentions not stated or confirmed

with crew being consulted, changes plan if necessary

changes plan without informing crew or follows plan blindly

clearly states goals and boundaries for task completion

goals and boundaries remain unclear

Workload management distributes tasks among the crew, checks and corrects appropriately

flying “solo” without other crewmembers involved

secondary operational tasks are prioritised to retain sufficient resources for primary flight duties

secondary operational tasks interfere with primary flight duties

allots adequate time to complete tasks

workload is increased through inadequate planning

notifies signs of stress and fatigue ignores signs of stress and fatigue

3.3.7References - Leadership and managerial skills

Blake, R.R. & Mouton, J.S. (1978). The New Managerial Grid. Houston: Gulf.

Fiedler, F.E. (1978). The contingency model and the dynamics of the leadership process. In L. Berkowitz (Ed.) Advances in Experimental Social Psychology (Vol. 11). New York: Academic Press.

Ginnett, R.C. (1993). Crews as groups: Their formation and their leadership. In: E.L. Wiener, B.G. Kanki & R.L. Helmreich (Eds.) Cockpit Resource Management. San Diego: Academic Press.

Hackman, J.R. (1993). Teams, leaders, and organizations: New directions for crew-oriented flight training. In: E.L. Wiener, B.G. Kanki & R.L. Helmreich (Eds.) Cockpit Resource Management. San Diego: Academic Press.

Hersey, P. & Blanchard, K.H. (1977). Management of Organizational Behaviour: Utilizing Human Resources. Englewood Cliffs, NJ: Prentice-Hall.

ICAO (1989a). Human factors digest no. 1. Fundamental human factors concepts. Montreal, Canada: International Civil Aviation Organization.

ICAO (1989b). Human factors digest no. 2. Flight crew training: Cockpit resource management (CRM) and line-oriented flight training (LOFT. Montreal, Canada: International Civil Aviation Organization.

Kanki, B.G., Palmer, M.T. (1993). Communication and crew resource management. In: E.L. Wiener, B.G. Kanki & R.L. Helmreich (Eds.) Cockpit Resource Management. San Diego: Academic Press.

Pettit, M.A. & Dunlap, J.H. (1997). Understanding leadership/followership skills. Paper presented at the Ninth symposium on Aviation Psychology. Columbus Ohio: Ohio State University.

Trollip, S.R. & Jensen, R.S. (1991). Human Factors for General Aviation. Englewood: Jeppesen Sanderson.

3.4Situation awareness

Lynne Martin & Rhona Flin

3.4.1DefinitionSituation awareness is ‘one’s ability to accurately perceive what is in the cockpit and outside the aircraft’ (ICAO, 1989, p13)

‘the perception of the elements in the environment within a volume of time and space, the comprehension of their meaning, and the projection of their status in the near future’ or ‘knowing what is going on’ (Endsley, 1995a, p36).

The ICAO Circular (1989) lists the following areas which should be taught in CRM training of situation awareness:∑ total awareness of surrounding environment∑ reality vs. perception of reality∑ fixation∑ monitoring∑ incapacitation (partial/total, physical, psychological).

To give some general background on situation awareness, a selective survey of the literature was conducted. Themes which emerged underlined that there is no agreed definition of situation awareness and, indeed, that which constitutes situation awareness is disputed, with some researchers stressing the importance of attention allocation and others emphasising comprehension and projection of ideas into the future. The above definitions appear to reflect current thinking, Endsley being one of the leading researchers in this field. Shrestha, Prince, Baker and Salas (1995) suggest: “situation awareness is a dynamic, multifaceted construct that involves the maintenance and anticipation of critical task performance events. They must also have temporal awareness, anticipating future events based on knowledge of both the past and the present. It is crucial that individuals monitor the environment so that potential problems can be corrected before they escalate”

Pilots might define situation awareness as ‘being ahead of your aircraft’. This implies that the pilot is prepared and is in control, has command of both the aircraft and the current tasks. It is not only knowing something that is crucial to good situation awareness, it is being able to fit this information into the right place in a mental model of the situation, so that if it is important it triggers problem recognition. This definition stresses the ever-changing nature of the cockpit environment and the need for the crew to continually monitor and update their situation by collecting information from all the sources that are available to them.

3.4.2Debate over definitionGenerally there is a confusion of terms, which may be due to a lack of a common understanding, although others argue that despite different definitions, the underlying conceptualisations are basically the same (see Flach, 1995; Noble, 1993; Smith & Hancock, 1995). Whilst some researchers focus on cognitive aspects, others include a broader range of issues including training, experience, preconceptions and workload. Shrestha et al (1995) give ten definitions of situation awareness which have been set out over the last 17 years. They distinguish between situation awareness and situation assessment, where situation awareness is the first part of the decision process and situation assessment is the second, leading on to the decision (see Prince & Salas, 1997). In the literature, the terms situation awareness and situation assessment have been used interchangeably, the former is used for the NOTECHS system as it appears to reflect current usage and is not meant to imply any distinction from the latter.

One of the problems with defining situation awareness (Sarter & Woods, 1991) is that the relevance of any given situational information depends on context. Thus the particular information an individual has to pay attention to is not useful for generic models and a cognitive perspective has to be more general, i.e. focusing on information gathering, attention and filtering in changing environments.

They cite potential moderators of situation awareness as: cognitive and spatial abilities; communication; assertiveness; technological advances. Brannick et al (in Strestha et al, 1995) would support the importance of experience as they found instructor pilots had significantly ‘more’ situation awareness in a task than student pilots. Lipshitz and Shaul (1997) collected data through a sea combat simulation and compared expert to novice performance which showed that experts collected more information on the situation before making a decision, a finding supported by other work.

Endsley (1997) describes situation awareness as involving three levels - (i) perception of critical factors in the environment; (ii) understanding what these factors mean when integrated together with the person’s goals and (iii) understanding what will happen with the system in the near future. These are different processes that the individual has to perform on the same set of data in an active search for meaning.

In essence, the information derived from situation awareness (or situation assessment) is used to guide behaviour and predict future events. Inaccurate situation awareness can be due to a poor match between an individual’s ‘mental model’ of the situation and the environment (Shrestha et al., 1995). However, Sarter and Woods, (1991) make a clear distinction between situation awareness and mental models, claiming in essence that mental models are static whereas situation awareness is dynamic. As mentioned above, there is a confusion between the terminology in this area which sometimes makes researchers’ standpoints on situation awareness unclear.

3.4.3Measuring cognitive processesThe problem with cognitive processes like situation awareness is that they are not directly observable, the behaviours that indicate situation awareness are only reflections of the process or mental model the pilot is using. This means that it is only possible to ‘measure’ situation awareness by inference. Even methods like looking for eye scan patterns do not relate directly to situation awareness, because they do not reveal exactly which information the brain is processing. However certain behaviours such as confirming, cross-checking information, sharing information are indicators of pilots’ situation awareness (Salas, Prince, Baker & Shrestha, 1995).

Salas et al (1995) discuss the methods available for measuring situation awareness including self report, verbal protocols, explicit and implicit measures. There are several packaged techniques which have been developed, such as: SAGAT - situation awareness global assessment technique (Endsley, 1995b) or SART - situation awareness rating technique (Selcon & Taylor, 1990). This is an area of current research and each method has relative strengths and weaknesses in terms of reliability and validity. For example:

Observations: Sarter and Woods (1991) recommend embedding events into complex scenarios to elicit key situation assessment behaviours and processes. Observers can then document and rate these behaviours (see Waag & Bell, 1997). This is essentially the method used in NOTECHS and the issues of rating scale design and training to maximise inter-rater reliability have to be taken into account.

Ratings: Subjective measures of current situation awareness can be collected from self-report methods. These have the typical problems associated with self rating data.

Reports: Retrospective reports and those using memory probes can also be used. Retrospective accounts may not distinguish between what actually occurred for the subject during the event and what the subject inferred after the event. Memory probes overcome some of these problems.

Freeze: Endsley (1995b) reviews measurement techniques from physiological to personal ratings and discusses limitations. She suggests that the freeze technique is the best - this is where the scenario or task is stopped. During the freeze of the situation she uses SAGAT to assess participants’ levels of situation awareness. She concludes that the technique is sound, does not appear to interfere with situation awareness and that participant situation awareness lasts for about 5 minutes. She claims the SAGAT does not impede pilot decision making, because the relevant schema remain active during the pause and data collection period. However, Sarter and Woods (1991) argue these measures are intrusive and in fact may be most suitable for research purposes.

3.4.4Model/ TheoryNOTECHS decided to use the SHELL model (Edwards, 1972; Hawkins, 1985) as the theoretical underpinning for situation awareness due to its widespread adoption by CRM trainers (see CRM Developers Group http://www.caar.db.erau.edu/crm/)software - awareness of aircraft systems’ modes hardware - awareness of aircraft states such as fuelenvironment - awareness of the external situation including traffic and weatherliveware - the pilot(liveware - this is an awareness of the other team members and is covered in the leadership category)

The SHELL model only covers the awareness elements of situation awareness. It does not specify how the pilot integrates these to gain situation awareness, thus it only covers information inputs the pilot needs to build a sufficient situation awareness. Again the argument that we are focusing on the inputs and take no account of the processes of situation awareness could be raised but again the integration cannot be observed directly and thus has to be inferred from communication or other behaviours.

3.4.5FrameworkThe pilot needs to know about a number of things in the situation, from which NOTECHS identified a number of key items under three headings:

1. awareness of aircraft systems - systems (mode and state), energy states (e.g. fuel)2. awareness of environment - position, weather, air traffic, terrain3. awareness of time (anticipation)

For all of these factors the pilot needs to not only know their present state but also to be able to predict their future states, so anticipation applies to systems and environment.

3.4.6Behavioural examplesCommunication is the medium through which situation awareness can be assessed.

Behavioural examples of these elements were initially selected from the list of assessment items compiled for the UK CAA behavioural markers project (Flin & Martin, in press). These were amended for NOTECHS to comply with its aim of behaviour observability. Situation awareness, as a cognitive process, is not directly detectable in a working environment such as the cockpit where measurement has to be non-interventionist. The observation of situation awareness therefore has to be inferred through the interaction of the crew, using behavioural examples which indicate that the crew member has situation awareness to be interacting in this way. For each of the elements of the situation awareness category, examples of behaviour were selected to reflect the multiple processes of gathering and sharing appropriate information.

Table 6 Examples of good and poor situation awareness

Element good practice poor practice

System Awareness Monitors and reports changes in systems’ states

does not ask for updates

Acknowledges entries and changes to systems

does not signal awareness of changing systems

Environmental Awareness Collects information about environment (position, weather and traffic)

does not enquire about environmental changes

Shares information about the environment with others

does not comment on relevant environmental factors, or is surprised by them

Contacts outside resources when needed (to maintain situation awareness)

operates a ‘closed shop’

Anticipation Discusses contingency strategies

does not discuss relationship between past events and present/future

Identifies possible future problems

is surprised by outcomes of past events

3.4.7References - Situation awareness

Edwards, E.(1972) Man and machine. Systems for safety. BALPA Medical Symposium, London.

Endsley, M. (1995a) Toward a theory of situation awareness in dynamic systems. Human Factors, 37, 32-64.

Endsley, M. (1995b) Measurement of situation awareness. Human Factors, 37, 65-84.

Endsley, M. (1997) The role of situation awareness in naturalistic decision making. In C. Zsambok & G. Klein (Eds.) Naturalistic Decision Making. Mahwah, NJ: LEA.

Flach, J. (1995) Situation awareness.Human Factors, 37, 149-157.

Flin, R. & Martin, L. (in press) Behavioural Markers for CRM. CAA Report, London.

Hawkins, F. (1987) Human Factors in Flight. Aldershot: Gower.

ICAO (1989) Flight crew training: CRM & LOFT. Human Factors Digest, 2. Montreal.

Lipshitz, R. & Shaul, O. (1997) Schemata and mental models in recognition-primed decision making. In C. Zsambok & G. Klein (Eds.) Naturalistic Decision Making.

Noble, D. (1993) A model to support development of situation awareness aids. In G. Klein, J. et al (Eds.) Decision Making in Action. New Jersey: Ablex.

Prince, C. & Salas, E. (1997) The role of situation assessment in the conduct of flight and in decision making. In D. Harris (Ed.) Engineering Psychology and Cognitive Ergonomics (Vol 1). Aldershot:Ashgate.

Salas, E., Prince, C., Baker, D. & Shrestha, L. (1995) Situation awareness in team performance: Implications for measurement and training. Human Factors, 37, 123-136.

Sarter, N. & Woods, D. (1991) Situation awareness: A critical but ill-defined phenomenon. International Journal of Aviation Psychology, 1, 45-57.

Selcon, S. & Taylor, R. (1990) Evaluation of the situational awareness rating technique (SART). Farnborough: DERA.`

Shrestha, L., Prince, C., Baker, D. & Salas, E. (1995) Understanding situation awareness: Concepts, methods and training. In W. Rouse (Ed) Human/Technology Interaction in Complex Systems. Vol 7. Greenwich, CT: JAI.

Smith, K. & Hancock, P. (1995) Situation awareness is adaptive, externally directed consciousness. Human Factors 37, 137-148.

Waag, W. & Bell, H. (1997) Situation awareness and decision making in skilled fighter pilots. In C. Zsambok & G. Klein (Eds.) Naturalistic Decision Making. NJ: LEA.

3.5Decision making

Lynne Martin & Rhona Flin

3.5.1DefinitionDecision making is - the process of reaching a judgement or choosing an option.

The definition of ‘decision making’ is not generally disputed in the aviation literature although it may be labelled aeronautical decision making (Keampf & Klein, 1994) or pilot judgement (Jensen, 1996). Unlike situation awareness (see above), where the problem is choosing a definition from the multitude which have been suggested, the problem for decision making is finding one. This is because pilot decision making cannot be pinned down to any one cognitive strategy - different types of decisions are made at different times. Decision events differ in what they demand of the crew, what options and supports exist in standard procedures for making the decisions, and in task features that may make the situation difficult or error prone.

Psychologists studying decision making have demonstrated that there are different processes or strategies used by individuals to reach a decision, depending on the demands of the decision event. Orasanu (1993), a NASA research psychologist has studied the different styles of decision making used by pilots in different situations, “crew decision-making is not one thing. Crews make many different kinds of decisions, but all involve situation awareness, choice among alternatives, and assessment of risk. However, the decisions differ in the degree to which they call on different types of cognitive processes. The nature of the processes involved in a decision depends on the structure of the decision task and the conditions surrounding it.” (p138) She also points out, ‘ Crew decision making is managed decision-making. The captain has responsibility for the decisions but is supported by input from the crew, both in the cockpit and on the ground (air traffic control dispatch, maintenance).” (p139).

Within the NOTECHS system, decision making is narrower than aeronautical decision or pilot judgement as defined by Jensen (1996). His model encompasses decision making in its wider context, taking into account all the contributory factors of the individual (e.g. motivation) as well as the situation. Likewise the ICAO (1989) circular on CRM training has a very broad category labelled “Problem Solving/Decision Making/Judgement.” The focus was for NOTECHS for decision making was on the observable behaviours which indicate cognitive decision processes or outcomes.

3.5.2Types of decisionThere are two basic classes of decision making that pilots are called upon to engage in. The first are routine decisions, for which the airline will normally have written a procedure. In these cases the pilot just has to identify the situation and apply the appropriate actions. Many pilots do not consider this decision making. Key skills for effective decision making by procedures are that the crew follows the checklists systematically and complies with the rules the airline has specified.

Hence the ability to recognise a situation which requires the application of a procedure is covered under technical skills and compliance with procedures is included under managerial skills of the NOTECHS framework. A distinction was drawn between procedure compliance, which is covered both in the assessment of technical skills and in the managerial aspects of non-technical skills, and non-routine decision making.

The second broad class of decisions are non-routine, for which there are no complete written procedures. Here the pilot not only has to identify the situation, which may be difficult in itself, but also to assess the parameters and the risks and then select an option or course of action (CoA). These kinds of decisions are less frequent and, although they are not always emergency/ crisis decisions, are the types of decisions the crew will have to make under high pressure.

Orasanu and Fischer (1997) have labelled these classes as Well structured decisions - where the problem is understood and responses are available, and Ill defined problems the problem is not understood and it required diagnosis, there may be no response options readily available. They also distinguish between rule-based and knowledge-based decisions using Rasmussen’s (1983) taxonomy. The crucial aspect of the decision process for rule based decisions lies with accurate situation awareness - rapid recognition of the situation as a precondition for retrieving a matching rule from memory. High performing crews tend to make rule based decisions earlier (to give a greater safety margin) because they have attended to preceding cues (therefore have better situation awareness). They identified six main types of pilot decision making styles, shown below. There are two categories of rule-based decisions and three types of choice decisions. In choice problems, the emphasis is on awareness of the constraints and competing goals and evaluation of options in the light of these. The crew must retrieve options, identify constrains, determine if they have adequate information and evaluate the alternatives. Errors can arise in any one of these processes (Orasanu, Fischer & Tarrel, 1993). The most difficult and most infrequent type of decision is creative, where the pilots must work out a new solution to an unfamiliar problem.

Table 7

Rule/knowledge structurego/ no go rule - binary well-structuredcondition-action rule - if then well-structuredchoice knowledge well-structuredchoice - scheduling knowledge well-structuredchoice - procedural management

knowledge ill-structured

creative knowledge ill-structured

3.5.3Selected Research FindingsWithin aviation decision making research, characteristics of high performing teams have been compared with those from low performing teams (e.g. Fischer, Orasanu & Montalvo, 1993). Findings suggest that high performing teams more often talk about their concerns, consider the strengths and weaknesses of the team and look at the management of the workload that decisions generate. NOTECHS considered these interaction patterns when generating behavioural examples for the decision making category.Decision behaviours of effective crews are varied, there are not simple patterns. The most effective crews tailor their decision strategies to the situation. Thus the problems crews encounter, determine the decision strategy they use. In some cases, fast decisions buy time whereas in others fast decisions reflect a lack of consideration of the options (Orasanu, Fischer & Tarrel, 1993). Low performing crews were less able to distinguish the two cases compared to high performing crews. A trade off exists between making a default decision quickly with little effort and taking more time and effort to gather information to make a better decision. The key to seeking more information is an awareness of the adequacy of the default decision in the light of constraints, time and resources available (Orasanu et al, 1993). Time seems to be a major determinant of the decision strategy, along with judgements of risk and problem complexity (Fischer, Orasanu & Wich, 1995).

Research suggests that high performing crews verbalise concerns, gather more information and take longer to make knowledge based decisions. Because they have planned for tasks, they begin tasks earlier, complete them faster and therefore give themselves a cushion of time. Lower performing crews do not appear to distinguish among the various types of decisions, applying the same strategies in all cases regardless of variations in their demands (Orasanu & Fischer, 1997; Orasanu et al, 1993). One study showed that in high performance crews there was little talk at critical points. Communication was mainly to set a plan in place. In lower performance crews there was much communication, mainly in the form of direct orders to do specific things. This reflects shared problem models (Orasanu, 1993) in the high performing crews. Shared models are organising frameworks. By articulating goals plans and strategies, effective captains create a context for interpreting their commands, observations and information requests. The shared model enables other members to contribute. In essence, effective decision making is as a result of situation awareness, planfulness, metacognition, shared mental models - based on explicit communications - and efficient resource management. These do not constitute decision making but create a context within which effective decisions can be made.

Mosier (1991) found that most crews did not wait until they had a complete understanding of the situation to make and implement decisions. They made a recognitional, almost reflexive, judgement based on a few critical items of information and spent additional time and effort verifying its correctness through continued investigation. If later information changed the situation a second option was generated and implemented. Only more effective crews took steps to prevent misjudgement and fruitless problem solving by filling in extra checklists. Often decisions involved a sequence of monitoring -> preliminary decision -> monitoring -> final decision (Fischer et al. 1993). Less effective crews tended to trade one problem for another but more effective ones kept one problem running in the background whilst they focused on the other problem. They created a situation where they had time to consider several options others evaluated serially.

High performing crews’ captains have a better understanding of the strengths and weaknesses of the team and are able to use these to best advantage. From this they develop strategies which buy the team time and increase flexibility: they showed greater sensitivity to timing and better planning. The best strategies depend on crew size and hence the resources available. Generalised strategies of high performing crews are:1. monitored environment closely and appreciated the significance of cues that signalled a problem.2. used more information in making decisions and manipulated the situation if necessary to obtain

additional information3. they adapted their strategies to the requirements of the situation - a flexible repertoire4. they planned for contingencies and kept their options open when possible5. they did not overestimate their own capabilities or the resources available to them6. they appreciated the complexity of decision situations and managed their workload to cope with

them.

3.5.4Models of pilot decision makingTraditional models of decision making (such as Multi-Attribute Utility Theory) tend to be prescriptive, that is stating the steps to reach the ‘perfect decision’ (Arkes & Hammond, 1986). These rather complex frameworks which were originally intended for economic and business decisions have tended to be rejected by researchers studying naturalistic decision making in demanding real world domains, such as aviation, medicine or emergency management as well as in aviation (Flin et al 1997; Zsambok & Klein, 1997). Orasanu (1993) explains that whilst the traditional type of model is not bad, it is lacking for pilots in several ways - it is not sensitive to what experts actually do, does not exploit the knowledge and experience of the expert, in many cases would take too long to arrive at a solution, it is not sensitive to significant differences in problems situations and it focuses on an individual decision maker rather than on a crew. Naturalistic or descriptive models are more popular and several have been constructed for the aviation domain.

They typically include stages of situation assessment, action and review. Closing the loop of activity to check the action outcome against the plan is stressed and serves to emphasise that decision making is not distinct from situation awareness but is both preceded and succeeded by it. Decision making in the cockpit is not a unitary event but involves repeated cycles of situation assessment and reaction.

a. Orasanu’s model consists of two major components - situation assessment and choosing a course of action (CoA)(Orasanu & Fischer, 1997). Situation assessment requires definition of the problem and assessment of risk level and time available to make the decision. These factors determine which of the six decision strategies outlined above, the pilots are most likely to use.

b. FOR-DEC (DLR) is used as a framework for the training of judgement and decision making in CRM courses for airline pilots (Hormann, 1995). ‘FOR-DEC’ represents the 6 stages the pilot has to consider to make effective decisions.

Facts collect information, define situationOptions generate possible responsesRisks assess risks and benefits. Leads to >Decisions select an optionExecution carry out optionCheck look to see whether everything is OK

As a simple model it is easily remembered and can counteract certain cognitive mechanisms that can adversely affect the quality of decision making.

c. DECIDE model designed originally for fire fighters has been taken up by some aviation trainers e.g. the Canadian military (Stewart, 1995) and is referred to by Jensen (1996). It is almost the same as the FOR-DEC model:

Detect changeEstimate significance of changeChoose outcome objectivesIdentify plausible action optionsDo best optionEvaluate progress

d. Jensen (1996) argues a model of good judgement has 2 parts, an intellectual and a motivational part. There are 8 phases to judgement:

problem vigil - sensation awarenessrecognition - perceptiondiagnosis - cognitionalternative identification - recollectionrisk assessment - calculationbackground factor - motivationdecision making - managementaction - implementation

Essentially models a-d are the same - they advocate assessing the situation, acting and then reviewing these actions.

3.5.5FrameworkGenerally the FOR-DEC style model was used for the NOTECHS framework in combination with Orasanu’s model. This approach emphasises that decision making takes place within constraints, that the pilot will have to assess the risks, such as the resources available and their condition, in the decisions s/he makes. It is also consistent with our arguments about the framework. The execution of the decision will be the technical skill of flying the aircraft and so is covered in technical assessments. It is argued that which option to select is indicated as a result of the risk assessment and therefore these two stages have been combined to give the model of decision making in NOTECHS.following procedurestaking a decision in slow time, pilot has time to compare optionstaking a decision in emergencies where time is limited.

3.5.6ElementsThe elements which drop out of the NOTECHS combined model are from the FOR-DEC/Orasanu combination 1Problem definition/ diagnosis2Option generation3Risk assessment & option selection4Outcome review

3.5.7Behavioural ExamplesThe debate on the measurement of decision making follows the same lines as that for situation awareness. As decision making is a cognitive process, it is not possible to observe it directly and will have to be inferred from other pilot behaviours. Again, this will be through communication - the crew telling each other what point their thinking has reached. For each of the four stages of decision making some examples of behaviour have been selected as shown in the following table.

Table 8 Examples of good and poor decision making


Problem definition/ diagnosis

gathers information to identify problem

nature of problem not stated or failure to diagnose

reviews causal factors with other crew members

no discussion of probable causes

Option generation states alternative options does not search for information

asks crew members for options

does not ask crew for alternatives

Risk assessment considers and shares estimated risk of alternative options

inadequate discussion of limiting factors with crew

talks about possible risks for action in terms of crew limits

no consideration of limiting factors

Option selection Confirms and states selected option / agreed action

does not inform crew of decision path being taken

Outcome review Checks outcome against plan

Fails to check selected outcome against goal

3.5.8References - Decision making

Arkes, H. & Hammond, K. (1986) (Eds.) Judgment and Decision Making. Cambridge: Cambridge University Press.

Fischer, U., Orasanu, J. & Montalvo, M. (1993) Efficient decision strategies on the flight deck. In Proceedings of the Seventh Intenational Symposium on Aviation Psychology. Columbus: Ohio State University.

Fischer, U., Orasanu, J. & Wich, M. (1995) Expert pilots’ perceptions of problem situations. In Proceedings of the Eighth Intenational Symposium on Aviation Psychology. Columbus: Ohio State University.

Flin, R., Salas, E., Strub, M. & Martin, L. (1997) (Eds.) Decision Making Under Stress. Aldershot: Ashgate.

Hormann, J. (1995) FOR-DEC: A prescriptive model for aeronautical decision making. In R. Fuller, N. Johnston & N. McDonald, (Eds.) Human Factors In Aviation Operations. Aldershot: Avebury.

Jensen, R. (1996) Pilot Judgment. Aldershot: Avebury.

Keampf, G. & Klein, G. (1994) Areonautical decision making: The next generation. In N. Johnston, N. McDonald & R. Fuller (Eds.) Aviation Psychology in Practice. Aldershot: Avebury.

Mosier, K. (1991) Expert decision making strategies. In Proceedings of the Sixth Intenational Symposium on Aviation Psychology. Columbus: Ohio State University.

Orasanu J. & Fischer, U. (1997) Finding decisions in natural environments: The view from the cockpit. In C. Zsambok & G. Klein, (Eds.) Naturalistic Decision Making.

Orasanu, J. (1993) Decision making in the cockpit. In R. Helmreich, E. Weiner & B. Kanki, (Eds.) Cockpit Resource Management. New York: Academic Press.

Orasanu, J., Fischer, U. & Tarrel, R. (1993) A taxonomy of decision problems on the flight deck. In Proceedings of the Seventh Intenational Symposium on Aviation Psychology. Columbus: Ohio State University.

Rasmussen, J. (1983) Skill, rules and knowledge. IEEE Transactions on Systems, Man & Cybernetics, SMC 13, 3.

Stewart, J. (1995) Pilot decision making training: A Canadian application. In R. Fuller, N. Johnston & N. McDonald, (Eds.) Human Factors In Aviation Operations. Aldershot: Avebury.

Zsambok, C. & Klein, G. (1997) (Eds.) Naturalistic Decision Making. NJ: LEA.

PART D.Workpackage 4 : Using the NOTECHS evaluation system

edited by J. van Avermaete, E. Kruijsen (NLR), and C. Valot (IMASSA)

4Workpackage 4 : Using the NOTECHS evaluation system

4.1IntroductionIn this section the proposals for AMC and IEM are presented. These proposals are based on the general principles stated in part A, and reflect further discussions on the method of evaluation within the NOTECHS group as described in parts B and C. On some points one could come to a different choice or solution. Where appropriate an explanation on the choice made here will be given.

4.2Rationale behind the proposed AMC and IEMThe rationale for the use of the NOTECHS system is to rate the applicant on each of the four categories identified in part C of this report. The elements provide guidance on what each category consists of, and the behavioural markers or descriptors present the unacceptable behaviour which should lead to a negative marking on that particular category.

A choice was made to describe the behavioural markers using a negative formulation. Where a positive wording is more appropriate in a training setting (the training goal is to bring the trainee to the level of the positive descriptor), in a check situation the opposite situation applies. The only issue here, also from the perspective of the pilot, is whether he/she will pass or fail the check. Use of both a positive and a negative descriptor was considered, but not adopted. It is argued that this would leave the examiner with too much room for interpretation, and hence induce variability in decisions where the border between acceptable and unacceptable lies. Note however, that objective (quantitative) data are lacking to support this statement. Another option would be to describe each behavioural marker with a neutral statement (instead of the negative wording). Again, this was thought to lead to too much variability in ratings given by different instructors.

It is proposed that a negative rating on non-technical skill behaviour should not lead to a fail on the check when no negative technical outcome is observed. A negative technical outcome in this context has to be translated with potentially dangerous for the safety of the flight. The examiner should give a written explanation on each negative rating given, and indicate the technical group where the unacceptable non-technical behaviour has lead to problems. The reason to propose not to fail applicants on non-technical aspects alone is in the difficulty to judge non-technical behaviour without seeing it in the context of the situation. Preliminary results from KLM suggest a lot of variability in situation assessment, even when instructors from one company are shown exactly the same filmed LOC scenarios. Without empirical data on the amount of variability in NOTECHS ratings given by different examiners it is felt not to be acceptable to fail people on their non-technical skills only. If there is a lot of variability in examiner ratings this would enhance the idea of arbitrariness, which is felt to some extent by the pilot community. Without objective data to support the opposite, the concerns expressed by pilots should not be ignored. User acceptance should always be treated as an important aspect, but especially in the area of non-technical skill evaluation.

One might argue what the benefit of assessing non-technical skill proficiency is if negative technical outcomes are still required before unacceptable non-technical behaviour can have any consequences. Referring back to the general principles discussed in part A, it is argued that the assessment on non-technical skills can help to indicate the reasons for technical failure, i.e. can point to a common cause behind seemingly unconnected technical behaviours. The distinction into four different categories of non-technical skills, and the use of a common terminology (elements and descriptors) to explain the

shortcomings, will hint towards the areas where additional training is required/recommended.

Observed negative behaviour in one of the elements should always lead to a negative rating for the category, and for the non-technical group as a whole. Because of the fact that the non-technical skills are used as a means of explaining reasons for failure, the examiner should not weigh one category or element against any other or try to find multiple elements to present a stronger “case for failure”.

4.3Proposal for AMC and IEM

4.3.1The AMC

AMC-FCL 1.240Evaluation of Non-technical SkillsSee JAR-FCL 1.240, AMC-FCL 1.261, AMC-FCL 1.365 & IEM-FCL 1.261& IEM-FCL 1.240 (as proposed by NOTECHS: see paragraph 9.3.2 of this report)

1 Non-technical skills (NOTECHS) refer to all pilots' attitudes and behaviours in the cockpit not directly and immediately connected with aircraft system control, technical outcomes and Standard Operating Procedures (SOPs) which are referred to as technical skills.

2 The evaluation should focus on and distinguish between four areas of pilots’ non-technical skill : (see IEMxxx for example of concepts and markers)2.1 Co-operation2.2 Leadership and Managerial skills2.3 Situation Awareness2.4 Decision Making

3 The evaluation should be guided on the following principles :

3.1 Principle no.1: “Need for technical consequences” : Non-technical skills cannot be rated and cannot provoke a FAILED condition out of the context of a related objective technical consequence leading to compromised flight safety in the short or long term.

3.2 Principle no. 2: “Use of a two-point rating scale” : Whatever the Non-technical skill to be noted, the form used by examiners should only offer two possibilities for the rating on non-technical skills: ACCEPTABLE or UNACCEPTABLE. This logic applies to the evaluation of non-technical skills only.

3.3 Principle no. 3: “Explanation of unacceptable rating required”: For each negative rating on a non-technical skill category an explanation by the examiner shall be given on the exam form, using standard phraseology as provided in IEMxxx. The examiner shall indicate both the related element as detailed in the NOTECHS system (see IEMxxx), as well as the technical consequences resulting from the observed unacceptable non-technical behaviour.

3.4 Principle no. 4: “Repetition required” : A leitmotiv of similar behaviours during the check should be observable to conclude that the pilot has problems in this area. In addition to this general idea of leitmotiv, the examiner should give a second chance to the applicant anytime this applicant has engaged flight safety due to the exhibition of unacceptable behaviour on a non-technical skill, unless any other JAR-paragraph on the consequences of technical failure dictates otherwise.

3.5 Principle no. 5 : “No evaluation of personality traits”: Any evaluation should be based only on observable behaviours. When inferences (interpretation of facts) are required to have access to social or cognitive skills, they should be limited and based on obvious observable facts and behaviours. The grid must exclude any reference to pilots’ personality or to emotional attitudes. This is the reason why a set of behavioural markers is described in the IEMxxx. These behavioural markers are designed to support an objective judgement of the trainee.

4 The examiners/Check pilots using the grid should4.1 have successfully passed the Human Performance and Limitation certificate for the ATPL, or4.2 have followed a theoretical Human Performance and Limitation course covering the whole syllabus of the Human Performance and Limitation certificate; and4.3 have received additional education in the field of personal and group awareness, decision making, group management, group dynamics, conflict management and 4.4 have received education on the specific use of the evaluation grid and rationale behind it.

5. It has to be made sure that applicant and check pilot are both familiar with the terminology used in the non-technical skill evaluation system. For the applicant this should mean that the same terminology is used in the CRM training.

4.3.2The proposal for IEMs

4.3.2.1Proposal for IEM 1.240

IEM FCL 1.240Evaluation of Non-technical SkillsSee JAR-FCL 1.240, AMC-FCL 1.240 (as proposed in this report)

1 The IEM presents a comprehensive system for LOC evaluation of pilots’ non-technical skills.2 The evaluation grid is divided into categories, elements (sub-categories) and markers

(descriptors).3 The ACCEPTABLE/UNACCEPTABLE evaluation is done at the category level. An

unacceptable rating on one of the categories should be interpreted as an indication of a need for additional training in that specific area.

4 With respect to AMC-FCL 1.240, there are four evaluated categories which are subdivided into two social skill categories (Co-operation; Leadership and Managerial skills) and two cognitive categories (Situation Awareness; Decision making).

5 In addition to those criteria stipulated in AMC-FCL 1.240, the following principles were used to guide the final choice of categories, components and descriptive terms.

5.1 The basic categories and elements should be formulated with the maximum mutual exclusivity.

5.2 A rule of parsimony was applied, in that each category should contain the minimum number of elements in order to encompass the critical behaviours.

5.3 The terminology used should reflect everyday language for behaviour, rather than psychological jargon.

5.4 The skill listed at the behaviour level should be directly observable in the case of social skills or could be inferred from communication, in the case of the cognitive skills.

6 The following table summarises the categories, elements and markers which can lead to a FAILED rating, provided the deficiency in non-technical skill has endangered flight safety.

Table 9

Categories(areas to be evaluated)

Sub-categories or elements

Descriptors or markers (poor practices)

CO-OPERATION“Ability to work effectively in a team/crew”

Team-building and maintaining

∑ Blocks open communication∑ Keeps barriers between CM∑ Competes with others

∑ Consideration of others ∑ Ignores suggestions of other CM∑ Does not take account of the condition

of other CM∑ Shows no reaction to other CM

∑ Support of others ∑ Hesitates to help other CM in demanding situations

∑ Does not offer assistance∑ Conflict solving ∑ Overreacts in interpersonal conflicts

∑ Sticks to own position without considering a compromise

∑ Accuses other CM of making errors∑ LEADERSHIP AND

MANAGERIAL SKILLS

The ability to achieve joint task completion within a motivated, fully functioning team through

Use of authority and assertiveness

∑ Hinders or withholds crew involvement∑ Passive, does not show initiative for

decisions, own position not recognisable

∑ Does not show appreciation for the crew, coaches very little or too much

∑ Co-ordination and persuasiveness

Providing and maintaining standards

∑ Does not comply to SOPs, does not monitor crew for SOP compliance

∑ Does not intervene in case of deviations

∑ Applies non-standard procedures without

announcement or consultation of crew members

∑ Planning and co-ordination

∑ Plans only for self, crew not involved∑ Intentions not stated or confirmed∑ Changes plan without informing crew or

follows plan blindly∑ Goals and boundaries remain unclear.

∑ Workload management ∑ Flying solo without other crew members involved

∑ Secondary operational tasks interfere with primary flight duties

∑ Workload is increased through inadequate planning

∑ Ignores signs of stress and fatigue.∑ SITUATION

AWARENESSAwareness of aircraft systems

∑ Does not ask for updates∑ Does not signal awareness of changing

systems∑ The ability to

accurately perceive what is in the cockpit and outside the aircraft

Awareness of external environment

∑ Does not acknowledge / repeat ATC directions

∑ Does not enquire about environmental changes

∑ Does not comment on relevant environmental factors, or is surprised by them

∑ Operates a “closed shop”∑ Awareness of time ∑ Does not set priorities with respect to

time limits∑ Anticipation of future

events∑ Does not discuss relationship between

past events and present/future∑ Is surprised by outcomes of past events

∑ DECISION MAKINGThe process of reaching a judgement or choosing an

Problem definition / diagnosis

∑ Nature of the problem not stated or failure to diagnose

∑ No discussion of probable causes∑ Option Option generation ∑ Does not search for information

∑ Does not ask crew for alternatives∑ Risk assessment ∑ Inadequate discussion of limiting factors

with crew∑ Option selection ∑ Does not inform crew of decision path

being taken∑ Outcome review ∑ Fails to check selected outcome against

goal.

4.3.2.2Modifications to the form of IEM FCL 1.240 (b)(1)The form defined in the IEM FCL 1.240 (b)(1)- should be modified by adding a grid for Notechs

assessment. The actual format of the material used by the companies may differ depending on their specific needs or house style. The format shown here should therefore be seen as an example.

Table 10 Skill test

Non Technical skillsCo-operation

Leadership & managerial skills

Situation Awareness

Decision making

Rating for observed behaviour on category

+ / - + / - + / - + / -

For each category rated unacceptable indicate the element which has lead to the negative rating:In case of negative rating on one or more categories: Indication of technical area or group where inadequate non-technical skill (potentially) had safety consequences:In case of negative rating on one or more categories: free text explanation by examiner on observed behaviour using standard phraseology provided by the company

4.3.2.3Modifications to Appendix 2 to JAR FCL 1.240 and 1.295Appendix 2 to JAR FCL 1.240 and 1.295 define the list of points to check.A section no 7 should be added for the categories proposed for Notechs assessment: Co-operation, Leadership & managerial skills, , Situation Awareness, and Decision making.

Table 11

Section 7 Non-technical skills

OTD FTD FS A/C instructor initial when training completed

Chkd inFS A/C

Examiner's initial when test completed

7.1 Co-operation FS only7.2 Leadership & managerial skills

FS only

7.3 Situation awareness FS only7.4 Decision making FS only

PART E.General conclusions

5Conclusions

Non-technical skills (NTS) are defined as pilots’ attitudes and behaviours in the cockpit not directly related to aircraft control, system management and Standard Operating Procedures (SOPs) – the latter are referred to as technical skills (TS). Classic examples of non-technical skills are in the field of cockpit authority, crew co-ordination and co-operation, communication and collective decision making, human error and conflict management, stress and workload management, attention, vigilance and confidence.

It is explicitly assumed that poor exercise of non-technical skills may facilitate errors in technical areas and has the potential to endanger the flight. This is the reason why training NTS is expected to improve flight safety. There is a general consensus that training in this area is important, and in every pilot training the topic is dealt with. However, the manner in which this is done and the wording used when discussing non-technical skills vary widely. Also the extent of the training may range from field introductions to being part of the company evaluation. Nevertheless, it is good to realise that the topic of non-technical skills is not a new invention: the aviation community has long realised that the classic examples mentioned above are important

Non-technical skills training and checking requirements have recently become part of JAR-FCL and JAR-OPS. This is the first step of a significant effort to further increase crew members’ awareness of these skills. The goal of this effort is to reduce the risk (ultimately: to ensure) that no individual behaviour in flight could lead to an incident or accident.

JAR-FCL may ask for the assessment of non-technical skill proficiency, but it is not very clear in its directives on acceptable means of compliance. The NOTECHS project was set up to study exactly that. The goal of the NOTECHS project is not to debate on the need for regulations, which is the responsibility of the ad-hoc JAA committees. The scope of the NOTECHS project was restricted to NTS evaluation, not training. Nevertheless, the material presented in this report has relevance to more training oriented requirements as well. With little modifications the evaluation method presented should be applicable to many requirements (JAR-FCL 1.030, 1.240, 1.250, or JAR-OPS 1.943, 1.945, 1.955, 1.965).

This report starts with a summary of present JAR-OPS and JAR-FCL requirements concerning non-technical skills. It has to be concluded that while referring to the same matter both documents use their own terminology. The main difference is that JAR-FCL uses a more flight-oriented definition of NTS (based on the Multi Crew Co-ordination concept) when JAR-OPS uses a more psychologically oriented definition (based on the definition and concept of CRM). An initiative to harmonise all JAR-regulations in their approach to non-technical aspects would be welcomed. It is proposed to use the NOTECHS framework that categorises non-technical skills as a starting point for such an initiative.

From a comparative description of NTS training and evaluation systems currently in use with European airlines, the NOTECHS group made an inventory of all the labels used in these systems. This resulted in a total of almost 150 unique, although sometimes almost identical labels. NOTECHS has reduced these to a comprehensive and manageable framework covering all aspects necessary to consider when assessing non-technical behaviour. Compared to the existing systems, some descriptors have been left out, reformulated or placed within a different category. Although the NOTECHS

framework is the result of thorough and detailed literature reviews and discussions within the project group, it should be noted that in some cases the choice to put an element in one or in the other category has been very difficult. The essence is that all relevant elements are present somewhere in the system, allowing examiners to describe the behaviour observed. Whether the element falls under category A or B is of lesser importance. A consequence of this approach is that all elements in the framework should be given equal weight. Any one unacceptable rating in one of the elements should always lead to an unacceptable rating in the category and hence to an overall unacceptable result on the non-technical skills group. This may seem a strict rule, but it is considered undesirable to adopt rules that e.g. require observed unacceptable behaviour on more than one element within one category. The goal of the NTS assessment is to identify behaviour that may contribute to an incident or accident, and every opportunity to identify existing training needs should therefore be seized. The safeguards present in the proposed NOTECHS method (as discussed below) prevent pilots being failed on coincidental, trivial observations.

The NOTECHS NTS framework consists of the following four categories, each of which are further subdivided into three to four elements, with behavioural markers (2-4) for each element:

∑ Co-operation∑ Leadership and managerial skills∑ Situation Awareness∑ Decision Making The exact wording of the different labels (on the category, element and behaviour level) has been chosen so that the meaning and relevance to the operational practice are immediately clear for all users of the system, the majority of whom are not educated psychologists. This means that in some cases another wording may be more correct from the point of view of a psychologist. As an example: it could be argued that Situation Awareness rather is a state or a result than a skill. However, changing the wording (e.g. into Situation Assessment) would make the topic less recognisable for the pilot community, thereby overshooting the mark of the NOTECHS exercise. Most people will expect Situation Awareness to be one of the aspects of the non-technical issue: it is also listed in each of the existing systems reviewed. Clear and unambiguous definitions of all terms, categories and elements in an NTS system are necessary for proper, consistent fail-pass decisions and clear pilot debriefings, especially when the system will be used by several airlines. Pilots will have to be taught the exact meaning of the terms used in the NTS assessment system.

An evaluation method to accompany the NOTECHS framework is proposed. From this proposal it should be feasible to construct the AMC and IEM for JAR-FCL 1.240. Project constraints precluded a test of the proposed method. Therefore there is no evidence that use of the framework and accompanying evaluation method will lead to a robust and valid assessment method. A follow-on project, named JAR-TEL, will work on the validation of the evaluation system. Although it is possible that a modification to the evaluation method will be needed on the basis of the JAR-TEL outcome, it is considered appropriate to include the NOTECHS material in the JAR-regulations before the end of JAR-TEL. The NOTECHS method was set up knowing that it would not be possible to validate it within the project. Therefore, safeguards were built into the proposed evaluation method, which guarantee that no pilot will be failed on a lapse in the system. These safeguards are formulated

in the five basic NOTECHS principles:

1. the need for technical consequences2. use of two-point rating scale3. explanation of unacceptable rating required4. repetition required5. no evaluation of personality traits

It should be said that the first principle is controversial. Without empirical data on the amount of variability in NOTECHS ratings given by different examiners it is felt not to be acceptable to fail people on their non-technical skills only. A great variability in examiner ratings would enhance the notion of arbitrariness, which is felt to some extent by the pilot community. Without objective data to support the opposite, the concerns expressed by pilots should not be ignored. User acceptance should always be treated as an important aspect, but especially in the area of non-technical skill evaluation.

One might argue, and in fact the JAA-PAG has argued, what the benefit of assessing non-technical skill proficiency is if negative technical outcomes are still a necessity to fail a candidate. The NOTECHS group understands the concern, but the advice based on the results of this project is not to remove this safeguard until the method has proven itself. The answer to the question raised is that there still is a benefit to non-technical skills assessment even with the technical consequences requirement in place. A strong argument in favour of the proposed evaluation method (with or without the technical relationship requirement) is that it may provide a reason for technical failure. The technical failure itself may be reason enough to fail the candidate, but the additional evaluation of non-technical aspects as proposed by NOTECHS, may reveal underlying reasons which would otherwise have been ignored. A clear and concise formulation of such underlying reasons for failures in technical skills will certainly make it easier to set up additional training for that pilot. For that reason it is recommended to list the unacceptable NTS rating on the examination form also when it does not have direct consequences for the outcome of the exam. It is up to the candidate and the company to take action and arrange for additional training.

In addition, the NOTECHS project group wishes to emphasise that it is up to the authority of each company and its Flight Standards department to decide what definition they will use of “safety engaged”, and therewith “technical consequences”. For example, a check policy may prescribe that the combination of poor non-technical skills and repetitive and related technical flaws, even if these technical shortcomings are individually not enough to provoke a failure, may justify at the very end a failed exam.

From the comparative description of existing NTS evaluation systems, the NOTECHS group concluded that most airlines keep non-technical aspects separate from technical skills. In the course of the NOTECHS project the realisation has grown that it is very difficult, if not impossible, to draw a line between technical and non-technical aspects. Apart from considerations regarding content, pilot acceptance is also an important factor necessitating an integrated TS and NTS evaluation (the first NOTECHS basic principle as described above further illustrates this problem). KLM, in its newest evaluation system, indeed no longer makes a distinction between technical and non-technical aspects.

Building on JAR-FCL, where the distinction is made, NOTECHS had to accept that technical and non-technical aspects are dealt with separately. It is suggested to investigate the merits of integrating the requirements regarding evaluation of technical and non-technical skills in JAR-regulations.

As mentioned before, the project objective to produce a descriptive NTS framework was successfully met. However, the importance of training examiners and instructors in the use of the proposed common terminology should not be underestimated. Confusion and misinterpretation of reasons leading to the unacceptable behaviour will be avoided when the examiner is obliged to describe the observed negative behaviour in phraseology that is clear to both the examiner and the candidate.

The NOTECHS NTS description and assessment system does not comprise the definition of a clear-cut division between behaviours leading to a FAILED test condition and those resulting in a PASSED test (i.e. a test standards document for NTS was not composed). The project group recognised however the importance of a sharp and unambiguous definition of this borderline. It is felt that only a practical test of the NOTECHS assessment system can provide proof for a workable FAILED/PASSED definition. Adjustment of the rating scale content (performance standards for each rating) is foreseen following field tests of the rating scale as scheduled in JAR-TEL for 1999. None of the existing systems reviewed seems to have an effective translation from their NTS categories and elements into unambiguous and objectively defined rating scale anchors. It is difficult (and undesirable) to define these anchors without knowing the exact situation to which they will be applied. Therefore, and because in examinations ultimately only the PASS/FAIL decision is of interest, the NOTECHS group elected to describe only the unacceptable behaviour. If that behaviour is observed, the corresponding element rating should be rated unacceptable. It is suggested to further investigate KLM’s approach in their new SHAPE system, where the PASS/FAIL-decision is made following a flowchart that starts with the outcome of the exercise. Knowing the outcome of the exercise (safe or unsafe) it becomes much easier to determine whether the observed behaviour on a specific element was DESIRED or NOT DESIRED (to use SHAPE terminology).

In summary it is stated that NOTECHS provides a descriptive framework for NTS together with a method for using the framework in a skill assessment situation. Undoubtedly, one can have questions or concerns about the end result as to whether this approach is too tolerant (or too strict for that matter). And of course the results of JAR-TEL should be awaited to see how many adjustments are still needed. Nevertheless, a plea is made to include an AMC and IEM in JAR-FCL based on the material and method provided by NOTECHS. Even before a formal validation has been conducted, the NOTECHS approach will offer advantages above just keeping quiet about non-technical aspects. There is no reason to evade the question: it seems that the proposed requirement to have technical (safety) consequences will become a point of contention. It is proposed here to keep this requirement in for reasons explained, but it is of course up to the ad-hoc JAA committees to decide in this matter.

NOTECHS - MEDESIM€¦ · Web viewThis study resulted in a proposal for a new non-technical skill...

Documents

Transcript of NOTECHS - MEDESIM€¦ · Web viewThis study resulted in a proposal for a new non-technical skill...