Staff Instruction - Transports Canada

Staff Instruction Subject: Development of Flight Crew Examinations

Issuing Office: Standards Document No.: SI 400-006-Y File Classification No.: Z 5000-32 Issue No.: 01 RDIMS No.: 2476427-V30 Effective Date: 2013-07-10

TABLE OF CONTENTS

1.0 INTRODUCTION.............................................................................................................................. 4

1.1 Purpose ............................................................................................................................................ 4

1.2 Applicability ...................................................................................................................................... 4

1.3 Description of Changes .................................................................................................................... 4

2.0 REFERENCES AND REQUIREMENTS ......................................................................................... 4

2.1 Reference Documents ..................................................................................................................... 4

2.2 Cancelled Documents ...................................................................................................................... 4

2.3 Definitions and Abbreviations .......................................................................................................... 4

3.0 BACKGROUND ............................................................................................................................... 5

4.0 NEW QUESTIONS........................................................................................................................... 5

4.1 Develop a new question ................................................................................................................... 7

4.2 Develop alternatives ......................................................................................................................... 9

4.3 Develop deficiency statements ...................................................................................................... 10

4.4 Enter questions in Flight Training and Aviation Education (FTAE) ................................................ 11

4.5 Format questions ........................................................................................................................... 12

4.6 Quality review of questions ............................................................................................................ 16

4.7 Approve questions ......................................................................................................................... 17

4.8 Translate questions ........................................................................................................................ 17

5.0 AMENDING ACTIVE QUESTIONS OR DEFICIENCY STATEMENTS ........................................ 18

6.0 INACTIVATING QUESTIONS ....................................................................................................... 18

7.0 DELETING QUESTIONS ............................................................................................................... 18

8.0 NEW EXAMINATION SERIES ...................................................................................................... 19

8.1 Create an examination plan (Blue Print) ........................................................................................ 19

8.2 Create a new examination series in Flight Training and Aviation Education (FTAE) .................... 20

8.3 Assemble the examination ............................................................................................................. 20

8.4 First independent review ................................................................................................................ 21

8.5 Second independent review ........................................................................................................... 21

8.6 French vetting ................................................................................................................................ 22

8.7 Format the examination ................................................................................................................. 22

8.8 Approve examination ..................................................................................................................... 22

8.9 Activate the examination ................................................................................................................ 23

8.10 Examination Storage ...................................................................................................................... 23

Development of Flight Crew Examinations

9.0 NEW EXAMINATION VERSION ................................................................................................... 23

9.1 Create a new examination version in Flight Training Aviation Education ...................................... 23

9.2 Replace questions .......................................................................................................................... 24

9.3 Independent review ........................................................................................................................ 24

9.4 French vetting ................................................................................................................................ 25

9.5 Format the examination ................................................................................................................. 25

9.6 Approve the examination ............................................................................................................... 26

9.7 Activate the new examination version ............................................................................................ 26

9.8 Examination Storage ...................................................................................................................... 26

10.0 ANALYSIS OF EXAMINATION PERFORMANCE ....................................................................... 27

11.0 REVIEW A CHALLENGED QUESTION ....................................................................................... 28

12.0 INACTIVATE AN EXAMINATION ................................................................................................. 28

13.0 GRAPHICS .................................................................................................................................... 28

13.1 Design graphics ............................................................................................................................. 28

13.2 Design approach plates – Canada Air Pilot (CAP) ........................................................................ 29

13.3 Design aerodrome directory/facility pages - Canada Flight Supplement ....................................... 30

13.4 Amend a graphic ............................................................................................................................ 30

14.0 APPENDICES ................................................................................................................................ 31

14.1 Assemble an appendix ................................................................................................................... 31

14.2 Amend an appendix ....................................................................................................................... 32

15.0 STUDY AND REFERENCE GUIDE .............................................................................................. 33

15.1 Review knowledge requirements ................................................................................................... 33

15.2 Obtain a Transport Canada Publication number ........................................................................... 33

15.3 Create draft .................................................................................................................................... 33

15.4 Review by working group ............................................................................................................... 33

15.5 Fine tune ........................................................................................................................................ 33


15.7 Translation ..................................................................................................................................... 33

15.8 Study and Reference Guide format ............................................................................................... 34

15.9 Post on the Internet and Intranet ................................................................................................... 34

16.0 AMEND STUDY AND REFERENCE GUIDE ................................................................................ 34

16.1 Download working copy from RDIMS ............................................................................................ 34

16.2 Amend working copy ...................................................................................................................... 34


16.4 Translation ..................................................................................................................................... 35

16.5 Study and Reference Guide format ............................................................................................... 35

16.6 Final approval ................................................................................................................................. 35

16.7 Post on Intranet and Internet ......................................................................................................... 35

17.0 ABBREVIATION LIST ................................................................................................................... 36

18.0 WEAK KNOWLEDGE AREA REPORT ........................................................................................ 36


18.1 Examination’s Analysis .................................................................................................................. 36

18.2 Produce the Report ........................................................................................................................ 37

18.3 Weak Knowledge Areas Report format .......................................................................................... 37

18.4 Final Approval ................................................................................................................................ 37

18.5 Post on Intranet and Internet ......................................................................................................... 38

19.0 ANNUAL SELF-PACED STUDY PROGRAM ............................................................................... 38

19.1 Develop new questions .................................................................................................................. 38

19.2 Review Committee ......................................................................................................................... 39

19.3 Perfect the questions ..................................................................................................................... 39

19.4 Final Review Committee ................................................................................................................ 39

19.5 Self-Paced Study Program Format and Storing ............................................................................ 39

19.6 Submission of the Final Self-Paced Study Program ...................................................................... 39

19.7 Translation ..................................................................................................................................... 40

19.8 Submission of the translated Self-Paced Study Program .............................................................. 40

19.9 Posting the Self-Paced Study Program on the Flight Crew Examination Web site ....................... 40

20.0 LIBRARY ....................................................................................................................................... 40

20.1 Administration of the Library .......................................................................................................... 40

20.2 How to borrow books from library .................................................................................................. 40

21.0 CHARTS ........................................................................................................................................ 41

21.1 Chart Selection Meeting ................................................................................................................. 41

21.2 Order charts ................................................................................................................................... 41

21.3 Monitor charts ................................................................................................................................ 41

22.0 INFORMATION MANAGEMENT .................................................................................................. 42

23.0 DOCUMENT HISTORY ................................................................................................................. 42

24.0 CONTACT OFFICE ....................................................................................................................... 42

APPENDIX A — SPELLING....................................................................................................................... 43


2013-07-10 4 of 45 SI 400-006-Y Issue 01

1.0 INTRODUCTION

1.1 Purpose

(1) The purpose of this Staff Instruction (SI) is to set out the procedures for developing written examinations for flight crew permits, licences and ratings.

1.2 Applicability

(1) This document applies to all those in Transport Canada Civil Aviation (TCCA) Headquarters (HQ) responsible for development of written examinations for flight crew permits, licences and ratings.

1.3 Description of Changes

(1) Not applicable.

2.0 REFERENCES AND REQUIREMENTS

2.1 Reference Documents

(1) It is intended that the following reference materials be used in conjunction with this document:

(a) ITP Nelson Canadian Dictionary 1997, Revised and expanded – The Gage Canadian Dictionary;

(b) Oxford University Press Canada, 1998 – The Canadian Oxford Dictionary;

(c) Supplement to the Canada Gazette Part II. Vol. 130, No. 2 (2 October 1996) – The Canadian Aviation Regulations (CARs);

(d) Transport Canada Publication TP 11958 – Glossary for Pilots and Air Traffic Services

Personnel, which can be found at: http://www.tc.gc.ca/eng/civilaviation/opssvs/secretariat-terminology-glossary-744.htm;

(e) A Guide to Writing and Editing - The Canadian Style.

2.2 Cancelled Documents

(1) As of the effective date of this document, the following document is cancelled:

(a) Guidelines and Recommended Practices – Flight Crew Examinations, Seventh Edition, February 2005.

(2) By default, it is understood that the publication of a new issue of a document automatically renders any earlier issues of the same document null and void.

2.3 Definitions and Abbreviations

(1) The following definitions are used in this document:

(a) Alternatives: A number of suggested or offered answers, or completions of statement, to the stem;

(b) Distractor: An alternative that incorrectly answers the stem;

(c) Full Set Examination: Includes English and French Examinations (covers, instructions, questions), references, answer key, deficiency statements and appendices;


2013-07-10 5 of 45 SI 400-006-Y Issue 01

(d) FTAE: Flight Training and Aviation Education. The computer system used for development and delivery of flight crew examinations, the recording of flight tests, flight training units and instructor refresher courses;

(e) Keyed response: The alternative that correctly answers/completes the stem;

(f) MCI: Master Coding Index is used in FTAE to classify questions based on subjects. It is useful for examination specialists to search for existing questions regarding a specific subject;

(g) Review committee: Question review process with at least 3 examination specialists;

(h) Stem: The part of the multiple choice question that poses the question or problem. It may be in the form of a direct question or incomplete statement.

(2) The following abbreviations are used in this document:

(a) ADF: automatic direction finder;

(b) AEI: Authorized Examination Invigilator;

(c) CFS: Canada Flight Supplement;

(d) FTAE: Flight Training and Aviation Education;

(e) MCI: Master Coding Index;

(f) OPI: Office of Primary Interest;

(g) RDIMS: Records, Documents and Information Management System;

(h) SI: Staff Instruction;

(i) TCCA: Transport Canada Civil Aviation;

(j) TP: Transport Canada Publication; and

(k) VOR: Very high frequency omnidirectional range.

(3) The abbreviations used in flight crew examination are detailed in Records, Documents and Information Management System (RDIMS) 1707754.

3.0 BACKGROUND

(1) The Canadian Aviation Regulations (CARs) set out in Part IV, Subpart 1, the requirements for flight crew permits, licences and ratings. This includes the knowledge requirements and, where written examinations are required, these are specified in the standards. Flight crew examinations are developed by specialist inspectors in the Standards branch. Together, these inspectors are responsible for more than 130 separate examinations.

4.0 NEW QUESTIONS

(1) Questions are considered PROTECTED B documents. Unauthorized disclosure could reasonably be expected to cause serious injury to the government. The storage must be in an approved security container with an integral lock in an approved operations zone. An approved security shredder, in accordance with the Information Management Directive Transport Canada Publication TP 13953E must be used to destroy copies of the questions. Questions shall be marked PROTECTED B on all pages. Questions shall have a reference document easily accessible to the candidate that supports the correct answer.


2013-07-10 6 of 45 SI 400-006-Y Issue 01

Figure 1 – (Question development process)

START

Develop new question

(OPI)

Design graphic(Appendix Specialist)

Approve graphic(OPI)

New graphic required?

Review of question and appendices

(Review Committee)

Accepted

- Make corrections as suggested by

Review Committee(OPI)

Final approval of question and lock question in FTAE

(Examination Specialist)

A

A

Send questions and graphics for

translation(OPI)

Enter translated questions into

FTAE (Examination

Officer) / Enter translation in

graphics (Appendix Specialist)

END

YES

NO

YES

NO


2013-07-10 7 of 45 SI 400-006-Y Issue 01

4.1 Develop a new question

(1) The questions should be safety-related, either directly relevant to safe operations or pertaining to knowledge that has an impact on the understanding and management of safety. The aim is to ensure that pilots have achieved the minimum standard of aeronautical knowledge in order to operate safely under the conditions the pilot will encounter when exercising the privileges of the permit, licence or rating they are seeking.

(2) Questions shall have a reference document easily accessible to the candidate that supports the correct answer.

(3) Mandatory flight test items should not be included in the written examination. Consult with the Office of Primary Interest (OPI) for the flight test standard, if a flight test is also required for the permit, licence or rating in order to avoid duplication.

(4) Most questions assess knowledge at the first two levels of Bloom’s taxonomy. Some questions will assess higher levels of learning. This can be achieved by situation-based questions that require synthesis of knowledge.

(5) Aviation involves technical terminology or language that has been recognized and accepted and pilots are required to know this terminology or language. Question writers are expected to use common aviation technical terms/phrases. See Appendix A for spelling of common terms.

(6) Each question shall “stand alone”, which means the scenario, data and information essential to the answering of a particular question must be contained either within the question itself or in an accompanying document, such as charts, tables, diagrams or graphs. This essential data must never be in, or dependent on, another question.

(7) Construct each question to assess a single objective. Questions that are not written with a specific objective in mind often end up measuring lower-level material that is of little value.

(8) After reading the stem, the candidate should know exactly what the problem is and what is expected to solve it. If the candidate has to infer what the problem is, the question will measure the ability to draw inferences from vague descriptions rather than measuring aeronautical knowledge.

(9) “Trick” questions are not acceptable. These can take many forms – loaded questions, suggestive questions, two questions in one, complex questions, etc. The aim is to test aeronautical knowledge, not reading comprehension or the ability to solve problems in logic.

(10) All questions are allocated one mark.

(11) The use of abbreviations is encouraged whenever possible. It is not mandatory to use abbreviations in place of the full word(s) as their use depends on the readability of the text. It is acceptable to ask “How many feet (rather than “ft.”) above sea level is the Halifax airport?”. Generally, when numerals are used, the abbreviation of the unit is used, i.e. 10 ft, 100 NM. A list of abbreviations is provided to every candidate at the beginning of the examination. See G:Drive\AARR\AARRF\Appendices\Abbreviations for the current list. Where required, non-standard abbreviations could be used, however the abbreviation should be introduced immediately after the definition with brackets. e.g.: Depressurized Operation (DO).

(12) Avoid using names or terms that require geographical or colloquial knowledge when developing questions. It is not the aim to test the geographical knowledge of Canada. e.g. “the Maritimes”, “the Prairies”.

(13) Questions are developed using one of the five following types of Multiple Choice Questions:

(a) Direct Question. This is the preferred question type in Transport Canada flight crew examinations. e.g.:


2013-07-10 8 of 45 SI 400-006-Y Issue 01

What does a steady green light directed at your aircraft in flight mean?

(1) Cleared to land.

(2) Return for landing.

(3) Cleared to base leg.

(4) Cleared for final.

(b) Incomplete Statement. This may be used in Transport Canada flight crew examinations. e.g.:

Refer to the VFR Navigation Chart (VNC).

The highest obstacle within 5 NM either side of your track from Lindsay to Gananoque is:

(1) 857 feet ASL.

(2) 600 feet ASL.

(3) 475 feet ASL.

(4) 246 feet AGL.

(c) Negative Question. This is a question that uses “not” in the stem. Because of the language challenge of a negative question and the desire to test aeronautical knowledge and not reading comprehension, Transport Canada uses this format only when the learning objective cannot be measured by other forms of questioning. e.g.:

Which is not required for the Private Pilot License (PPL) ground school course?

(1) Radio Navigation.

(2) Flight Operations.

(3) Weather.

(4) Regulations.

(d) Combined Response Question. This is a question in which one or more of the alternatives are correct answers. Transport Canada uses some combined response questions on flight crew examinations but they are not a preferred option because of the language challenge. Generally, they are used when trying to test for higher levels of knowledge and used more on professional examinations than on recreational examinations. The number of responses to be combined must be manageable. e.g.:

Refer to the Appendix: CFS – Brampton, Ont.

Select the correct statements regarding the airport information.

There is an FSS on the field.

A. Airport variation is 10ºW.

B. There are VASIS on both runways.

C. The airport has aircraft radio controlled runway lights.

D. The control zone extends for 5 NM.

E. The NDB is located 5.2 NM East of the airport.

(1) A, B, D.

(2) A, C, D.


2013-07-10 9 of 45 SI 400-006-Y Issue 01

(3) B, D, E.

(4) C, E, F.

(e) Fill-in-the Blank Question. This is a question with one or more blank spaces that must be completed. Questions with one blank space are used on Transport Canada flight crew examinations. More than one blank space may be used but language complexity for the candidate is greater, which risks testing language instead of aeronautical knowledge. e.g.:

During an approach to land on an upsloping runway, the pilot may experience the illusion that the aeroplane is. . . . . than it actually is.

(1) higher.

(2) lower.

(3) closer.

(4) approaching faster.

4.2 Develop alternatives

(1) Three or four alternatives are used in flight crew examination questions. Three alternatives may be used for a set of examination where authorized by the Chief, Commercial Flight Standards.

(2) Include as much of the question as possible in the stem, but do not include irrelevant material. This avoids repeating words and phrases in each alternative.

(3) Keep the alternatives mutually exclusive. Overlapping alternatives may be easily identified as distractors. As well, if the overlap includes the intended answer, there may be more than one alternative that provides a correct answer.

(4) Keep the alternatives free from clues as to which alternative is correct. Poorly-written questions often contain clues, which will measure how clever the candidate is at finding clues rather than whether they have the required aeronautical knowledge.

(5) Keep the grammar of each alternative consistent with the stem. Inconsistent grammar is often a sign of a distractor.

(6) Each question shall have only one correct answer. The technique of having two or more correct answers with one being “more correct” than the other is not used in flight crew examinations. To ensure that there is only one correct and indisputable answer to each question, every distractor must contain an element of “incorrectness”. Where a situation may require a certain number of items (equipment, observance of rules/procedures, criteria, etc.) listing only one or a few, but not all, of these items does not make the distractors incorrect unless the question stem specifically requires ALL such items to be listed. Usually a question, whose keyed response requires the complete list of several mandatory items, will result in poor distractors. Thus, the test objective should be re-examined, and the question then re-written differently.

(7) Keep the alternatives similar in length. An alternative noticeably longer or shorter than the others is frequently assumed to be the answer, which is often the case.

(8) Use plausible alternatives. The distractors should look as plausible as the answer. Unrealistic distractors increase the candidate’s chance of guessing the correct answer.

(9) Word the alternatives clearly and concisely. Clear wording reduces confusion and concise wording reduces the reading burden on the test candidate.

(10) Avoid using “all of the above” or “none of the above” in the alternatives. While some questions of this type may remain in the question bank, Transport Canada no longer develops questions using “all of the above” or “none of the above” as an alternative.


2013-07-10 10 of 45 SI 400-006-Y Issue 01

(11) Avoid using specific determiners such as "always", "never", "everyone", etc. These will serve as clues to the alert candidate.

(12) Present the answer in each of the alternative positions approximately an equal number of times. There is a tendency to avoid placing the answer in the first or last alternative, preferring instead to “bury the answer in the middle”. Candidates will look for this pattern, or any pattern to the positions of the answers that will help them make their selections.

(13) Alternatives should be as homogeneous as possible with consistent structure and sequencing.

(14) If possible sequence numerical alternatives from least to greatest or greatest to least. e.g.:

If a heading of 250°M maintains your outbound track of 242°M, the required heading to maintain the reciprocal track back to your departure point would be:

(1) 078°M.

(2) 070°M.

(3) 062°M.

(4) 054°M.

(15) When dealing with numerical values, “Rounding Out” is a standard practice for homogenization. Thus, the stem shall qualify the answer and the distractors as being “closest to” the actual value. When “Rounding Out” the actual value shall be clearly closest to the keyed response. E.g. Actual answer is 31.5, responses shall not be (1) 31, (2) 32. This is particularly true when interpreting from graphs. e.g.:

Your estimated time en route from Ottawa to Toronto is closest to:

(1) 31 minutes.

(2) 35 minutes.

(3) 43 minutes.

(4) 59 minutes.

(16) In a question where all the responses begin with one or more words that are the same, move the "common words" up into the stem. Common words that are not at the beginning of the responses do not necessarily go in the stem. The readability of the question and answers is of primary importance.

4.3 Develop deficiency statements

(1) At the end of an examination, the candidate receives a feedback letter with its examination result, which includes the deficiency statements of failed questions. Deficiency statements are worded so that the candidate will have a reasonably accurate picture of the knowledge expected, but will avoid question specifics, such as “state the flight test standards for altitude in a steep turn”. Deficiency statements start with verbs, which apply to the levels in the cognitive domain. See Bloom’s Taxonomy table below for the list of verbs that apply to the levels in the cognitive domain. Each question should have a deficiency statement. Deficiency statements, or feedback, are worded to complete a statement of expectation, i.e. a private pilot must be able to…“decode a Aviation Routine Weather Report (METAR)” or a flight instructor must be able to… “state the flight test standards for a steep turn”.


2013-07-10 11 of 45 SI 400-006-Y Issue 01

Table 1 - Bloom”s Taxonomy of the Cognitive Domain

Category Key Words

Knowledge

Recall data or information.

Arrange, define, duplicate, label, list, memorize, name, order, recognize, relate, recall, repeat, reproduce.

Comprehension Understand the meaning, translation, interpolation, and interpretation of instructions and problems. State a problem in one’s own words.

Classify, describe, discuss, explain, express, identify, indicate, locate, recognize, report, restate, review, select, translate.

Application Use knowledge or generalization in a new situation.

Apply, choose, demonstrate, dramatize, employ, illustrate, interpret, operate, practice, schedule, sketch, solve, use.

Analysis Break down knowledge into parts and show relationships among parts.

Analyze, appraise, calculate, categorize, compare, contrast, criticize, diagram, differentiate, discriminate, distinguish, examine, experiment, inventory, question, test.

Synthesis Bring together parts of knowledge to form a whole and build relationships for new situations.

Arrange, assemble, collect, compose, construct, create, design, formulate, manage, organize, plan, prepare, propose, set up, write.

Evaluation Make judgments on basis of given criteria.

Appraise, argue, assess, attach, choose, compare, defend, estimate, evaluate, judge, predict, rate, score, select, support, value.

4.4 Enter questions in Flight Training and Aviation Education (FTAE)

(1) Select Q. Bank

(2) Click on “Add” button. A new question will be created with an Identification (ID) number. Some information will be automatically displayed, such as: date, user and the status of the question, which will be “Dev” (Development) by default. The status will be changed to “Locked” status when the quality review is done and changed again to “Active” status by the terminology specialist when the language vetting is completed.

(3) By default the English text will be selected. “French” text or “English/French split” can be selected on the right side by clicking on the appropriate tab.

(4) Add a Master Coding Index (MCI) code that corresponds to the subject of the question. The MCI can be found by clicking on “Q-Bank button” selecting “MCI codes” and then selecting “MCI list”. When you have found the “MCI code” that matches the question, go back to the question and type the MCI code in the MCI field.

(5) Mark the “Check French” box if a French vetting is required prior to activation the question.

(6) References must be attached to the question in Flight Training and Aviation Education (FTAE). In order to attach a reference, click on the “Reference” tab on the right side of the screen and then select “Add reference” on the left top of the screen. The list of reference will show up, click on the


2013-07-10 12 of 45 SI 400-006-Y Issue 01

appropriate reference and click the “attach” button. You can add several references for one question; two or more is better than one.

(7) Attach a graphic number when required. In order to attach the graphic to a question, click on the “Appendix” tab on the right side of the screen, the appendix list will show up and click on “add Appendix” on the left top of the screen, select the appropriate graphic and attach it.

(8) The ”Remarks” field is used to enter information such as: Final approval, copied questions or any information about the question that you want to keep or make other people aware of.

(9) The ”Details” field is used to explain the calculation behind the distractors.

(10) Add a deficiency statement by clicking on ”Deficiency” and typing the deficiency statement.

(11) Add a category/licence level in which the question can be used. The same question can be used on several examinations. It can be used for different categories; licences, ratings and can touch different levels of knowledge. In order to categorize a question in FTAE, click on the “CAT/LIC/LEV” tab on the right side of the screen and select the appropriate category, licence/rating and level.

4.5 Format questions

(1) Use the Arial font, size 11.

(2) If the stem contains more than one sentence, leave only one space after each period.

(3) Leave one line space between the stem and the first response.

(4) Each response numbers should be in brackets, e.g.:

(1) A, B.

(2) A, C.

(3) B, C.

(4) C, D.

(5) Leave a tab between the response numbers and the responses.

(6) Leave no line spaces between responses.

(7) Use a period after each response that completes an open ended statement or answers a direct question.

(8) Use five dots with a space between each dot to represent a blank in the stem. If those five dots are at the end of the stem, do not follow it with a period. Use no period after responses that fill in a blank except if the fill in a blank is at the end of the sentence, e.g.:

The altitude of the cloud layer is . . . . .and is concealed by . . .. .

(1) 500; 8 octas of cloud.

(2) 1,000; 6 octas of cloud.

(9) If those five dots are in the middle of a sentence, the responses should not have a period, e.g.:

The cloud base at CYOW is . . . . .ft.

(1) 500

(2) 1,000


2013-07-10 13 of 45 SI 400-006-Y Issue 01

(10) Use a semi-colon to separate more than one answer, e.g.:

The cloud base at CYOW is . . . . . ft at . . . . . UTC (Z).

(1) 500; 1200.

(11) Use the following example to format multiple-answer questions. Arrange the letters for each response in either alphabetical order or reverse alphabetical order; once you have chosen which order to use, maintain consistency throughout the question. When possible avoid these complex type questions. e.g.:

What are the first four steps in the Examination Development/Amendment Process?

A. Task assignment.

B. Examination plan and approval.

C. Regional vetting.

D. Independent check.

E. Industry trial.

F. Final check and release

(1) A, B, C, E.

(2) A, B, D, F.

(3) B, C, D, E.

(4) C, D, E, F.

(12) Keep numerals on the same line as the terms, abbreviations or symbols they are used with, e.g.:

(a) 5,000 ft, RWY 23,

not

(b) 5,000 ft, RWY 23

(13) Use a comma to group numbers with more than three figures in English. In French use a space, e.g.:

(a) English: 3,000 ft.

(b) French: 3 000 pieds.

(14) A space is not required between FL and the number, e.g.:

(a) English: FL340.

(b) French: FL340.

(15) Exception: numbers above 999 representing a transponder code or similar type of item are presented without a comma/space, e.g.:

4096

(16) Qualify navigation heading as magnetic or true, except when referring to a very high frequency omnidirectional range (VOR) or an automatic direction finder (ADF), e.g.:

(a) English: 180ºM, 360ºT.

(b) French: 180 ºM, 360 ºT.


2013-07-10 14 of 45 SI 400-006-Y Issue 01

(17) When the degree symbol “º” is used with a number only, there is no space in English or in French, e.g.:

90º.

(18) Otherwise, a space is not required between the number and a “º” symbol in English; however, a space is required in French, e.g.:

(a) English = 2ºC.

(b) French = 2 ºC.

(19) A space is not required between the symbol # in English and the number, however, a space is required in French between the symbol Nº and the number, e.g.:

(a) English = #1.

(b) French = Nº 1.

(20) Use font 8 for superscript.

(21) When the coordinates are not provided in the appendix the latitude/longitude coordinates of relevant positions will be given in the stem to assist the candidate pinpoint the locations among the congested details of the charts. Use an alphanumeric format when referring to latitude and longitude. Use the following example to write the coordinates in English and in French:

45°19”21”N 75°40”09”W.

(22) Cardinal directions are not capitalized and should always be spelled out. Exceptions are when it is part of a name; when it is capitalized and possibly abbreviated, e.g.:

(a) southwest, south-southwest.

(b) North Battleford, Queen St. E.

(23) Measurement abbreviations don’t take an “s” for the plural form, e.g.: lb not lbs.

(24) All the other Abbreviations in English take a small “s” at the end for the plural form however, abbreviation in French don’t take an “s” for the plural form, e.g.:

(a) English = GFAs.

(b) French = GFA.

(25) The abbreviations for the provinces are used in the examination questions. However, the Canada Flight Supplement (CFS) uses the symbols, therefore, when a graphic is created from the CFS, symbols will be used in the graphic only to keep the same format as the CFS, e.g.:

(a) Refer to the Appendix: CFS – TOFINO, B.C.

(b) Référez-vous à l’appendice : CFS – TOFINO, C.-B.

English English & French French

Province/Territory Abbrev. Symbols Abrév. Province/Territoire

Alberta Alta. AB Alb. Alberta

British Columbia B.C. BC C.-B. Colombie-Britannique

Manitoba Man. MB Man. Manitoba

New Brunswick N.B. NB N.-B. Nouveau-Brunswick

Newfoundland and

Labrador

N.L. NL T.-N.-L. Terre-Neuve-et-

Labrador


2013-07-10 15 of 45 SI 400-006-Y Issue 01

Northwest Territories N.W.T. NT T.N.O Territoires du Nord-

Ouest

Nova Scotia N.S. NS N.-É. Nouvelle-Écosse

Nunavut - NU - Nunavut

Ontario Ont. ON Ont. Ontario

Prince Edward Island P.E.I. PE Î.-P.-É. Île-du-Prince-Édouard

Quebec Que. QC Qc Québec

Saskatchewan Sask. SK Sask. Saskatchewan

Yukon Y.T. YT Yn Yukon

(26) In English, use a colon when referring to time in a specific time zone. In French use the abbreviation «h». e.g.:

(a) English: 14:30 EST

(b) French: 14 h 30 HNE

(27) Do not use a colon when referring to time in Zulu or UTC, e.g.:

ETA 2048Z

(28) For estimated time enroute use the following format:

2 hours 30 minutes.

(29) Use double quotation marks (“_”) in English and, double guillemets («_») in French to highlight a technical term, an expression or a direct quotation. Don’t leave a space between the quotation marks and the term in English. Leave a space between the guillemets and the term in French, e.g.:

(a) English: With respect to aeroplane, “extended over-water operations” means…

(b) French : En ce qui concerne les avions, « survol prolongé d’un plan d’eau » désigne…

(30) Underline the word “not” to emphasize a negative statement, e.g.: … an aircraft shall not be flown… This practice is used to ensure that a thought reversal is recognized. Underlining other words for question clarity is possible, however must be kept to a minimum. Question clarity and readability is the overriding priority. The use of this (underlining) practice in a question requires a review committee. The use of negative in the stem and the answer and/or distractors should be avoided since it causes confusion to the candidate.

(31) In the questions requiring the use of charts, tables and/or diagrams, such questions shall begin with a reference to the appropriate appendix. Name the appendix using the same wording as it appears in the graphic title. If desired for clarity, the question may refer the candidate to a specific part of a multiple page appendix (i.e. Weather Synopsis #xx, METAR). The aim is to assist the candidate to zoom in quickly to the relevant part/page of the appendix, e.g.:

(a) Refer to the VNC.

(b) Refer to the Appendix: TAKE-OFF PERFORMANCE (Graph #3).

(32) If an item contains supplemental information, list the information above the stem. Align the information with the stem, and leave one line space between the supplemental information and the stem, e.g.:

Average fuel consumption 5.7 gph. Elapsed time from X to Y 55 minutes.

Using the above data, the minimum…


2013-07-10 16 of 45 SI 400-006-Y Issue 01

(33) When a note is used within a question, use capital letters, not bold, and a colon. Leave 2 spaces in English, and one space in French, between the colon and the information that follows, e.g.:

NOTE: Use an average fuel burn of 13.5 gph for cruise.

(34) When referring to the Canadian Aviation Regulations (CARs), use one of the following formats:

(a) Section 401.42 of the CARs;

(b) CAR 401.42; or

(c) CARs require that an operator...

(35) Format weight and balance. Use the following examples to format questions with weight and balance supplemental information:

(a) Sample No. 1:

Item Weight lb Moment in. lb Pilot 180 — Passengers 260 — Fuel 20 — Oil 20 — Baggage 40 —

Using the above…

(b) Sample No. 2:

*includes oil

Calculations based on the above table…

4.6 Quality review of questions

(1) Once questions have been developed and, when possible, checked by an examination specialist other than the one who developed the question, they are presented to a review committee.

(2) The review committee consists of three to five specialists (preferably specialists who work on aviation examinations or in the applicable subject area) convened for the purpose of giving feedback and ensuring the quality of the question.

(3) When possible, proposed questions should be distributed to participants at least the day before the scheduled review committee. Where possible, questions should be accompanied by copies of the reference material.

(4) The review committee is convened to review questions, not examinations.

(5) Review committee sessions should not exceed 2 hours.

Load Details: Weight lb

Arm in.

Moment in. lb

Aeroplane weight empty* 3,000 − 140,200 Pilot and front Passengers

370 30 −

Rear seat passengers 380 − 26,000 Fuel main tank − 60 5,400


2013-07-10 17 of 45 SI 400-006-Y Issue 01

(6) Print enough copies for each participant. The computer can be used in the boardroom to display the question on a screen instead of using hard copies. Bring all the necessary books, materials and equipment.

(7) Proposed questions submitted to the committee for approval include their references, deficiency statement and appendix. (A rough copy is sufficient).

(8) The committee meets to approve each question and deficiency statement on the basis of technical and editorial accuracy. They also check the accuracy of the answers, the validity of the distractors, the references, the MCI code and the category of examination in which the question can be used.

(9) During the review, if, after a maximum of 15 minutes of discussion on a given question, significant progress has not been made, the question is to be discarded or returned to the originator for further work. However, if a consensus has almost been reached and a usable question is imminent, an additional 10 minutes may be allotted subject to the concurrence of the group.

(10) If a problem with a question is identified, it should be resolved with a minimum of discussion. If full agreement cannot be obtained but two of the three specialists are in agreement, then the question can be accepted.

(11) Provided the question meets all the principles of good multiple-choice questions, participants must be prepared to accept the consensus of the group. Excessive discussion of minor points will probably result in a rejected or returned question.

4.7 Approve questions

(1) When the question is accepted as suggested by the review committee, the originating examination specialist writes in the “Remarks” page of or the question: Review committee done on (date), by (name/initials).

(2) The originating inspector submits the approved question to an independent specialist for final approval. That independent specialist gives final approval to the question and deficiency statement by writing in the “Remarks” page of the question “Final approval” on (date), by (name/initials) and by changing the question status box on the text page to “Locked”.

(3) When the independent specialist does not final approve the questions for any reason, the question is returned to the originator with appropriate suggestions. Depending on the work required, the question is either resubmitted to the review committee or is returned for final approval.

4.8 Translate questions

(1) Approved questions and deficiency statements are saved on an USB key in a word document, the USB key or a hard copy is brought directly in the Translation Coordinator inbox in two envelopes.

(2) Send the Service Request Form along with the Word Document and any bilingual references where possible. The Service Request Form can be found on the Transport Canada Web: Forms # 26-0804 Download the Service Request Form, fill in the information and send it electronically to the Translation Services at: AARCD Translation / Traduction AARCD.

(3) When the originating inspector receives the translated questions from translation, the inspector has the translation entered into FTAE.

(4) French vetting by the terminology specialist is done during the French vetting of the examination approval process.


2013-07-10 18 of 45 SI 400-006-Y Issue 01

5.0 AMENDING ACTIVE QUESTIONS OR DEFICIENCY STATEMENTS

(1) Active questions will not normally be amended.

(2) Active questions or deficiency statements can be amended only where spelling, formatting, translation, coding, or numerical changes that do not affect the outcome are made. The question does not need to go to the review committee as it has already been approved and vetted. When the active question can be amended, the examination officer corrects the question or the deficiency statement directly in the “Active”status. If it is a translation amendment, the correction is done by the terminology specialist. The correction must be checked by at least one other examination specialist for accuracy.

(3) Questions where answers/distractors are rearranged or rotated must be copied to a new question ID and will require review by at least one other examination specialist prior to being changed to an active question.

(4) A question with other changes need to be copied to a new question ID and will need to be approved in the review committee.

6.0 INACTIVATING QUESTIONS

(1) Sometimes a question must be inactivated because of a change in the regulation, a change in a procedure or the question having two good answers.

(2) No question shall be inactivated without the agreement of every inspector whose active examination contains the question.

(3) The examination officer inactivates the question in FTAE and indicates the reason why the question has been inactivated in the “remarks tab”.

(4) Once a question has been inactivated, it cannot be re-activated. If this question could be useful, it can be copied as a new question ID.

(5) The instruction pages of the affected examinations shall be updated to show the number of questions that remain active (e.g.: 99 questions instead of 100).

(6) An email is sent to the regional contacts and a “CC” to the examination specialists that have this question on their examinations. A template of the email can be use by searching in Records, Documents and Information Management System (RDIMS) under document name: “E-MAIL – NOTICE OF INACTIVE QUESTIONS / QUESTIONS INACTIVES”

(7) The email is saved in RDIMS under the name E-MAIL – NOTICE OF INACTIVE QUESTIONS / QUESTIONS INACTIVES” and the RDIMS number is indicated in the “Amend Rec” in FTAE in each affected examination.

(8) The question is then stamped “inactive” in the black binder paper copy. A copy of the email is inserted in the correspondence tab of that binder and a note is written on the amendment sheet.

(9) Replace the inactivated question in the examinations. Every examination that is not at full capacity shall be re-established at its maximum capacity as soon as possible.

(10) A new version of the examination shall be developed as per the procedure set in this Staff Instruction (SI)

7.0 DELETING QUESTIONS

(1) Questions should be kept for at least 5 years after being inactivated, as Federal Government policy.


2013-07-10 19 of 45 SI 400-006-Y Issue 01

(2) Questions that are inactivated or in DEV mode for a period over 5 years will be deleted at the beginning of January of each year.

8.0 NEW EXAMINATION SERIES

Figure 2 - Examination development process

8.1 Create an examination plan (Blue Print)

(1) Make an examination plan (blue print) based on the learning objectives (if developed), subject areas and topics required by the appropriate Study and Reference Guide.


2013-07-10 20 of 45 SI 400-006-Y Issue 01

(2) The specialist decides which subject areas and topics should be assessed based on different criteria such as weak knowledge areas, new procedures, new regulations, Transport Security Bureau (TSB) recommendations, or stakeholders” feedbacks.

(3) The examination specialist will strive to have the same level of difficulty as the other examinations of the same title (e.g. Private Pilot, Aeroplane).

8.2 Create a new examination series in Flight Training and Aviation Education (FTAE)

(1) The examination officer is responsible to create new examination series template in FTAE.

(2) A new series will typically replace an older series, which will be inactivated at the same time as the new examination series is activated.

(3) A new examination series is issued when there is a significant change in the older series. Usually more than 15% of the questions will be new to the examination.

(4) On the main menu in FTAE, click the “Exams” icon.

(5) Select “Examination list”.

(6) Select or type the examination name, series and version.

(7) Select (highlight) the examination and press the icon “Copy Exam” on the top menu.

(8) A question will appear asking you if you wish to copy the cover page, instruction page and all the questions with remarks and deficiency statements, select ”Yes”.

(9) You will then be asked for the version #. Enter the desired version # and select “Proceed”.

(10) A new version of the desired examination is then created in “Development” status.

8.3 Assemble the examination

(1) In FTAE click on the “Exams” button on the left top of the page. Select the examination list and write the name, the series and the version of the examination in the fields, or select it from the list, then press enter.

(2) Complete all the examination fields by clicking on the appropriate tab on the right side of the page such as:

• Criteria button: Enter all the fields as displayed.

• Cover page: Enter the English name of the examination in the English tab using Arial 12 and capital letters. Enter the French name using the French tab using the same format.

• Instructions page: Copy instruction page from another examination and change the time permitted, number of questions and marks in the English and French tabs.

(3) Assemble the examination by entering the question’s number (ID) in the “Question Field”. Selectively retrieve questions from the question bank in FTAE or by developing new questions as required.

(4) Keep questions of similar nature (same MCI) together wherever possible. There may be exceptions, such as regulation-related questions in the navigation section.

(5) Balance the number of answers 1, 2, 3 and 4.

(6) Use an examination NOTE between questions to introduce questions relating to a cross-country scenario or a weather scenario. Place the “NOTE” directly above the first question in the series. The word “NOTE” should be capitalized, in bold type, and followed by a colon. Leave a minimum of one line space above and below the note, and leave two spaces in English and one space in French between the colon and the information that follows. For example:


2013-07-10 21 of 45 SI 400-006-Y Issue 01

NOTE: For questions 22 to 40 inclusive, refer to the Appendix CROSS-COUNTRY FLIGHT #28.

(7) Check for duplicate/similar questions. This check can be done easily in FTAE. Every time a new question is entered and saved in the question field, FTAE offers the option to check for duplicate/similar questions.

(8) Complete the written examination approval checklist which is located in G:Drive\AARR\AARRF\Checklists.

8.4 First independent review

(1) Submit the proposed examination, complete with the answer key, references, deficiencies, appendices and the checklist to an examination specialist for an independent review or submit the checklist alone if the examination specialist doesn’t want a hard copy.

(2) In the case of sectionalized examinations, differences between the main examination and the partial examinations should be pointed out on the checklist for the benefit of the examination specialists (reviewers).

(3) The examination specialist checks for conflicting questions and verifies if there is a logical flow to the examination. The level of knowledge should be adequate for the licence and the topics should match the Study and Reference Guide. Some old active questions in FTAE don’t have a deficiency statement therefore, it is important to verify that no deficiency statements are missing. The graphics should match the questions and the references should be checked for accuracy.

(4) The examination specialist makes suggestions, comments or corrections. The suggestions could be entered directly into FTAE in the “REMARKS” field for each individual question, on paper, or on the “written examination approval checklist”. If the suggestions, comments or corrections are entered in the “REMARKS” field of FTAE, the examination specialist writes the question numbers, which require an attention on the “written examination approval checklist”, for the benefit of the originating inspector.

(5) When the independent review is completed, the examination specialist signs the “written examination approval checklist” and returns it to the originating inspector.

(6) The originating inspector makes the appropriate corrections according to the independent review.

8.5 Second independent review

(1) The originating inspector submits the proposed examination to a second examination specialist for a second independent review.

(2) The examination specialist checks for conflicting questions and verifies if there is a logical flow to the examination. The level of knowledge should be adequate for the licence and the topics should match the Study and Reference Guide. Some old active questions in FTAE don’t have a deficiency statement therefore, it is important to verify that no deficiency statements are missing. The graphics should match the questions and the references should be checked for accuracy.

(3) The examination specialist makes suggestions, comments or corrections. The suggestions could be entered directly into FTAE in the “REMARKS” field for each individual question, on paper, or on the “written examination approval checklist”. If the suggestions, comments or corrections are entered in the “REMARKS” field of FTAE, the examination specialist writes the question numbers, which require attention on the “written examination approval checklist”, for the benefit of the originating inspector.

(4) When the second independent review is completed, the examination specialist signs the “written examination approval checklist” and gives it back to the originating inspector.


2013-07-10 22 of 45 SI 400-006-Y Issue 01

(5) The originating inspector makes the appropriate corrections according to the second independent review.

8.6 French vetting

(1) The originating inspector submits the new examination to the terminology examination specialist, who then verifies each “Locked” question on the basis of French/English editorial accuracy. The deficiencies and appendices should also be verified for French/English editorial accuracy.

(2) The terminology examination specialist changes the status of questions from “Locked” to “Active”, clears the “Check French” box, and removes any French comment in the remarks section.

(3) If required, the terminology specialist coordinates with the originating inspector and the appendix specialist to make the appropriate corrections.

(4) The terminology specialist signs the “written examination approval checklist” and gives the examination to the examination officer.

8.7 Format the examination

(1) The formatting of a new examination series is done by the examination officer.

(2) The officer checks the cover page to make sure it is the correct examination name and verifies the formatting.

(3) The officer checks the instruction page for accuracy and formatting.

(4) The officer formats each question as per this SI (Format questions).

(5) The officer assures that the date on the examination matches the date on the appendix (only the month is displayed on the appendix).

(6) The officer asks the appendix specialist to date the appendix and to make a Portable Document Format (PDF) version with a password security as per this SI (Assemble an appendix) and saves it in the G:Drive\AARR\AARRF\Appendices.

(7) The officer dates the examination. The date shall be after the examination final approval date and shall match the date on the email.

(8) The officer sets FTAE examination date for automatic activation.

(9) The officer prints one complete set of original examinations (French and English) with appendix, references, deficiency statements, and answer key.

(10) The officer initials the “written examination approval checklist”.

(11) The officer prepares an email to be sent to the regional contacts. This email has to be initialled by the originating inspector before being sent to the regional contacts.

(12) The officer prepares a notice to Approved Examination Invigilators (AEIs) if a new appendix is available for printing and use.

8.8 Approve examination

(1) The complete set of examinations, with the email and the “written examination approval checklist” is submitted to the originating inspector for approval.

(2) The originating inspector executes the final quality review of the examination (French and English), appendices (French and English), references, deficiency statements, and answer key.

(3) The “written examination approval checklist” and the email are initialed and returned to the examination officer for activation.


2013-07-10 23 of 45 SI 400-006-Y Issue 01

8.9 Activate the examination

(1) The activation of a new examination series is done by the examination officer.

(2) The examination officer assures that the new or updated appendix (in Adobe Acrobat format) is in the MS Outlook folder.

(3) An email is sent to the regional contacts and “CC” to the examination specialist, one day prior to the activation of the examination if the examination has no appendix. The examination officer may allow 3 to 4 days advance notice if this examination requires printing of the appendix for their inventory. A template of the email can be use by searching in RDIMS under document name: “E-MAIL – NOTICE OF NEW EXAMINATION*. The email is saved in RDIMS under the name: E-MAIL – NOTICE OF NEW EXAMINATION – (NAME OF THE EXAMINATION)

(4) The distribution list for regional contacts consists of all FTAE users that have “CDE” access in FTAE. Other people, like managers may wish to be added to the list so they can be informed of all examination updates. This request is normally done by them sending the examination officer an email.

(5) All FTAE users with Computer Delivered Examinations (CDE) access should also have access to the appendices in MS Outlook folder. Their names are added by: MS Outlook – pf ott Apendices (reviewers) – modify members – add – select or type in name – the name appears in the “Add” line at the bottom – click OK (3 times).

(6) If the examination is used by AEIs and a new appendix is required, the examination officer may have to retain the examination until the appendix is available on the web. Then the examination officer sends a notice to AEIs via FTAE to advise them that a new appendix is available for printing. The examination officer allows 3 to 4 days advance notice to AEIs to print the appendix for their inventory.

(7) The new examination series is automatically activated in FTAE at 00:01 EST on the date specified in the email.

(8) Any examinations that are replaced are automatically inactivated at the same time.

(9) The examination officer initials the “written examination approval checklist”.

8.10 Examination Storage

(1) The paper copy of the full set of examinations is stored in the black binder in the cabinet located in the library. All information pertaining to the examination is recorded on the amendment sheet located at the front of the binder.

(2) The email is stored in RDIMS and the RDIMS # is written in the amendment sheet.

9.0 NEW EXAMINATION VERSION

(1) A new examination version should not change more than 15% of the questions from the original version. A new version is used to replace problem questions or fine-tune an examination.

(2) A new version will replace an older one, which will be automatically inactivated at the same time that the new examination version is activated.

9.1 Create a new examination version in Flight Training Aviation Education

(1) The examination officer is responsible to create new examination version template in FTAE.

(2) On the main menu in FTAE, click the “Exams” button.


2013-07-10 24 of 45 SI 400-006-Y Issue 01

(3) Select “Examination list”.

(4) Select or type the examination name, series and version.

(5) Select (highlight) the examination and press the button “Copy Exam” on the top menu.

(6) A question will appear asking you if you wish to copy the cover page, instruction page and all the questions with remarks and deficiency statements, select ”Yes”.

(7) You will then be asked for the version #. Enter the desired version # and select “Proceed”.

(8) A new version of the desired examination is then created in “Development” status.

9.2 Replace questions

(1) In the FTAE main menu, click on the “Exams” button located on the top left corner. Select “examination list” and select or type in the name, the series and the version of the examination in the fields, or select it from the list, then press enter.

(2) Click on the question list tab on the right side of the screen.

(3) Select approved questions from the question bank in FTAE or develop new questions as required to replace the questions that need to be changed.

(4) Enter the new question’s ID over top of the question that needs to be replaced in the “Question” ID Field.

(5) When you press ”Save”, FTAE automatically offers to check for duplicate/similar questions. It is strongly recommended to do a duplicate/similar check at least once before releasing the examination.

(6) Ask an appendix specialist to update the appendix (French and English), if required for this new examination version. When completed, the appendix specialist initials the checklist. (See the process to amend an appendix).

(7) If no change is required in the appendix, select the box “keep existing appendix” on the checklist and the appendix specialist’s signature is not required.

(8) Complete the “written examination approval checklist” located on the G:ºDrive\AARR\AARRF\Checklists.

9.3 Independent review

(1) Submit the proposed new examination version, complete with answer key, references, deficiencies and appendices to an examination specialist for an independent review, or submit the written examination approval checklist alone if the examination specialist doesn’t want a hard copy.

(2) In the case of sectionalized examinations, differences between the main examination and the partial examination should be pointed out on the checklist for the benefit of the examination specialist (reviewer).

(3) The examination specialist skims the whole examination and pays more attention to new questions, checks for conflicting questions and verifies if there is a logical flow to the examination. The level of knowledge should be adequate for the licence and the topics should match the Study and Reference Guide. Some old active questions in FTAE don’t have a deficiency statement therefore, it is important to check that no deficiency statement is missing. The graphics should match the questions and the references should be checked for accuracy.

(4) The examination specialist makes suggestions, comments or corrections. The suggestions could be entered directly into FTAE in the “REMARKS” field for each question, on paper, or on the


2013-07-10 25 of 45 SI 400-006-Y Issue 01

checklist. If the suggestions, comments or corrections are entered in the “REMARKS” field of FTAE, the examination specialist writes the question IDs that require correction on the checklist for the benefit of the originating inspector (OPI).

(5) When the independent review is completed, the examination specialist initials the “Written Examination Approval Checklist” and returns it to the originating inspector.

(6) The originating inspector makes the appropriate corrections according to the independent review.

9.4 French vetting

(1) If only active questions are used, the terminology examination specialist’s signature is not required.

(2) The originating inspector submits the new examination to the terminology examination specialist, who then verifies each “Locked” question on the basis of French/English editorial accuracy. The deficiencies and appendices should also be verified for French/English editorial accuracy.

(3) The terminology specialist changes the status of questions from “Locked” to “Active”, clears the “Check French” box, and removes any French comment or suggestions in the remarks section.

(4) If required, the terminology specialist coordinates with the originating inspector and the appendix specialist to make the appropriate corrections.

(5) The terminology specialist signs the checklist and gives the examination to the examination officer.

9.5 Format the examination

(1) The formatting of a new examination version is done by the examination officer.

(2) The examination officer checks the cover page to make sure it is the correct examination name and verifies the formatting.

(3) The officer checks the instruction page for accuracy and formatting.

(4) The officer checks “notes” attached to the examination.

(5) The officer formats each question as per the process included in this SI (Format questions).

(6) The officer assures that the date on the examination matches the month on the appendix.

(7) The examination officer ask the appendix specialist to date the appendix and make a PDF version with a password security as per the process included in this SI (Assemble an appendix) and saves it in the G:Drive\AARR\AARRF\Appendices.

(8) The date on the examination shall be the day after the examination final approval and shall match the date on the email.

(9) FTAE will automatically activate the examination at 00:01 on the date specified.

(10) The officer prints one complete set of original examination (French and English) with appendix, references, deficiency statement, and answer key.

(11) The officer initials the checklist.

(12) The officer prepares an email to be sent to the regional contacts. The examination officer has the POI initial the email before sending it to the regional contact.


2013-07-10 26 of 45 SI 400-006-Y Issue 01

9.6 Approve the examination

(1) The complete set of examinations with the email and the checklist is submitted to the originating inspector for final approval.

(2) The originating inspector executes the final quality review of the examination (French and English), appendices French and English), references, deficiency statements, and answer keys.

(3) The checklist and the email are initialled and returned to the examination officer for activation.

9.7 Activate the new examination version

(1) The activation of a new examination version is done by the examination officer.

(2) The examination officer assures that the new or updated appendix (in Adobe Acrobat format) is in the MS Outlook folder.

(3) The email is sent to the regional contacts and “CC” to the examination specialist, one day prior to the activation of the examination if the examination has no appendix. The examination officer may allow 3 to 4 days advance notice if this examination requires printing of the appendix for their inventory. A template of the email can be use by searching in RDIMS under document name: “E-MAIL – NOTICE OF NEW VERSION”. The email is saved in RDIMS under the name: “E-MAIL – NOTICE OF NEW VERSION – (NAME OF THE EXAMINATION)”.

(4) The distribution list for regional contacts consists of all FTAE users that have “CDE” access in FTAE. Other people, like managers may wish to be added to the list so they can be informed of all examination updates. This request is normally done by them sending the examination officer an email.

(5) All FTAE users with “CDE” access should also have access to the appendices in MS Outlook folder. Only the examination officer can add new names to the list. Their names are added by: MS Outlook – pf ott Apendices (revewers) – modify members – add – select or type in name – the name appears in the “Add” line at the bottom – click OK (3 times).

(6) If the examination is used by AEIs and a new appendix is required, the examination officer may have to retain the examination until the appendix is available on the web. Then the examination officer sends a notice to AEIs via FTAE to advise them that a new appendix is available for printing. The examination officer allows 3 to 4 days advance notice to AEIs to print the appendix for their inventory.

(7) The new examination version is automatically activated in FTAE at 00:01 EST on the date specified in the email.

(8) Any examinations that are replaced are automatically inactivated at the same time.

(9) The examination officer initials the “written examination approval checklist”.

9.8 Examination Storage

(1) The paper copy of the full set of examinations is stored in the black binder in the cabinet located in the library. All information pertaining to the examination is recorded on the amendment sheet located at the front of the binder.

(2) The email is stored in RDIMS and the RDIMS # is written in the amendment sheet.


2013-07-10 27 of 45 SI 400-006-Y Issue 01

10.0 ANALYSIS OF EXAMINATION PERFORMANCE

(1) A first analysis of a new examination is conducted when there are sufficient examinations completed to provide useful indications. This analysis may be conducted as soon as ten examinations have been written. It provides a quick indication of an issue with a question.

(2) The analysis is done using Ease Index and Discrimination Index available in FTAE. Click the “Exams” button, select “Analysis”. Choose the examination, click reports and select “Question Analysis by Examination”.

(3) The Ease Index in FTAE is a ratio between 0.0 and 1.0 that indicates the proportion of candidates who answered the question correctly. In some other institutions, it might be called the Difficulty Index. It is a ratio of total correct choices divided by the number of candidates – for example, if 15 of 20 candidates chose the correct response, the Ease Index would be .75, or 75%.

(4) The Discrimination Index in FTAE is a comparison of the performance of strong and weak candidates, and gives an indication of how the question discriminates between the most knowledgeable and the least knowledgeable candidates. It indicates whether the candidates who answered the question correctly were among the top performers on that exam (in the case of FTAE, the top 50% of exam candidates). If those who answered correctly were mostly among the top performers, the number will be positive, between 0.0 and 1.0. If those who answered correctly were mostly among the bottom performers, the number will be negative.

(5) Flight Crew examinations are designed to be criterion referenced, indicating a satisfactory level of knowledge, not norm referenced which aim to discriminate the more knowledgeable from less knowledgeable candidates. As such, the Discrimination Index is only used to signal potential problems with the question.

(6) An Ease or Discrimination Index may be obtained for the question’s performance on every examination on which the question is used – the results may vary from examination to examination.

(7) Neither index is an absolute indicator of a problem question; they serve only to signal that an investigation into unusual numbers is required. A low Ease Index may indicate a question problem or a knowledge problem. The index of similar questions may help the investigation. A large negative Discrimination Index may signal a problem with the question, but the magnitude of the index is dependent on the Ease Index (if no-one answers correctly, the discrimination will be zero).

(8) Verify the Ease Index and Discrimination Index for each question. Every time the Ease Index is less than 50% or the Discrimination Index is a negative value, the question is reviewed and revised as required.

(9) If a question is found not to be in accordance with the procedure set in this SI, the question should be amended or inactivated as per the process described herein.

(10) Another analysis is conducted to compare the level of difficulty between the new examination and the other examinations of the same category. This analysis may be conducted after one hundred examinations are completed.

(11) This analysis is done using the “Average score pass rate” report for each examination of the same type.

(12) If the examination is found to be harder or easier than the other examinations of the same type, the examination shall be adjusted to the same level of difficulty and a new version shall be issued.


2013-07-10 28 of 45 SI 400-006-Y Issue 01

11.0 REVIEW A CHALLENGED QUESTION

(1) On occasion, a candidate may decide to challenge a question’s validity. A candidate usually challenges a question directly to the examination clerk or to the AEI after writing the examination.

(2) The regional examination clerk forwards the question with the appendix (if required) to a Regional Inspector for verification or,

(3) The AEI forwards the name of the examination and the number of the question to the Regional Transport Canada Office for verification by a Regional Inspector.

(4) The Regional Inspector reviews the question and if an issue is found, the issue is forwarded to Headquarters.

(5) The HQ examination specialist reviews the question on the basis of technical and editorial accuracy, and checks the accuracy of the answers, the validity of the distractors and the references. An analysis is conducted in FTAE to verify how the question performs nationally using Ease Index and Discrimination Index. Other examination specialists may be consulted.

(6) If the question is found to have an issue, the question is amended or inactivated as per the process included in this document.

(7) The candidate’s result may be corrected as per the process included in SI – 400-003 (RDIMS 2309934), Administering Examinations for Flight Crew, Chief Pilots, Operations Managers and Dispatcher.

12.0 INACTIVATE AN EXAMINATION

(1) An examination is automatically inactivated when it is replaced by a new version.

(2) In some cases an examination is inactivated when it has been compromised due to cheating.

(3) The examination officer will inactivate the examination in FTAE and enter the information – date and reason for inactivation in the “Amend” tab in FTAE.

(4) The examination officer advises the Regions by email.

(5) The examination officer saves the e-mail in RDIMS and records the RDIMS # in the “Amend” tab.

(6) The examination officer removes the examination from the cabinet and disposes of it.

13.0 GRAPHICS

(1) When a graphic is not already designed, an examination specialist competent in using the CorelDRAW software can create one.

(2) The appendix coordinator appoints the design of the graphics to an appendix specialist according to their availability. CorelDRAW software and Adobe Acrobat (full version, not just Reader) is required.

13.1 Design graphics

(1) Get a hard copy or an electronic copy of the graph, diagram, chart or table that needs to be designed.

(2) Open the G: Drive\AARR\AARRF\Graphics, and find a similar graphic. This will give you the general size of the outline box, text, format etc. Use standard format 8x14 inches. Save this as your new graphic file in the G: Drive\AARR\AARRF\Graphics with the new graphic number. The


2013-07-10 29 of 45 SI 400-006-Y Issue 01

new graphic number is given by the FTAE coordinator and is based on the FTAE Graphic Database. Choose an appropriate title in consultation with the FTAE coordinator.

(3) Ungroup the graphic and edit the title, keeping the box and deleting the existing graphics. Edit the graphic number at the button.

(4) Design the graphic according to the CorelDRAW training manual.

(5) When the graphic is ready, the POI executes the quality review on the basis of technical and editorial accuracy and provides comments, suggestions or corrections to the appendix specialist.

(6) The appendix specialist corrects the graphics.

(7) The appendix specialist brings a hard copy of the Graphic, along with the Service Request Form and any bilingual references where possible, to the Translation Coordinator inbox. The Request Form can be found on the Transport Canada Web: Forms # 26-0804 Download the Request Form and fill in the information. The Request Form can also be sent electronically along with a text document to the Translation Services at: AARCD Translation / Traduction AARCD.

(8) When the appendix specialist receives the translation, the appendix specialist enters the translation into the graphic and saves it in the G: drive as a new document with the same number but with the letter “E” for English and “F” for French. E.g.: 0402E or 0402F.

(9) The appendix specialist make a PDF version of the English and French graphics and saves them in the G:Drive\AARR\AARRF\Graphics.

13.2 Design approach plates – Canada Air Pilot (CAP)

(1) Go the Transport Canada Technical Reference Centre (http://tcinfo2/dbtw-wpd/navcane.htm) – NAV CANADA Publications.

(2) Open the most recent version of the Electronic Publications.

(3) Select the appropriate CAP and save the document to your computer’s desktop. This will be an Acrobat file. Quit the internet browser.

(4) Open the CAP document on your desktop (in Acrobat) and “extract” the desired approach plate. Save that page/plate to your desktop.

(5) Open a CorelDRAW graphic already used as a CAP graphic (such as 0096 or 0100). This will give you the general size of the outline box, text etc.

(6) Ungroup this graphic, edit the title, graphic number, keep the outline box, and delete the existing CAP graphic. The naming format of approach plate graphics is in a Word document in the “…AARRF\Procedures” folder.

(7) Import your new CAP graphic (the Acrobat file that you just saved on your desktop). Use Arial fonts to replace missing fonts such as Swiss font. Match the plate’s size to the typical size using the guidelines in CorelDRAW.

(8) Touch up text (watch for bold, italics and line attributes, check the dashes on the overshoot lines). Consult a real copy of the CAP for guidance. Check that characters (such as the “symbol for feet) have not been lost or changed.

(9) Delete the “Effective” dates and “Change” remarks. Add the note “FOR EXAMINATION PURPOSES ONLY” (AUX FINS D’EXAMENS SEULEMENT) in the lower, left corner.

(10) Approach plates from Quebec are available in French and should be imported from the Transport Canada Virtual Library using the French documents. Regardless of the published language, the appendix specialist will create both an English version (number XXXXE) and a French version (number XXXXF), even if only the title and notation “appendix” change languages between the English and French graphics.


2013-07-10 30 of 45 SI 400-006-Y Issue 01

(11) Approach plates themselves should not be translated if they do not appear in the other language.

(12) When the graphic is ready, the examination specialist (who requested the approach plate graphic) executes the quality check on the basis of technical and editorial accuracy and provides comments, suggestions or corrections to the appendix specialist.

(13) The appendix specialist corrects the graphic if required.

(14) The appendix specialist saves a PDF version of the English and French graphics in the same G drive folder.

13.3 Design aerodrome directory/facility pages - Canada Flight Supplement

(1) Go the National Aircraft Certification Technical Reference Centre http://tcinfo2/dbtw-wpd/navcane.htm NAV CANADA documents – Electronic Publications CD.

(2) Open the most recent version of the CD ROM online.

(3) Select the Canada Flight Supplement (CFS), and save the document to your computer’s desktop. This will be an Acrobat file. Quit the internet browser.

(4) Open the CFS document on your desktop (in Acrobat) and “extract” the desired aerodrome page. Save that page to your desktop.

(5) Open a CorelDRAW graphic already used as a CFS graphic (such as 0251, 0326 or 0482). This will give you the general size of the outline box, text, etc.

(6) Ungroup this graphic, edit the title, graphic number, keep the outline box, and extracted page that you just saved. As an alternative, you can just import the airport diagram and edit the old graphic text. Save this graphic immediately under the number/name of your new graphic.

(7) Import your new CFS graphic (the Acrobat file that you just saved on your desktop) – you can import the entire page, or just the aerodrome diagram. Match the CFS page size to the typical size using the guidelines in CorelDRAW.

(8) Touch up the text and line attributes if necessary. Consult a real copy of the CFS for guidance. Check that characters (such as the symbol for daylight savings times) have not been lost or changed.

(9) Aerodrome pages from Quebec are available in French and should be imported from the Transport Canada Virtual Library using the French document. CFS pages will be translated if they do not appear in the other language.

(10) When the graphic is ready, the examination specialist (who requested the CFS graphic) executes the quality check on the basis of technical and editorial accuracy and provides comments, suggestions or corrections to the appendix specialist.

(11) The appendix specialist corrects the graphic if required.

(12) The appendix specialist saves a PDF version of the English and French graphics in the same G: drive folder.

13.4 Amend a graphic

(1) Before requesting that a graphic be amended, the POI must review the questions that it is used with. If the change impacts active questions, steps must be taken to correct any impact.

(2) Open the CorelDRAW graphic in G: Drive\AARR\AARRF\Graphic.

(3) Amend the graphic in both languages and save the amendment.


2013-07-10 31 of 45 SI 400-006-Y Issue 01

(4) The graphic is then passed to the examination specialist (POI) for a quality review. The POI provides comments, suggestions or corrections to the appendix specialist.

(5) The appendix specialist corrects the graphic as required in the CorelDraw version.

(6) Make a new PDF version of the graphic (English and French) and save it in the G: Drive\AARR\AARRF\GRAPHICs.

14.0 APPENDICES

(1) Appendices are made available to the regions through Transport Canada’s intranet, in a MS Outlook folder specific for this purpose. Access to the Appendices folder is controlled in HQ. When a new examination user is given the authority to print/view examinations, they are also to be given access to the Outlook folder containing Appendices. The list of people with access to that Outlook folder is named “PF OTT Appendices (Reviewers)”, and can be found in the Global Address List. Only few people have privileges that allow changing the members of that list.

(2) Authorized Examination Invigilators (AEIs) in the industry also use appendices. All the appendices are included in a booklet that is accessible to the AEIs through the GALRO web site.

(3) Although not a Protected B document, it is expected that appendices will be treated in the same manner as Protected B documents in terms of physical storage and disposal. Adobe Acrobat versions of the appendix documents have security setting that allows only printing of the document to prevent copying or tampering with the document.

(4) The appendix coordinator appoints the assembly of an appendix to an appendix specialist according to their availability. CorelDRAW software and Adobe Acrobat (full version, not just Reader) is required.

(5) Prior to creating or amending an appendix used by AEIs, coordination shall be done with the appendix coordinator. The production of new AEI appendix should be limited to once a year to minimize the impact on the AEIs.

14.1 Assemble an appendix

(1) In FTAE, click on the “exam” button on the left top of the page. Select the examination by typing the name, the series and the version of the examination in the fields. e.g.: PPAER 020 01or select the exam from the list, and “enter”.

(2) Click on “Reports” and select “Examination Appendices”. This report gives all the graphics that should be included in the appendix for that particular examination.

(3) In CorelDRAW, open an existing appendix that is used for the same type of examination in G: Drive\AARR\AARRF\Appendices. Use a French appendix to assemble a new French appendix and, an English appendix to assemble an English appendix. This will give you the general size of the outline box, text etc.

(4) Save this as your new appendix file in the G: Drive\AARR\AARRF\Appendices. The title should be the name of the examination with the series and E for English and F for French at the end. e.g.: PPAER 016E. Old files can be archived in sub-folders.

(5) Edit the front page and edit the title/header on each page.

(6) Delete the old graphics on each page and import the new graphics. If possible, the order of the graphics in an appendix should be the same as the order on the Appendices Report. Complete the table of contents.

(7) The appendix should be assembled in both languages.


2013-07-10 32 of 45 SI 400-006-Y Issue 01

(8) The appendix is then passed to the examination specialist (OPI) for a quality review. The OPI provides comments, suggestions or corrections to the appendix specialist.

(9) The appendix specialist corrects the appendix as required in the CorelDraw versions.

(10) When the examination officer is ready to release a new examination, the examination officer asks an appendix specialist to date the appendix with the same month as the examination. The appendix specialist will make a PDF version of both English and French appendices and save them in the G: Drive\AARR\AARRF\Appendices - beside the CorelDRAW version.

(11) Security is set on the PDF version with “only printing allowed”. The password should be the library locker combination, and the PDF filed saved beside the CorelDRAW version.

(12) When a new AEI appendix is issued, the appendix coordinator shall amend the appendix booklet on the GALRO web site and advise the AEIs via a notification on the GALRO web site. Refer to the “Industry Invigilation of Computer Delivered Examinations” RDIMS #44522.

14.2 Amend an appendix

(1) Open the CorelDRAW appendix in G: Drive\AARR\AARRF\Appendices.

(2) Change the date on the front page to the month and year of amendment/issue.

(3) Import the new graphics. Amend the appendix in both languages.

(4) Edit the table of contents, if required.

(5) The appendix is then passed to the examination specialist (OPI) for a quality review and provides comments, suggestions or corrections to the appendix specialist.

(6) The appendix specialist corrects the appendix as suggested by the examination specialist and makes a PDF version of both English and French appendices and saves it in the G: Drive\AARR\AARRF\Appendices - beside the CorelDRAW version.

(7) Security is set on the PDF version with “only printing allowed”. The password should be the library locker combination, and the Acrobat file is saved beside the CorelDRAW version.

(8) An email is sent to the Regional offices to advise them to reprint the new appendix. The new Acrobat version is copied to the MS outlook appendices folder. A template of the email can be use by searching in RDIMS under document name: “E-MAIL – NOTICE OF NEW APPENDIX”. The email is saved in RDIMS under the name: E-MAIL – NOTICE OF NEW APPENDIX/NOUVEL APPENDICE – (NAME OF THE APPENDIX)

(9) When a new AEI appendix or an amendment appendix is issued, the appendix coordinator shall amend the appendix booklet on the GALRO web site and advise the AEIs via a notification on the GALRO web site.


2013-07-10 33 of 45 SI 400-006-Y Issue 01

15.0 STUDY AND REFERENCE GUIDE

15.1 Review knowledge requirements

(1) Convene a working group from industry associations and interested/affected individuals within Transport Canada.

(2) Establish knowledge requirements and the format of the Study and Reference Guide (learning objectives, topic list or sample questions).

15.2 Obtain a Transport Canada Publication number

(1) Transport Canada Publication’s (TP) numbers are assigned by Multimedia Products and Services (AARA-MPS).

(2) They are requested and assigned through the Product Development Checklist (PDC) form 26-0656.

(3) Keep the title consistent with other Study and Reference Guides.

15.3 Create draft

(1) Create the new guide consistent with the look of existing Study and Reference Guides.

(2) Save the document in RDIMS. RDIMS is the permanent storage location for Study and Reference Guides. The title begins with the TP number of the document.

15.4 Review by working group

(1) The Study and Reference Guide is reviewed to confirm the intent has been met and to include any new items deemed necessary.

15.5 Fine tune

(1) The Study and Reference Guide is finalized, including contact information and reference material.

(2) Complete the “Study and Reference Guide approval checklist” located on the G: Drive\AARR\AARRF\Checklist.


(1) An examination specialist verifies the Study and Reference Guide on the basis of technical and editorial accuracy. The level of knowledge should be adequate for the licence, permit, rating and position.

(2) The examination specialist makes suggestions, comments or corrections.

(3) The examination specialist signs the “Study and Reference Guide approval checklist” and returns it to the POI.

(4) The POI makes the appropriate corrections according to the independent review.

15.7 Translation

(1) The examination specialist brings the Study and Reference Guide along with the Service Request Form and any bilingual references where possible in the Translation Coordinator inbox. The Request Form can be found on the Transport Canada Web: Forms # 26-0804 Download the


2013-07-10 34 of 45 SI 400-006-Y Issue 01

Request Form and fill in the information. The Request Form can also be sent electronically to the Translation Services at: AARCD Translation / Traduction AARCD.

(2) When the examination specialist receives the translated Study and Reference Guide from translation, the examination specialist then formats it and saves it as a separate RDIMS document.

(3) The Study and Reference Guide is then sent to the terminology specialist for vetting. If required the terminology specialist coordinates with the examination specialist to make appropriate corrections.

(4) The examination specialist makes the corrections suggested by the terminology specialist.

(5) The terminology specialist signs the “Study and Reference Guide” approval checklist.

15.8 Study and Reference Guide format

(1) The examination officer verifies the format.

(2) The examination officer removes the OPI’s edit rights in RDIMS to prevent any changes to the document during the AARA-MPS work. If the OPI wants to start working on a new version of the Study and Reference Guide, it shall be under another RDIMS number.

(3) The formatted Study and Reference Guide is submitted to the OPI for approval

(4) The OPI executes the quality review of the guide in both French and English.

(5) The OPI signs the checklist and returns it to the examination officer.

15.9 Post on the Internet and Intranet

(1) The examination officer sends an email with the RDIMS references for both the French and English Study and Reference Guides, and the appropriate web pages links, to the web contact for posting.

(2) Once the confirmation is received from the web team that the new Study and Reference Guides have been posted on the web, the examination officer imports the AARA-MPS version into RDIMS and marks it as a final document.

(3) The OPI proofreads the web copies for accuracy.

(4) An email is sent to advise the Regions that a new Study and Reference Guide has been issued.

16.0 AMEND STUDY AND REFERENCE GUIDE

16.1 Download working copy from RDIMS

(1) The examination specialist, who is the OPI for the Study and Reference Guide, makes a copy of the last approved Study and Reference Guide located in RDIMS and saves it as a new RDIMS number.

(2) The RDIMS number of the previous version shall be entered as a cross-reference.

16.2 Amend working copy

(1) The working copy is amended by the examination specialist, including updating web links and reference materials. It may be helpful to have MS Word track the changes made.

(2) Regional Inspectors who work in related fields can be consulted if necessary.


2013-07-10 35 of 45 SI 400-006-Y Issue 01

(3) The “Study and Reference Guide Approval Checklist” located on the G: Drive\AARR\AARRF\Checklist is completed and given to another examination specialist.


(1) Submit the proposed amended copy of the Study and Reference Guide to an examination specialist for an independent review on the basis of technical and editorial accuracy. The level of knowledge should be adequate for the licence. The examination specialist makes suggestions, comments or corrections.

(2) The examination specialist checks and signs the “Study and Reference Guide Approval Checklist” and returns it to the OPI.

(3) The OPI makes the appropriate corrections according to the independent review.

16.4 Translation

(1) The examination specialist sends the Study and Reference Guide along with the Service Request Form and any bilingual references where possible to the Translation Coordinator inbox. Send the Request Form along with the Word Document and any bilingual references where possible. The Request Form can be found on the Transport Canada Web: Forms # 26-0804 Download the Service Request Form and fill in the information. The Request Form can also be sent electronically to the Translation Services at: AARCD Translation / Traduction AARCD.

(2) When the examination specialist receives the translated Study and Reference Guide from translation, the examination specialist then formats it and saves it as a separate RDIMS document.

(3) The Study and Reference Guide is then sent to the terminology specialist for vetting. If required the terminology specialist coordinates with the examination specialist to make appropriate corrections.


(5) The terminology specialist signs the Study and Reference Guide approval checklist.

16.5 Study and Reference Guide format

(1) The examination officer verifies the format.

(2) The examination officer removes the POI”s edit right in RDIMS to prevent any changes to the document during the AARA-MPS work. If the POI wants to start working on a new version of the Study and Reference Guide, it shall be under another RDIMS number.

16.6 Final approval

(1) The Study and Reference Guide is sent to the OPI for approval.

(2) The quality review is executed by the OPI on both the French and English guides.

(3) The OPI signs the checklist and returns it to the examination officer.

16.7 Post on Intranet and Internet

(1) The examination officer sends an email, with the RDIMS references for both the French and English Study and Reference Guides and the appropriate web page links, to the web contact for posting.


2013-07-10 36 of 45 SI 400-006-Y Issue 01

(2) Once the confirmation is received from the web team that the new Study and Reference Guide has been posted on the web, the examination officer imports the AARA-MPS version in RDIMS and marks it as a final document.

(3) The OPI proofreads the web copies for accuracy.

(4) An email is sent to advise the Regions that a new Study and Reference Guide has been issued.

17.0 ABBREVIATION LIST

(1) A list of abbreviations is a separate supplement that is handed out with written examinations. The abbreviation list is saved in the G: Drive\AARR\AARRF\Appendices\Abbreviations list, and it is published on the flight crew examination web site and available with the appendices in a public “MS Outlook” folder. Make sure that all abbreviations used in an examination are on this list.

(2) The abbreviations list can be updated as required.

(3) When the abbreviations list is updated, the new one shall be saved in RDIMS and the public folder shall also be updated. The Regions and the AEIs shall be notified to print the new version of the abbreviations list.

18.0 WEAK KNOWLEDGE AREA REPORT

(1) This document is produced to inform flight instructors, pilots and student pilots of weak knowledge area on written examinations.

(2) A new report is posted on the Flight Crew Examinations Web Site every year and covers a 12–month period.

(3) The process should begin in January in order to be posted on the web site as soon as possible.

(4) Every inspector is responsible to analyze his or her own examinations and produce a partial part of the report. However, one inspector is in charge of gathering all the partial reports from each inspector and produces a final report.

(5) The examination’s analysis is required and the weak knowledge area report is presented where there are sufficient examinations completed to provide useful results.

18.1 Examination’s Analysis

(1) The examination’s analysis is done via FTAE by left clicking on the “Exams” button and selecting “Analysis”.

(2) Type the name of the examination in the “Exam” field.

(3) Click on the “Reports” button and select “Weak Knowledge Areas”.

(4) Select “Results with MCI Codes and Descriptions”.

(5) Type 50% in the “Ease Index Threshold” field and click on “Produce Report”.

(6) Using the MCI code from the report find the question that is on your examination and scores less than 50%. In order to do that, click on the “Q-Bank” button then “Question”, enter the MCI in the MCI field and click on search. Find which question is on your examination and make an analysis of the question using the “analysis” tab on the right side to see if the question performs less than 50%.


2013-07-10 37 of 45 SI 400-006-Y Issue 01

(7) Confirm that all the questions performing less than 50% are accurate, the phraseology is good, the level of knowledge is appropriate for the licence or rating and the answer and distractors are correct.

(8) If the question has no issues, the question is considered to be a weak knowledge area and will be placed into the weak knowledge area report.

(9) If the question needs to be amended or inactivated, refer to section explaining the process of amending active questions or inactivating questions in this SI.

18.2 Produce the Report

(1) Copy the previous weak knowledge areas report from the Flight Crew Examinations Web Site into a word document.

(2) Update the old report with the new weak knowledge areas.

(3) Provide the inspector in charge of the final report with your examination’s weak knowledge areas analysis.

(4) The inspector in charge of the final report gathers all the inspector’s reports and produces a final report in a word document. The introduction paragraph should be updated with the new date.

(5) The examination specialist sends the weak knowledge areas guide, along with the Service Request Form and any bilingual references where possible, to the Translation Coordinator inbox. The Request Form can be found on the Transport Canada Web: Forms # 26-0804. Download the Request Form, fill in the information and send it electronically to the Translation Services at: AARCD Translation / Traduction AARCD.

(6) When the examination specialist receives the translated weak knowledge areas report from translation, the examination specialist then formats it and saves it as a separate RDIMS document.

(7) The weak knowledge areas report is then sent to the terminology specialist for vetting. If required the terminology specialist coordinate with the examination specialist to make appropriate corrections.


18.3 Weak Knowledge Areas Report format

(1) The working copies (English and French) are e-mailed to the examination officer, preferably with the “track changes” turned off, who then formats the document.

(2) The document is saved in RDIMS version and a remark is added to not make any more changes, as it is the final version.

18.4 Final Approval

(1) The examination officer submits the report to the inspector in charge of the final version for approval.

(2) The inspector executes the final quality review on both the French and English report.

(3) The inspector provides it to the examination officer for its release.


2013-07-10 38 of 45 SI 400-006-Y Issue 01

18.5 Post on Intranet and Internet

(1) The examination officer sends the Weak Knowledge Areas Report to the web contact for upgrading the report from the last year with the new report on the appropriate pages with links from other sites as necessary.

(2) The inspector confirms that the new report is posted on the web and careful proof reading of the results is recommended.

19.0 ANNUAL SELF-PACED STUDY PROGRAM

(1) The annual Self-Paced Study Program is produced by Flight Crew Examinations and appears in the fall Aviation Safety Letter. The process should begin in January or February in order to be ready for the layout of the Safety Letter in the fall.

(2) The object of the program is to update the pilot’s knowledge.

(3) The target audience is recreational Visual Flight Rules (VFR) pilots who have no other regular training program (i.e. no Instrument Flight Rules (IFR) or trans-oceanic topics).

(4) The Self-Paced Study Program typically contains 30 questions which are found in every section of the Transport Canada Aeronautical Information Manual (TC AIM) (GEN, AGA, COM, MET, RAC, FAL, SAR, MAP, LRA, AIR), and 2 questions for each category of aircraft using any references, including possibly the AIM as a reference: aeroplane (including ultra-light), helicopter, gyroplane, glider and balloon). The TC AIM is chosen because it is commonly available and covers a large range of topics.

(5) If possible, topics include significant changes to the TC AIM.

(6) Another source of topics might be the annual examination weak knowledge areas report.

(7) Questions should be different than those used in the previous 2 years.

(8) Each inspector produces a number of short-answer questions in the assigned topics. However, one inspector is in charge of gathering all the questions and producing the final version of the Self-Paced Study Program.

19.1 Develop new questions

(1) Each inspector chooses a section of the TC AIM and produces a number of short-answer questions in the assigned topics.

(2) Questions are not multiple-choice questions. The suggested question format is: 1) The pilot briefing service is provided by _______________________. (AIM-MET 1.1.2) Answer: NAV CANADA Flight Information Center (FIC).

(3) Questions must be correctly and clearly written, supported by established/current reference texts.

(4) The examination specialist only uses standard terminology, which pilots are required to know.

(5) The language used shall be as simple as is practical. The question should not be a test of the language itself. The solution of the problem posed by the question should not hinge on language or grammatical nuances. To assist the participating pilot in recognizing the correct response, wording of the question should be similar to the wording of the references.

(6) Questions are to be e-mailed to the inspector in charge of the final version of the Self-Paced Study Program.


2013-07-10 39 of 45 SI 400-006-Y Issue 01

19.2 Review Committee

(1) Once questions have been developed, they are presented to a review committee.

(2) The review committee consists of all the examination specialists.

(3) Review committee sessions should be approximately 1 ½ hours in length but never longer than 2 hours.

(4) Print enough copies for each participant and bring all the necessary books, materials and equipment to the review committee.

(5) Examination specialists meet together and review all the proposed questions on the basis of technical and editorial accuracy. They also check the accuracy of the answers and the references.

(6) The examinations specialists rank the questions as A, B or C level. A-level questions will be significant enough to be included in the exam. B-level questions will be included if possible. C-level questions will be low significance or difficult to perfect and will not progress further.

19.3 Perfect the questions

(1) Each examination specialist perfects his or her questions as suggested by the review committee.

(2) Reviewed questions are e-mailed to the inspector in charge of the final version of the Self-Paced Study Program.

19.4 Final Review Committee

(1) When the questions have been corrected, they are presented to a final review committee.

(2) The final review committee consists of the same participant as the first review committee.

(3) Print enough copies for each participant and bring all the necessary books, materials and equipment to the review committee.

(4) Examinations specialists review all the corrected questions and selected questions for inclusion in the Self-Paced Study Program.

19.5 Self-Paced Study Program Format and Storing

(1) Format the questions and the examination. The order of the questions should follow the TC AIM sections (GEN-AGA-COM-MET-RAC-FAL-SAR-MAP-LRA-AIR).

(2) An introductory paragraph should be placed at the beginning of the Self-Paced Study Program that follows the same format as previous years.

(3) Ensure the length of the lines for the answers are long enough to accommodate the answers.

(4) Save the examination in RDIMS.

(5) Save the questions in the G drive G:\AARR\AARRF\Self-Paced Recency for reference.

(6) Enter the questions in the FTAE “Recency” question bank for future reference.

19.6 Submission of the Final Self-Paced Study Program

(1) Send the proposed Self-Paced Study Program to the editor of the Aviation Safety Letter.

(2) The edited Self-Paced Study Program is sent back to the OPI for approval.


2013-07-10 40 of 45 SI 400-006-Y Issue 01

19.7 Translation

(1) Safety Services is responsible for translation.

(2) When the inspector in charge of the final version of the Self-Paced Study Program Examination receives the translated document, the inspector forwards the document to the examination terminology specialist who ensures the language is consistent between English & French, and with industry usage.

(3) When the French vetting is completed, save the document as a separate RDIMS document.

(4) Save the translated questions in the G: Drive\ AARR\AARRFSelf-Paced Recency for reference.

(5) Enter the translated questions in the FTAE “Recency” question bank for future reference.

19.8 Submission of the translated Self-Paced Study Program

(1) The inspector in charge of the final version of the Self-Paced Study Program sends the proposed examination to Safety Services for layout in the Aviation Safety Letter.

19.9 Posting the Self-Paced Study Program on the Flight Crew Examination Web site

(1) The examination officer sends the Self-Paced Study Program to the web contact for upgrading the report from the last year with the new report on the appropriate pages with links from other sites as necessary.

(2) The inspector confirms that the new program is posted on the web and careful proof reading of the results is recommended.

20.0 LIBRARY

(1) Reference material is available in the Flight Training and Examinations Library.

20.1 Administration of the Library

(1) An inspector is appointed to manage the library.

(2) This inspector is responsible for ordering new books, maintaining a list of active library books and, circulates pamphlets and catalogues among the examination specialists.

(3) The list of library material is kept in the G: Drive\AARR\AARRF\library. The list is a MS Access file.

(4) The inspector orders new books through the examination officer as requested by other examination specialists, this includes obtaining spending approval.

(5) New books are identified with a tag on the spine, according to their subject group.

20.2 How to borrow books from library

(1) When a book is borrowed from the library, it must be signed out using the recording system in place.

(2) To improve availability, borrowers are requested to return the book as soon as possible.


2013-07-10 41 of 45 SI 400-006-Y Issue 01

21.0 CHARTS

(1) Several examinations require an aeronautical chart.

(2) AARA-MPS is responsible to purchase charts from NAV CANADA and to sell them exclusively to Transport Canada Centres and AEIs.

(3) In order to have sufficient charts in stock to sustain the examinations for a long period of time, Flight Crew Examinations order a large quantity of charts from AARA-MPS. 1300 copies of VNCs Charts and 500 copies of the related LO Charts are typically required.

(4) When the inventory falls below 200 charts, the ordering process of this chart or the selection process for a new chart should commence.

(5) Examination specialists shall advise the person responsible for charts when a replacement is required due to validity of the information and presentation.

(6) A list of all examinations and charts is available in RDIMS # 1297306.

21.1 Chart Selection Meeting

(1) A selection meeting consists of all the examination specialists that work with examinations that require a chart.

(2) Proposed charts should be distributed to participants in advance of the selection meeting.

(3) Specialists choose a new chart based on the possibilities to create cross-country routes appropriate for the examinations, the presence of different Class of airspace, the quantity of information, navigation aids and the other charts already in use.

(4) The legend of the charts shall be provided in English and French.

21.2 Order charts

(1) A work order is sent to AARA-MPS. This form (26-0482) is only available on paper format at the Communication Centre.

(2) A PDC form (Product Development Checklist) is sent to AARA-MPS. This form (26-0656) is available in the forms catalogue.

(3) A special request shall be made to produce stickers that say “For Examination Purposes Only”/ “Aux fins d’examens seulement”, and the stickers shall be placed on each chart in a clearly visible location.

(4) AARA-MPS advises when the charts have arrived and when they are ready to be sold.

(5) Regions and AEIs shall be advised by email that the new charts are available for purchase.

(6) Every specialist that plans on using the new chart shall have a copy of the chart.

(7) The examination officer buys about 20 copies of the new chart and stores them in the library for reference.

21.3 Monitor charts

(1) The quantity of each chart is monitored on a regular basis to predict the need for replacements and the trends in usage.

(2) AARA-MPS will advise Flight Training and Examinations when the inventory falls under 200 charts.

(3) An inventory report can be requested from AARA-MPS when required


2013-07-10 42 of 45 SI 400-006-Y Issue 01

22.0 INFORMATION MANAGEMENT

(1) Examinations and results are PROTECTED material and are not to be saved in RDIMS. They are saved in the secure FTAE computer system.

(2) Study and Reference Guides are to be saved in RDIMS. In keeping with the naming convention, their name will follow the format “Study and Reference Guide – Flight Instructor – Gyroplane – TP 15209E V1”. The file number 5802-33-1 will be used for Study and Reference Guides.

(3) Common Transport Canada file numbers for examination material are: 5802-33 Examination Procedures 5802-33-1 Flight Crew Examinations – General 5802-33-2 Flight Crew Examinations – Authorized Invigilators

(4) A file that has been used for generic aviation examination correspondence in the past is: 5802-4 Written Examinations

23.0 DOCUMENT HISTORY

(1) Not applicable.

24.0 CONTACT OFFICE

For more information, please contact: Commercial Flight Standards (AARTF)

Phone: 613-990-6077 Fax: 613-990-6215 E-mail: [email protected]

Suggestions for amendment to this document are invited, and should be submitted via: [email protected]

Original signed by Aaron McCrorie on July 5, 2013

Aaron McCrorie Director, Standards Civil Aviation Transport Canada

--------------------------- Transport Canada documents or intranet pages mentioned in this document are available upon request through the

Contact Office


2013-07-10 43 of 45 SI 400-006-Y Issue 01

APPENDIX A — SPELLING

The spelling for the terms shown here shall be used on flight crew examinations

A aerodynamic aerofoil after-landing (adj) airborne airflow airspeed air time air-to-ground air traffic control unit all-up (adj) aluminum ambient ammeter amperage ampere ancillary aneroid angle of climb anticlockwise anticollision anticyclone antidrag anti-icing antiservo tab antitorque approach path articulated asymmetric automatic direction finder automatically auxiliary axis (pl: axes) B backfire base leg Bernoulli buildup (n) built-in (adj) built-up (adj) bulkhead bypass by-product C cadmium cantilever

capacitor carburetor centring centre centre line (n) centrifugal characteristic check flight checklist check ride checkpoint chord circumferential clockwise cockpit coincident collocated constant speed continuous loop controllable co-ordinate counteract counterbalance counterclockwise counterweight crankshaft cross-border cross-check cross-country cross-section crosswind cruise altitude cruise attitude cutoff (adj) (n) cutout (n) cut out (v) cut-out (adj) D daytime debrief de-energize de-ice

desirable detergent detonation detrimental diaphragm

dihedral direction finder direction-finding equipment disc dispersant dissimilar dissipate downdraft downgoing (adj) downwind E en route equidistant F false start fibre fire warning (n) fixed-pitch (adj) flashlight flight time floatplane float-equipped flying boat free wheeling full throttle full-throttle height G generator go-around (n) go around (v) governor ground loop (n) (v) ground school groundspeed gust load gyro-operated grey (adj. colour) H half-hour half-turn (n) hang glider head wind heavier-than-air high intensity high-speed (adj)


2013-07-10 44 of 45 SI 400-006-Y Issue 01

high-wing hold-off (n) hold off (v) hook-up (n) hook up (v) hot section hydraulic hydraulically hydoplaning I idling imbalance impeller incurred inflammable inbound in-flight (adj) influence infrared intake (adj) (n) interconnector into wind (n) into-wind (adj) iso-octane J judgment K kinetic L landing gear landing path landing roll landmarks landplane lead-acid (adj) left-hand (adj) let-down (adj) (n) level-off (adj) level off (v) licence (n) license (v) licensed licensing life jacket liftoff (n) lift off (V) lightweight (adj) load sharing logbook lookout (n) low-intensity (adj) low-level (adj) life saving

M mach magnetism main wheel manoeuvre manoeuvring map reading mechanism metre microburst mid-air (adj) (n) mid-field (adj) (n) midday midpoint midway mileage multi-engine (adj) N never exceed speed non-detergent non-instrument non-magnetic non-standard non-structural non-turbulent nose-down (adj) nose-high (adj) nose-low (adj) nose-over (n) nose over (v) nose-up (adj) nose-wheel (adj) nose wheel (n) no wind NOTAM NOTAMs noticeable O obscured occasional occurred occurrence off-airport (adj) off-centre oleo omnibearing selector omnidirectional omnirange omnitest one-half one-in-sixty onshore (adj) oscillation outbound

offset overall overbank overcast overconfident overcrowded overrich overriding oversensivity overshoot overspeed overtemperature overuse P parallel pickup (adj) pick up (v) pilot-in-command pin-point Plexiglas pneumatic porpoising polarity power-on/off (adj) practise (v) practice (n) prearrange precaution precede predetermine pre-flight (adj) pre-heat (adj) pre-start pressurization preheat (v) pre-ignition prescribe pre-solo Q R radio-equipped rate of climb rate-one-turn reaction reciprocating recover (v) redirect reprint reset residual resistance reteach


2013-07-10 45 of 45 SI 400-006-Y Issue 01

rewrite rheostat right-hand (adj) right of way right side up rigidity in space roll-in (n) roll in (v) roll-out (n) roll out (v) roundout (n) round out (v) run-up (n) S safetying devices scavenge schematic seaplane seat belt seesaw self-contained semi-rigid serviceable servo-tab set-heading (adj) set heading (v) shock strut shortcut (n) (v) short-field (adj) shutdown (n) shut down (v) shut-off (adj) (n) shut off (v) slide rule slipstream snow-covered (adj (n) soft-field (adj) (n) speed-sensitive (adj) speed-up spool-up (adj) (n) spool up (v) S-turn sideslip (n) (v) sideways simultaneously single-engine/single-rotor skiplane starter-generator step-down stopover straight-in (adj)

supercharged susceptible synthetic T takeoff (n) take-off (adj) take off (v) tail plane tail wheel (n) tail-wheel (adj) tail wind taxiing teardrop (adj) thermostat thrustline tie-down (n) tie down (v) top-heavy torque touch-and-go (adj) (n) touchdown (n) touch-down (adj) touch down (v) track error track made good trailing edge transformer troubleshoot turbine-powered turbocharged turboprop turn and bank indicator turn co-ordinator turnoff (n) turn off (v) two-way (adj) U ultrahigh ultra-light ultrasonic unco-ordinated underbank under read undershoot underslung under way unfeather unfiltered unobstructed

unpressurized unserviceable unsupercharged unusable updraft upgoing (adj) upwind usable vapour venturi vernier vice versa viscosity vortic W walk-around (n) walk around (v) warm-up (n) (adj) warm up (v) wash-in (n) wash-out (n) weathercock wheelbarrow wheel brakes wheel-equipped wheel-ski whiteout (n) wind cone windrows wind shear windshield wind sock wind velocity wing tip X Y Z Zigzag Zyglo test

Staff Instruction - Transports Canada

Documents

Transcript of Staff Instruction - Transports Canada