Course Schedule (cont) two Principles of Testing Principles of Testing Principles of Assessment...
-
Upload
haley-garcia -
Category
Documents
-
view
224 -
download
3
Transcript of Course Schedule (cont) two Principles of Testing Principles of Testing Principles of Assessment...
Course Schedule Course Schedule (con’t)(con’t)
twotwo
Principles of TestingPrinciples of Testing Principles of AssessmentPrinciples of Assessment Conducting a PPCConducting a PPC
PPC TolerancesPPC Tolerances
Course ScheduleCourse Scheduletwotwo
Principles of TestingPrinciples of Testing Principles of AssessmentPrinciples of Assessment Conducting a PPCConducting a PPC
PPC TolerancesPPC Tolerances
Outline Outline General PrinciplesCommon Rating ErrorsSelf-study handout
Principles of TestingPrinciples of Testing
Outline General PrinciplesCommon Rating ErrorsSelf-study handout
Principles of TestingPrinciples of Testing
General PrinciplesGeneral Principles A good test should be:A good test should be:
1) Valid2) Reliable3) Comprehensive4) Discriminating5) Objective
Principles of TestingPrinciples of Testing
Outline General PrinciplesCommon Rating ErrorsSelf-study handout
Principles of TestingPrinciples of Testing
Validity
Does test measure what it is Does test measure what it is supposedsupposed
to measure?to measure?1) make the test realistic2) make the test
complete3) apply standards
carefully4) no test of items
outside std
ValidityValidity
Which is the Which is the mostmost valid? valid?1) describe action in
briefing2) simulate action in
aircraft3) do action in aircraft4) do action in simulator
Validity ExampleValidity Example
Outline General PrinciplesCommon Rating ErrorsSelf-study handout
Principles of TestingPrinciples of Testing
Reliability
How accurately does it How accurately does it measuremeasure
what it measures?what it measures?Must be consistent:1) Avoid extremes
- distractions- aircraft- weather
2) Plan compensations (eg. V1)- beforehand- carefully
ReliabilityReliability
Which test is Which test is mostmost reliable? reliable?
1) PPC in marginal VFR weather
2) PPC in good weather, light winds
3) PPC in good weather, strong winds
4) PPC in IFR weather near minima
Reliability ExampleReliability Example
Outline General PrinciplesCommon Rating ErrorsSelf-study handout
Principles of TestingPrinciples of Testing
Comprehensive
The test must include all theThe test must include all therelevant sequencesrelevant sequences
- important "need to know" info
- do not focus on details- vary the program from test to
test- change approaches,
sequences, emergencies (where
discretion)
ComprehensivenessComprehensiveness
Which test is Which test is mostmost comprehensive?comprehensive?
1) departure, steep turns, hold, NDB, ILS
2) dep, hold, ILS, ILS, VOR3) dep, hold, ILS, VOR4) dep, steep turns, LOC (BC), PAR
Comprehensiveness ExampleComprehensiveness Example
Outline General PrinciplesCommon Rating ErrorsSelf-study handout
Principles of TestingPrinciples of Testing
Discriminating
Detects different levels ofDetects different levels ofachievementachievement
- separate qualified from unqualified
- this is main goal of test- rank those who are qualified- use overall performance for
borderline cases
DiscriminationDiscrimination
Which test is Which test is mostmost discriminating?discriminating?
1) no oral test before flight2) oral test before flight with
general briefing3) oral test and specific test
sequences 4) oral test before and during
flight
Discrimination ExampleDiscrimination Example
Outline General PrinciplesCommon Rating ErrorsSelf-study handout
Principles of TestingPrinciples of Testing
Objective
Test outcome must not be Test outcome must not be affected byaffected by
personal examinerpersonal examiner
1) do not impose your own standards2) adhere to standards in FM, SOP, AOM and DCP Manual3) do not let rating errors affect your
assessment4) carefully record events as they occur
ObjectivityObjectivity
Which written assessment is Which written assessment is most objective?most objective?
1) poor judgment during emergency2) excessive swing during engine fail3) altitude 100ft low inbd to beacon4) airspeed unstable on final
Objectivity ExampleObjectivity Example
Outline General PrinciplesCommon Rating
ErrorsSelf-study handout
Principles of TestingPrinciples of Testing
1) Error of Standards2) Error of Central Tendency3) Error of Halo4) Error of Logic5) Error of Familiarity6) Error of Narrow Criterion
Common Rating ErrorsCommon Rating Errors
Outline General PrinciplesCommon Rating
ErrorsSelf-study handout
Principles of TestingPrinciples of Testing
Error of centraltendancy
Rate of performance at centre of scale
Hesitant to give excellent or poor ratings
Affects all assessments by this observer
Most common with inexperiencedobservers
Error of Central TendencyError of Central Tendency
Outline General PrinciplesCommon Rating
ErrorsSelf-study handout
Principles of TestingPrinciples of Testing
Error ofStandards
Use personal standards to rate others
If good at it - rate harderIf poor at it - rate easierAffects all assessments by this
observer
Error of StandardsError of Standards
Outline General PrinciplesCommon Rating
ErrorsSelf-study handout
Principles of TestingPrinciples of Testing
Error of Halo
Allow general impression to affect rating
Can be favourable or unfavourableAffects only certain persons rated Can include stereotypes
Error of haloError of halo
Outline General PrinciplesCommon Rating
ErrorsSelf-study handout
Principles of TestingPrinciples of Testing
Error of Logic
When several similar traitsTraits are related in mind of
observerCould relate engine fail at V1 to SE
landingObserver generally not aware of
error
Error of LogicError of Logic
Outline General PrinciplesCommon Rating
ErrorsSelf-study handout
Principles of TestingPrinciples of Testing
Error of Familiarity
Error of FamiliarityError of Familiarity
Grading objectivity weakenedGrading objectivity weakened Accustomed to weaknesses and Accustomed to weaknesses and
eccentricitieseccentricities Occurs over long period of timeOccurs over long period of time Stop and regain perspectiveStop and regain perspective
Outline General PrinciplesCommon Rating
ErrorsSelf-study handout
Principles of TestingPrinciples of Testing
Error of Narrow Criterion
Error of Narrow CriterionError of Narrow Criterion
Related to assigning ratingRelated to assigning rating Often occurs in assessing two-crew or Often occurs in assessing two-crew or
multi-crew assessmentsmulti-crew assessments One who does the best of perhaps a One who does the best of perhaps a
marginal performance obtains a high ratingmarginal performance obtains a high rating
Outline General PrinciplesCommon Rating ErrorsSelf-study handout
Principles of TestingPrinciples of Testing
Course ScheduleCourse Scheduleday 2day 2
Principles of TestingPrinciples of Testing Principles of AssessmentPrinciples of Assessment Conducting a PPCConducting a PPC
PPC TolerancesPPC Tolerances
Outline Outline PPC general guidelinesAssessment formatOverall assessment
Principles of AssessmentPrinciples of Assessment
Outline PPC general
guidelinesAssessment formatOverall assessment
Principles of AssessmentPrinciples of Assessment
1) Take note soon after event2) Terminate test if failure3) Allowance for turbulence4) No advice, criticisms, training
or demonstrations
5) Create realistic scenarios6) Do not simulate multiple,
unrelated failures
PPC general guidelinesPPC general guidelines
7) PPC - Use Designated Check Pilot Manual
8) Simulated weather at or below minimums
9) Need a "U" to fail10) Take weather info into
account in scenarios11) PNF responsible if gross
deviations, violations
PPC general guidelines (con’t)PPC general guidelines (con’t)
According to the following According to the following publications:publications:
• DCP ManualPPC - AFM
- AOM, Pilot Operating Handbook
- Operations Manual- SOP Manual
IFT - Instrument Procedures Manual- Approach Procedures (Jepp)- Company Procedures
Performance AssessmentPerformance Assessment
Outline PPC general guidelinesAssessment formatOverall assessment
Principles of AssessmentPrinciples of Assessment
S exercise flown within acceptable tolerances
and procedures correct and effective
SB a brief excursion from the criteria ora deviation from standard practices.Corrected by acknowledging the
deviationduring de-briefing; or regaining the
criteriaduring flight
U gross excursions from acceptable tolerances or a violation of airspace or clearance or a fault in procedure that could create an unsafe or ineffective situation
PPC Assessment FormatPPC Assessment Format
S - Minor errors only
- Airspeed and altitude control acceptable
- Handling and knowledge are acceptable and safe considering the experience of the candidate
PPC Assessment FormatPPC Assessment Format
U - Endanger aircraft, passenger, crew;
- It results in a crash;- Multiple errors made in one exercise;- Violates ATC clearance or an altitude;- Major deviation from standard
procedures or practices or safety was jeopardized;
- Candidate required continued prompting or help from other candidate;
- It exceeds aircraft limitations; or- Candidates demonstrates
unsatisfactory knowledge of systems, equipment, or procedures
PPC Assessment FormatPPC Assessment Format
SB - Aircraft handling and knowledge are safe but of a lower standard that can be corrected by briefing;
- Brief excursion from tolerances but corrected promptly;
- Deviated from SOP but did not create a more hazardous situation and is clarified by candidate
- Candidate experienced some difficulty or required slight prompting to accomplish the task
PPC Assessment FormatPPC Assessment Format
Outline PPC general guidelinesAssessment formatOverall assessment
Principles of AssessmentPrinciples of Assessment
• is it safe and effective?• are criteria broken?• are any fail points done?• is crew concept used?• are crew in control of situation?• are procedues known and executed?• are limitations adhered to?• does an error create an unsafe situation?• are errors recognized and corrected?• would you send your family with this
pilot?
Overall PerformanceOverall Performance
Course ScheduleCourse Scheduleday 2day 2
Principles of TestingPrinciples of Testing Principles of AssessmentPrinciples of Assessment Conducting a PPCConducting a PPC
PPC/IFT TolerancesPPC/IFT Tolerances
Outline Outline Flight test requirements
- PPCPre-flight preparation/briefingFlight test scenariosSafe checking practicesPost-flight briefingWritten comments
Conducting a PPCConducting a PPC
Outline Outline Flight test requirementsPPCPre-flight preparation/briefingFlight test scenariosSafe checking practicesPost-flight briefingWritten comments
Conducting a PPCConducting a PPC
PRE-FLIGHTPRE-FLIGHT- Flt planning - weather, fuel, NOTAMs,
W&B- IFR x-country exercise for initials- Cockpit checks and ident of NAVAIDS- Read back and interpretation of an
ATC clearance- Take-off and passenger briefings
DEPARTURE AND ENROUTEDEPARTURE AND ENROUTE- Transition to instruments- Climb and level flight- Use of NAVAIDS and adherence to
ATC clearance
PPC RequirementsPPC Requirements
AIRWORK - Applicant Should AIRWORK - Applicant Should Perform:Perform:
• Medium and steep turns• Hold• Tracking on VOR and NDB• Failure of AI and DI
- timed turns, S&L- climbs and descents- optional for renewals
PPC Requirements (con’t)PPC Requirements (con’t)
TERMINAL PROCEDURES• Transition to approach should be
flown• At least 2 approaches to minima
- ILS and NPA required for initials
- different approaches if possible
- NDB recommended• Missed approach• Circling (optional)• Landing from an instrument
approachEMERGENCIES
• One asymmetric approach should be done
• No multiple unrelated failures
PPC Requirements (con’t)PPC Requirements (con’t)
Typical Large Type Aircraft• 2 T/O's• 2 landings (one
asymmetric)• asymmetric instrument
approach• engine fail after T/O• RTO (mandatory in SIM)• engine fail on rejected
ldg/MAP• circling if company
limits<1000/3refer to DCP Manual Chap 7
Schedule I
PPC Additional RequirementsPPC Additional Requirements
Typical Small Type Aircraft• 2 T/O's• 2 landings (one assymetric)• steep turns• approach to stall - ldg and
T/O• RTO (mandatory in SIM)
refer to DCP Manual Chap7Schedule II
PPC Additional RequirementsPPC Additional Requirements
Outline Outline Flight test requirements
- PPCPre-flight
preparation/briefingFlight test scenariosSafe checking practicesPost-flight briefingWritten comments
Conducting a PPCConducting a PPC
• Before meeting candidate- check weather- applicant's info- plan IFR x-country exercise- plan oral questions
• After meeting candidate- ensure admission req'ts met- ensure aircraft is ready for IFR
flight- check accuracy of log, flt plan,
W&B- test knowledge of weather and
NOTAMs- test IFR knowledge and aircraft limitations
Pre-Flight PreparationPre-Flight Preparation
• Valuable part of any flight test• Can be used to identify changes• Can provide feedback on weak
training areas• Ask "why" questions• Cover only "need-to-know" material• Vary questions from test to test
Oral QuestionsOral Questions
"Need to know" a/c speeds if not "Need to know" a/c speeds if not familiar with type:familiar with type:
Vref VrHold VmcaEngine out approach
• Use these briefed speeds to assess speed control
Aircraft LimitationsAircraft Limitations
- Radio, 2 nav radios, aircraft instruments
- C of A, C of R,- Journey log book- Compass swing (12 months)- Static pressure check (24 months)- Altimeter certified (24 months)- Weight and Balance report- Radio station license
Aircraft IFR RequirementsAircraft IFR Requirements( general aviation requirements)
• private location, calm atmosphere• entire flight is IMC at minimum• clear who is PIC• description of major test
components• responsibility for radios and clncs• cockpit checks to be done per SOP• crew responsibilities and
coordination• emergencies - use of
"SIMULATED"• how engine failure to be simulated• "FIELD IN SIGHT" or overshoot• candidates QUESTIONS
Pre-Flight BriefingPre-Flight Briefing
Outline Outline Flight test requirements
- PPCPre-flight preparation/briefingFlight test scenarios -
ExerciseSafe checking practicesPost-flight briefingWritten comments
Conducting a PPCConducting a PPC
• Make efficient use of time• Create realistic scenarios• Liaise with ATC early• No overloading• Minimize re-positioning in Sim• Use proper ATC terminology
Flight Test ScenariosFlight Test Scenarios
Outline Outline Flight test requirements
- PPCPre-flight preparation/briefingFlight test scenariosSafe checking practicesPost-flight briefingWritten comments
Conducting a PPCConducting a PPC
• Check Pilot assuming crew duties• RTO• Engine fail after V1
• Rejected landing with engine fail• Engine out landing• Approach to stall
Safe Checking PracticesSafe Checking Practices
• Advise points in order• Include location, atmosphere, tone• Quiet location no distraction
De-Brief After Flight TestDe-Brief After Flight Test
Outline Outline Flight test requirements
- PPCPre-flight preparation/briefingFlight test scenariosSafe checking practicesPost-flight briefingWritten comments
Conducting a PPCConducting a PPC
Post-Flight BriefingPost-Flight BriefingPassPass• advise applicant of outcome• establish a positive atmosphere• highlight above-average performance• de-brief as per written notes• identity major (possibly minor) errors
FailFail• advise applicant of outcome• establish a positive atmosphere• highlight above average performance• explain reasons for
failure• de-brief other major errors• advise regarding training and re-test
Outline Outline Flight test requirements
- PPCPre-flight preparation/briefingFlight test scenariosSafe checking practicesPost-flight briefingWritten comments
Conducting a PPCConducting a PPC
• Should be factual• Clear and concise• Must support assessment• Required for "U" and "SB"• No sermons or teaching• Relate assessments to PPC
tolerances
Written CommentsWritten Comments
Course ScheduleCourse Scheduleday 2day 2
Principles of TestingPrinciples of Testing Principles of AssessmentPrinciples of Assessment Conducting a PPCConducting a PPC
PPC TolerancesPPC Tolerances
Outline Outline StatisticsDCP ManualPPC fail pointsDCP Manual tolerances: 6.8.8
PPC TolerancesPPC Tolerances
Outline Outline StatisticsACP ManualIFT fail pointsPPC fail pointsACP Manual tolerances
PPC/IFT TolerancesPPC/IFT Tolerances
IFT Failure ItemsIFT Failure Items
RENEWALRENEWAL
NDB 28%HOLD 23%ILS 13%TRANSITION 8%LOC 5%VOR 5%DEPARTURE 5%
INITIALINITIAL
NDB 29%HOLD 26%ILS 26%DEPARTURE 7%TRANSITION 6%FLT. PLANNING 6%HANDLING 4%
Outline Outline StatisticsDCP ManualPPC fail pointsDCP Manual tolerances
PPC TolerancesPPC Tolerances
from DCP Manual 6.8.8
-assigned headings within 10 degrees-assigned tracks and bearings within 10 degrees-altitude within 100 ft except at MDA when accurate
altitude control is required-airspeed within 10 knots for holding, approach, and
missed approach;and-not more than half scale deflection, as appropriate to
aircraft type, of course deviation indicators during instrument approaches
-assumes no unusual circumstances; wx,turbulence, malfunction(s), and type of approach may modify tolerances
DCP CriteriaDCP Criteria
• Outside DCP Manual, AFM, IFR Procedures• No check of flt instruments before flight• No check or incorrect use of icing systems• No check of electrical charging system• Wrong altimeter setting• Vital actions checklist item missed• Non-compliance with ATC clearance• No communication with ATC when required• No ident or monitor of NAVAIDs• Comm fail procedures• Inability to establish cleared holding pattern
fail Pointsfail Points
• Loss of directional control• Wrong corrective action for swing• Exceeding airframe or engine
limitations• Rotation before Vr• Failure to comply with departure
clearance• Vital action check missed
Tolerance Example 1Tolerance Example 1
Take-Take-offoff
• Loss of separation due to failure to follow clearance
• No prompt corrective action when DCP criteria exceeded
• Unable to complete a successful landing
• Exceeding airframe or engine limitations
Tolerance Example 2Tolerance Example 2
Intstrument Intstrument ApproachesApproaches
Systems MalfunctionsSystems Malfunctions • Vital action check missed• Missing a check which could create an unsafe situation• Exceeding airframe or engine limitations
Missed ApproachMissed Approach• Not following published procedure or clearance
Tolerance Example 3Tolerance Example 3
Questions?