Risk Analysis Tool Process (RAT/RAP) Meetings Seminars and Workshops/Safety... · 1 Risk Analysis...
Transcript of Risk Analysis Tool Process (RAT/RAP) Meetings Seminars and Workshops/Safety... · 1 Risk Analysis...
1
Risk Analysis Tool & Process (RAT/RAP)The way the harmonise assessment of ATM incidents over EUROPE and FAA
Tony LICU
Gilles LE GALO
Radu CIOPONEA
Florin CIORAN
EUROCONTROL
Art SHEFFIELD
James C. BEADOW
Jon JONES
Mike McFADYEN
FAA
The European Organisation for the Safety of Air Navigation
Federal AviationAdministration
WORKSHOP ON IMPLEMENTATION OF THE ICAO SAFETY MANAGEMENT REQUIREMENTS IN STATES
Paris, France – 14/15 February 2011
ICAO Annex 13
• First edition September 1951
• 1st‐3rd edition (04/1973) called“Aircraft Accident Inquiry”Aircraft Accident Inquiry
• 4th‐7th edition (05/1988) called“Aircraft Accident Investigation”
• 8th‐9th edition (07/2001) called“Aircraft Accident and IncidentInvestigation”
Federal AviationAdministration
SMS Workshop – ICAO Paris, France 14‐15 Feb 2011 2
2
Annex 13 First Edition
Federal AviationAdministration
SMS Workshop – ICAO Paris, France 14‐15 Feb 2011 3
ATM Safety reporting and investigation evolution in Europe
• ESARR 2 (ATM Occurrence Reporting and Investigation) and the first Risk Classification Scheme ‐ 1999
• Risk Wizard 2004/2005• Risk Wizard – 2004/2005
• RAT – Risk Analysis Tool ‐ 2009
• Implementing rule on Performance EC691/2010
Federal AviationAdministration
SMS Workshop – ICAO Paris, France 14‐15 Feb 2011 4
1999 20062004 2007 2009 2010
Safety KPI
Roadmap
SAFREP &
RAT
ESARR 2 &
EAM2GUI1
Risk Wizard IR on Performance
3
European Regulatory environment change
•Need to know what occurs at European level
•States /ANSPs report to EUROCONTROL as of 1999
•EUROCONTROL to analyse the data
•Action Plans
•SESII PRB Performance framework
Federal AviationAdministration
SMS Workshop – ICAO Paris, France 14‐15 Feb 2011 5
HARMONISATION
•SESII, PRB, Performance framework
ESARR 2
Need to know about undesired events that have had or might have had an impact on safety
Reporting systems
Requires a common
TAXONOMY
Reported Safety Occurrences
Findings, RecommendationsSeverity & Risk Assessment
Need to determine to what extent ATM has contributed to the occurrences and severity of a safety risk
Data collectionAnalysis
Severity & Severity & Risk assessmentRisk assessmentATM contribution
Requires
HARMONISED PROCESSES
Federal AviationAdministration
SMS Workshop – ICAO Paris, France 14‐15 Feb 2011 6
Severity & Risk Assessment
Need to share experiences
Trends, KRA, ATM improvementsGLOBAL SOLUTIONS
Annual Summary Template
Agreements(bilateral orregional)
Requires
PROCEDURES
CONFIDENTIAL ASSURANCE
4
Status in Europe back in 1999
• State shall ensure “that the severity of occurrences is determined, the risk posed by occurrences classified, and the results recorded.” (See ESARR 2 –section 5.1.6)
• The assessment/investigation of the occurrence shall enable the determination of the severity level of the occurrence (ESARR 2‐ A‐3.1)
• EAM 2‐ GUI 1 states Supplementary material will be developed with experience to decrease the level of subjectivity inherent to severity
Federal AviationAdministration
SMS Workshop – ICAO Paris, France 14‐15 Feb 2011 7
p j y yassessment
D1 E1 C1 B1 A11
very frequen
tnt
ESARR2 in Details ‐ Classification (Incidents)Risk Classification Scheme for occurrences involving aircraft and their occupants (European version)
D4
D3
D2
E4
E3
E2
C4
C3
C2
B4
B3
B2
A4
A3
A22
3
4
frequen
occasional
rare
Federal AviationAdministration
SMS Workshop – ICAO Paris, France 14‐15 Feb 2011 8
D5 E5 C5 B5 A55
extrem
ely rare
ABCD E
seriousmajorsignificantno determined No safety effect
5
D2
D1
E2
E1
C2
C1
B2
B1
A2
A11very frequen
ten
t
AA2
AA1
Risk Classification Scheme for occurrences covering the capability to deliver safe ATM services (European version)
D4
D3
D2
E4
E3
E2
C4
C3
C2
B4
B3
B2
A4
A3
A22
3
4
freque
occasional
rare AA4
AA3
AA2
Federal AviationAdministration
SMS Workshop – ICAO Paris, France 14‐15 Feb 2011 9
D5 E5 C5 B5 A55
extrem
ely rare
ABCD E
Partial inabilityto provide safeATM services
Ability to providesafe but degraded
servicesNot determined
No effect on ATM services
AA
AA5
Serious inabilityto provide safeATM services
Total inabilityto provide safeATM services
RAT and the performance IR EC691/2010
• (2012‐2014)• The second European Union‐wide
safety key performance indicator shall be the percentage of application of the severity classification of Risk Analysis Tool as defined in Section 2, point 1(b) below in States where this Regulationbelow in States where this Regulation applies, to allow harmonised reporting of severity assessment of Separation Minima Infringements, Runway Incursions and ATM Specific Technical Events
• The second national/FAB safety KPI shall be the application of the severity classification of the Risk Analysis Toolto allow harmonised reporting of severity assessment of Separation Minima Infringement, Runway
Federal AviationAdministration
SMS Workshop – ICAO Paris, France 14‐15 Feb 2011 10
Incursions and ATM Specific TechnicalEvents at all Air Traffic Control Centres and airports with more than 150 000commercial air transport movements per year within the scope of this Regulation (yes/no value).
6
Why RAT?
The RAT is a tool meant for:
• Service Providers – as part of their SMS (The ATS Unit• Service Providers – as part of their SMS (The ATS Unit Manager perspective: the safety performance of his unit )
• Regulators – as part of the Aviation‐wide sector oversight (Representing the airline passenger perspective: how safe is it to travel by air)
Federal AviationAdministration
SMS Workshop – ICAO Paris, France 14‐15 Feb 2011 11
Why Risk Analysis?
The Risk Analysis allows setting priorities:
ability
Urgent action
May like/think to do something
Federal AviationAdministration
SMS Workshop – ICAO Paris, France 14‐15 Feb 2011 12
severity
repeata
Do nothing
Look Into it
7
Risk Analysis Tool Aim
Ambition was to have a tool that is:
•Simple
•Easy to use
•Efficient/reliable (at least for 80/90% of cases)
Realism is that the tool:
•Will not provide the golden truth
Federal AviationAdministration
SMS Workshop – ICAO Paris, France 14‐15 Feb 2011 13
•May not work well for complex incidents
•will still rely on experience and knowledge of experts
Tool principles
RAT (Excel sheet) works as follows:
•Requires the user to answer questions looking as much as
A. SEVERITY
1. Risk of collision ATM ground
ATM airborne
ATM overall
Minimum separation achieved 0 0Separation + 75% minimum 1 1Separation >50%, <=75% minimum 3 3Separation >25%, <=50% minimum 7 7Separation <=25% minimum 10 10
Total separation (a) 0Rate of closure NONE 0 0Rate of closure LOW (<=85knots, <=1000ft/mn) 1 1Rate of closure MEDIUM (>85 and <=205 knots, >1000
d 2000 ft/ ) 2 2
B. REPEATABILITY
3. Systemic issues ATM ground
ATM airborne
ATM overall
Procedures DESIGN 12 12Procedures IMPLEMENTATION 8 8Procedures LACK OF 8 8 0Equipment DESIGN 12 12Equipment IMPLEMENTATION 8 8Equipment LACK OF 8 8 0Human resources management (staff planning, staff assignment, training) DESIGN 12 12q q g
possible at the facts (from investigation report)
•Based on these answers RAT computes proposed Risk classifications for ATM Ground, as well as for ATM (i.e. ground and airborne part)
and <=2000 ft/mn) 2 2Rate of closure HIGH (>205 and <=700 knots, >2000 and <=4000 ft/mn) 4 4Rate of closure VERY HIGH (>700knots, >4000ft/mn) 5 5
Total rate of closure (b) 0
TOTAL (1-ATM) Risk of Collision (a)+(b) 0TOTAL (1-ATM Ground) Risk of Collision (a)+(b) 0
2. Controllability ATM ground
ATM airborne
ATM overall
Conflict detected 0 0Conflict detected late 3 0Conflict NOT detected 5 0 0Plan CORRECT 0 0Plan INADEQUATE 3 0NO plan 5 0 0Execution CORRECT 0 0Execution INADEQUATE 3 5NO execution 5 10 0Loss of separation detected because of STCA 3 0No detection (including by STCA) 5 0 0Recovery CORRECT 0 0
Human resources management IMPLEMENTATION 8 8Human resources management LACK OF 8 8 0
4. Non-Systemic/ Human Involvement issues ATM ground
ATM airborne
ATM overall
Non-Systemic/ Human Involvement issues with Contextual Conditions 12 12Non-Systemic/ Human Involvement issues without Contextual Conditions 8 8 0
TOTAL 3&4a 0
TOTAL 3&4b 0 0
Total (3&4-ATM) = (3&4a)+(3&4b) 0Total (3&4-ATM Ground) = (3&4a) 0
5. Window of OpportunitySituation
Methods
Daily routine
Workload peak
Emergency/ Unusual
situationsnormal 7 5 3degraded mode 6 4 2contingency 3 2 1
Total (5)
Federal AviationAdministration
SMS Workshop – ICAO Paris, France 14‐15 Feb 2011 14
Recovery INADEQUATE 5 6NO recovery or the ATM ground actions for recovery have worsened the situation or ATM airborne has worsened the situation 10 15 0TCAS triggered (useful RAs only to be considered) or see and avoid pilot decision (in the absence of TCAS) 10 0NO TCAS RA 0 10 0Pilot(s) followed RA (or, in absence of RA, took other effective action, as a result of see and avoid decision) 0 0Pilot(s) INSUFFICIENTLY followed RA 0 10Pilot(s) INCORRECTLY followed RA (or, in the absence of RA, took other inadequate action) 0 15 0
TOTAL (2-ATM
Ground) 0
TOTAL (2-ATM
Airborne) 0 0
TOTAL SEVERITY :SEVERITY ATM =(1) + (2-ATM) 0
SEVERITY ATM Ground = (1) + (2-ATM Ground) 0
TOTAL REPEATABILITY :ATM =(3-ATM)+(4) 0
ATM Ground = (3-ATM GROUND)+(4) 0
8
•You can savecomments orremarks about yourselections.•It’s limited to 250 characters.
Federal AviationAdministration
SMS Workshop – ICAO Paris, France 14‐15 Feb 2011 15
Click to open Causal / Contributing FactorswindowOverall ATM and ATM
Ground ReliabilityIndicators with colourdegraded progress bars
How does RAT work?
(barrier model)
ACCIDENT
(barrier model)
ACCIDENT
Separation(V or H)
Rate of closure(V or H)
CONTROLLABILITYRISK of COLLISION
Systemic & Non systemic issues
Window of opportunity
ATC PILOT
PROVIDENCECONTROL
Separation(V or H)
Rate of closure(V or H)
CONTROLLABILITYRISK of COLLISION
Systemic & Non systemic issues
Window of opportunity
ATC PILOT
PROVIDENCECONTROL
Federal AviationAdministration
SMS Workshop – ICAO Paris, France 14‐15 Feb 2011 16
RISK
SEVERITY REPEATABILITY
RISK
SEVERITY REPEATABILITY
9
Where do the figures (used for computation) come from?
A large number of incidents were classified using expert judgement
Numbers used for computation could thenNumbers used for computation could then be worked out (kind of reverse engineering)
ValidationA large number of incidents was used (same and then different from first set)
Federal AviationAdministration
SMS Workshop – ICAO Paris, France 14‐15 Feb 2011 17
Comparison and adjustments
Group 1 NOT using RAT
Group 2 using RAT
Next incident(s)Swap groups
Rules for organisation(s) SOP?
Tool limitations Usage biais
Possible inconsistent results/data set
RULES & PROCEDURES
What will RAT be used for? ALL incidents? During and after investigation only after?
Federal AviationAdministration
SMS Workshop – ICAO Paris, France 14‐15 Feb 2011 18
What will RAT be used for? ALL incidents? During and after investigation, only after?
Who will be entitled to use the tool? Experience and training requirements
Who will have the final word? Panel chair
Who will use the results and what for? Finding KRAs, trends and follow‐up
How will a panel work? Consensus?
Etc…
10
Rules for organisation(s) SOP – The FAA Example
ApplicabilityThe Risk Analysis Tool is to be applied to radar incidents involving two or more airborne aircraft with a Measure of Loss (MOL) of less than 66 percent. The tool contains separate spreadsheets for wake and non‐wake eventscontains separate spreadsheets for wake and non‐wake events.
In order to prevent the introduction of inadvertent bias to Risk Analysis Tool values, data entry should be completed by a group of experts in possession of as much investigatory and factual information as possible.
Risk Analysis Moderation PanelThe Office of Quality Assurance will determine those events eligible for Risk A l i ll d Ri k A l i E t (RAE ) d ill ll t d t f li bl
Federal AviationAdministration
SMS Workshop – ICAO Paris, France 14‐15 Feb 2011 19
Analysis, called Risk Analysis Events (RAEs), and will collect data for reliable analysis.
The panel will consist of three to five experts trained in the use of the tool.These experts will have diverse backgrounds from ATC, human factors, and/or flight operations..
Rules for organisation(s) SOP – The FAA Example
The FAA example (2)Panel Qualification and CurrencyThere are multiple requirements for experts to participate on a Risk Analysis Moderation Panel, including experience, judgment, objectivity, and reliability. In addition, participants must have a clear understanding of the concept of risk as defined in this document and its application in the ATC environment;
Nomination ProcessThe Manager of Quality Assurance will nominate, in writing, candidates for panel participation. Nominations should include a brief summary of the characteristics and qualifications noted above and will be submitted to the Director of Quality Assurance, who will select appropriate candidates from the pool of nominees;;
Member Training and CurrencyTo ensure quality and standardization, panel members must be properly trained and qualified. Once qualified individuals are responsible for maintaining their currency requirements as detailed in this
Federal AviationAdministration
SMS Workshop – ICAO Paris, France 14‐15 Feb 2011 20
qualified, individuals are responsible for maintaining their currency requirements as detailed in this section. Panel members must notify the chair if their currency lapses;
In order to receive initial qualification as a Risk Analysis Moderation Panel member, candidates must:• Read and become thoroughly familiar with the most current SOP and otherdocumentation • Attend a one‐day training session conducted by the chair or his/her designee• Observe at least three RAE analyses.
11
FAA FY 2010 non RAEs vs RAEs (Risk Analysis Events)
FY10 loSS and RAEs
Non RAEs
RAEs19%
RAEs
Non RAEs
Federal AviationAdministration
SMS Workshop – ICAO Paris, France 14‐15 Feb 2011 21
Number of Events
Non RAEs81%
FY10 Monthly RAE and LoSS count
350
400
FAA FY 2010 non RAEs vs RAEs (Risk Analysis Events) cont’d
100
150
200
250
300
LoSS Events
Number of RAEs
Federal AviationAdministration
SMS Workshop – ICAO Paris, France 14‐15 Feb 2011 22
0
50
100
Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep
12
November 26, 2010
• 552 Total Risk Analysis Events (RAEs) FY10
365 were Operational Errors (OEs)365 were Operational Errors (OEs)
187 were Pilot Deviations (PDs)
• 84 Total RAEs for FY11
51 were Operational Errors (OEs)
33 were Pilot Deviations (PDs)
Federal AviationAdministration
SMS Workshop – ICAO Paris, France 14‐15 Feb 2011 23
Leading Causal/Contributing Factors
Federal AviationAdministration
SMS Workshop – ICAO Paris, France 14‐15 Feb 2011 24
13
High Risk
Actual results of the 513 events reviewed (Nov 09 – Oct 10), using the ATO SMS risk
i d i k
FAA Risk Assessment Results
Risk Analysis Event (RAE)
1 2 3 4 5
5
Increasing Severity
# OF RAEs
ASSIGNED VALUE
(%OF RAEs
ASSIGNED VALUE)
Medium RiskLow Risk
matrix and risk assessment program jointly developed by FAA & EUROCONTROL
Detailed analyses are triggered by a loss of separation greater than 34% of standard
5
4
3
2
Increasing Likelih
ood
27 123(23 98%)
52(10 14%)
103(20 08%)
118(23%)
4(.78%)
24(4.68%)
18(3.51%)
7(1.36%)
11(2.14%)
1(0.19%)
4(.78%)
4(0.78%)
4(.78%)
3(.58%)
1( 19%)
2(.39%)
1(.19%)
Federal AviationAdministration
SMS Workshop – ICAO Paris, France 14‐15 Feb 2011 25
1(0.5%)
separation.
Serious Loss Event = High Risk Matrix Event (Red)
2
1
(5.26%)
3(.58%)
(23.98%) (10.14%)(20.08%)
2(.39%)
1(0.19%)
(.19%)
>=31 18‐30 10‐17 0‐9
not enough
Information
Severityhigh low
Number of Separation Minima Infringements (Risk ATM GND) in RAT‐Risk‐Matrix
Sample of European ANSP Risk Assessment Results
Serious Major Significant No safety effect Not determined
A B C E D‐‐> 4 15 83 124
>=32
1 0 0 0 5SMI
ATM GND
total
24‐31
2 0 1 2 8 226
17‐23
3 1 3 11 23 21
pe
ata
bil
ity
hig
h
for
SM
I, a
s th
ere
is a
lway
s io
n av
aila
ble
for
inve
stig
atio
n as
sess
men
t of r
isk
Federal AviationAdministration
SMS Workshop – ICAO Paris, France 14‐15 Feb 2011 26
11‐16
4 1 4 27 34 51
0‐10
5 2 7 43 54 154
Risk to be monitored
Risk to be mitigated
Re
low
No further action
Not
rel
evan
t fen
oug
h in
form
atan
d a
14
FAA Initial results ‐ Issues Identified using RAT
• OJTI performance
• ATC recovery methods• ATC recovery methods
• RNAV procedures and mitigation strategies
• Misapplication of Visual Separation
• Turns to Final
Federal AviationAdministration
SMS Workshop – ICAO Paris, France 14‐15 Feb 2011 27
Causal Factors Watch List
• Controller recovery actions to address immediate aircraft conflicts
‐ 41% involved ATC recovery issues
‐ 55% of the recovery weighted events were poor
‐ Out of the most severe RAE 87% involve ATC recovery issues
‐ Most severe score: 5/5
Federal AviationAdministration
SMS Workshop – ICAO Paris, France 14‐15 Feb 2011 28
Most severe score: 5/5
15
Causal Factors Watch List
• Turns to final (same altitude on parallel turn‐ons, poor )vectoring techniques, etc.)
‐ 5% of RAE
• On‐the‐Job Training instructor performance
‐ 10% of RAEs
Most severe score: 5/3
Federal AviationAdministration
SMS Workshop – ICAO Paris, France 14‐15 Feb 2011 29
‐ Most severe score: 5/3
Causal Factors Watch List
• PDs and mitigation designs for simultaneous Area Navigation (RNAV) departure applications
f‐ 13% of PD RAEs
‐ Most severe score: 4/3
• Misapplication of Visual Separation
‐ 9% of RAEs
Most severe score: 4/3
Federal AviationAdministration
SMS Workshop – ICAO Paris, France 14‐15 Feb 2011 30
‐ Most severe score: 4/3
16
Top Five Causal Contributing Factors
7948
FAA Top 5 ATC Causal Contributing Factors
5150
49
Federal AviationAdministration
SMS Workshop – ICAO Paris, France 14‐15 Feb 2011 31
A1-2. Not See A2-1. Forgot to monitor
D3-9. Controller / pilot under training D6-14. Lack of personal perception of risk
A3-2. Incorrect decision / plan
Top 5 Pilot Causal Contributing factors
33 23
FAA Top 5 Pilot Causal Contributing Factors
8636
35
Federal AviationAdministration
SMS Workshop – ICAO Paris, France 14‐15 Feb 2011 32
B1-2. Correct pilot readback followed by incorrect action
B1-9. Failed to follow ATC procedure
A2-1. Forgot to monitor
B1-6. CRM issues
B2-2. Pilot failed to / or slow to respond
17
What are the advantages of RAT?
RAT provides for:
•A means to reduce subjectivity
•Harmonisation
•Transparency
•Easier communication/discussion
•Quick process
Federal AviationAdministration
SMS Workshop – ICAO Paris, France 14‐15 Feb 2011 33
•Works as “push” to enhance investigation
•Provide hard facts in backing up Recommendation for improvement – see Recovery example
Federal AviationAdministration
SMS Workshop – ICAO Paris, France 14‐15 Feb 2011 34