1
Safety Assessment
February 2006
2
SAFETY ASSESSMENTSAFETY ASSESSMENT
A Safety Assessment is essentially a process for finding answers to three fundamental questions:
What could go wrong? What would be the consequences? How often is it likely to occur?
Once we know the answers this automatically raises the next question:
Is this acceptable? What can we do if not?
3
SAFETY ASSESSMENTSAFETY ASSESSMENT
Consequently, the objective of Safety Assessments is to:
ensure that the system operates normally and without exposing unacceptable risks to anyone;
reduce and prevent incidents and accidents and;
limit the consequences of any occurrence that might occur.
The Scope of the Safety Assessments includes: Safety Assessment on Air Navigation Systems
covering people, procedures and equipment; … does not address Air Navigation System
“certification” issues; … does not address organisational and
management aspects related to safety assessment.
4
SAFETY ASSESSMENTSAFETY ASSESSMENT
Safety A condition in which the risk of harm or
damages is limited to an acceptable level
Risk The probable rate of occurrence of a hazard
causing harm and the degree of severity of the harm
Risk = Severity * likelihood
Need to define severity and likelihood Need to define acceptability
5
SEVERITY CLASSIFICATIONSEVERITY CLASSIFICATION
Severity Classification Scheme
1 Accident One or more catastrophic accident One or more mid-air collision One of more collisions on ground between two aircraft No independent source of recovery mechanism, such as surveillance or ATC / Flight Crew procedure, can
reasonably be expected to prevent the accident(s)
2 Serious Incident large reduction in separation (e.g. a separation of less than half the separation minima), without crew or ATC fully controlling the situation or able to recover from the situation.
one or more aircraft deviating from their intended clearance, so that abrupt manoeuvre is required to avoid collision with another aircraft or with terrain (or when an avoidance action would be appropriate).
3 Major Incident large reduction in separation (e.g. a separation of less than half the separation minima), without crew or ATC fully controlling the situation or able to recover from the situation.
Minor reduction in separation (e.g. a separation of more than half the separation minima), without crew or ATC fully controlling the situation, or able to recover from the situation, jeopardising the ability to recover without use of collision or terrain avoidance manoeuvres
4 Significant Incident Increased workload on ATCO or Flight Crew or slightly degrading capability of the CSN system Minor reduction in separation (e.g. a separation of more than half the separation minima), without crew or
ATC fully controlling the situation, or able to recover from the situation and fully able to recover the situation
5 No immediate effect on safety
No immediate direct or indirect impact on operations
6
LIKELIHOOD CLASSIFICATIONLIKELIHOOD CLASSIFICATION
Likelihood Classification Scheme
1 Frequently Likely to occur frequently (often)
2 Probable Likely to occur several times during the life-time of the system (2-5 occurrences per year)
3 Occasional Occurs sometimes during the life-time of the system (1 occurrence per year)
4 Remote Unlikely to occur sometimes during the life-time of the system (1 occurrence per 5 years)
5 Improbable Very unlikely to occur (1 occurrence per 20 years)
6 Extremely Improbable Extremely unlikely to occur (1 occurrence per 100 years)
7
RISK CLASSIFICATIONRISK CLASSIFICATION
Risk Classification
Probability Severity
Probability Qualitative Definition Quantitative
Definition 1 2 3 4 5
Frequently Likely to occur frequently. > 5*10-4 A A A A C
Probable Likely to occur several times during system life.
< 5*10-4 A A A B D
Occasional Occurs sometime during system life. < 1*10-5 A A B C D
Remote Unlikely to occur sometimes during system life.
< 1*10-6 A B C D D
Improbable Very unlikely to occur. < 1*10-7 B C D D D
Extremely Improbable
Extremely unlikely to occur. < 1*10-8 C D D D D
Likelihood
Likelihood
8
AS LOW AS REASONABLE PRACTICABLEAS LOW AS REASONABLE PRACTICABLE
The risk is less than the pre-determined unacceptable limit,
The risk has been reduced to a level which is as low as reasonable practicable (ALARP) and
The benefits of the proposed system or changes are sufficient to justify accepting the risk
All three of the above criteria should be satisfied before a risk is classed as tolerable
9
SAFETY ASSESSMENTSAFETY ASSESSMENT
ICAO SEVEN STEP APPROACH Hazard Identification and Estimation steps
Step 1 – System and Environment Description Step 2 – Hazard Identification Step 3 – Hazard Severity Step 4 – Hazard Likelihood
Mitigation steps Step 5 – Risk Evaluation Step 6 – Risk Mitigation
Documentation Step 7 – Safety Assessment Documentation
10
STEP 1 - DESCRIPTIONSTEP 1 - DESCRIPTION
Before a safety assessment can be performed, we need to describe the ATM system being assessed. For that purpose we need (as a minimum):
System Description;
Operational Environment Description.
11
STEP 1 - DESCRIPTIONSTEP 1 - DESCRIPTION
A detailed system description should include:
the purpose of the system; how the system will be used; a description of system functions; the system boundaries and the external interfaces; where appropriate, the transition procedures from the previous
system to the new system, including any hazards associated with the decommissioning of the previous system;
description of contingency procedures and other procedures for non-normal operations;
other input such as other safety assessment results, occurrence and investigation reports, lessons learnt etc.;
regulatory framework and applicable standards.
12
STEP 1 - DESCRIPTIONSTEP 1 - DESCRIPTION
A detailed operational environment description should include:
traffic characteristics; weather characteristics & weather-related factors (e.g. average
frequency of diversions due to severe weather); topography; aircraft performance and equipment; infrastructure modes and limitations including e.g. runway in use,
closed taxiways etc; environmental constraints; characteristics of the users of the system; adjacent centre capabilities; …and other input concerning the environment in which the system
is to be operated.
13
HAZARD IDENTIFICATION AND ESTIMATION PROCESS
hazard
hazard
hazardhazard
hazardhazard
Brainstorming – Hazard Identification
hazard
hazard
hazardhazard
hazardhazard
hazard
hazard
hazardhazard
hazardhazard
hazard
hazard
hazardhazard
hazardhazard
Brainstorming – Hazard Identification
hazard1 can lead to?hazard2 can lead to?hazard3 can lead to?
--
hazard1 can lead to?hazard2 can lead to?hazard3 can lead to?
--
Identification of Hazard Consequences
1. Introduction
2. Methodology
3. Operational Environment
4. Scenario
5. Classification Schemes
6. Example
Briefings what are the potentialwhat are the potentialconsequences?consequences?
what can go wrong?what can go wrong?
Catastrophic ?Major Incident ?
Negligible?--
Catastrophic ?Major Incident ?
Negligible?--
Identification of Severities
How severe can it become?How severe can it become?
Frequently ?Occasionally ?
Negligible ?--
How often can it occur?How often can it occur?
List of 10 most safety-critical hazardsIdentification of Likelihood of Occurrence
1
2 3
4
6
5
14
STEP 2 – HAZARD IDENTIFICATIONSTEP 2 – HAZARD IDENTIFICATION
Purpose
…to identify what could go wrong!(- or anticipate problems before they occur…)
….to identify the consequences (on safety) of the hazards
A hazard is defined as any condition, event or
circumstances which could induce an accident
or incident (ICAO DOC 9422)
The equipment (hardware and software);
The operating environment; The human operators; The human machine interface (HMI); Operational procedures; Maintenance procedures; External services.
15
STEP 2 – HAZARD IDENTIFICATIONSTEP 2 – HAZARD IDENTIFICATION
…to identify the consequences of the hazards on operation!
A hazard consequence is defined as the potential effects on operation that a hazard may create
The operational consequences list the effects the hazard will have on the operation and emphasise the impact / changes the hazard will introduce compared with “normal operation”.
The safety consequences are derived from the operational consequences by deciding the impact on the safe provision of ATS. E.g. potential loss of separation.
- increased receive/transmit- increased co-ordination
- increased receive/transmit- increased co-ordination
- potential loss of separation- potential loss of separation
16
17
18
STEP 2 – HAZARD IDENTIFICATIONSTEP 2 – HAZARD IDENTIFICATION
The hazard identification step should consider all the possible sources of system failure. Depending on the nature and size of the system under consideration these could include:
The equipment (hardware and software); The operating environment (including physical
conditions, airspace and air route design); The human operators; The human machine interface (HMI); Operational procedures; Maintenance procedures; External services.
19
STEP 2 – HAZARD IDENTIFICATIONSTEP 2 – HAZARD IDENTIFICATION
Methodologies
Brainstorming;
Vision Conferences;
Historical Records of Incidents;
Checklists;
Other systematic methods.
20
STEP 2 – HAZARD IDENTIFICATIONSTEP 2 – HAZARD IDENTIFICATION
Preferred Methodology
Brainstorming because: Easy and straightforward process. No need to
complicate or make too academic! Such group sessions are usually good at
generating ideas and identifying issues – mutual inspiration;
The interactions between participants with varying experience and knowledge tend to lead to broader, more comprehensive and more balanced consideration of safety issues.
21
STEP 2 – HAZARD IDENTIFICATIONSTEP 2 – HAZARD IDENTIFICATION
WHAT IF?MODERATOR
ATCO
SYSTEMEXPERT
SAFETYEXPERT
Brainstorming Process
interactive session facilitated by a moderator experts encouraged to bring
forward any safety-related issue they can think of
based upon pre-developed scenarios
first step: identify hazards second step: identify
consequences of the hazards
22
STEP 2 – HAZARD IDENTIFICATIONSTEP 2 – HAZARD IDENTIFICATION
Participants participants should be chosen for their expertise in fields
relevant to the project being assessed.
Such experts usually include System users/operational experts: ATCOs and Flight
Crew (where necessary), to assess the consequences of hazard(s) from an operational perspective;
System technical experts, to explain the system purpose, interfaces and functions;
Safety and human factors experts, to guide in the application of the FHA methodology itself and to bring wider experience of the consequences of hazards.
23
STEP 2 – HAZARD IDENTIFICATIONSTEP 2 – HAZARD IDENTIFICATION
EXAMPLE
24
STEP 3 – SEVERITY ASSESSMENT
The severity expresses the impact on operation or the harm an individual may suffer.
Severity Classification is a gradation, ranging from "worst case/accident" to "no safety impact" – expressing the magnitude of the consequence of the hazard.
Thus, a severity is allocated each hazard consequence in accordance with the agreed severity classification scheme.
25
STEP 3 – SEVERITY ASSESSMENT
Severity Classification Scheme
1 Accident One or more catastrophic accident One or more mid-air collision One of more collisions on ground between two aircraft No independent source of recovery mechanism, such as surveillance or ATC / Flight Crew procedure, can
reasonably be expected to prevent the accident(s)
2 Serious Incident large reduction in separation (e.g. a separation of less than half the separation minima), without crew or ATC fully controlling the situation or able to recover from the situation.
one or more aircraft deviating from their intended clearance, so that abrupt manoeuvre is required to avoid collision with another aircraft or with terrain (or when an avoidance action would be appropriate).
3 Major Incident large reduction in separation (e.g. a separation of less than half the separation minima), without crew or ATC fully controlling the situation or able to recover from the situation.
Minor reduction in separation (e.g. a separation of more than half the separation minima), without crew or ATC fully controlling the situation, or able to recover from the situation, jeopardising the ability to recover without use of collision or terrain avoidance manoeuvres
4 Significant Incident Increased workload on ATCO or Flight Crew or slightly degrading capability of the CSN system Minor reduction in separation (e.g. a separation of more than half the separation minima), without crew or
ATC fully controlling the situation, or able to recover from the situation and fully able to recover the situation
5 No immediate effect on safety
No immediate direct or indirect impact on operations
26
STEP 4 – LIKELIHOOD ASSESSMENT
The likelihood of occurrence expresses how often the consequence of a hazard is likely to occur.
Likelihood Classification is a gradation, ranging from "frequently" to “extremely improbable".
Thus, a likelihood is allocated each hazard consequence in accordance with the agreed likelihood classification scheme.
27
STEP 4 – LIKELIHOOD ASSESSMENT
Likelihood Classification Scheme
1 Frequently Likely to occur frequently (often)
2 Probable Likely to occur several times during the life-time of the system (2-5 occurrences per year)
3 Occasional Occurs sometimes during the life-time of the system (1 occurrence per year)
4 Remote Unlikely to occur sometimes during the life-time of the system (1 occurrence per 5 years)
5 Improbable Very unlikely to occur (1 occurrence per 20 years)
6 Extremely Improbable Extremely unlikely to occur (1 occurrence per 100 years)
28
STEP 3 & 4 – SEVERITY AND LIKELIHOODSTEP 3 & 4 – SEVERITY AND LIKELIHOOD
EXAMPLE
29
STEP 5 & 6 – RISK EVALUATION AND MITIGATION
Is this risk acceptable?
We have a risk
with a defined likelihood
and severity
Acceptablerisks
No
Yes
Notacceptable
risks
One of the causes
training of
Discussion of causes and failures
What are the potential causes
could be insufficientThis consequence
prevented if
How can we resolve it?
Discussion of Risk Mitigation
could be reduced or
Risk Mitigation Plan
Mitigation willremove risk
Mitigation willnot remove risk
Residualrisk
acceptable?
Riskmitigation
impracticable?
Mitigation impracticable
Openrisks
Discussion of acceptability
30
STEP 5 – RISK EVALUATION
Determine what is / is not acceptable Acceptable level of Safety
Determine acceptability of identified risks Clearly unacceptable Clearly acceptable May be / may be not acceptable
Risk Classification
Probability Severity
Probability Qualitative Definition Quantitative
Definition 1 2 3 4 5
Frequently Likely to occur frequently. > 5*10-4 A A A A C
Probable Likely to occur several times during system life.
< 5*10-4 A A A B D
Occasional Occurs sometime during system life. < 1*10-5 A A B C D
Remote Unlikely to occur sometimes during system life.
< 1*10-6 A B C D D
Improbable Very unlikely to occur. < 1*10-7 B C D D D
Extremely Improbable
Extremely unlikely to occur. < 1*10-8 C D D D D
likelihood
likelihood
31
STEP 5 – RISK EVALUATION
Performed by a small group System users/operational experts: ATCOs and Flight Crew
(where necessary), to assess the consequences of hazard(s) from an operational perspective;
System technical experts, to explain the system purpose, interfaces and functions;
Safety and human factors experts, to guide in the application of the FHA methodology itself and to bring wider experience of the consequences of hazards.
May need to be extended with specialists in areas relevant for the ALARP assessment
32
STEP 5 – RISK EVALUATIONSTEP 5 – RISK EVALUATION
EXAMPLE
33
STEP 6 – RISK MITIGATIONSTEP 6 – RISK MITIGATION
Identify potential causes for a risk to occur Some causes are identified during the hazard
identification Ensure that we have identified all causes
Identify potential mitigation Remove the risk (remove the cause of the risk) Reduce the risk
Reduce severity and/or probability
Identify preferred mitigation approach
34
Risk Classification
Probability Severity
Probability Qualitative Definition Quantitative
Definition 1 2 3 4 5
Frequently Likely to occur frequently. > 5*10-4 A A A A C
Probable Likely to occur several times during system life.
< 5*10-4 A A A B D
Occasional Occurs sometime during system life. < 1*10-5 A A B C D
Remote Unlikely to occur sometimes during system life.
< 1*10-6 A B C D D
Improbable Very unlikely to occur. < 1*10-7 B C D D D
Extremely Improbable
Extremely unlikely to occur. < 1*10-8 C D D D D
likelihood
likelihood
STEP 6 – RISK MITIGATIONSTEP 6 – RISK MITIGATION
35
STEP 6 – RISK MITIGATIONSTEP 6 – RISK MITIGATION
Risk mitigation should be sought in any of the three components of a system:
People Procedures Equipment
The possible approaches to risk mitigation include:
revision of the system (or airport) design; modification of operational procedures; changes to staffing arrangements; and training of personnel to deal with the hazard.
36
STEP 6 – RISK MITIGATIONSTEP 6 – RISK MITIGATION
To identify causes a number of techniques may be required
Brainstorming sessions Fault tree analysis - Effect tree analysis Common cause failure identification (Single point
failure) Task, Fail-Safe & Error Tolerance Analysis Failure Mode and Criticality Analysis Reliability, Availability and Maintainability Analysis
Focus on components giving: Highest likelihood Highest degree of severity
37
STEP 6 – RISK MITIGATIONSTEP 6 – RISK MITIGATION
Performed by a small group System users/operational experts System technical experts Safety and human factors experts
Different experts may be required to: Performed detailed studies of the causes of a risk
Study system design to determine component potentially causing, e.g. loss of air situation display
Study procedures to determine where e.g. misunderstandings can arise
Ways to remove those causes
38
STEP 6 – RISK MITIGATIONSTEP 6 – RISK MITIGATION
SW
Hazard
S
F
S
S
F
F
Effect 1
Effect 2
Effect 3
Effect 4
P=Likelihood
E = Severity
PR
P=Likelihood
Failure Recovery
Fault Tree and Effect Tree Analysis
39
STEP 6 – RISK MITIGATIONSTEP 6 – RISK MITIGATION
Procedure Assurance Level Procedure development effort should be proportional to the
potential Risk associated with the Procedure. To achieve this, objective PAL should be determined and satisfied.
PAL is setting some objectives to be met during the different phases of the procedure life cycle – Table 1.
PAL objectives are applicable to the entire Procedure, not only to some part of it.
40
STEP 6 – RISK MITIGATIONSTEP 6 – RISK MITIGATION
Level Definition Design and validation
Implementation Transfer in operations
Operations
3 Other/own experience benchmarking
Specification quality assurance
Fast time simulation Qualitative risk
assessment Pre-implementation
trials
Dedicated training Staff acceptance
argumentation Quality assurance of
implementation
Competency argument for the staff to perform transfer
Contingency plan
Regular proficiency checks
4 Other/own experience benchmarking
Specification quality assurance
Fast time simulation Qualitative risk
assessment Pre-implementation
trials
Quality assurance of implementation
Contingency plan Regular proficiency checks
Procedure Assurance Level
41
STEP 6 – RISK MITIGATIONSTEP 6 – RISK MITIGATION
Software Assurance Level Software development effort should be proportional to the
potential Risk associated with the Software. To achieve this, objective SWAL should be determined and satisfied.
SWAL is setting some objectives to be met during the different phases of the software life cycle.
SWAL objectives are applicable to the software component is question (only some part of of the total software).
42
STEP 6 – RISK MITIGATIONSTEP 6 – RISK MITIGATION
LevelRequirement
1 2 3 4
37.3 Unit, integration and system testing
37.3.1 Unit and integration tests shall be conducted on individual units and on partially integrated units to demonstrate that the software is executable and that it produces the expected results for the specified test cases.
M M M M
37.3.3 Integration tests shall as a minimum demonstrate the correctness of all interfaces.
J1 J2 M M
M Mandatory requirement to the development processJ1 Justification is to be provided if the clause or part of the clause is not followedJ2 Justification for the omission or non-compliance is to be provided
Extract from DEF-STAN-55
43
STEP 6 – RISK MITIGATIONSTEP 6 – RISK MITIGATION
Mitigation actions (safety requirements) should be carefully analysed:
Will the mitigation remove the risk or reduce the risk (what will be remaining risk be)
Will the implementation introduce any new hazards (repeat step 3, 4 and 5)
Mitigation actions shall be documented Risk Mitigation Plan
44
STEP 6 – RISK MITIGATIONSTEP 6 – RISK MITIGATION
EXAMPLE
45
STEP 7 - SAFETY ASSESSMENT DOCUMENTATION
The purpose: To provide a permanent record of the final result of
the safety assessment To provide the arguments and evidence
demonstrating that the risks associated with the implementation of the proposed system or change:
have been eliminated, or have been adequately controlled and reduced to a
tolerable level.
46
STEP 7 - SAFETY ASSESSMENT DOCUMENTATION
Should contain a summary of: Methods used Safety criteria (the agreed safety levels) Results of the hazard identification process (including Hazard Logs) Risk mitigation required (safety requirements) Follow-up actions Evidence of compliance with safety requirements
References should be included Evidence of validity of assumptions
47
DIFFICULTIES – DIFFICULTIES – SAFETY ASSESSMENTSAFETY ASSESSMENT
General Complex, resource-demanding activity
Target Levels of Safety (Severity and Likelihood) Complexity No guidelines or recommendation – in most cases not even
statistics No guidelines to apportioning Safety Targets to lower levels No guidelines to who does what (Regulator Provider
Supplier)
48
DIFFICULTIES – DIFFICULTIES – SAFETY ASSESSMENTSAFETY ASSESSMENT
Risk Mitigation Very demanding concepts (software assurance
levels, procedure assurance levels) Very demanding activities for risk mitigation Analyses required beyond reach for many
organisation
49
RECOMMENDATIONSRECOMMENDATIONS
Start with low level of ambition Even simple Safety Assessment provides quite
efficient risk mitigation Introduce more advanced features once the simple
version works Start with quantitative likelihood classification while
data are collected to establish qualitative figures Make sure assumptions are well-defined
and traced
50
RECOMMENDATIONSRECOMMENDATIONS
Don’t forget to design a follow-up system for (ICAO 2.26.5)
Hazards (likelihood for different causes) Assumptions, e.g.:
Capacity figures Reliability figures
Should be extracted from the reporting system
51
SUPPORTING SLIDESSUPPORTING SLIDES
52
Target Level of SafetyTarget Level of Safety
MET NAV/Enr NAV/Term
Ground TWR APP ACC
Safety factor for Accidents (1,55 10-8 per Flight hour)
Mid-air collision ÷
Controlled flight into terrain
÷
Accident on ground with fatalities
÷ ÷ ÷
……
Safety Factors for Serious Incidents
Separation minima infringement (less than
50%)
÷
Runway incursion with avoiding action
÷ ÷ ÷
……
Top Related