1ST INTERNATIONAL CROSS-INDUSTRY SAFETY CONFERENCE …

Lean Six-Sigma in Aviation Safety:

An implementation guide

for measuring aviation system’s safety performance

Ilias Panagopoulos*, Chris Atkin, Ivan SikoraCity University of London, United Kingdom

1ST INTERNATIONAL CROSS-INDUSTRY

SAFETY CONFERENCE (ICSC)

Amsterdam, 3-4 November 2016

Agenda

Introduction and Motivation

The Conceptual Framework

Implementation Guide

Findings and Conclusions

1ST INTERNATIONAL CROSS-INDUSTRY SAFETY CONFERENCE (ICSC)Amsterdam, 3-4 November 2016

Motivation:

What we are measuring….

B777- Malaysia MH370, 227pax +12crewB777-Malaysia MH17, 283pax +15crewA320- Air Asia QZ8501, 162pax + 7 crewA320- Germanwings 4U9525, 150pax + 6crew

Total Damage: 4 airliner aircrafts – 862 fatalities

Calendar Year 2014….the best year ever for airline safety….. (Ascend-Flightglobal)

January’14 December’14

Calendar Year 2015….the safest ever year…. (Aviation Safety Network)

January’15 December’15


March 2014 – March 2015

The worst year ever…..

Motivation:

Do we normalise the data…?

- Turbojets or Turboprop? - Passenger or Cargo?- Boeing or Airbus?- Embraer or Bombardier?- Western or Eastern built jet?- During take off or Landing?- Climb, descend or cruise?


Flight Duty Period

Flight TimeLimitations Runway

Excursion

Top Root CauseLeading Indicator

UnstableApproach

Precision orNon Precision

Root CauseLeading Indicator

UnstableApproachContinuedfor landing


Crew Fatigue

TimePressure

Root CauseLeading Indicators


Origin FlightDelay

Major Accidentsor

Serious Incidents

Worst credible outcomeLagging Indicator

PROACTIVEPROACTIVEPROACTIVEPROACTIVE

What do we really need to measure?

SevereWeather

(Wx) Conditions

Unscheduled Maintenance

(Mx)



EquipmentFailure


FDPFTL

Runway Excursion


UnstableApproach




Crew Fatigue

TimePressure



Origin FlightDelay

Major Accidentor

Serious Incident



Assuming, we are measuring Unstable Approaches

What is our Acceptable Level of Safety (ALoS)?

SevereWeather

Conditions

Unscheduled Mx



EquipmentFailure


2 out of 10..?5 out of 30..?8 out of 50..?10 out of 1000FH..?

FDPFTL

Runway Excursion


UnstableApproach




Crew Fatigue

TimePressure



Origin FlightDelay

Major Accidentor

Serious Incident



Assuming, we have defined ALoS

What is our Acceptable practical drift?

SevereWeather

Conditions

Unscheduled Mx



EquipmentFailure


Safety Target: No more than 10 out of 1000FH

+/-10%..?+20 %... ?+2 or +3..?By when..?

Air Asia A320, Flight QZ8501, 28 December 2014

ALoS???


The plane's rudder control system malfunctioned 4 times during the flight - a fault that occurred 23

times in the preceding year

Malaysia B777, Flight MH370, 8 March 2014

ALoS???

IAW Cargo manifest:2.5 tonnes of Lithium-ion batteries in the cargo hold of a pax a/c…!!!

‘Aviation safety is the state in which risks associated with aviation activities are reduced and controlled to an acceptable level’

ICAO Annex 19

Since 2009, in an effort to achieve an Acceptable Level of Safety (ALoS) performance

in the Air Transport industry, regulatory authorities mandate air operators

to implement a Safety Management System (SMS).

In sequence, SMS mandates the continuous measuring and monitoring

of safety performance


According to ICAO each service provider shall, as a minimum:

1) Establish an SMS

2) Identify safety hazards.

3) Ensure remedial action to maintain agreed safety performance.

4) Provide continuing monitoring and regular assessment of safety performance.

5) Aim at a continuous improvement of the overall performance of the SMS


Safety Performance (SP) is the State or a service provider’s safety achievement as defined by its SP Targets (SPTs) and SP Indicators (SPIs).

Nevertheless, in Air transport Industry the process for measuring System’s Safety Performance has not yet been introduced or standarised.

Research twofold Problem

How could aviation organisations have a proactive and performance-based approach to safety that focuses on desired,

measurable outcomes and on the management of operational risks?

How could an aviation organisation measure its overall safety effectiveness against performance goals and examine

safety performance variability from core organisational objectives?

Key Research Questions

What methodology could proactively measure system safety performance and improve the safety performancemeasurement process?Could a conceptual framework assist the continuous improvement of the safety performance measuring process?


To introduce a conceptual framework for measuring system’s safety performance and performance variability from core organizational objectives.

In addition, the study aims to provide an implementation guide on how air operators coulddesign and develop a proactive, performance-based methodology for measuring AcceptableLevels of Safety Performance (ALoSP) at sigma (σ) level, a statistical measurement unit.

Research Aim


Agenda







The conceptual framework

The Safety-Performance Indicator Lean Sigma (Safety-PILS) model embedded within the

Define-Measure- Analyse-Improve-Control (DMAIC) continuous improvement process.

The Safety-PILS model

provides a holistic view on how organisations could set:

-leading indicators and monitor metrics on the top of identified root-causes

-lagging indicators and feedback metrics on the top of safety outcomes


The Safety-PILS model

aims to control and maintain safety performance within agreed Upper or Lower Specification Limits and to develop an objective methodology that will proactively investigate and measure system performance variability within ±1.5 sigma (σ) from an ALoS Performance target.


Through DMAIC process

the operator could apply Lean-Six Sigma (L6S) methodology for measuring both the performance of each established indicator and system safety performance variability

at sigma level from core safety objectives.


Agenda






The implementation guide is divided in two phases,

Phase-I and Phase-II.

Phase-I is mainly the utilisation of the Safety-PILS model

Phase-II the practical implementation of the DMAIC

process.

Both Phase I and Phase II are forming the conceptual

framework.



for measuring system’s safety performance

Phase-I.

Safety-PILS model utilisation

1. Design Safety-PILS model for VOB

- Define the driven KPI for the VOB

- Set the VOB Targets and the LSL-USL based on industry standards

- Set SPIs on the VOB- Critical to Safety (CTS) characteristics

- Set metrics on each VOB SPI and the associated LSL-USL

2. Correlation and Multiple Regression Analysis (or Pareto Analysis)

- Identify correlation between cause/effect


Phase-II. : Apply Six Sigma-DMAIC methodology

3. Data Collection Planning (DCP) for Hypotheses tests

- Hypothesis Testing - Data normalization

4. Control Chart selection - road map

- Control Chart selection for each VOB SPI and Metric

- Identify special causes: If none the process is In-Control

5. Measurement System Analysis (MSA)

- Where does the variation of data comes from?

- Is the process Accurate and Precise?

6. Process Capability

- Is the process capable (i.e. efficient)?

- At what sigma level?

7. Analyse the data

- Identify root cause and attractive areas for improvement

- Identify best and feasible solutions

8. Pilot solutions

- Demonstrate that piloted solution provides a Return of Investment (ROI)

9. Define Control Plan and Roll-out improvement

- Monitor the Control Plan to sustain the change

10. Measure total system safety performance1ST INTERNATIONAL CROSS-INDUSTRY SAFETY CONFERENCE (ICSC)



for measuring system’s safety performance

Phase-I.

Safety-PILS model utilisation

1. Design Safety-PILS model for VOB

- Define the driven KPI for the VOB

- Set the VOB Targets and the LSL-USL based on industry standards

- Set SPIs on the VOB- Critical to Safety (CTS) characteristics

- Set metrics on each VOB SPI and the associated LSL-USL

2. Correlation and Multiple Regression Analysis (or Pareto Analysis)

- Identify correlation between cause/effect


The Table shows two indicative indicators, with their associated metrics. SPI_02 and SPI_06

have been selected as a research sample for further explaining and validating the

conceptual framework implementation step guide.

SPI_02: Runway excursions (RE) SPI_06: Loss of Control (LOC)

Metric 02.1: Deep Landing events Metric 06.1: Stick-shake and alpha floor events

Metric 02.2: Unstable-De-stabilized approaches (all) Metric 06.2: Take off Configuration warnings

Metric 02.3: Unstable-De-stabilized approaches (all) continued for

landing

Metric 06.3: Low speed during cruise events

Metric 02.4: High speed touchdown events Metric 06.4: Low speed during approach events

Metric 02.5: High speed rejected take-off events Metric 06.5: Percentage of pilot’s readiness rate for proficiency

Metric 02.6: Take-off landing events involving loss of aircraft directional

control caused by contaminated runway surface

Metric 06.6: Pilot’s utilisation effectiveness

Metric 02.7: Runway and Overrun events due to runway contamination Metric 06.7: Percentage of pilots received upset recovery training

Metric 02.8: Proportion of licensed aerodromes using new reporting

criteria for runway surface condition

Metric 06.8: Percentage of qualified and current pilot’s availability

rate


Safety-PILS model utilization for the design of the performance indicators SPI_02 & SPI_06

and its associated metrics. For the purpose of this study,

Metric 02.2: Unstable-De-stabilized approaches (all)

Metric 06.2: Take-off configuration warnings events

have been selected as an indicative sample for validating the conceptual framework implementation step guide


The research study examined the correlationexist among VOB indicators and metrics andrevealed a moderate to strong correlation,since all examined R values are ranging from0.6- 0.8.

In addition, the multiple regression analysisof the study examined the VOB indicatorsand metrics and revealed with 95%confidence that the selected VOB SPIs andmetrics are important or very importantfactors to the VOB process, since allexamined R-Squared values are ranging from44.74 - 83.38%.

Also, the study revealed that Metric 02.2 (i.e.Unstable approaches), is the one thataccounts the most for the variation in theSAC VOB process output.

C8 = 1.0536 - 0.279 X1 + 0.0768 X1^2

Step Change Step P Final P

1

Add X1^2

Add X1

0.021

0.263

0.021

0.263

1007550250

R-Squared(adjusted) %

1.1876484560

0.5938242280

151050

Increase in R-Squared %

43210

1.00

0.75

0.50

0.5938242280

C8

1.1876484560

0.5938242280

100500

R-Squared %

X1: Unstable App X2: T/O Conf. Warnings

X1: 0.5938242280 X2: 1.1876484560

Final Model Equation

Model Building SequenceDisplays the order in which terms were added or removed.

Incremental Impact of X VariablesLong bars represent Xs that contribute the most new

information to the model.

Fitted Line Plot for 0.5938242280

Shows the relationship between C8 and 0.5938242280.

Each X Regressed on All Other Terms

Gray bars represent Xs that do not help explain

additional variation in Y.

A gray bar represents an X variable not in the model.

Multiple Regression forVOBModel Building Report



Phase-II.

Apply Six Sigma-DMAIC methodology

3. Data Collection Planning (DCP) for Hypotheses tests

- Hypothesis Testing – Data are normalised

4. Control Chart selection - road map

- Control Chart selection for each VOB SPI and Metric

- Identify special causes: If none the process is In-Control

5. Measurement System Analysis (MSA)

- Where does the variation of data comes from?

- Is the process Accurate and Precise?

6. Process Capability

- Is the process capable (i.e. efficient)?

- At what sigma level?

7. Analyse the data

- Identify root cause and attractive areas for improvement

- Identify best and feasible solutions

8. Pilot solutions

- Demonstrate that piloted solution provides a Return of Investment (ROI)

9. Define Control Plan and Roll-out improvement

- Monitor the Control Plan to sustain the change

10. Measure total system safety performance

- Voice of the safety Process (VOP)= VOB


Data Collection Planning (DCP)

for hypotheses test

Decide on:

• what type of data is most appropriate to

collect for measuring the VOB SPI and

metrics,

• what resolution is needed,

• what statistical tool should be used to

interpret the data and

• what the sample size and frequency

should be.1ST INTERNATIONAL CROSS-INDUSTRY SAFETY CONFERENCE (ICSC)



Hypotheses test: Normalise the data



Hypotheses test

43210-1

4

3

2

1

0

Mean 1.568

StDev 1.185

N 23

0.5938242280285035

Freq

uen

cy

Normal

Histogram of Unstabilised App

3210-1

5

4

3

2

1

0

Mean 1.220

StDev 0.9272

N 23

1.187648456057007

Freq

uen

cy

Normal

Histogram of T/O Configuartion Warnings


2321191715131197531

4

2

0Su

bg

rou

p M

ean

_X=1.394

UCL=3.559

LCL=-0.771

2321191715131197531

2

1

0MR

of

Su

bg

rou

p M

ean

__MR=0.814

UCL=2.660

LCL=0

2321191715131197531

4

2

0

Sample

Sam

ple

Ran

ge

_R=1.262

UCL=4.124

LCL=0

I-MR-R/S (Between/Within) Chart of Unstable App and T/O Conf, warnings

Control Chart selection: Process is in-control



Process Capability: Is the process capable?


Data Analysis

• Cp> Cpk meaning that the potential process for all metrics was centered.

• Cpk is less than 1, meaning that not only a special cause but also a common cause

of variation was going to produce unacceptable variation (i.e. defects).

• Cp=0.17 means that only 17% of the process fit within USL/LSL and Cpk=-0.08

means the process was 80% over one specification limit.

• The control charts revealed that there was no special cause of variation,

meaning that this process was in-control.

• However, neither the actual process nor the potential was capable.

• At this point the operator needs to apply solutions or to take mitigation measures


Process NOT in control


Normality Test

Results Pass

P-value 0.448

(Anderson-Darling)

1.2

0.9

0.6

Mea

n

121110987654321

0.30

0.15

0.00

Ran

geXbar-R Chart

Confirm that the process is stable.

Normality Plot

The points should be close to the line.

Capability Analysis for VOB before (C8)Diagnostic Report

Before implementing Solutions• The actual process is centered

• The process is performing at 0.98+1.5 sigma level (i.e. z actual

value), which is equal to 2.75 sigma performance.

• After implementing solutions, the process will perform at

2.21+1.5 sigma level (i.e. z potential value), which is equal to

3.71 sigma performance.

• The data are following normal distribution pattern since the P

value=0.441which is greater than the threshold value of 0.05.

• The defect rate is 16.43% which estimates the percentage of parts

from the process that are outside the specification limits.

• The DPMO is 164248, meaning that the operator currently

experiences 164248 defects per 1 million flying hours or 164 defects

per 1000 flying hours related to unstable approaches and take-off

configuration warning events.

• The actual process mean which is 0.90783 does not differ

significantly from the target which is 0.94.1ST INTERNATIONAL CROSS-INDUSTRY SAFETY CONFERENCE (ICSC)



• It appears that the actual capability is

improving, with 7.97% of the process being

out of the specification limits with 79711

DPMO.

• In addition, the process is centered still, since

Cp>Cpk.

• This report verifies that data are continuing to

follow a normal distribution since P

value=0.564 which is greater than the

threshold value of 0.05.

• Besides, the Xbar-R chart shows none special

causes for the existing variation.



After implementing Solutions

Agenda






Before / After Capability Comparison



Summary Results

The process performance report shows the improvements achieved

`AFTER the implementation of solutions as follows:

• The percentage out of the specification limits has been significantly reduced by 51%, meaning from

16.43% before to 7.97%.

• The STDV (i.e. variation) was significantly reduced by 7%.

• The actual process performance has been increased by 0.52%

• The sigma level has been increased by 43%,

• The DPMO have been reduced by -84577.

• The actual sigma level performance is 1.41+1.5 sigma which equals to 2.91 sigma level performance.


Measuring total system safety performance

Voice of the Safety Process (VOP) = VOB = Total system’s safety performance

At the end of Sep 2016 the VOB (i.e.

occurrences rate) had achieved its target 0.06,

meaning that the total percent defective is

0.068% and the percentage yield or

acceptance rate is 99.32%.

The results of 0.046% or 99.954% indicate that

the total system’s safety effectiveness was

approaching 4 sigma performance with the

potential to have 6210 defects per million

opportunities (i.e. occurrences per flying

hours).


Recommendations


The paper recommends the application of the conceptual framework to:

- different settings,

- different sample,

- different type of SPIs/metrics, and

- qualitative validate the results by interviewing Subject Matter Experts (SMEs).

As further research the study recommends:

- the application of Genetics Algorithms and Simulation to the implementation

guide. In this case, a metaheuristic procedure could sample a set of solutions

and select or generate an algorithm that provides a sufficiently good solution to

this safety performance optimization problem.

Lean Six-Sigma in Aviation Safety:

An implementation guide

for measuring aviation system’s safety performance

Ilias Panagopoulos*, Chris Atkin, Ivan SikoraCity University of London, United Kingdom

1ST INTERNATIONAL CROSS-INDUSTRY

SAFETY CONFERENCE (ICSC)


Thank you for your attention!

1ST INTERNATIONAL CROSS-INDUSTRY SAFETY CONFERENCE …

Documents

Transcript of 1ST INTERNATIONAL CROSS-INDUSTRY SAFETY CONFERENCE …