Risk Assessment using Flight Data Analysis

Dr. Thom MayerAustin Digital Inc.

International Aviation Safety ConferenceJune 4, 2008

• Measuring Accident Risk using Flight Databased on work done 2005 to present(No actual Airline results are shown. Any results shown are not real data.)

• Future Directions for Analytical Risk AssessmentSuggestion of where to go next with the technique

Risk Assessment using Flight Data Analysis

FDM Provides the Ability to Measure every Flight

the mean and standard deviation of the distribution tell about “standard flight operations”

but to study safety risk we need to

look at the tails of the distribution

Risk Assessment – 2006Company Confidential - Do Not Distribute

For risk measurement purposes we need measures for closeness to each accident category

• Controlled Flight into Terrain (CFIT) ………...………………………………....... • Landing Overrun ……………………………………………………………............• Fuel Exhaustion ………………………………………………………………..........• Stall ………….………………………………………………………………….........• Hard Landing …………………………………………………………………..........• Takeoff Runway Overrun …………………………………………………….........• Landing Offside ………………………………………………………………..........• Landing Short ……….……………………………………………………...............• Runway Incursion ……….………………………………………………...............• Fuel Tank Explosion ……….……………………………………………...............

For some accident categories the measure is relatively straight-forward, but for other accident categories a good measure is more

involved.

Why does one believe that a flight that only came close to an accident represents risk?

Because variations in other variables could have caused a different outcome.( i.e. variations in winds, weight & balance, training, reaction times, …)

So it is believable that there is a relationship between accident rates (i.e. risk)

and the frequency and severity of “near misses”

a “near miss”accide

ntthresho

ldmeasure of closeness to accident category

distribution of how the event could have played out

Our approach takes advantage of a large population of flight data (i.e. 107) and the multi-year accident rate statistics

tail of distribution measured from large sample of flights

(actually it is sparse)

accidentthreshold

measure of closeness to accident category

Theoretical distribution that one could measure if we had infinite data

actualaccident“rate”

we define a mathematical measure of the tail of the distribution and “calibrated” it using a large pool of flights and the accident rate statistics for that population.

This enables one to compare risk between sub-populations

• The “weight” of the tail of the entire population represents the accident rate statistics for that population.

• The “weight” of the tail of a sub-population, compared to the “weight” of the tail for the entire population, give relative risk.

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

200 250 300 350 400

% o

f F

ligh

ts w

ith

les

s F

ue

l B747

B757

A320

A340

Airline Combined Fleets

EMS Combined Fleets

Percentage of "No Option" Fuel Remaining at Touchdown

Measure of Fuel Exhaustion Risk(Q2-2004 to Q1-2005)

< 300 % Remaining# %

B737 no data no dataB747 8 0.010B757 14 0.022A320 0 0A340 4 0.007Airline 26 0.009EMS Group 68 0.008

This Analysis is Performed for each Accident Category

and each sub-Population Drill Drown

0

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0.16

0.18

0.2

25 30 35 40 45 50

% o

f F

ligh

ts t

ha

t E

xc

ee

d

B737

B747

B757

A320

A340

Airline Combined Fleets

EMS Combined Fleets

Measure of Landing Overrun Risk(Q2-2004 to Q1-2005)

Portion of the Available Deceleration Needed during Roll Out (% )

> 25 % Decel Needed# %

B737 680 0.6B747 229 0.3B757 342 1.3A320 297 0.6A340 218 0.7Airline 1766 0.6EMS Group 9125 0.5

> 35 % Decel Needed# %

B737 2 0.005B747 1 0.002B757 4 0.010A320 6 0.012A340 2 0.005Airline 15 0.008EMS Group 39 0.005

The calculated risk is reported per sub-population(in this example the accident category is CFIT and the sub-populations are fleet type)

Risk of CFIT by Fleet (Q2-2004 to Q1-2005)

0 50 100 150 200 250 300 350 400 450 500

B737

B747

B757

A320

A340

Airline Combined Fleets

EMS Combined Fleets

Lowest Risk of any EMS Fleet

Highest Risk of any EMS Fleet

Probability of (total loss) Accident / Billion Flights

When the sub-populations are time intervals, the technique provides risk trending

This data has been corrected to remove variations resulting from the addition or removal of fleets to the FOQA program from one quarter to the next.

0

1

2

3

4

5

6

7

8

9

Q1-03 Q2-03 Q3-03 Q4-03 Q1-04 Q2-04 Q3-04 Q4-04 Q1-05

Flight Date

Pro

ba

bil

ity

of

(to

tal

los

s)

Ac

cid

en

t /

Bil

lio

n F

lig

hts

Airline Combined Fleets

EMS Combined Fleets

Airline + 1 std dev

Airline - 1 std dev

Trend in Fuel Exhaustion Risk

One can drill down into specific runways and fleet combinations(note that as you drill down, the error bars become larger)

Risk of Landing Overrun by Runway & Fleet Group(Q2-2004 to Q1-2005)

0 500 1000 1500 2000 2500 3000

BWI 33L (A340)

NGU 10 (B737)

LGA 22 (A320)

BOS 04R (B757)

ELP 22 (A340)

IND 23L (B737)

JFK 31L (A320)

MFR 32 (B747)

PDX 28 R(B747)

ROC 22 (B757)

TPA 36L (A340)

AUS 17R (A320)

CMH 28R (B737)

FLL 09L (B757)

JAN 16L (B737)

Probability of (total loss) Accident / Billion Flights

Risk Assessment – 2006Company Confidential - Do Not Distribute

Using this technique we generate an Annual Risk Assessment Report to Participating Airlines

Risk Assessment 2006

Risk Assessment – 2006Company Confidential - Do Not Distribute

CFIT44.3%

Misc0.0%

Short Landing0.6%

Struct Fail2.9%

Decompression0.4%

Engine Failure8.7%

Fire (other)5.8%

Windshear3.9%

Offside Landing1.4% Stall

3.8%

FCS Failure5.6%

Midair5.2%

Instr Fail3.9%

Rwy Overrun (RTO)0.3%

Rwy Incursion3.0%

Hard Landing1.0%

Rwy Overrun (Land)1.0%

Fire (fuel tank)1.4%

Fuel Exhaust1.3%

Rwy Overrun (T/O)5.3%

63 %

What Types of Accident Risk are Assessed for this Report?

Accident categories, representing 63% of American and European commercial jet passenger fleet fatalities were analyzed for this report.

Risk Assessment – 2006Company Confidential - Do Not Distribute

This Report is the Second Step of the Annual Risk Assessment Cycle Use this report to set goals (perhaps the reduction of a specific accident risk for each fleet)

2) Risk Assessment Report•Find “high risk” areas•Set goals•Trend progress toward meeting past goals

4) Corrective Action•Change how we operate

1) Process Risk Metrics

3) Investigation•Review “high risk” flights•What is their cause?

Risk Assessment – 2006Company Confidential - Do Not Distribute

Airline Risk Overview

For each accident category, the report to the participating Airline includes a trending of risk for the group and the airline compared to the group,

and a drill down of risk by fleet type and airport/runway.

What FDM capabilities do you need to support this type of Analysis?

• Ability to generate sophisticated measurese.g. CFIT or risk of running off end of runway

• Ability to support very low false positive rates if accident rate is 10-8 then “close call” rate might is 10-5, so false positives rate can’t be 10-3

• Ability to handle large numbers of flightsWe are getting close to 107 flights in the EMS group

What implications does this have for Data Aggregation from multiple Airlines?

• Identical processing for all dataEither the processing systems at the Airlines need to be identical,

or aggregation needs to be at the flight data level

• No “Cherry Picking” of data setParticipants need to provide complete data set. Filtering of “Incident” or “Sensitive” flights would invalidate the data set.

• Large Data Sets Requiredto get any resolution on drill-down