Proposed Performance Metrics Block Time & Predictability
Transcript of Proposed Performance Metrics Block Time & Predictability
F E D E R A L A V I A T I O N A D M I N I S T R A T I O N • A I R T R A F F I C O R G A N I Z A T I O N 1
ProposedPerformance Metrics
Block Time & Predictability
Marc Rose, MCR/SETA-IIGabriela Rohlck, MCR/SETA-II
15 March, 2006
Presented atNEXTOR - Asilomar
F E D E R A L A V I A T I O N A D M I N I S T R A T I O N • A I R T R A F F I C O R G A N I Z A T I O N 2
Objective
An evaluation of two specific metrics: Scheduled Block Time and PredictabilityA general discussion of types of metrics
This work has been sponsored by two ATO Planning offices:1) The Office of Strategy2) The Office of Performance Analysis
F E D E R A L A V I A T I O N A D M I N I S T R A T I O N • A I R T R A F F I C O R G A N I Z A T I O N 3
Three Primary Types of Metrics
System Activity or Workload– Number of flights per month, overall average block time– Provides an estimate of workload, generally NOT performance– Impacted by non-performance based changes
• Avg Flight length - more longer flights, not performance related
Performance – Generally a system activity metric in constrained times– Normalized block time, arrival delay (sometimes)
Diagnostic– Needed to understand changes in performance
F E D E R A L A V I A T I O N A D M I N I S T R A T I O N • A I R T R A F F I C O R G A N I Z A T I O N 4
Three Primary Types of Metrics
ALL metrics have flaws/limitations:Identification of these limitations is vital to understanding
and proper usage
F E D E R A L A V I A T I O N A D M I N I S T R A T I O N • A I R T R A F F I C O R G A N I Z A T I O N 5
Data and DefinitionData Sources
– Airline Service Quality Performance (ASQP)• Data available from 1/1995 to 12/2005
– Aviation System Performance Metrics (ASPM)• Data available from 1/2000 to 12/2005
– Official Airline Guide (OAG)• Data available from 1/1982 to 9/2005
Definition:– Block Time: Airborne+Taxi-Out+Taxi-In (Gate-Out to Gate-in)– Total Flight Time: Block Time + Departure Delay
• Or: Actual arrival (in) - Scheduled Departure– Scheduled: Airline published departure/arrival times
• Based on OAG or ASPM data (pre departure)– Actual: departure/arrival times as flown (post departure)
• Based on ASQP or ASPM data
F E D E R A L A V I A T I O N A D M I N I S T R A T I O N • A I R T R A F F I C O R G A N I Z A T I O N 6
Scheduled Block Time
A Performance Metric (a lagging indicator)Metric based on OEP to OEP flights
– A current FAA Executive Council decisionConsidered the “NAS Level” (All* flights) and the OEP-OEP level (only flights to/from an OEP 35 airport)Used ASPM data, limited to those flights that have OAG data
– OAG=“Y” flag– Does not include cancelled flights
Two metrics proposed1) “Normalized” average block time
– Normalization consists of identical set of O-D pairs weighted using the average number of flights/month across all datasets
2) Delta metric - A relative change from the overall average
*Not truly ALL flights – as will be seen in the development
F E D E R A L A V I A T I O N A D M I N I S T R A T I O N • A I R T R A F F I C O R G A N I Z A T I O N 7
Normalization of Block TimeMathematical Details
∑∑
=
jiij
ijji
ijk
k n
n
,
,
ˆ
ˆµµ
N
nn
N
kijk
ij
∑== 1ˆ
ijk
lijkl
ijk n
x∑=µ
Where x is the block time for flight lfor O-D pair ij within month k andn is the number of flights for O-D pairWithin all (majority*) of data sets
Combining into a normalized NAS level “Performance” measure:
Two intermediate values are calculated1) the mean time 2) the average number of flights
µijn̂
Where N is the number of months of data
Where ij is an O-D pair,and k is month/year
F E D E R A L A V I A T I O N A D M I N I S T R A T I O N • A I R T R A F F I C O R G A N I Z A T I O N 8
Normalized ScheduledBlock Time (OEP to OEP)
Moving average (ttm)
140
141
142
143
144
145
146
147
148
149
150
Jan-0
0Apr-
00Ju
l-00
Oct-00
Jan-0
1Apr-
01Ju
l-01
Oct-01
Jan-0
2Apr-
02Ju
l-02
Oct-02
Jan-0
3Apr-
03Ju
l-03
Oct-03
Jan-0
4Apr-
04Ju
l-04
Oct-04
Jan-0
5Apr-
05Ju
l-05
Oct-05
Source: ASPM
F E D E R A L A V I A T I O N A D M I N I S T R A T I O N • A I R T R A F F I C O R G A N I Z A T I O N 9
Normalized ScheduledBlock Time (NAS Level)
Moving average (ttm)
Easy to see missing data
80
85
90
95
100
105
Jan-
82
Jan-
83
Jan-
84
Jan-
85
Jan-
86
Jan-
87
Jan-
88
Jan-
89
Jan-
90
Jan-
91
Jan-
92
Jan-
93
Jan-
94
Jan-
95
Jan-
96
Jan-
97
Jan-
98
Jan-
99
Jan-
00
Jan-
01
Jan-
02
Jan-
03
Jan-
04
Jan-
05
Nor
mal
ized
Ave
rage
Blo
ck T
ime
(min
)
Source: OAG
F E D E R A L A V I A T I O N A D M I N I S T R A T I O N • A I R T R A F F I C O R G A N I Z A T I O N 10
Scheduled Block Times A Lagging Metric
130
135
140
145
150
155
160
2000 2001 2002 2003 2004 2005
Flig
ht T
imes
(min
)
Total Flight Time (ttm)
Scheduled Block Time (ttm)
Actual Block Time (ttm)
0.97
0.98
0.99
1
1.01
1.02
1.03
1.04
1.05
Dec-95
Apr-96
Aug-96Dec
-96Apr-9
7Aug-97Dec
-97Apr-9
8Aug-98Dec
-98Apr-9
9Aug-99Dec
-99Apr-0
0Aug-00Dec
-00Apr-0
1Aug-01Dec
-01Apr-0
2Aug-02Dec
-02Apr-0
3Aug-03Dec
-03Apr-0
4Aug-04Dec
-04Apr-0
5
Rel
ativ
e C
hang
ing
Blo
cktim
e
Scheduled
Actual
Source: ASPM, OAG and ASQP
F E D E R A L A V I A T I O N A D M I N I S T R A T I O N • A I R T R A F F I C O R G A N I Z A T I O N 11
Delta Metric – Another Approach
ijk
n
lijkl
ijk n
xijk
∑== 1µ
Where x is the block time for flight land O-D pair ij within month k andn is the number of flights
∑∑==
kijk
N
kijkijk
ij n
n1ˆµ
µ A multi-month average block time
∑∑ −
=
ijijk
ijijkijijk
k n
n)ˆ( µµδ
Which is a relative performance metric for the block time that includesALL flights
F E D E R A L A V I A T I O N A D M I N I S T R A T I O N • A I R T R A F F I C O R G A N I Z A T I O N 12
Scheduled Block Time Delta Metric (OEP-OEP)
-2.5
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
2.5
Jan-00
Jul-0
0Ja
n-01
Jul-0
1Ja
n-02
Jul-0
2Ja
n-03
Jul-0
3Ja
n-04
Jul-0
4Ja
n-05
Jul-0
5
Min
utes
Data Source: ASPM
F E D E R A L A V I A T I O N A D M I N I S T R A T I O N • A I R T R A F F I C O R G A N I Z A T I O N 13
Scheduled Block TimeDelta Metric (NAS Level)
Data Source: ASPM
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
Jan-00
Apr-00
Jul-0
0Oct-
00Ja
n-01Apr-0
1Ju
l-01
Oct-01
Jan-02
Apr-02
Jul-0
2Oct-
02Ja
n-03Apr-0
3Ju
l-03
Oct-03
Jan-04
Apr-04
Jul-0
4Oct-
04Ja
n-05Apr-0
5Ju
l-05
Oct-05
Min
utes
F E D E R A L A V I A T I O N A D M I N I S T R A T I O N • A I R T R A F F I C O R G A N I Z A T I O N 14
Historical Scheduled Block Time Delta Metric (NAS Level)
-8
-6
-4
-2
0
2
4
6
Jan-
82
Jan-
83
Jan-
84
Jan-
85
Jan-
86
Jan-
87
Jan-
88
Jan-
89
Jan-
90
Jan-
91
Jan-
92
Jan-
93
Jan-
94
Jan-
95
Jan-
96
Jan-
97
Jan-
98
Jan-
99
Jan-
00
Jan-
01
Jan-
02
Jan-
03
Jan-
04
Jan-
05
Blo
ck T
ime
Cha
nges
(min
)
Moving average (ttm)
Data Source: OAG
F E D E R A L A V I A T I O N A D M I N I S T R A T I O N • A I R T R A F F I C O R G A N I Z A T I O N 15
Scheduled Block Time ChangesAccounting for Aircraft Type
-4
-3
-2
-1
0
1
2
3
Jan-
82
Jan-
83
Jan-
84
Jan-
85
Jan-
86
Jan-
87
Jan-
88
Jan-
89
Jan-
90
Jan-
91
Jan-
92
Jan-
93
Jan-
94
Jan-
95
Jan-
96
Jan-
97
Jan-
98
Jan-
99
Jan-
00
Jan-
01
Jan-
02
Jan-
03
Jan-
04
Jan-
05
Data Source: OAG
F E D E R A L A V I A T I O N A D M I N I S T R A T I O N • A I R T R A F F I C O R G A N I Z A T I O N 16
Total Flight Time Variabilitya Predictability Metric?
F E D E R A L A V I A T I O N A D M I N I S T R A T I O N • A I R T R A F F I C O R G A N I Z A T I O N 17
Percentiles as a Predictability Metric
• Start at origin-destination level:– How does measured block time of airport-pair (O-
D) differ from average timewhere:
is measured time for O-D pair (i, j) is mean time for O-D pair (i, j)
• Flight-level Mk is generated for all flights and the Selected percentiles are output as a NAS level metric
ijijk xM µ−=
ijµijx
F E D E R A L A V I A T I O N A D M I N I S T R A T I O N • A I R T R A F F I C O R G A N I Z A T I O N 18
NAS PredictabilityPerformance Metric(Total Flight Time – includes gate delay)
Source: ASQP
-30
-20
-10
0
10
20
30
40
50
60
1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005
Pred
icta
bilit
y M
etric
(min
)
F E D E R A L A V I A T I O N A D M I N I S T R A T I O N • A I R T R A F F I C O R G A N I Z A T I O N 19
Predictability Metric OEP Arrival Airports – Sep/2005
-40
-30
-20
-10
0
10
20
30
40
50
ATL
BO
S
BW
I
CLE
CLT
CVG DC
A
DEN
DFW DTW
EWR
FLL
HN
L
IAD
IAH
JFK
LAS
LAX
LGA
MC
O
MD
WM
EM MIA
MSP
OR
D
PDX
PHL
PHX
PIT
SAN
SEA
SFO
SLC
STL
TPA
F E D E R A L A V I A T I O N A D M I N I S T R A T I O N • A I R T R A F F I C O R G A N I Z A T I O N 20
Observations• Using 10th, 90th percentiles
–Eliminate significant outliers–Analysis shows distribution right-skewed.
•Average closer to lower bound than upper bound.
–Metric defined as difference between lower, upper percentiles.
•Can be calculated with one month of data.–Does not require historical data.
• Seasonality–This Metric partially accounts for known seasonality
(e.g., winter headwinds) by using current months averages rather than historical
F E D E R A L A V I A T I O N A D M I N I S T R A T I O N • A I R T R A F F I C O R G A N I Z A T I O N 21
Backup
F E D E R A L A V I A T I O N A D M I N I S T R A T I O N • A I R T R A F F I C O R G A N I Z A T I O N 22
Metric with Normalized Average Flight Time
0
10
20
30
40
50
60
70
80
90
100
Jan-95
Jul-95
Jan-96
Jul-96
Jan-97
Jul-97
Jan-98
Jul-98
Jan-99
Jul-99
Jan-00
Jul-00
Jan-01
Jul-01
Jan-02
Jul-02
Jan-03
Jul-03
Jan-04
Jul-04
Jan-05
Jul-05
Met
ric (m
in)
0
20
40
60
80
100
120
140
160
Ave
rage
Tot
al F
light
Tim
e (m
in)
F E D E R A L A V I A T I O N A D M I N I S T R A T I O N • A I R T R A F F I C O R G A N I Z A T I O N 23
Actual Block Time Variability:Percentiles and Weighted Average(10th and 90th percentiles)
100
110
120
130
140
150
16019
95
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
Blo
ck ti
me
(min
)
F E D E R A L A V I A T I O N A D M I N I S T R A T I O N • A I R T R A F F I C O R G A N I Z A T I O N 24
Predictability Metric Calculation - Example
Origin Destination Flight Time Avg Flt tm Delta Sorted DeltaA B 90 98.0 -8.0 -29.9A B 110 98.0 12.0 -14.9A B 92 98.0 -6.0 -14.8A B 100 98.0 2.0 -9.8C D 97 81.8 15.2 -8.0C D 67 81.8 -14.8 -7.9C D 72 81.8 -9.8 -7.8C D 101 81.8 19.2 -6.0C D 80 81.8 -1.8 -4.9C D 74 81.8 -7.8 -1.8E F 200 174.9 25.1 2.0E F 202 174.9 27.1 5.1E F 160 174.9 -14.9 12.0E F 145 174.9 -29.9 15.2E F 170 174.9 -4.9 19.2E F 180 174.9 5.1 25.1E F 167 174.9 -7.9 27.1
Metric = -14.8 to +21.5