Data-Driven Precursor Detection Algorithm for Terminal ... · Data-Driven Precursor Detection...
Transcript of Data-Driven Precursor Detection Algorithm for Terminal ... · Data-Driven Precursor Detection...
-
School of Aeronautics and Astronautics
Data-Driven Precursor Detection Algorithm for Terminal Airspace Operations
Raj Deshmukh, Dawei Sun and Inseok Hwang
School of Aeronautics and AstronauticsPurdue University
Thirteenth USA/Europe Air Traffic Management Research and Development Seminar (ATM2019)
-
Presentation Outline
• Research background and motivation
• Algorithm development: Precursor detection in terminal airspace
– Preprocessing
– Anomaly detection and precursor detection model generated in the
form of signal temporal logic (STL)
– Test and analysis
• Conclusion and future work
-
Data-driven Learning for ATM• Modern ATM technologies record detailed information regarding states of aircraft,
and status of airport as time-series datasets
• Detecting outliers (called anomalies) in such time-series datasets allows understanding air transportation system complexity and behaviour, while getting insight into emergent behaviour Anomalies have correlation to operational or safety issues Anomaly detection by manually hunting through datasets is not feasible, requiring the
use of machine learning (data-driven) algorithms
ASPM
ASRS
TAIS
ASDE-XADS-B
ETMSSWIM
ITWS
ITWS
METAR
TBFM
TFMS
TAIS
ASDE-X
-
Level-set Definitions
• Anomaly: a rare event with a meaningful impact. Anomaly is different from coincidence, and must have potential for some operational or safety related impact
• Precursor: Events or conditions of the systems that have correlation to occurrence of anomaly and precede it
-
• Develop machine learning models to identify safety-related and operationally-significant anomalies in time-series observations of surveillance datasets in terminal airspace
• Use results from anomaly detection to predict anomalies before they occur, using precursor detection (prognosis)
• Terminal airspace (or terminal control area) Controlled airspace surrounding major airports, with highest volume of traffic
Problem Setup
Data preprocessingvia clustering
Anomaly detectionfrom time-series data
Model describing temporal, and physical relations
Surveillance data
TempAD
Anomalous
Normal
Precursor detection model
Precursor detectionfrom time-series data
Reactive TempAD
• R. Deshmukh and I. Hwang, “Anomaly Detection Using Temporal Logic Based Learning for Terminal Airspace Operations”, In AIAA Scitech 2019 Forum (p. 0682). (Finalist for ‘Intelligent Systems’ Best Student Paper Competition)
• R. Deshmukh and I. Hwang, “Incremental Learning Based Unsupervised Anomaly Detection Algorithm for Terminal Airspace Operations ”, AIAA Journal of Aerospace Information Systems, 2018 (revised version submitted; under review)
• R. Deshmukh, D. Sun and I. Hwang, “Anomaly Detection Using Surveillance Data and Airport Conditions in the Terminal Airspace”, in preparation
-
Machine Learning• Machine learning approaches
Supervised Unsupervised
SVM, ANN, decision tree, Naïve-Bayes, random forest, max. entropy classifier
k-means, DBSCAN, OCSVM, hierarchical clustering, autoencoders
6
• Anomaly detection: unsupervised learning
• Precursor detection: supervised learning
-
Objectives
Proposed anomaly detection algorithm: Temporal logic learning based AnomalyDetection (TempAD)
– Signal temporal logic (STL) is learned from data via learning algorithm to segregatedata as normal and anomalous
– TempAD model is generated such that they bound the states of all normal aircraft• Thus, any violation of any of these bounds makes flight anomalous
Proposed precursor detection algorithm: Reactive Temporal logic learning basedAnomaly Detection (reactive TempAD)
– Generate temporal logic models using supervised learning which can classify systemproperties before the anomaly occurred
– Results from reactive TempAD algorithm should deliver a high correlation betweenthe occurrence of precursor and anomaly – minimum missed detection and falsealarms
– Demonstrate application of the developed algorithm using real air traffic surveillancedata
-
Signal Temporal Logic
• Proposed algorithms infer a signal temporal logic (STL) in a human-readable format
• STL model can express system properties such as bounds (calledpredicates) on time and physical parameters
• For example,
𝜑𝜑𝜃𝜃 = 𝐹𝐹 0,320 𝑥𝑥𝑠𝑠 > 21.73 ∧ 𝐺𝐺 308,313 𝑦𝑦𝑠𝑠 < 34.51
can be interpreted as “for a system behaving normally, its 𝑥𝑥-coordinate should beeventually greater than 21.73 between 0 and 320 time steps and its 𝑦𝑦-coordinateshould be always less than 34.51 between 308 and 313 time steps”
Predicate
Time stepsGlobalFinal
Z. Kong, A. Jones, A., A.M. Ayala, E.A. Gol, and C. Belta, “Temporal logic inference for classification and prediction from data,” Proceedings of the 17th international conference on Hybrid systems: computation and control, pp. 273-282, April 2014.
-
Background: Anomaly Detection
• Input: altitude data during final 20 nautical miles before touchdown, for arrivals to a runway at LGA during a specific day
• Output:
Vertical anomaly detection model
𝐺𝐺 7.5,0 −284𝑥𝑥 + 𝑦𝑦 − 500 > 0 ∧ 𝐺𝐺 20,7.5 −164𝑥𝑥 + 𝑦𝑦 − 1880 > 0∧ 𝐺𝐺 8,0 −420𝑥𝑥 + 𝑦𝑦 − 500 < 0 ∧ 𝐺𝐺 20,8 −306.7𝑥𝑥 + 𝑦𝑦 − 1066.7 < 0
where 𝑥𝑥: distance to touchdown [nm]; 𝑦𝑦: altitude [ft]
• Natural language interpretation:Any approaching aircraft should have a glideslope angle between 1.5o and 2.8o from 20nm to 7.5 nm away from touchdown pointand then between 2.6o and 3.9o in the last 7.5nm of final approach.
-
Input: Surveillance Data• Both ASDE-X data and TAIS data are surveillance data
– Recorded variables: time, flight ID, longitude, latitude, altitude, and groundspeed
• Differences between ASDE-X data and TAIS data
ASDE-X Airport Surface Detection Equipment - Model XTAIS Terminal Automation Information Service
ASDE-X data TAIS data
Period 19 days(Apr 6 – 24, 2016)3 months
(Sep – Nov, 2016)
Detection range ~20 nm from airport ~141 nm from airport
Sampling rate 1 sec 5 sec
No. of arrivalflights
LGA 9,748 36,243
JFK 11,909 46,852
EWR 10,841 45,471
-
0 5 10 15 20 25 30 35 40-100
-50
0
50
Algorithm Development for Precursor Detection
TempAD
Signal 𝑠𝑠𝑖𝑖 (𝑎𝑎 𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓)
𝑇𝑇𝑇𝑇 − �̃�𝑓0
𝒔𝒔𝒊𝒊,𝒄𝒄 𝒔𝒔𝒊𝒊,𝒆𝒆
Effect model violationCause model violation
• Model describing normal behaviors:EFFECT formula (𝜑𝜑𝑒𝑒)
• Labels (normal 𝒑𝒑𝒊𝒊 = 𝟏𝟏/abnormal 𝒑𝒑𝒊𝒊 = 𝟎𝟎)
Unlabeled time-series data𝑠𝑠𝑖𝑖,𝑒𝑒
Anomaly Detection
Reactive TempAD
• Model separating normal/abnormal behaviors: CAUSE formula (𝜑𝜑𝑐𝑐)
Labeled time-series data𝑠𝑠𝑖𝑖,𝑐𝑐 , 𝑝𝑝𝑖𝑖
Precursor Detection
Unsupervised Learning
Supervised Learning
“If and only if 𝒔𝒔𝒊𝒊,𝒄𝒄violates 𝝋𝝋𝒄𝒄 in [𝟎𝟎,𝑻𝑻 −
Assumption: normal or abnormal behaviors occur in the last �̃�𝑓 seconds of the observed signals, i.e., the effect formula 𝝋𝝋𝒆𝒆 can adequately classify 𝒔𝒔𝒊𝒊 based on the signal 𝒔𝒔𝒊𝒊,𝒆𝒆over time interval 𝑻𝑻 − �𝒕𝒕,𝑻𝑻
-
Reactive TempAD for Precursor Detection
• Reactive TempAD: Given a set of labeled signals 𝒔𝒔𝒊𝒊,𝒄𝒄,𝒑𝒑𝒊𝒊 𝒊𝒊=𝟏𝟏𝑵𝑵
generated by
TempAD over [0,𝑇𝑇 − �̃�𝑓), find the cause formula 𝝋𝝋𝒄𝒄 that distinguishes between classes of signals 𝑠𝑠𝑖𝑖,𝑐𝑐 over time interval [0,𝑇𝑇 − �̃�𝑓)
• Supervised learning block which is responsible for separating precursor to normal and anomalous trajectories can comprise of any classification algorithm, such as Support Vector Machine (SVM), Neural Network (NN), random forests, naïve Bayes classifier, etc.
• It is possible that cause model and effect model may be required to be implemented using different features
– Essential for air traffic applications, as cause in one dimension can manifest as an effect in another dimension
– e.g.: go-around/missed approach anomaly detected using TempAD of an effect model 𝜑𝜑𝑒𝑒,𝑎𝑎𝑎𝑎𝑎𝑎 (altitude), but its precursor can be 𝜑𝜑𝑐𝑐,𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆 (energy dimension) or a mixture of several dimensions
-
Reactive TempAD: Support Vector Machine
• SVMs maximize the margin around the separating hyperplane
• The decision function is fully specified by a subset of training samples, the support vectors
• SVM classifier hyperplane is determined using constrained optimization
Support vectors
MaximizesmarginNarrower
margin
-
Reactive TempAD: Artificial Neural Network• ANN approach is loosely patterned on a neuron in
the human brain, which fires when it encounters sufficient stimuli
• A node combines input from data with a set of coefficients (weights), that either amplify or dampen that input, thereby assigning significance to inputs regarding the task the algorithm is trying to learn
– Basic function of neuron (node) is to sum inputs, and produce output given sum is greater than a threshold (activation)
Input WeightsOutput
• Each layer’s output is simultaneously the subsequent layer’s input, starting from an initial input layer receiving the data
• Pairing adjustable weights with input features is how we assign significance to those features with regard to how the neural network classifies and clusters input
-
Input Features (Dimensions)• Basic features used for anomaly detection are in horizontal (longitude,
latitude), vertical (altitude), and speed (groundspeed) dimensions
• Derived features: Energy features which are computed from basicfeatures
– Energy management is important for aircraft operation in terminalairspace Better metric for detection of energy excess or energy deficitanomalies Homogenous across a fleet of varying aircraft types
Energy Features FormulaSpecific Total Energy (STE) 𝑓 + 𝑣𝑣2/2𝑓𝑓Specific Kinetic Energy (SKE) 𝑣𝑣2/2𝑓𝑓Specific Potential Energy (SPE) 𝑓Specific Total Energy Rate (STER) �̇� + �̇�𝑣𝑣𝑣/𝑓𝑓Specific Kinetic Energy Rate (SKER) �̇�𝑣𝑣𝑣/𝑓𝑓Specific Potential Energy Rate (SPER) �̇�
-
Go-around Anomalies• Go-around anomaly: Pilot aborts landing during short final, climbs using
missed approach procedures, circles around and then attempts landing again
Horizontal Vertical
Speed STE
SPER
Go-around at LGAResults in violation of TempAD
models for all dimensions
Go-around at JFK-74.35 -74.3 -74.25 -74.2 -74.15 -74.1 -74.05 -74 -73.95 -73.9
40.65
40.7
40.75
40.8
40.85
40.9
40.95
Go-around at EWR
EWR
LGA
JFK
Go-around anomaly is common across all three airports
Airport No. of arrivals (Sept.-Nov.)No. of go-arounds
LGA 36,243 201
JFK 46,852 348
EWR 45,471 364
-
Precursor Detection for Go-around Anomalies• We use support vector machine (SVM) algorithm to classify anomaly
precursors and normal flights
• To determine feature which can deliver best go-around precursor detectionperformance, we search across several dimensions and their combinations andchoose the feature that grants the highest F1 classification score:
Precision = ratio of true anomaly detections over total precursor detections
Recall = ratio of anomalies detected by precursors over total true anomalies
F1 score = 2 × 𝑆𝑆𝑃𝑃𝑒𝑒𝑐𝑐𝑖𝑖𝑠𝑠𝑖𝑖𝑃𝑃𝑃𝑃×𝑆𝑆𝑒𝑒𝑐𝑐𝑎𝑎𝑎𝑎𝑎𝑎𝑆𝑆𝑃𝑃𝑒𝑒𝑐𝑐𝑖𝑖𝑠𝑠𝑖𝑖𝑃𝑃𝑃𝑃+𝑆𝑆𝑒𝑒𝑐𝑐𝑎𝑎𝑎𝑎𝑎𝑎
• From all tested features, a derived feature (𝑓𝑓) which was a combination ofaircraft energies and energy rate resulted in the highest F1 score:
𝒇𝒇 =𝑺𝑺𝑺𝑺𝑺𝑺𝑺𝑺𝑺𝑺𝑺𝑺𝑺𝑺 × 𝑺𝑺𝑺𝑺𝑺𝑺 =
�̇�𝒉𝒉𝒉 ×
𝒗𝒗𝟐𝟐
𝒈𝒈
-
Precursor Detection for Go-around Anomalies• Results of precursor detection using feature 𝑓𝑓:
• Thus, if arrival flights violate cause (or precursor) model (𝜑𝜑𝑐𝑐), it is guaranteedwithin a margin of error that effect (or anomaly) model (𝜑𝜑𝑒𝑒) will be violated
Anomaly DetectionPrecursor Detection
𝜑𝜑𝑐𝑐 = 𝐺𝐺[0,55)𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆
𝑆𝑆𝑆𝑆𝑆𝑆 < 37 𝜑𝜑𝑒𝑒 = 𝐺𝐺[55,60) 1650 × 𝑓𝑓 + 75 × h < 123750
-
Precursor Detection for Go-around Anomalies• Look-ahead time: Time difference between detection of precursor and
occurrence of anomaly– Desirable that this time is as long as possible, to allow more time for ATCs and
pilots to make decision and to accommodate pilot input lag
• Preliminary precursor detection with SVM (reactive TempAD-SVM) andfeature 𝑓𝑓 resulted in average look-ahead time of 7 seconds, which is aborderline feasible reaction time for pilot and ATC to react to theanomaly
• To improve precursor detection performance, we need to allowlearning technique to automatically search the best combination offeatures
– We enhance reactive TempAD to use artificial neural network (ANN), whichexplores and weighs combinations of diverse features to give best test results
– Furthermore, we enhance the algorithm by making feature space richer byintroducing a new feature – distance to preceding flight, which is the horizontaland vertical distance to the nearest preceding aircraft
Demonstration of preliminary application of go-around precursor detection
T. Blajev, and W. Curtis, “Go-Around Decision Making and Execution Project: Final Report to Flight Safety Foundation," Flight Safety Foundation, 2017.
-
Precursor Detection for Go-around AnomaliesReactive TempAD-SVM
• Average look-ahead time: 7s• F1 score: 79.55%• Confusion matrix:
Reactive TempAD-ANN• Average look-ahead time: 11s (+57%)• F1 score: 87.23%• Confusion matrix:
PredictedTrue
Anomalousflights
Normalflights Total
Anomalousflights
True positive35
False negative13 48
Normalflights
False positive5
True negative9,581 9,596
Total 40 9,594 9,634
PredictedTrue
Anomalousflights
Normalflights Total
Anomalousflights
True positive41
False negative7 48
Normalflights
False positive5
True negative9,581 9,596
Total 46 9,588 9,634
Number of false negatives (FNs) using the reformulated reactive TempAD-ANN has nearly halved. This is important, as minimizing instances of FNs (i.e., missed detections) is vital in safety-critical applications such as air traffic management (ATM)
-
Precursor Detection for S-turn Anomalies• S-turn anomaly: Pilot undertakes an extension of flight path, and a visible path
stretch manifests in horizontal dimension
• Major precursors:– Separation from preceding aircraft– Excess approach speed– Runway occupancy
Horizontal view of S-turn anomaly in arrivals to RWY4 at LGA
Horizontal view of S-turn anomaly in arrivals to RWY22 at LGA
-
• We generate features and use them as training input for ANN classification. The features correspond to:
• Aircraft separation: vertical and horizontal distance to preceding arriving and departing flight
• Unstable approach: Flight speed and altitude
• Analysis showed that S-turn anomaly was often preceded by sequence of precursors before anomaly occurred
• Look-ahead time can be further increased by analyzing this sequence of precursors• Objective Determine most distant precursor from anomaly (earliest occurring
precursor)
• Average look-ahead time: 8.7 seconds• F1 score: 83.01%
Precursor Detection for S-turn Anomalies
PredictedTrue
Anomalousflights
Normalflights Total
Anomalousflights
True positive66
False negative18 84
NormalFlights
False positive9
True negative3,257 3,266
Total 75 3,275 3,350
-
56789101112
Distance to touchdown (nm)
1500
2000
2500
3000
3500
4000
4500
5000
Altit
ude
(ft)
• New average look-ahead time: 12.3 seconds
• Objective Determine most distant precursor from anomaly
Aircraft approaches closely to preceding aircraft
Aircraft is below normal glideslope
Aircraft undertakes path stretch maneuver
Vertical precursor precedes S-turn
Loss of separation forces trailing flight to undertake
vertical maneuver
Algorithm determines loss of separation as true
precursor
Distribution of look-ahead times for arrivals to LGA over 11 days
Precursor Detection for S-turn Anomalies
-
Conclusions• Developed a temporal logic based precursor detection algorithm to
detect precursor to anomalies in unlabeled aviation surveillancedatasets
• Demonstrated the algorithm using two air traffic surveillance datasets ASDE-X and TAIS
• Effectively identified precursors to anomalous flights (go-around and S-turn anomalies) detected by TempAD
• Apply precursor detection algorithm to predict a wider variety of anomalies• Anomalies in vertical dimension (glide slope intercept mismatch), underspeed,
overspeed, energy deficit or excess anomalies
• Acquire a richer and larger dataset to improve precursor detectionperformance
Future WorkLimitations• Precursor detection would benefit from a richer input dataset – adding voice data,
pilot intent data as input could grant a better insight into precursor detection• Precursor detection models are not specific to runway/airports. Using a larger
input dataset could allow us to find precursor to, say, go-around anomalies forarrivals to RWY22 at LGA, making precursor detection very accurate
-
Acknowledgement
• This research is funded by the National Aeronautics and Space Administration(NASA) through the NASA Big Data Analytics project (NNH15ZEA001N).
• The authors are grateful to Dr. Andrew Churchill at Mosaic ATM, Megan Hawleyat Honeywell, and Dr. Nikunj C. Oza at NASA Ames for their valuable commentsand continued support.
Data-Driven Precursor Detection Algorithm for Terminal Airspace OperationsPresentation OutlineData-driven Learning for ATMLevel-set DefinitionsProblem SetupMachine LearningObjectivesSignal Temporal LogicBackground: Anomaly DetectionInput: Surveillance DataAlgorithm Development for Precursor DetectionReactive TempAD for Precursor DetectionReactive TempAD: Support Vector MachineReactive TempAD: Artificial Neural NetworkInput Features (Dimensions)Go-around AnomaliesPrecursor Detection for Go-around AnomaliesPrecursor Detection for Go-around AnomaliesSlide Number 19Slide Number 20Slide Number 21Slide Number 22Slide Number 23ConclusionsSlide Number 25