1 st Annual Workshop NAS-Wide Simulation in Support of NextGen, 12/10/08 Probabilistic NAS Platform...
-
Upload
sophie-davidson -
Category
Documents
-
view
218 -
download
0
Transcript of 1 st Annual Workshop NAS-Wide Simulation in Support of NextGen, 12/10/08 Probabilistic NAS Platform...
1st A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
Probabilistic NAS Platform
George Hunter, Fred WielandBen Boisvert, Krishnakumar Ramamoorthy
Sensis Corporation
December 10, 2008
2
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
Outline
• What is PNP? • Team and development history• Example uses of the model • Software processes and testing• Validation
3
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
Outline
• What is PNP? • Team and development history• Example uses of the model • Software processes and testing• Validation
4
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
What is PNP?
• An fast-time and flexible NAS-wide simulation tool– Real-time or fast-time modes
• Half-hour runtime on a laptop, to simulate a day in the NAS– Physics-based: trajectories computed through integrating aerodynamic energy
balance equations by varying the time-step size– System uncertainties (weather, security, operations …)– Plug-and-play architecture
• Dynamic clients (TFM, DAC, AOC, …)– An ATC community resource– Formal software development processes in place– Adaptable to current system or NextGen future concepts
• Uses– Environment in which to design, build and test decision support tools
• TFM, DAC, AOC, …• Fast-time, real-time, shadow-mode
– Potential NAS tool• Service provider, operator, collaborative uses
– Benefits assessment tool• Fast-time tool to evaluate improved
infrastructure, technology, procedures …• Evaluates historic and future traffic
scenarios in weather
5
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
PNP Architecture
Probabilistic NAS Platform
(PNP)Weather DataWeather Data
ReportsReports
MATLAB®
ScriptingInterface
MATLAB®
ScriptingInterface
NASDatabase
NA
S S
imu
latio
n
Performance DataPerformance Data
Flight DataFlight Data
Graphical User InterfacePlan View Display
Graphical User InterfacePlan View Display
A fast-time physics-based (trajectory-based) NAS-wide
modeling tool
6
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
PNP Architecture
Probabilistic NAS Platform
(PNP)Weather DataWeather Data
ReportsReports
SimObjects
MATLAB®
ScriptingInterface
MATLAB®
ScriptingInterface
NASDatabase
MATLAB® ClientMATLAB® Client
External Client(Any Language)External Client(Any Language)
ClientAs Middleware
ClientAs Middleware
Java ClientJava Client
Decis
ion
makin
gN
AS
Sim
ula
tion
Performance DataPerformance Data
Flight DataFlight Data
Graphical User InterfacePlan View Display
Graphical User InterfacePlan View Display
Prob-TFMProb-TFM
A fast-time physics-based (trajectory-based) NAS-wide
modeling tool
7
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
PNP Client Development
• TFM client development– ProbTFM (Sensis internal development)
• TFM client integration– C2 (algorithms from and used with permission of Bob Hoffman,
Metron)– Constrained LP (algorithms from and used with permission of
NASA, Joey Rios) in progress
• DAC client integration– MxDAC (algorithms from and used with permission of Min Xue,
NASA/UARC)
• AOC client development– Gaming behaviors (collaboration with GMU/Lance Sherry) in
progress
8
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
Capabilities Summary• Real-time• Fast-time• Airport weather impact models• Airspace weather impact models• Weather-integrated decision making• Probabilistic modeling / decision making• Traffic flow management• Dynamic airspace configuration• Surface traffic modeling• Terminal area modeling• Super density operations• Fuel burn modeling• Emissions modeling• Trajectory-based operations• Separation assurance• Plug-n-play• Fast run-time
Existing Can Support
√√√√√√√√√√√√√√√√
√√
√√
√√√√
√√√√
√√√√√√
9
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
Outline
• What is PNP? • Team and development history• Example uses of the model • Software processes and testing• Validation
A fast-time physics-based (trajectory-based) NAS-wide
modeling tool
10
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
Team and Development History
People
Env’nment
Data
Funct’ality
Projects
Users
Matlab Java/real-time
WSI collaboration for real-time weather feed
Dynamicclients
NAS-wide,probabilistic
Wx modelingand routing
Clientarchitecture
Internal NWA GMU
KrishnakumarRamamoorthy
BenBoisvert
DiegoEscala
TaeLee
MichelleLu
HuinaGao
GeorgeHunter
2004 2005 2006 2007 20082003
JPDO FAA NASPACNASA NRAs
Web 2.0
Project SystemSystem lead SystemSoftware lead Software Software
11
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
Outline
• What is PNP? • Team and development history• Example uses of the model • Software processes and testing• Validation
A fast-time physics-based (trajectory-based) NAS-wide
modeling tool
12
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
Example Project Uses
• JPDO Modeling and Analysis– NextGen performance evaluation with weather
• FAA NASPAC Weather Modeling– Convection impact modeling for NASPAC
• NASA Gaming NRA– Evaluation of NextGen gaming with AOC clients
• NASA MetaSimulation NRA– Investigation of TFM + DAC interactions
• NASA SLDAST RFA– Evaluation of NextGen TFM concepts and models
• NASA Market-Based TFM NRA– Evaluation of NextGen market-based TFM concepts
13
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
NextGen Sensitivity StudiesNextGen Performance Sensitivity Analysis
Benefit of Improved Wx Forecasts
Benefit of Using Clear Weather Forecasts
Persistence forecast11/16/06
Case 2: No distinction between clear and heavy weather forecast accuracy
Case 1: Take advantage of improved forecast accuracy in clear weather
NAS Performance Sensitivity
Kris Ramamoorthy, George Hunter, "Evaluation of National Airspace System Performance Improvement With Four Dimensional Trajectories," AIAA Digital Avionics Systems Conference (DASC), Dallas, TX, October, 2007
George Hunter, Fred Wieland " Sensitivity of the National Airspace System Performance to Weather Forecast Accuracy," Integrated Communications, Navigation and Surveillance Conference (ICNS), Herndon, VA, May, 2008
14
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
Market-Based TFM Studies
UAL233 Delay Cost
NAS Access Valuation Models
SCC
Delay
George Hunter, et. al., "Toward an Economic Model to Incentivize Voluntary Optimization of NAS Traffic Flow," AIAA ATIO Conference, Anchorage, AK, September, 2008.
15
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
Dynamic Airspace Configuration
Nov 12, 2006, LAT=2, #Gen=40
ZFW FAA sectors
George Hunter, "Preliminary Assessment of Interactions Between Traffic Flow Management and Dynamic Airspace Configuration Capabilities," AIAA Digital Avionics Systems Conference (DASC), St. Paul, MN, October, 2008.
16
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
AOC Dispatch Use Case
Reroute with low probability of delay
17
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
Outline
• What is PNP? • Team and development history• Example uses of the model • Software processes and testing• Validation
A fast-time physics-based (trajectory-based) NAS-wide
modeling tool
18
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
Unit and System Testing
Regression Testing
Trunk Configuration Management
Branch Configuration Management
Project Monitoring & Control
Development Tracking
Quantitative Project Management
Processes and Testing Cycle
19
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
Project Monitoring and Control
• JIRA is used to track issues– Project Manager and Lead Software Engineer assign task priorities, due dates,
and personnel.
• Weekly telecoms keep distributed team apprised of PNP and communications open
• Project Manager maintains a master schedule in MS-Project
20
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
Development Tracking
• Software engineers use JIRA to track and status development efforts.
21
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
Branch Configuration Management
• Software Engineers are responsible for creating branches from the trunk to develop fixes/enhancements.
• The Configuration Management of the software is accomplished with Subversion– Subversion is an open source version control system
(http://subversion.tigris.org/)
22
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
Unit and System Testing
• Software Engineers are responsible for creating unit tests to verify the correctness of their code. The JIRA issue number is to be used throughout the code and unit tests for tracking purposes.
• Software Engineers are responsible for running their own system/function tests to verify their software.
• Once testing is validated, code is merged back on to the trunk.
23
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
Trunk Configuration Management
• Once all validated JIRA issues are merged unto the trunk, regression testing is performed.
24
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
Regression Testing
• Regression testing• Aggregate results
– Total delay
– Total congestion
– Traffic volume
– #TFM initiatives
– Runtime
• Different scenarios– Truncated demand set
– Full demand set
– Weather
• Automated– Weekly or as required
• Archived• Graphical quick-look
25
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
Quantitative Project Management
• Regression testing validation is performed and the release letter is updated.
• Release is tagged in Subversion.• JIRA issues are closed.• Documentation is updated to reflect changes in
software.
26
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
Outline
• What is PNP? • Team and development history• Example uses of the model • Software processes and testing• Validation
A fast-time physics-based (trajectory-based) NAS-wide
modeling tool
27
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
System-Level Engineering Validation
• ASPM / ETMS verification tests– Compare ASPM/ETMS data with simulation data
• Calibrate concept to match aggregate field observations
– Models• Trajectory data• Airport capacities (VMC / IMC)• Sector capacities in weather
– Aggregate performance• Mean flight delay• Sector and airport overloadings
– Detailed performance• Flight delay by airport and time of day• Overloading and delay patterns (Spatial and temporal)
Delays by airport and time of day Sector and airport loading by time of day Spatial loading patterns
– Light and heavy weather days
√
28
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
System-Level Software Verification
• Cross check sums– Flights = Operations at all airports
– Flight time = Minutes from sector loads
– Sector load by sector = Sector load by time
– Airport ops = Flights using the airport in demand set
– Delays by flight = Delays by time; and reroutes
• Weather data checks– Compare PNP/Metar airport capacity with ASPM AAR/ADR
– Compare PNP/Metar airport capacity with ASPM IFR periods
– Ensure En route convection versus time of day is smooth
– Ensure WxMAP ≤ MAP for all sector time bins
• Graphical– Ensure reroutes overlaid on weather make sense
• TFM Performance– Number of delays per flight, min and max flight delay
– Maximum airport and sector overloading (ensure are reasonable)
29
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
System-Level Engineering Validation
• ASPM / ETMS verification tests– Compare ASPM/ETMS data with simulation data
• Calibrate concept to match aggregate field observations
– Models• Trajectory data• Airport capacities (VMC / IMC)• Sector capacities in weather
– Aggregate performance• Mean flight delay• Sector and airport overloadings
– Detailed performance• Flight delay by airport and time of day• Overloading and delay patterns (Spatial and temporal)
Delays by airport and time of day Sector and airport loading by time of day Spatial loading patterns
– Light and heavy weather days
√
30
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
Trajectory Model Validation
• Compared to ETMS flight data (May 2008)
N: 316Mean: 0.321 minStd dev: 11.95 min
Mean: 0.80 minStd dev: 6.51 minR2: 0.012
Detrended for Range
George Hunter, Ben Boisvert, Kris Ramamoorthy, "Advanced Traffic Flow Management Experiments for National Airspace Performance Improvement," 2007 Winter Simulation Conference, Washington, DC, December, 2007
31
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
ProbTFM Performance
• ASPM / ETMS verification tests– Compare ASPM/ETMS data with simulation data
• Calibrate concept to match aggregate field observations
– Models• Trajectory data• Airport capacities (VMC / IMC), actual and forecasted• Sector capacities in weather, actual and forecasted
– Aggregate performance• Mean flight delay• Sector and airport loadings
– Detailed performance• Flight delay by airport and time of day• Overloading and delay patterns (Spatial and temporal)
Delays by airport and time of day Sector and airport loading by time of day Spatial loading patterns
– Light and heavy weather days
√
32
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
Compare With Field Observations
0
500
1000
1500
2000
2500
3000
0 5 10 15 20 25
Average Delay (minutes per aircraft)
Sec
tor
Co
ng
esti
on
January 7, 2007(Similar resultswith other days)
LAT = 0
LAT = 60 minutes
LAT = 30 mins
(14.5,1657)
• Compare to ETMS/ASPM– Forecast accuracies, Decision making horizon, Delay distribution
33
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
Verification of Results
• ASPM / ETMS verification tests– Compare ASPM/ETMS data with simulation data
• Calibrate concept to match aggregate field observations
– Models• Trajectory data• Airport capacities (VMC / IMC), actual and forecasted• Sector capacities in weather, actual and forecasted
– Aggregate performance• Mean flight delay• Sector and airport loadings
– Detailed performance• Flight delay by airport and time of day• Overloading and delay patterns (Spatial and temporal)
Delays by airport and time of day Sector and airport loading by time of day Spatial loading patterns
– Light and heavy weather days
√
34
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
System Loading Patterns
ProbTFM predicted, 14:45 GMTETMS Actual, 14:45 GMT
ETMS
ProbTFM
ETMSUnderloading Overloading
ProbTFM loading
35
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
Verification of Results
• ASPM / ETMS verification tests– Compare ASPM/ETMS data with simulation data
• Calibrate concept to match aggregate field observations
– Models• Trajectory data• Airport capacities (VMC / IMC), actual and forecasted• Sector capacities in weather, actual and forecasted
– Aggregate performance• Mean flight delay• Sector and airport loadings
– Detailed performance• Flight delay by airport and time of day• Overloading and delay patterns (Spatial and temporal)
Delays by airport and time of day Sector and airport loading by time of day Spatial loading patterns
– Light and heavy weather days, control days√
36
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
Conclusion
• The development of PNP has benefited from lessons learned over past two decades in NAS system wide modeling– Plug and play simulation architecture– Supports both analytical and HITL studies– Adaptable to simulate current system as well as NextGen future
concepts– Fast-time, physics-based– Formal software development processes in place– Probabilistic decision making and extensive weather modeling
explicitly incorporated in tool
37
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
Publications1. George Hunter, "Preliminary Assessment of Interactions Between Traffic Flow Management and Dynamic Airspace
Configuration Capabilities," AIAA Digital Avionics Systems Conference (DASC), St. Paul, MN, October, 2008.2. George Hunter, et. al., "Toward an Economic Model to Incentivize Voluntary Optimization of NAS Traffic Flow," AIAA ATIO
Conference, Anchorage, AK, September, 2008.3. George Hunter, Fred Wieland " Sensitivity of the National Airspace System Performance to Weather Forecast Accuracy,"
Integrated Communications, Navigation and Surveillance Conference (ICNS), Herndon, VA, May, 2008.4. George Hunter, Kris Ramamoorthy, "Integration of terminal area probabilistic meteorological forecasts in NAS-wide traffic
flow management decision making," 13th Conference on Aviation, Range and Aerospace Meteorology, New Orleans, LA, January, 2008.
5. Kris Ramamoorthy, George Hunter, "The Integration of Meteorological Data in Air Traffic Management: Requirements and Sensitivities," 46th AIAA Aerospace Sciences Meeting and Exhibit, Reno, NV, January, 2008.
6. George Hunter, Ben Boisvert, Kris Ramamoorthy, "Advanced Traffic Flow Management Experiments for National Airspace Performance Improvement," 2007 Winter Simulation Conference, Washington, DC, December, 2007.
7. Kris Ramamoorthy, George Hunter, "Evaluation of National Airspace System Performance Improvement With Four Dimensional Trajectories," AIAA Digital Avionics Systems Conference (DASC), Dallas, TX, October, 2007.
8. Kris Ramamoorthy, Ben Boisvert, George Hunter, "Sensitivity of Advanced Traffic Flow Management to Different Weather Scenarios," Integrated Communications, Navigation and Surveillance Conference (ICNS), Herndon, VA, May, 2007.
9. George Hunter, Ben Boisvert, Kris Ramamoorthy, "Use of automated aviation weather forecasts in future NAS," The 87th American Meteorological Society Annual Meeting, San Antonio, TX, January, 2007.
10. Kris Ramamoorthy, George Hunter, "Probabilistic Traffic Flow Management in the Presence of Inclement Weather and Other System Uncertainties," INFORMS Annual Meeting, Pittsburgh, PA, November, 2006.
11. Kris Ramamoorthy, Ben Boisvert, George Hunter, "A Real-Time Probabilistic TFM Evaluation Tool," AIAA Digital Avionics Systems Conference (DASC), Portland, OR, October, 2006.
12. George Hunter, Kris Ramamoorthy, Alexander Klein "Modeling and Performance of NAS in Inclement Weather," AIAA Aviation Technology, Integration and Operations (ATIO) Forum, Wichita, KS, September 2006.
13. Kris Ramamoorthy, George Hunter, "A Trajectory-Based Probabilistic TFM Evaluation Tool and Experiment," Integrated Communications, Navigation and Surveillance Conference (ICNS), Baltimore, MD, May, 2006.
14. Kris Ramamoorthy, George Hunter, "Avionics and National Airspace Architecture Strategies for Future Demand Scenarios in Inclement Weather," AIAA Digital Avionics Systems Conference (DASC), Crystal City, VA, October, 2005.
15. George Hunter, Kris Ramamoorthy, Joe Post, "Evaluation of the Future National Airspace System in Heavy Weather," AIAA Aviation Technology, Integration and Operations (ATIO) Forum, Arlington, VA, September 2005.
16. James D. Phillips, “An Accurate and Flexible Trajectory Analysis,” World Aviation Congress (SAE Paper 975599), Anaheim, CA, October 13-16, 1997.
38
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
Questions?
39
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
Backup
40
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
PNP Systems Requirements
• System requirements– PNP is a Java application– Hardware
• Memory: minimum 1GB, preferred 2GB
• CPU: Pentium (4) 3.2 GHz or better
• Video card: 128MB memory, preferred 256MB
– Software• Java JDK 6 http://java.sun.com/javase/downloads/index.jsp
• MySQL Server 5.0 http://dev.mysql.com
– Third party licenses• Eurocontrol BADA usage license
41
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
Weather Days
• Ten weather days, two control days
42
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
Weather Days
• Weather days– Spectrum of weather days
• Variation in weather type and intensity• Variation in season
– Support real-world comparison• Support same sector data• Variation in traffic demand volume and structure
Different days of week, holidays
• Control days
43
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
44
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
NextGen PerformanceSensitivity Analysis
45
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
En Route and Terminal Area Combined Sensitivities - 2025
46
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
47
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
Benefit of ImprovedConvection Forecasts
48
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
Investment Analysis
49
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
50
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
Benefit of Using Clear Weather Forecasts
51
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
Benefit Evaluation
Persistence forecast, 11/16/06
Case 2: No distinction between clear and heavy weather forecast accuracy
Case 1: Take advantage of improved forecast accuracy in clear weather
52
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
53
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
Market-Based TFM:Valuation of NAS Access
54
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
Congestion-Delay Relationship
• Unconstrained sector congestion cost (SCC) for zero lookahead time (blue) and PNP-ProbTFM simulated delay (black) time histories for all en route NAS sectors and flights, respectively.
SCC
Delay
55
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
Aggregate Delay Model
• Hypothesize a first-order lag transfer function
K
1
1
sSCC(s) Delay(s)
Simulated delay
Modeled delay
56
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
• Hypothesize a second-order transfer function
Simulated delay
Modeled delay
Aggregate Delay Model
57
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
Transfer Functions Summary
58
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
Explicit Cost Model
• Evaluate cost of NAS access by removing the flight• Remove one flight
– 11/16/06, UAL233, A320– Morning departure from Bradley International (KBDL) to Chicago
O’Hare airport (KORD)– Relatively high cost flight
• 90.02 SCC
59
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
Remove UAL233
• Delay reduction by time bin in simulation run– Delay reduction of 8141 minutes
60
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
61
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
NAS Performance Sensitivity Studies• Performance sensitivity to:
• Delay distribution policy (most important factor)
• TFM system agility
• System forecasts (least important factor)
Nov 12, 2006
ETMS/ASPM
Minimum Delay
Non AgileMinimum Delay
Delay Distribution
Non AgileDelay Distribution
62
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
63
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
Dynamic Airspace Configuration
64
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08 NAS Sectorization
• Nov 12, 2006
65
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08 MxDAC Afternoon Sectorization
• Nov 12, 2006, LAT=6, #Gen=20
66
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08 MxDAC Midday Sectorization
• Nov 12, 2006, LAT=2, #Gen=40
Coeff_peak_ac_var=0.0Coeff_avg_ac_var=0.0Coeff_crossings=0.0Coeff_transition_time=0.0Coeff_residual_capacity=1.0
67
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08 Delay-Congestion Performance
MxDAC off
MxDAC on, LAT = 4 hrs
MxDAC on, LAT = 2 hrs
68
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
69
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
Equity Analysis:Cost of Delay Distribution
70
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
Cost of Distributing Delay
• RMS delay can be reduced by spreading delay to more flights– But at the cost of increased total delay
Nov 12, 2006$65/minute
Increased delay distribution
71
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
72
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
AOC Dispatch Use Case
73
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
Dispatcher Successfully Finds a Reroute
Reroute with low probability of delay
74
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
75
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
Project Monitoring and Control
• JIRA is used to track issues– Project Manager and Lead Software Engineer assign task priorities, due dates,
and personnel.
• Weekly telecoms keep distributed team apprised of PNP and communications open
• Project Manager maintains a master schedule in MS-Project
76
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
Development Tracking
• Software engineers use JIRA to track and status development efforts.
77
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
Branch Configuration Management
• Software Engineers are responsible for creating branches from the trunk to develop fixes/enhancements.
• The Configuration Management of the software is accomplished with Subversion– Subversion is an open source version control system
(http://subversion.tigris.org/)
78
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
Unit and System Testing
• Software Engineers are responsible for creating unit tests to verify the correctness of their code. The JIRA issue number is to be used throughout the code and unit tests for tracking purposes.
• Software Engineers are responsible for running their own system/function tests to verify their software.
• Once testing is validated, code is merged back on to the trunk.
79
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
Trunk Configuration Management
• Once all validated JIRA issues are merged unto the trunk, regression testing is performed.
80
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
Regression Testing
• Regression testing• Aggregate results
– Total delay
– Total congestion
– Traffic volume
– #TFM initiatives
– Runtime
• Different scenarios– Truncated demand set
– Full demand set
– Weather
• Automated– Weekly or as required
• Archived• Graphical quick-look
81
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
Quantitative Project Management
• Regression testing validation is performed and the release letter is updated.
• Release is tagged in Subversion.• JIRA issues are closed.• Documentation is updated to reflect changes in
software.
82
1s
t A
nn
ua
l W
ork
sh
op
NA
S-W
ide
Sim
ula
tio
n i
n S
up
po
rt o
f N
ex
tGe
n,
12
/10
/08
Risk Management
• Lessons learned analysis– A wrap up meeting is held to discuss all issues on a project in
which proactive steps can be documented to avoid the same mistakes