Converging Measures: Federal Aviation Administration ...

Federal AviationAdministration

Converging Measures:

Integrating Diverse Data

Sources in ATM Human-

in-the-Loop Simulations,

FAA

Ben Willems, FAA Kenneth Allendoerfer, Ph.D.

Federal Aviation Administration

Human Factors Branch, ANG-E25

Ben Willems


ATO Program Management Office, AJM-1310

FAA-EUROCONTROL

Technical Interchange Meeting –

Modeling & Simulation

October 16-17, 2019

2

William J. Hughes Technical Center


October 2019


•Why

Why conduct human-in-the

loop simulations at all?

3



October 2019


What do we want to learn from ATM HITL

simulations?

• Effects of proposed…

– Procedure or operational concept

– Airspace or airport design

– Technology or equipment

– Training program

• On controllers’…

– Physical and physiological states

– Cognition

– Decisions

– Actions

• And on…

– overall human-system performance.

4



October 2019


Why conduct HITL simulations at all?

• Human behavior is hard to predict especially in complex and novel situations

– Fast-time models typically do not capture this unpredictability very well.

• Human actions are driven by physiological and cognitive processes and states

– What’s going on is a different question than what does the human know about what’s going on.

• Project may have explicit human-performance goals

– “… improve pilot awareness of hazardous weather that may affect the flight…”

• Humans are going to be asked to work in this new environment and to live with the consequences

– Rhetorical impact & credibility

5



October 2019


SAFETYEFFICIENCY

& CAPACITY

WORKLOAD

& EFFORTACCEPTANCE

& USAGE

High-Level Constructs

6



October 2019


Analytic Weight

• Analysis-HEAVY

– Incident reviews, replays, watch and code videos

– Qualitative and quantitative data

– Requires human expertise to design and conduct the analysis

• Analysis-LITE

– Data collected automatically by system and processed by algorithms

– Quantitative data

– Requires human expertise to design the analysis, but the analysis itself can be conducted automatically

– Lend themselves to incorporation into fast-time models

• HITLs lend themselves to Analysis-HEAVY metrics

– Effects on humans (especially cognitive) are hard to measure automatically

7



October 2019


Efficiency & Capacity

• Fast-time modelers will be familiar with these

• Typically calculated from track data or output

of ATM automation systems

– Number of aircraft handled per hour

– Number of arrivals

– Average spacing over the arrival fix

– Median time between departures

– Total Distance flown

– Trajectory conformance

– Etc.

8



October 2019


Safety

• Operational errors

• Runway incursions

• Near misses

– How to define?

– Quantification of risk

• Unlikely to occur organically during a HITL sim

– Controllers see problems ahead of time and fix them

– Script 2x the conflicts that you want

9



October 2019


Safety

• Precursors – Increased Risk

– Workload (very high / very low)

– Fatigue

– Small errors

– Lack of experience / recent training

– “Sharpness”

• Awareness – Attention

– Mental model of situation vs. situation itself

– Situation Awareness (SA)

– Distraction / inattention

– “Tricky” situations – things look normal but aren’t –inattentional blindness, confirmation bias

10



October 2019


Eye Tracking to Measure

Awareness / Attention

• Point of gaze

• Fixations, saccades

– Time since last fixation

– Sequences and patterns of fixations

– Visual scan pattern

• Reaction time to alerts, surprises

• Pupil diameter (cognitive activity)

11



October 2019


Karsten, G., Goldberg, B., Rood, R., & Sulzer, R. (1975). Oculometer measurement of air traffic

controller visual attention (FAA-NA-74-61). Atlantic City, NJ: Federal Aviation Administration

National Aviation Facilities Experimental Center.

12



October 2019


Eye Tracking (Old School)

13



October 2019


Eye Tracking (Today)

14



October 2019


Eye Tracking

15



October 2019


Eye Tracking – Point of Gaze

16



October 2019


Analysis-HEAVY

• Because safety events are rare in sims, we tend to go “deep”

• Replays of incidents with SMEs

– Communications, screen-capture videos

– Over-the-shoulder videos

– Identify where controller’s mental model disconnected from the true state of the world

• Review of point-of-gaze data for a chosen aircraft pair

• We don’t have strong algorithms to identify a good scan pattern or a risky scan pattern

17



October 2019


FALCON

18



October 2019


Workload & Effort

• Subjective Ratings

– Air Traffic Workload

Input Technique

(ATWIT)

– NASA-TLX

• Ratings by SME

observers

• Post-Scenario

Ratings / Interviews

Workload Assessment Keypad (WAK)

19



October 2019


Task Metrics

• Control actions

– Altitude, speed, heading, route changes

• Communications

– Number, duration, errors

• UI interactions

– Number, duration, errors

20



October 2019


Physiological Metrics

• Neuroergonomics

– Electroencephalography (EEG)

– Functional Near Infrared Spectroscopy (fNIRS)

• Pupil diameter

• Heart rate variability

• Stress

– Galvanic skin response

– Hormone levels

– Heart rate

– Speech characteristics

21



October 2019


EEG

22



October 2019


fNIRS

23



October 2019


Sample of fNIRS Data

Ayaz, H., Shewokis, P. A., Bunce, S., Izzetoglu, K., Willems, B., Onaral, B. (2012). Optical brain monitoring for operator training and mental

workload assessment, NeuroImage, 59, 36–47.

24



October 2019


Acceptance & Usage

• Do the humans choose to use the new thing?

– Did controllers take the advice given by the decision-

support tool? Under what circumstances?

• Trust / Distrust

• Reliance / Over-reliance

• Does it feel intuitive? Do they enjoy using it?

• We are a long way from having models that can

predict these effects (but they are important).

25



October 2019


Complex Event Processing

• Must be able to analyze a stream of data

across different sources

– Identify patterns within a source but also across

sources

– Abstract the patterns into complex events

– Count or measure complex events

• How long did it take?

• How many steps did it take?

• What was the effect of the new thing on the complex event?

– Example: Controller issues a reroute.

26



October 2019


Complex Event Processing Example

TIME

Cognitive Activity (EEG)

↓ Cognitive activity starts to climb ↓ Cognitive activity reaches maximum

Attention (eye tracker)

↓ Controller visually attends to AAL1234 datablock ↓ Controller visually attends to route menu

Situation Awareness (SAGAT)

↓ Probe reveals high awareness

of AAL1234 position and route

System Events (UI states)

↓ TFM notifies that reroute is needed ↓ New route appears in route field

Taskload (UI interactions)

↓ C activates route menu ↓ C starts editing route field ↓ C finishes editing route field

Taskload (voice comm)

↓ C initiates A-G call ↓ C completes A-G call

Operational Events (aircraft states)

↓ AAL1234 enters sector ↓ AAL1234 begins to follow new route

27



October 2019


Considerations for the future

• Cheap biometric sensors

– Wearable technology

• Big data analytics and visualization

techniques

– Neuroergonomics tools provide a mountain of data

– Pattern derivation within and across metrics

– Potential application of machine learning & AI

28



October 2019


Considerations for the future

• Use of better operational performance metrics from fast-time sims into HITLs

• Fast-time models with more sophisticated human models

– Stochastic models of human decisions and actions

– Output metrics in human terms

– Use of HITLs to develop algorithms and parameters for fast-time models

• Use of fast-time sims to validate and expand results from HITLs

• Use of fast-time sims to identify areas that need additional investments in HITLs


Kenneth Allendoerfer, Ph.D.

(609) 485-4864

[email protected]

Ben Willems

(609) 485-4191

[email protected]



William J. Hughes Technical Center, Building 28

Atlantic City International Airport, NJ 08405

http://hf.tc.faa.gov

Questions &

Contact Information

mailto:[email protected]

http://hf.tc.faa.gov/

Converging Measures: Federal Aviation Administration ...

Documents

Transcript of Converging Measures: Federal Aviation Administration ...