DISTRIBUTION A. Approved for public release; distribution unlimited. WPAFB 07-0083

Cognitive Technologies for Teams

711HPW/RHCPT

September 2010

Benjamin A. Knott, Ph.D.

Human Effectiveness Directorate

711th Human Performance Wing

Air Force Research Laboratory

Report Documentation Page Form ApprovedOMB No. 0704-0188

Public reporting burden for the collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering andmaintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information,including suggestions for reducing this burden, to Washington Headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, ArlingtonVA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to a penalty for failing to comply with a collection of information if itdoes not display a currently valid OMB control number.

1. REPORT DATE SEP 2010

2. REPORT TYPE N/A

3. DATES COVERED -

4. TITLE AND SUBTITLE Cognitive Technologies for Teams711HPW/RHCPT

5a. CONTRACT NUMBER

5b. GRANT NUMBER

5c. PROGRAM ELEMENT NUMBER

6. AUTHOR(S) 5d. PROJECT NUMBER

5e. TASK NUMBER

5f. WORK UNIT NUMBER

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) Cognitive Technologies for Teams U.S. Air Force Research LaboratoryWPAFB, Dayton, USA

8. PERFORMING ORGANIZATIONREPORT NUMBER

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR’S ACRONYM(S)

11. SPONSOR/MONITOR’S REPORT NUMBER(S)

12. DISTRIBUTION/AVAILABILITY STATEMENT Approved for public release, distribution unlimited

13. SUPPLEMENTARY NOTES See also ADA560467. Indo-US Science and Technology Round Table Meeting (4th Annual) - Power Energyand Cognitive Science Held in Bangalore, India on September 21-23, 2010. U.S. Government or FederalPurpose Rights License

14. ABSTRACT The mission of the Cognitive Technology for Teams (CTT) research program is to conduct research anddevelopment that enhances the Air Forces capability to support teams that are effective, resilient, andadaptable within the context of command and control (C2). The program provides science and technologyleadership in two areas, a) the development and assessment of collaborative interfaces to extend theeffectiveness of battle managers working within a network-centric framework, and b) the development ofnew metrics for assessing team workload and performance. To this end the program conducts appliedresearch within two laboratories. The Collaborative Technology Testbed permits the systematic evaluationof advanced collaboration interface technologies, data visualization tools, and multi-modal interfacetechnologies and their effects on team performance, communication effectiveness, shared situationawareness, and decision effectiveness. Experiments in this lab typically employ high-fidelity simulatedwork environments for human-in-the-loop experimentation. The Augmented Team Workload AssessmentLab is designed to explore the application of physiologic-based operator state assessment technology to theobjective, online measurement of team states such as mental workload, stress, and fatigue. Research in thislab is focused on the development and validation of theory-driven, innovative subjective and behavioralmetrics for characterizing individual and team workload; and development of robust physiological indicesof team workload, with a particular interest in minimally invasive measures such as EEG, EOG, ECG eyemovement data and cerebral hemodynamics. Current research directions for the CTT program will be discussed.

15. SUBJECT TERMS

16. SECURITY CLASSIFICATION OF: 17. LIMITATION OF ABSTRACT

SAR

18. NUMBEROF PAGES

19

19a. NAME OFRESPONSIBLE PERSON

a. REPORT unclassified

b. ABSTRACT unclassified

c. THIS PAGE unclassified

Standard Form 298 (Rev. 8-98) Prescribed by ANSI Std Z39-18

2

Plans & Programs

(XP)

Directed Energy (RD)Kirtland AFB, NM

Space Vehicles (RV)Kirtland AFB, NMHanscom AFB, MA

Information (RI)Rome, NY

Wright-Patterson AFB, OH

Sensors (RY)Wright-Patterson AFB, OH

Hanscom AFB, MARome, NY

Propulsion (RZ)

Wright-Patterson AFB, OHEdwards AFB, CA

Air Vehicles (RB)

Wright-Patterson AFB, OH

Munitions (RW)

Eglin AFB, FL

711 Human Performance Wing

Wright-Patterson AFB, OH

Brooks AFB, TXMesa, AZ

Materials & Manufacturing

(RX)Wright-Patterson AFB, OH

Tyndall AFB, FL

Air Force Office of Scientific

Research (AFOSR)

Washington DC

Comptroller

(FM)Personnel

(DP)Legal

(JA)Small

Business

(SB)

Air Force Research Laboratory

Organization Structure

Maui, HI

Contracting

(PK)Comp Science &

Engineering (RC)Safety

(SE)

HQ AFRL

Human Effectiveness

(RH)Wright-Patterson AFB, OH

Brooks AFB, TXMesa, AZ

3

Battlespace Acoustics Branch (RHCB)Leading the discovery, evaluation, and transition of revolutionary auditory and communication technologies that optimize warfighter survivability and lethality across the full range of battlespace environments

Supervisory Control Interfaces Branch (RHCI)Conducting research to enhance the effectiveness of the integration of crew and/or operators with intelligent and autonomous systems to fully exploit the joint capabilities of the human-machine system.

Collaborative Interfaces Branch (RHCP)Leading the discovery of innovative technologies that optimize human-to-human and human-to-machine collaboration in a network-centric, distributed environment for both teams and individuals across all USAF domains

Battlespace Visualization Branch (RHCV)Advancing the science and technology associated with the collection, optimization, display, and assimilation of visually complex informationto enable accurate and effective decision making across the battlespace domains

Warfighter Interface Division

4

Collaborative Interfaces for C2

Program Goals• Collaborative Tools for Tactical C2

(FY 05-09)

– Design Tools – with multi-modal collaborative interface technologies

– To Enhance: Performance Efficiency,

Decision-Making, Situation Awareness,

Workload

• Augmented Team Workload Assessment (FY08-12)

– Develop Metrics to Assess: Team Cognitive Workload and Situation Awareness

– To Enhance: Distribution of Workload, Situation Awareness, Efficiency & Effectiveness of Decision-Making, Speed of Command

• Other Studies/Research Areas

5

DRAW Use Case 1: Dynamic Replanning

POP UP CAUSES CHANGE IN

CAP POSITION AIR DEFENCE

MOVES TO ACCOMMODATE

REPLANNED EGRESS

AD BENO

Can operators quickly communicate information changes (e.g. new EoB info that changes plan), and insure safety and success?

6

Experiment Players

•C2:

– Tactical – E-3

– Operational – CAOC (White Force)

•TST Strike Package:

– Strikers – GR-4s

– SEAD – F-16CJs

TST Coord

Ftr C2

7

Experimental Factors

• Joint Warrior-based scenario– TST and intel injects via WF

• US & UK operator participation – 1-5 Mar, Farnborough, UK

• Experimental focus:– Interflight coordination for support

– ROE & SEAD asset dependencies– Quickly communicating changes

– DRAW tool (US)– Dynamic mission replanning

– Mission management tools (UK)– Resulting TTPs

8

Chat Communication Study

Benefits to the Warfighter

Projected Research Schedule As of: 11 Dec 09Month Nov Jan Mar May July

Dec Feb Apr Jun Data Interview with E-3 SME’s

Synthetic Task Environment Scenario Design, Development & Testing

Live-fly Observation

Distribute Survey

Collect experimental data

Data Analysis & Write-up & Final Report

ApproachThree major components:

1.Operational Chat Survey

2.Live-fly Observation (if permissible)

3.Empirical (Lab) Study

Greater insight into the impact of chat communication

can help:

• Enhance implementation to maximize strengths

and minimize weaknesses

• Focus future training

• Make smart decisions on best practice of deploying

technology

Chat communication study to assess the impact chat

communication may have on communication processes,

shared understanding, and sensemaking behaviors; all of

which impact communication and coordination effectiveness.

Description

9

Chat Communication Study

Three Components of Study:

1.Operational Survey

- Objective: understand current practices, procedures, issues, usage, concerns, and operator requirements.

2.Field Observation (contingent on opportunity)

- Objective: understand chat usage and difficulties and domain challenges

3.Experimental Study

- Objective: empirically test the impact chat technology on how teams of operators use this tool to solve problems, coordinate, and communicate

Communication = a means to solve complex problems

Voice Keyboard

Transient Voice Only Chat Only (messages

disappear)

Permanent Voice + Archival

Chat Log

Chat + Archival Chat

Log

2x2 Experimental Design

10

Team Resource And Cognitive

Effectiveness (TRACE) MonitorObjective: Develop near-real-time behaviorally- and

neurophysiologically-based measures of team fitness

(operator functional states).

Approach

• Leverage advanced mathematical techniques to recognize

patterns in team behavioral and physiologic data

associated with effective or impaired team performance

– Statistical modeling of team communication and behavior using

Hidden Markov Models (HMMs)

– Applying nonlinear data analytic techniques (recurrence quantification

analysis, cross-recurrence quantification, fractal analysis, etc.) to

identify chaotic, emergent patterns in team communication and

physiologic data

• Apply online, neurophysiological measures to diagnose

likely drivers of team performance impairments (extreme

workload, inequitably distributed workload, stress, fatigue,

etc.)

– Potentially useful measures have been derived from EEG, ECG, eye-

gaze tracking, and cerebral hemodynamics and oximetry

Fractal analysis of human inter-beat interval data

Cross-recurrence analysis of eye-gaze data

HMM of operator UAV control (from Boussemart, Las Fargeas, Cummings, & Roy, 2009)

11

Relevance

• Future network-centric CONOPs require rapidly formed, distributed teams for missions such as

time-sensitive-targeting (e.g., Alberts & Hayes, 2003).

• Distributed teams may not have the opportunity to develop shared mental models that support

good team performance & SA (Salas et al., 1995)

• TRACE will allow remote mission commanders and adaptive aiding tools to perceive and

anticipate team “mental” fitness, allowing them to better direct team resources and improve

performance & SA

Payoffs

• Monitoring and diagnostic tools for dynamic assessment, management, and mitigation of

teams, improving performance and SA

• Provides critical metrics for understanding human-unmanned systems

– Such systems operate differently than human teams in many respects (e.g., issues of trust,

complacency, communication, etc.)

– TRACE provides additional/novel approaches to understand teams, team processes

• A diverse suite of validated team process metrics, allowing more accurate appraisal of team

effectiveness

– Allows us to treat teams as emergent systems, not simply collections of individuals

TRACE Monitor

12

Cerebral Hemodynamics

• Transcranial Doppler Sonography

(TCD)

– Utilizes ultrasound signals to monitor

intracranial arteries

– When a particular area of the brain

becomes metabolically active, by-

products of this activity will increase

• This results in increased blood flow to the

region to remove the unwanted by-products

• Near-Infrared Spectroscopy (NIRS)

– Utilizes tissue absorption of near-

infrared wavelengths to measure

cortical oxygen saturation levels or

regional saturation of oxygen (rSO2)

13

Voice Stress Analysis

Measures:

• Low Power

• Word Search

• Word Count

• Word Trimming

• Pitch Tracking

• Gender

Determination

• Fundamental

Frequency

• Jitter/Shimmer

Wave File

Low Power Algorithm

MrateAlgorithm

Pitch Tracking

Jitter/Shimmer Algorithms

Infrasonic Stress Measures

Speaking Rate Measures

f0 Measures

Jitter/Shimmer Measures

Voiced Segments

A sample of the data used for analysis. Raw data (left) is filtered and windowed (center) prior to power spectrum (right).

14

Non-Linear and Fractal Analysis

• Time series data

– EEG Data

– EKG

– Eye Tracking

– Communication data

• Recurrence quantification

analysis

– Behavior of systems over

time

– Visual representations of

patterns

• Power Spectral Density

Analysis

– Fast Fourier Transform

– Determine if the system is

fractal in nature

– Combined with control

theory to determine the

nature of the system

producing the time series

Goal: Determine the utility of dynamical systems

analysis for understanding team processes

Cross-recurrence analysis of eye-gaze data

y = 3E-09x-2.229

1.00E-04

1.00E-03

1.00E-02

1.00E-01

1.00E+00

1.00E+01

1.00E+02

1.00E-06 1.00E-05 1.00E-04 1.00E-03 1.00E-02 1.00E-01 1.00E+00

Power Spectral Density

-3 to -4 Hz -4 to -5 Hz

Fractal analysis of human inter-beat interval data

15

Sensitivity & Diagnosticity in

Predicting Team Performance

• HMM development extracts patterns of behavior

from large corpora of training data

– Future prediction is based on statistical

likelihoods of a chain of behavior derived from

patterns learned in training

– Provide a novel means to monitor and predict

individual and team performance

– Uncertain if predictive accuracy is improved

using separate HMMs for each team member,

or using a single “team” HMM

– HMM prediction under different levels of task

demand

• EEG-based measures of workload are particularly

promising (Gevins & Smith, 2003)

– Central assumption is that changes in brain

activity reflect ongoing mental work (Tsang &

Vidulich, 2006)

16

RHCPT Team

Research Scientists

– Gregory Funke, Ph.D.

– Benjamin Knott, Ph.D.

– Lt Connie Ambrose

– Becky Brown

– April Courtice*

– Matthew Funke*

– Maj Chris McClernon,

Ph.D.

– April Rose Panganiban*

– Sheldon Russell*

*students

Software Engineers

– Allen Dukes

– Brent Miller

– Jim Hyson

– Matt Middendorf

Program Managers

– Sam Kuper

17

• AFRL

– RHCPA – RHXS

– RHCB – RHCI

– RHA – RISA

• International Partners

– DSTL & QinetiQ, UK

– DSTO, AUS

• Academic Partners

– Massachusetts Institute of Technology (MIT), Humans and Automation

Laboratory (HAL)

– University of Cincinnati

– Wright State University

– West Point

– University of Central Florida

• Industry

– Boeing

RHCPT Collaborators

18

Collaborative Tools for C2

Questions?

19

RHCPT Spaces and Projects

• CTT Lab

– SDO Program – Sam Kuper

– Dynamic Cyber Security – Janet Peasant

– MATRIX Experiment

• DRAW & UK PA – Allen Dukes

• WCAS & MMC – Brent Miller

• SPO Chat Study – April Courtice

• Nonlinear Analysis – Sheldon Russell

• Workload Lab

– Voice Stress Analysis

• Algorithms and Metrics – Chris McClernon, Matt Middendorf

• Nonverbal Voice Stress Analysis –Mike Harter

– BioRadios & EEG – Becky Brown, April Rose Panganiban

• DART Lab

– Change Blindness Research – April Rose Panganiban, Becky Brown

• BMC2 Lab

– Transcranial Doppler Sonography Research

• TCD and Vigilance Research –Matt Funke

• Removal of Voice-Related Artifacts from TCD Recording – Connie Ambrose

– Overview of TRACE Research

• FaceLab Demo – Allen Dukes

• TRACE Workload Scale and Exchange Interfaces – Jim Hyson