The Impact of Coordination Quality

on Coordination Dynamics and Team Performance:

When Humans Team with Autonomy

by

Mustafa Demir

A Dissertation Presented in Partial Fulfillment

of the Requirements for the Degree

Doctor of Philosophy

Approved April 2017 by the

Graduate Supervisory Committee

Nancy J. Cooke, Chair

Jennifer Bekki

Polemnia G. Amazeen

Robert Gray

ARIZONA STATE UNIVERSITY

May 2017

i

ABSTRACT

This increasing role of highly automated and intelligent systems as team members has

started a paradigm shift from human-human teaming to Human-Autonomy Teaming

(HAT). However, moving from human-human teaming to HAT is challenging.

Teamwork requires skills that are often missing in robots and synthetic agents. It is

possible that adding a synthetic agent as a team member may lead teams to demonstrate

different coordination patterns resulting in differences in team cognition and ultimately

team effectiveness. The theory of Interactive Team Cognition (ITC) emphasizes the

importance of team interaction behaviors over the collection of individual knowledge. In

this dissertation, Nonlinear Dynamical Methods (NDMs) were applied to capture

characteristics of overall team coordination and communication behaviors. The findings

supported the hypothesis that coordination stability is related to team performance in a

nonlinear manner with optimal performance associated with moderate stability coupled

with flexibility. Thus, we need to build mechanisms in HATs to demonstrate moderately

stable and flexible coordination behavior to achieve team-level goals under routine and

novel task conditions.

ii

DEDICATION

This dissertation is dedicated to those who have helped my academic and

professional career. I want to express my deep and sincere gratitude to my advisor, Dr.

Nancy J. Cooke, for her encouragement and generous support throughout my study at

Arizona State University. Without her brilliant guidance, this dissertation would not have

been possible.

I would also like to thanks to my dissertation committee: Dr. Jennifer Bekki, Dr.

Polemnia G. Amazeen, and Dr. Robert Gray for their valuable comments on this

dissertation. I also would like to extend my gratitude to my industry collaborators for

their support throughout this research: Dr. Steven M. Shope, Dr. Christopher W. Myers,

and Paul Jorgenson.

I would like to convey thanks to Dr. Nathan J. McNeese of the Cognitive

Engineering Research Institute (CERI); and Dr. Aaron Likens and Cameron Gibbons of

the Dynamics of Perception, Action, and Cognition (DPAC) lab for their support and

invaluable suggestions throughout the course of this dissertation.

I am also deeply indebted to my friends, Dr. Nedim Yel and Stanley A. Lubanski

for their help, stimulating suggestions and encouragement, and for giving me a shoulder

to lean on whenever I need.

Finally, I am mostly grateful to my mother Dondu Demir, my father Selim Demir,

and my sister Zehra Demir for their love and support. Since the very beginning of my

academic career my family has provided an immense amount of support for me, and

without them, the completion of this work would not have been possible.

I dedicate this work to you all.

iii

ACKNOWLEDGMENTS

Portions of the work presented in this dissertation were supported by a research

grant from the Office of Naval Research (ONR) Award N000141110844 (Program

Managers: Dr. Marc Steinberg, Dr. Paul Bello). The ACT-R based synthetic agent, which

was used as a team member in this research, was developed by Dr. Christopher W. Myers

and his research team Dr. Jerry Ball, and Dr. Mary Friedman in the Air Force Research

Laboratory (AFRL). The Unmanned Air Vehicle - Simulated Task Environment (UAV-

STE) was developed by Dr. Steven M. Shope and Paul Jorgenson. The experiment was

conducted by the Cognitive Engineering Research Institute (CERI) research team, Cade

Bartlett, Ashley Knobloch, and Lourdes Reyes. In addition, this dissertation was

supported during the 2016 - 2017 academic year by the Graduate Education Dissertation

Fellowship Grant by Arizona State University Graduate Education College.

In this dissertation, some of portions of Chapters 2, 3, and 4 based on the

following published reports:

Cooke, N. J., Demir, M., & McNeese, N. J. (2016). Synthetic Teammates as Team

Players: Coordination of Human and Synthetic Teammates (Technical Report No.

N000141110844). Mesa, Arizona: Cognitive Engineering Research Institute.

Demir, M., McNeese, N. J., Cooke, N. J., & Myers, C. (2016). The Synthetic Teammate

as a Team Player in Command-and-Control Teams. Proceedings of the Human

Factors and Ergonomics Society Annual Meeting, 60(1), 116–116.

https://doi.org/10.1177/1541931213601026

iv

Demir, M., McNeese, N. J., & Cooke, N. J. (2016). Team situation awareness within the

context of human-autonomy teaming. Cognitive Systems Research.

https://doi.org/10.1016/j.cogsys.2016.11.003

v

TABLE OF CONTENTS

Page

LIST OF TABLES ............................................................................................................ vii

LIST OF FIGURES ......................................................................................................... viii

LIST OF ABBREVIATIONS ............................................................................................ ix

CHAPTER

1 INTRODUCTION .................................................................................................. 1

2 BACKGROUND LITERATURE ........................................................................... 3

Human-Autonomy Teaming ............................................................................. 4

Team Cognition .............................................................................................. 15

Nonlinear Dynamical Systems Approach ....................................................... 29

3 EXPERIMENT OVERVIEW AND HYPOTHESES ........................................... 36

Task and Roles ................................................................................................ 36

Hypotheses ...................................................................................................... 37

4 METHODOLOGY ............................................................................................... 41

Participants ...................................................................................................... 41

Materials and Apparatus ................................................................................. 41

Procedure ........................................................................................................ 43

Manipulation ................................................................................................... 46

vi

CHAPTER Page

Measures ......................................................................................................... 47

5 DATA ANALYSIS AND RESULTS ................................................................... 54

Attractor Reconstruction ................................................................................. 54

Stability Analysis ............................................................................................ 56

Predicting Team Performance and Situation Awareness via the Lyapunov ... 59

Surrogate Analysis .......................................................................................... 60

Joint recurrence Quantification Analysis (JRQA) .......................................... 61

Results Summary ............................................................................................ 68

6 DISCUSSION ....................................................................................................... 70

Contributions of This Research ...................................................................... 72

Limitations ...................................................................................................... 74

Future Work .................................................................................................... 76

REFERENCES ................................................................................................................. 78

APPENDIX

A COORDINATION SCRIPT ................................................................................. 89

B VARIABLES FOR TEAM PERFORMANCE SCORE....................................... 91

vii

LIST OF TABLES

Table Page

1. Levels of Automation, and Shifting from Automation to Autonomy ..................... 7

2. Making Automated System as a Team Player ........................................................ 9

3. Emprical Studies with Synthetic Teammate in the Context of HAT .................... 14

4. Experimental Session ............................................................................................ 46

5. Team Performance Score ...................................................................................... 48

6. Summary of Regression Analyses for Lyapunov Exponent Predicting Team

performance, Team Situation Awareness, and Target Processing Efficiency ...... 60

7. Summary of Regression Analyses for Determinism Predicting Team Performance,

Team Situation Awareness, and Target Processing Efficiency ............................ 68

8. Summary of Analysis of Variance Results ........................................................... 69

9. Team Coordination Dynamics Across the Conditions and Their Implications on

Team Performance and Situation Awareness ....................................................... 72

viii

LIST OF FIGURES

Figure Page

1. I-P-O: Team Mental Model to Team Performance. .............................................. 18

2. Proposed Relationship between Team Stability and Team Performance ............. 40

3. Team Coordination Sequence: Information-Negotiation-Feedback ..................... 50

4. Coordination Logger Software Tool ..................................................................... 51

5. The Intrinsic Geometry Coordination Score. ........................................................ 52

6. Situation Awareness Window ............................................................................... 53

7. Example Reconstructed Attractors from Three UAV Teams ............................... 57

8. Mean the Largest Lyapunov Exponents (λ) Across the Conditions ..................... 58

9. Example Joint Recurrence Plots for Three UAV Teams’ Interactions in Three

Conditions. ............................................................................................................ 66

10. Determinism Across the Conditions (Control < Experimenter < Synthetic) ........ 67

11. The Predicted Relationship between Team Stability and Team Performance for

Each Condition; Observed Example Coordination Dynamics for Each Condition

............................................................................................................................... 72

12. Results of the Real-Time Dynamical Analysis of Team Coordination Stability in

the UAV Simulator for One Mission .................................................................... 77

ix

LIST OF ABBREVIATIONS

ACT-R : Adaptive Control of Thought Rationale

AFRL : Air Force Research Laboratory

AMI : Average Mutual Information

ANOVA : Analysis of Variance

AVO : Air Vehicle Operator

C2 : Command-and-Control Environment

CAST : Coordinated Awareness of Situation by Teams

CERTT : Cognitive Engineering Research on Team Tasks Laboratory

CRP : Cross Recurrence Plot

CRQA : Cross Recurrence Quantification Analysis

DEMPC : Data Exploration, Mission Planning, and Communications

DET : Determinism

FNN : False Nearest Neighbors

m : Embedding Dimension

HAT : Human-Autonomy Teaming

INF : Information-Negotiation-Feedback

I-P-O : Input-Process-Output

ITC : Interactive Team Cognition

JRQA : Joint Recurrence Quantification Analysis

JRP : Joint Recurrence Plot

к : Kappa Score

x

λ : Lyapunov Exponent

MANOVA : Multivariate Analysis of Variance

NASA TLX : National Aeronautics and Space Administration Task Load Index

NDM : Nonlinear Dynamical Method

NDS : Nonlinear Dynamical Systems

PLO : Payload Operator

ROZ : Restricted Operating Zones

RP : Recurrence Plot

RQA : Recurrence Quantification Analysis

SA : Situation Awareness

STE : Synthetic Task Environment

𝜏 : Time delay

TSA : Team Situation Awareness

UAV : Unmanned Aerial Vehicle

1

CHAPTER 1

INTRODUCTION

When two or more people (who may have diverse training and background

knowledge) work together to accomplish a task, they must coordinate dynamically and

adaptively. This process of team member interaction (i.e., communication and

coordination) is team cognition. In this case, teams can be considered as dynamical

systems in which cognition emerges through interactions (Cooke, Gorman, Myers, &

Duran, 2013), which must be measured at the team level. Cooke and colleagues have

conducted nine experiments in the Unmanned Aerial Vehicle (UAV)-Synthetic Task

Environment (STE) with all human teams in order to observe the relationship between

team process behaviors—interactions, i.e., coordination, communication, and situation

awareness—and team performance (Cooke, Gorman, Myers, & Duran, 2013; Demir &

Cooke, 2014). More recently, in collaboration with the Air Force Research Lab (AFRL),

the human UAV pilot or Air Vehicle Operator (AVO) has been replaced with a

cognitively plausible ACT-R-based computational model that serves as a fully-fledged

synthetic teammate for a three-agent UAV ground control crew. The synthetic teammate

can potentially be used for training as a coach, white forces for team training, and even as

an operational teammate.

The findings from several of these studies (Cooke et al., 2013; Demir & Cooke,

2014; Demir et al., 2015) underline the importance of coordination, specifically the

coordination difficulties that may plague the synthetic teammate (e.g., lags in chat or

human expectations of autonomy). They also found that working with a synthetic

2

teammate changes subtle interactions and perception of workload (Demir & Cooke,

2014). These findings provide the basis for the current experiment, the focus of which is

the impact of quality of individual coordination capabilities on team coordination and

performance within the context of Human-Autonomy Teaming (HAT). The current

synthetic teammate served as an example of a poorly coordinated pilot and, at the other

end of the scale, an experienced member of the research team served as a highly

coordinated pilot (Demir et al., 2015).

Thus, the primary purpose of this dissertation is to examine team coordination

dynamics of HATs by comparing them with all-human teams. Teams must interact

effectively under routine conditions, but also have the ability to change their coordination

patterns under novel conditions in a cognitively demanding task environment. However,

adding an autonomous team member may lead teams to demonstrate different

coordination patterns with different characteristics in routine and novel conditions when

compared to all-human teams.

3

CHAPTER 2

BACKGROUND LITERATURE

Human team members’ work increasingly involves interaction with highly

automated systems (e.g., synthetic agents or robots) in cognitively demanding task

environments, such as Command-and-Control or surgical rooms. This increasing role of

highly automated systems as team members has started a paradigm shift from human-

human teaming to Human-Autonomy Teaming (HAT). Historically, researchers have

measured team cognition for human-human teams from the shared knowledge

perspective (a static perspective): relying on measures like “mental model similarity”

which fail to capture team dynamics. When trying to capture team cognition with the

HAT framework in a dynamic environment, the static approach—which mainly considers

individual knowledge or its sum—falls short in two respects. First, in the HAT context,

some of the autonomous agents (e.g., synthetic teammate) have limited knowledge (due

to limited declarative memory) and, second, the dynamic task environment requires

effective team interaction (i.e., communication and coordination) among team members

which is the locus of team cognition. Therefore, another perspective which addresses

these issues, called the interactionist perspective, was considered by several studies

(Gorman, Amazeen, & Cooke, 2010; Gorman, Cooke, Amazeen, & Fouse, 2012), which

focused on all-human teams in a Command-and-Control environment and viewed team

cognition as emanating from team interactions using nonlinear dynamical methods. In

this regard, to understand team cognition within the context of HAT in cognitively

demanding task environments, it is important to look at team coordination dynamics.

4

Therefore, in the HAT framework, using team interaction-based methods to

capture team cognition is more meaningful than using static, knowledge-based methods if

the task environment is dynamic and complex. Thus, in this section of the dissertation,

team coordination dynamics within the HAT is explained and then discussed via related

studies—in particular, ones focusing on team coordination.

Human-Autonomy Teaming

Technological innovations in high-tech industries have resulted in collaborations

between human team members and automation—a system that will do exactly what it is

programmed without independent action (Vagia, Transeth, & Fjerdingen, 2016)—and/or

autonomy, which is a system that acts independently of any human operator (e.g.,

synthetic agents or robots) (Cox, 2013; Demir & Cooke, 2014; Endsley, 2015; Demir et

al., 2015; Vagia et al., 2016).

In the literature, though these two terms (i.e., automation and autonomy) are

different in terms of their autonomous level, they overlap with each other in how they are

used in several studies. Table 1 considers whether the automation can be a team member

or not based on the four functions that Endsley (1999)’s study mentions: 1) Monitoring

(M) - scanning displays to perceive system status, 2) Generating (G) - formulating

options or strategies for achieving goals, 3) Selecting (S) – making decisions on a

particular option/strategy, and 4) Implementing (I) – carrying out the chosen option (p.

464: see Table 1). When the level of automation increases, the automation gets closer to

being autonomous and, in turn, being a team member. However, up to Level 10, the

automation is still not a highly automated system (i.e., autonomy), and as a result not a

5

team member (the reasons are specifically discussed in the following sections). For

instance, the first level of the automation, ‘Manual Control’ does not require assistance

from the automation, and a human carries out all of the four listed functions: acquisition

of information, conscious perception, decision making, and response selection (e.g.,

synthetic agent). Therefore, there is no human-autonomy teaming at all. Another example

on the table is Level 9 - ‘supervisory control’1 in which the automation is close to being a

team member, but still not fully independent from the human operators. In terms of HAT,

at the last level (Level 10), the automation is equivalent to an autonomy and can therefore

be considered a team member because it independently acts on each of the four functions

(Endsley & Kaber, 1999: see Table 1).

Thus, autonomy can be defined as systems that can function—at least partially in

a self-directed manner—outside of the sorts of situations that they were designed for by

using some intelligence-based capabilities (Endsley, 2015). Being more autonomous for

an agent requires several characteristics, including: the ability to achieve goals

independently, use verbal and non-verbal communication to perform better under

dynamical environments, and self-correct in the event of system failures (Schooley,

Zeigler, Cellier, & Wang, 1993; Krogmann, 1999; Endsley, 2015).

With that in mind, HAT can be defined as a team that has human team members

working with autonomy with some degree of intelligence (Schulte, Donath, & Lange,

2016). There are several studies which have examined the use of automated systems as a

teammate (even though it was called “automation” in the study) (Sycara, & Lewis 2004;

1 Supervisory control can be defined as “intermittent operator interaction with a computer that closes an

autonomous control loop” (Donmez, Nehme, & Cummings, 2010, p.1180).

6

Wijngaards, Kempen, Smit, & Nieuwenhuis, 2006; Cuevas, Fiore, Caldwell, & Strater,

2007; Langan-Fox, Canty, & Sankey, 2009; Schulte, Donath, & Lange, 2016). For

instance, Wijngaards et al. (2006) considers automated or intelligent systems as

teammates equivalent in status to humans and, therefore, they define ‘team’ as an “actor-

agent community” (p.35). In their study, they underline that automated (or intelligent)

systems may take the initiative and give orders to fellow teammates (either human or

automated). Another study done by Cuevas et al. (2007) again considers this actor-agent

community idea for HAT and they define it as “the dynamic, interdependent coupling

between one or more human operators and one or more automated systems requiring

collaboration and coordination to achieve successful task completion” (p. B64). Also,

these highly autonomous systems make their own decisions about their action during the

team task. Therefore, without outside intervention, an autonomous system can

independently achieve goals and maintain good performance in highly dynamic

environments by interacting with other humans or other agents (Schooley, Zeigler,

Cellier, & Wang, 1993; Krogmann, 1999; Endsley, 2015). Although these studies

consider autonomous systems as a team member, Klein, et al. (2004) underline that the

automated system’s lack of intelligence is a large obstacle on its path to becoming a team

member.

7

Note

. M

onit

ori

ng (

M),

Gen

erat

ing (

G),

Sel

ecti

ng (

S),

and I

mple

men

ting (

I);

Hum

an (

H),

Auto

mat

ed (

A)

Note

. A

dap

ted f

rom

“L

evel

of

auto

mat

ion e

ffec

ts o

n p

erfo

rman

ce, si

tuat

ion a

war

enes

s an

d w

ork

load

in a

dynam

ic c

ontr

ol

task

,” M

.R.

Ensl

ey a

nd D

.B.

Kab

er,

1999,

Erg

onom

ics,

42

, p

p. 4

64-4

65

; an

d “

A m

od

el f

or

typ

es

and l

evel

s of

hum

an i

nte

ract

ion w

ith a

uto

mat

ion,”

R. P

aras

ura

man

, T

.B. S

her

idan

and C

.D. W

icken

s, 2

000,

IEE

E

Tra

nsa

ctio

ns

on S

yste

ms,

Man a

nd C

yber

net

ics,

Part

A:

Sys

tem

s and H

um

ans,

30, p. 287.

Tab

le 1

Lev

els

of

Auto

mat

ion, an

d S

hif

tin

g f

rom

Auto

mat

ion t

o A

uto

nom

y

8

Challenges of autonomy being a team member. The challenges of making an

automated system a team member is amplified in highly dynamic contexts (such as

emergency medical care and military command-and-control).

In a number of contexts, the HAT members interact with each other in a

distributed manner, separated by time and space. In this kind of highly dynamic

environment many different factors can have an impact on HAT performance including:

task characteristics (time pressure, workload), automated agent’s characteristics

(behavioural characteristics), automated vehicle characteristics (air, ground, underwater),

environmental characteristics (terrain quality, obstacles, time of the day, weather), and

technological constraints (available bandwidth, communication delays, visual display

characteristics; Cuevas et al., 2007). This highly dynamic context also requires the

development of a shared understanding between team members in the current situation.

However, developing a shared understanding between team members is extremely

complicated in HATs, mainly due to the automated team member not having the

intelligence to develop and share knowledge to articulate an understanding of its

teammates.

Christoffersen and Woods (2002) and Dekker and Woods (2002) consider that

automated agents can be teammates, if they provide two fundamental characteristics:

observability (i.e. human operator is able to observe which display and control panel the

agent is attending to during, and after, the task) and directability (i.e. human operators

need to be able to re-direct machine activities fluently in instances where they recognize a

9

need to intervene; Christoffersen & Woods, 2002). Based on this perspective, good team

members make their activities observable to fellow team members and easy to direct

(Dekker & Woods, 2002) and these same characteristics are the driving forces in the

relationship between human and automated agents in any joint system (Christoffersen &

Woods, 2002, p. 10). However, filling the role of a human teammate is extremely

challenging for automated systems. Klein et al. (2004) proposed 10 challenges (see Table

2) in making automated systems into effective teammates when they interact with

humans in significant ways (p. 92). Based on these challenges, one of the main goals of

HAT is to formulate higher-level cognitive processes (i.e., team mental models, team

situation awareness, sense making).

Table 2

Making an Automated System a Team Player

Challenges Description

Engaging in

common

ground

activities

Fulfill the requirements of a basic compact to engage in common-grounding activities.

Basic compact, which is a mostly tacit agreement, consists of some requirements,

including: facilitating coordination, working toward shared goals, and preventing

breakdowns in team coordination.

Adequately

modeling

Be able to adequately model the other teammates’ intentions and actions vis-à-vis the

joint activity’s state and evolution (e.g. are they having trouble, are they on a standard

path proceeding smoothly?).

Predictability Be reasonably predictable and reasonably able to predict others teammates’ future

behavior. However this is a difficult process. According to Klein et al.’s (2004) study,

automated systems currently work directly against the confidence that people have in

their predictability (p. 92). There is a negative relationship between more adaptable

automated systems and predictability, because “the more a system takes the initiative

to adapt to human teammate’s working style, the more reluctant operators might be to

adapt their own behavior because of confusions that these adaptations might create”

(Klein et al., 2004, p.92)

Directability Be directable. That is, developing means for controlling aspects of automated systems

in a fashion that can be both dynamically specified and easily understood (Klein et al.,

2004, p. 93).

(continued)

10

Table 3 (continued)

Making an Automated System a Team Player

Challenges Description

Revealing

pertinent

aspects

Be able to make pertinent aspects of their status and intentions obvious to their

teammates. One of the examples for this case is the highest levels of automation on the

flight deck of commercial jet aircraft (flight management systems). In this case,

commercial pilots occasionally have issues with flight management systems, which

create questions about the automated systems, such as: “what is the automated system

currently doing?”, “why it is doing that?”, and “what it will do next?”

Observing and

interpreting

signals

Be able to observe and interpret signals of status and intentions. This means that the

automated systems need to interpret signals received and also form models of their

teammates. However, mostly automated systems can’t, “recognize the other

teammates’ stance, much less appreciate the operator’s knowledge, mental models, or

goals, giving the evolving state of the plan in progress and the world being controlled”

(Klein et al., 2004). It is also assumed that there is an inherent asymmetry in

coordination competencies between human and automated systems, thus it creates

issues for designing human-automation teams. It is possible to minimize this

asymmetry by stretching the automated systems’ performance. For instance,

“exploiting and integrating available channels from the agent to the human and,

conversely, sensing and inferring the human’s cognitive state through range of

psychological measures in real time” (Klein et al., 2004, p.93).

Goal

negotiation

Be able to engage in goal negotiation. In order for the appropriate team members to

engage in negotiations, automated systems must relay current and potential goals.

Supporting

technology

With a collaborative autonomy approach, there is an assumption that the

understanding, problem solving, and task execution processes build slowly over time,

are open for negotiation, and always tentative. Every element of an “autonomous”

system must be designed with this in mind, i.e., designed to make it easy for this sort

of natural, give-and-take teamwork to occur among team members.

Managing

attention

Be able to participate in managing attention. During coordination, teammates direct

each-other’s attention in order to get the most important signals, activities, and

changes. However this is not easy for automation.

Controlling

cost

Help control the costs of coordinated activity. All coordination activities (e.g.

providing signals, improving predictability, monitoring the others’ status) take time

and continual energy investment, otherwise they break down. In order to keep

coordination costs down, it is necessary to have good human-computer interface

design, and to have the agents follow the human teammates’ needs, and not vice-versa.

Note. Adapted from “Ten Challenges for Making Automation a “Team Player” in Joint

Human-Agent Activity,” by G. Klein, D.D. Woods, J.M. Bradshaw, R.R. Hoffman and

P.J. Feltovich, 2004, IEEE Intelligent Systems, 19, pp. 91–95.

11

Synthetic agent as a team member. Several studies have focused on developing

a synthetic agent to coach or serve a training function (Zachary, Santarelli, Lyons,

Bergondy, & Johnston, 2001; Ball et al., 2010; Priest & Stader, 2012), as well as to

model behavior of the appropriate coordination dynamics in a team (Cooke et al., 2013).

Here, a synthetic teammate is a cognitively plausible ACT-R (Adaptive Control of

Thought—Rational) based computational model/cognitive architecture (Anderson, 2007).

In the current computational model, five key components have been implemented and

integrated within the synthetic teammate, including (Rodgers, Myers, Ball, & Freiman,

2011, p.341): language comprehension (i.e., reading incoming texts, producing linguistic

representation, handling ambiguity), language generation (i.e., request and response

messages, selecting messages), agent-environment interaction (i.e., completing flight-

relevant goals, selecting goals using declarative memory), dialog management (i.e.,

controlling language generation via utilizing dialog norms), and situation modeling (i.e.,

providing context to language generation, agent-environment interaction, dialog

management, language analysis).

As for modeling teammates (i.e, synthetic teammates), if interactions between

teammates can be produced solely by cognitive capacities (i.e., five key components) at

the teammate level of analysis, then creating a synthetic agent inside of an all-agent or

agent-human team with the right cognitive capacities should result in a rigorous test for

producing team-level cognitive phenomena synthetically. If this test produces

coordination dynamics in teams with autonomy like those produced by all-human teams

12

(i.e., team-level learning and retention), then it would further validate the synthetic agent

as a teammate.

Examining the synthetic teammate within HAT. In the synthetic teammate

project, a synthetic agent played one of the roles in a three-agent Unmanned Air Vehicle

(UAV) team. In the analyses of the data from a recent synthetic teammate study (which

are summarized in Table 3), we mainly addressed the following two questions: (1)

whether teams communicated effectively or not, and (2) how their communication was

associated with their team performance.

The synthetic teammate’s communication with its team members and task

environment is crucial for good team performance in a dynamical task environment. The

recent analyses conducted for the synthetic teammate project underline the importance of

team communication behavior (Demir & Cooke, 2014; Demir et al., 2015) and its

relationship with team performance (Demir, McNeese, & Cooke, 2016; Demir, McNeese,

Cooke, et al., 2016; Demir, McNeese, & Cooke, 2016; McNeese, Demir, Cooke, & Myers,

2017).

From the perspective of communication with a synthetic teammate, Demir and

Cooke (2014) in an earlier study indicated that adding a faux-synthetic teammate (played

by a human) as a team member changed the human team members’ communication

behavior as the humans exerted more control on the “synthetic team member”. Later,

Demir et al. (2015) showed that, because the synthetic teammate had limited

communication behaviors, human team members either had to restrict their speech patterns

13

such that it could understand them (i.e., avoiding cryptic or misspelled language), or restrict

coordination with it and, thus, jeopardize the mission.

From the perspective of the relationship between team verbal behaviors and team

performance, Demir, McNeese, and Cooke (2016) showed that, even when human team

members communicated properly with the synthetic agent, team performance was

hampered by limited interaction capabilities within the team. This limited interaction

capability of the synthetic agent and, in turn among the team members in the HAT, led to

poor team adaptation in the dynamic environment, especially during roadblocks. Findings

from one of the studies (Demir, McNeese, & Cooke, 2016) highlight that the anticipation

of other team members’ behaviors and information requirements in synthetic teams was

lower than for the all-human teams. Therefore, developing team interaction mechanisms

within HAT is needed for effective team situation awareness and team performance

(Demir, McNeese, & Cooke, 2016; McNeese, Demir, Cooke, & Myers, 2017).

Overall these recent findings from our studies address the limited team interaction

capabilities of the synthetic teammate, and also limited interaction within the HAT and its

task environment. In order to overcome the autonomous agent’s communication and

coordination limitations, the synthetic agent needs to have interaction patterns that are

synchronous with humans in temporal state in terms of what (communication), when

(coordination), and how (communication and coordination) (Demir et al., 2015; Semin,

2007). This will also help to improve the HATs’ situational awareness, and in turn their

team performance.

14

Table 4

Empirical Studies with Synthetic Teammate in the Context of HAT

Note. Three conditions: Synthetic (Syn.), Control (Cont.), and Experimenter (Exp.)

Note. This table is adapted from the following studies (Demir, McNeese, & Cooke, 2016;

Demir, McNeese, Cooke, et al., 2016; Demir, McNeese, & Cooke, 2016; McNeese et al.,

2017).

Study Variable (Dependent &

Independent) Method Result

Demir,

McNeese, and

Cooke (2016)

Dependent Variables: Team

Performance (TP)

Independent Variables: Team

verbal behaviors, condition (syn.,

cont., and exp.), and role (pilot,

navigator, photographer)

LASSO

variable

selection

method

The effect of individual

team behaviors on TP

changed across the roles

and across the conditions.

Demir,

McNeese, and

Cooke (2016)

Dependent Variables: Team

Performance (TP) and Team

Situation Awareness (TSA)

Independent Variables: Pushing,

Pulling Information, and condition

(syn., cont., and exp.)

Split-Plot-

ANOVA

Growth

Curve

Modelling

Push: Exp > Cont > Syn

Pull: Exp > Cont > Syn

Pushing and TSA: Positive& significant

Pushing and TP: Positive

& significant

Demir,

McNeese, and

Cooke (2016);

McNeese,

Demir, Cooke,

and Myers

(Submitted)

Dependent Variables: Team

Performance (TP),

Target Process Efficiency (TPE),

Pilot Performance (PP), Team

Situation Awareness (TSA)

Independent Variable: Condition

(syn., cont., and exp.)

Split-plot

ANOVA

TP: Exp > Cont = Syn

TPE: Exp > Cont > Syn

PP: Exp > Cont > Syn

TSA: Exp > Cont = Syn

15

Summary. Moving from human-human teaming to Human-Autonomy Teaming

(HAT) is challenging, even when using advanced technology to build these highly

automated systems, (i.e., synthetic agents, robots). Based on the findings in Table 3, this

challenge can clearly be observed during the synthetic teammate’s interaction with its

human team members and its dynamic task environment. In this regard, it is possible that

adding a synthetic agent as a team member may lead teams to demonstrate different

coordination patterns with different characteristics. Therefore, the primary focus of this

dissertation is to explore team coordination dynamics in HATs in comparison with all-

human teams. More generally, team coordination can be considered one aspect of team

cognition. Therefore, in the following section, literature and perspectives surrounding

team cognition are reviewed.

Team Cognition

A team, which consists of two or more heterogeneous and interdependent

individual members, may be defined as, “a social entity composed of members with high

task interdependency and shared and valued common goals” (Salas, Cooke, & Rosen,

2008, p. 541; Cooke, Gorman, Duran, & Taylor, 2007). One of the primary activities that

teams engage in is called cognition, which refers to “cognitive processes or activities that

occur at a team level” (Cooke et al., 2013, p. 256). More specifically, “team cognition

emerges from the interplay of the individual cognition of each team member and team

process behaviors” (Cooke, Salas, Kiekel, & Bell 2003, p.4). In order to understand team

cognition, an important question remains: do individuals think within a team or do the

teams think as a whole? To answer this question, and others like it, two primary

16

perspectives of team cognition are discussed in this study (elaborated on below): 1) the

shared knowledge perspective (the predominant theoretical perspective) which focuses on

team cognition at the individual level, and 2) the interactionist perspective which focuses

on cognitive processes (i.e., interactions) at the team level.

Shared knowledge perspective - team knowledge within team cognition. Team

knowledge is “the collection of task and team related knowledge held by teammates and

their collective understanding of the current situation” (Cooke, Salas, Cannon-Bowers, &

Stout, 2000, p. 154). Therefore, team knowledge, which is shared, is the essential

component of team cognition. According to Cannon-Bowers et al. (1999)’s study, in

order to maximize team performance, team knowledge must be distributed sensibly

across team members, and arranged to ensure that each team member assesses task

situations congruently and develops effective team strategies (Cooke et al., 2000). Team

knowledge includes two high level cognitive constructs and processes: team mental

models and team situation models (Cooke, Stout, & Salas, 1998; Cooke et al., 2000;

Cooke, Kiekel, et al., 2003; Gorman, 2006; Pina, Cummings, Crandall, & Penna, 2008).2

2 In this study, the term, shared, is avoided, and the term team is used instead, because shared can mean

“to hold in common” or “to apportion”, and the ambiguity can create confusion (Cooke, Salas, Cannon-

Bowers, & Stout, 2000, p. 153).

17

Team mental model. The team mental models3 are considered “organized mental

representations of the key elements within a team’s relevant environment that are shared

across team members”, and can improve team performance if team members develop and

rely on them (Mohammed, Ferzandi, & Hamilton, 2010, p. 876). From this, it can be seen

that team mental models affect teamwork in two ways: (1) they enable “team members to

anticipate other team member behaviors and information requirements” (depite having

limited information channels), and (2) “team mental models of a team task enable team

members to perform functions from a common frame of reference” (Salas et al., 1995).

The concept of team mental models has been presented prominently in the Input –

Process – Output (I-P-O) approach of team performance (see Figure 1). Basically, the I-

P-O approach focuses on the relation between the input proporties of the team members

(a mental model from each team member), process (team process behaviors which

transform input properties into team outcomes), and products of team member interaction

(the team outcome, e.g., a decision or plan). Team mental models are an emergent state,

from the I-P-O perspective, and this state shapes (and is shaped by) each team’s

behavioral interaction processes. Taking this view, team mental models can be

considered mechanisms (inversely related to the processes) that carry the effects of input

variables (e.g., training) down to valued team outcomes (e.g., team performance) (Marks,

Mathieu, & Zaccaro, 2001).4

3 A mental model is a “mechanism whereby humans generate descriptions of a system’s purpose and form,

explanations of system functioning and observed system states, and predictions of future system states”

(Cannon-Bowers, Salas, & Converse, 1993, p. 226). 4 Besides, the locus of team cognition in the three components of the I-P-O is considered differently

(Cooke, Gorman, & Rowe, 2008): (1) collective cognition as an input (Mathieu, Heffner, Goodwin, Salas,

& Cannon-Bowers, 2000); (2) team cognition as process behaviors (e.g., leadership, adaptability,

18

Another definition, by Cooke et al. (2000), states that team mental models are

“the collective taskwork and teamwork knowledge that members bring to a situation” (p.

153). According to Figure 1, input of the I-P-O framework consists of two types of

knowledge, including taskwork and teamwork knowledge. Whereas taskwork knowledge

is “the knowledge about the individual and team task”, teamwork knowledge, is, “ the

knowledge about the roles, requirements, and responsibilities of team members” (Cooke,

et al., 2003, p. 11). Both of these types of knowledge can be obtained by individuals

within the team through formal training, experience, and other similar methods (Cooke et

al., 2000; Pina et al., 2008).

Figure 1. I-P-O: Team Mental Model to Team Performance.

Note. Adapted from the Design of Work Teams (p.316), by J.R. Hackman, 1987,

Handbook of Organizational Behavior, In J. W. Lorsch (Ed.), Englewood Cliffs, NJ:

Prentice-Hall.

In the Input-Process-Output (I-P-O) framework of team mental models

(Hackman, 1987), team coordination is thought to arise from the complementary

knowledge structures within team members’ heads, in which team knowledge is linked to

team outcomes through team coordination. This traditional approach calls for a

communications, and decision making) (Mohammed & Dumville, 2001), and 3) team cognition as an

output (Brannick, Prince, Prince, & Salas, 1995).

PROCESSCoordination

INPUTTeam Mental Model:

Individual taskwork knowledge

Individual teamwork knowledge

OUTPUTTeam Performance

19

quantified, pooled estimate of common knowledge (independent of team member roles)

where the knowledge from one team member is compared with the other team members’

knowledge (i.e., commonality) (Langan-Fox, Code, & Langfield-Smith, 2000).

Generally, in the shared-knowledge approach, effective team coordination is the

product of a team mental model (Cooke et al., 2000), and relates directly to how much

knowledge team members share (Gorman, 2014). The team mental model is one in which

individual mental models either overlap or complement each other in terms of knowledge

content and accuracy (Cannon-Bowers, Salas, & Converse, 1993). Thus, team members

can describe, explain, and predict each other through team mental models. The

development of team mental models allows team members to anticipate each other’s

needs and coordinate implicitly (in place of explicit interaction), a shift which improves

coordination and, in turn, affects team performance.

However, this approach is limited in that it does not capture the nuances of

relations between team member roles. The purpose of assembling a team of specialists is

not to maximize common knowledge, but, rather, to maximize the availability and

communication of specialty knowledge (Gorman et al., 2010). Also, this traditional

measurement perspective produces a static measure of coordination by aggregating across

time (e.g., for the dynamic task environment, looking at communication events averaged

over time). This means that, for any given point within the dynamic task environment,

this approach assumes that the team performs the “average” behavior.

Another crucial point that is ignored in this traditional approach, is how the

behavior is tied to the dynamics of the task environment. For instance, in a surgical team,

20

coordination can be changed during a routine procedure in the event of an anomaly

(Gorman et al., 2010). Thus, the team mental model approach focuses on inner mental

process, unlike team situation awareness which focuses on the dynamics of the task

environment (Gorman, 2014).

Team situation awareness. Team knowledge is a fundamental component of

Team Situation Awareness (TSA), which is the team’s understanding of a complex and

dynamic situation at any point in time (Cooke et al., 1998; Cooke, Kiekel, et al., 2003).

TSA has been adopted from theories of individual situation awareness in the aviation

field, which mainly examined Situation Awareness (SA) at the individual level (Fracker,

1989;Endsley, 1995; Durso & Gronlund, 1999). In general, SA can be defined as “the

perception of the elements in the environment within a volume of time and space, the

comprehension of their meaning and the projection of their status in the near future”

(Endsley, 1988, p. 97). Team situation awareness is considered the “input” in the I-P-O

framework, meaning that knowledge requirements are the starting point when making

decisions, planning, or performing other cognitive activities. (Cooke, Gorman, & Rowe,

2008). In the context of team knowledge, TSA (an aspect of team cognition), is a concept

wherein, “the team’s ability to assess the situation is supposedly influenced by the team’s

fleeting knowledge of the situation that the team processes” (Cooke, Kiekel, et al., 2003,

p. 180).

At any point in time, individuals on the team need to understand the current

situation and adapt their understanding when the situation changes (this is called dynamic

understanding). As such, team situation models can be considered the team’s collective

21

understanding of the characteristics of a specific situation, e.g., cues and patterns, and

these situation-dependent characteristics are what differentiate team situation models

from team mental models (Cooke et al., 2000). According to Cooke et al. (2000), under

the coordination of the team situation model, teams can make several judgments,

including: “assessing additional cues and patterns in the situation, determining the

strategies available to the team, assessing how the team is proceeding, predicting what

teammates will do and need, and selecting appropriate actions to take” (p. 154).

In the knowledge-based SA perspective, each individual team member’s SA is

measured using the Situation Awareness Global Assessment Technique (SAGAT)

(Endsley, 1995). In SAGAT, team members are questioned during team task performance

in a “frozen” task environment. The TSA researcher’s questions concern current or future

states of the team members’ heterogeneous and overlapping SA requirements (Bolstad &

Endsley, 2003). To achieve a TSA composite score utilizing SAGAT, each team

member’s query score is averaged (Bolstad & Endsley, 2003). In previous

administrations of SAGAT (Endsley, 1995), the roles’ accuracy varied on queries during

the task and was more independent than expected within the group. However,

information on performance was not provided and it remains unclear whether a common

understanding of the tested knowledge was required for teams to do their jobs. Another

challenge for measuring SA at the individual level in this manner is that changes to the

situation often occur before individuals can be queried (Gorman, Cooke, & Winner,

2006). This means that query-based metrics often interrupt the task (even removing the

22

effective stimulus) and are taken after processing has occurred (Gorman, Cooke,

Pederson, Olena, & DeJoode, 2005).

Team situation awareness and team mental models are compatible with the shared

knowledge perspective. In terms of the I-P-O framework, these two constructs are both

treated as inputs to decision making, planning, and other cognitive activities. This means

that the measures capture individual knowledge, but not cognitive processing across

individuals. Thus, as a unit of analysis, both of these constructs from the shared

knowledge perspective are static snapshots of cognition and focus on individuals, but not

the team. A new way of thinking about team cognition is to consider the team as the unit

of analysis.

Interactionist (ecological) perspective. An alternative view of team cognition is

the interactionist perspective which considers the locus of team cognition as an emergent

feature that results from a history of interactions (i.e., communication and coordination)

among team members. This stands in contrast to the shared knowledge perspective which

addresses knowledge within the individual team member (Cooke, Gorman, & Rowe,

2008). Because interaction among the team members is considered over time, the primary

focus of this perspective is interaction dynamics and team activity and, as such, the unit

of analysis is the team rather than individual.

According to this perspective, team cognition begins with the sum of individual

cognition, but then interactions (i.e., cognitive processes) happen among the individuals,

and, finally, dynamic changes happen within the team. Several studies that emphasize

team cognition as an emergent property of team interactions (Gorman, Cooke, & Winner,

23

2006; Cooke, Gorman, Duran, et al., 2007; Gorman et al., 2010; Cooke et al., 2013;

Gorman, 2014) conclude that interactions (i.e., communication and coordination) among

team members explain most of the variance in effective team performance. Therefore,

team cognition can be assessed by focusing on communication and coordination among

the team members (Cooke, Gorman, Duran, et al., 2007). Coordination requires

communication (which is a team level cognition) among the team members in order for

them to articulate their plans, actions, and responsibilities (MacMillan, Entin, & Serfaty,

2004).

Interactive team cognition. Individual cognition and team process behaviors,

which are two intertwined aspects of team cognition, are “analogous to cognitive

structures and cognitive processes at the individual level” (Cooke et al., 2003, p. 3).

Cooke et al. (2003) explains this situation with the concept of the knowledge structure

(p.3), from which individual knowledge can be explained. From the perspective of a

team, an individual’s knowledge can be transformed into team knowledge through team

member interaction, i.e., process behaviors like communication or coordination. This

transformation is the basis of effective team cognition (Cooke et al., 2000).

According to Cooke et al. (2013), the interactionist perspective primarily uses

measures like patterns of communication flow and content - as indices of team

coordination (Cooke & Gorman, 2009), and team situation awareness (Gorman et al.,

2005; Gorman, Cooke, & Winner, 2006). There are several studies (Gorman et al., 2006;

Cooke, Gorman, Pedersen, et al., 2007; Cooke, Gorman, & Kiekel, 2008; Cooke &

Gorman, 2009; Gorman et al., 2010; Cooke et al., 2013) which consider interactions

24

among the team members to be an important measure for team performance. Cooke et al.

(2013)’s observations over these previous studies show that the improvement of

communication and coordination is positively related to team performance (p. 262) and

introduces the theory of Interactive Team Cognition (ITC). This theory considers team

cognition as an emergent and dynamic activity at the team level (as opposed to shared

cognition which focuses on aggregation at the individual level). ITC theory posits that

team cognition is: 1) “an activity rather than a property or product” (p. 266), 2) “requires

team level measurement” (p. 266), and 3) is “context dependent” (p. 266; Cooke et al.,

2013). In this regard, ITC theory takes team cognition beyond team knowledge by

postulating team interactions as cognitive processes that are more directly tied to team

effectiveness than knowledge. If team members properly distribute their knowledge

within and among the team, yet lack effective coordination (due to minimal or failed

interaction), then it is likely the team will fail to meet their objective.

Thus, the main concentration of these two team cognition perspectives (i.e.,

knowledge-based and interaction-based) are different aspects of team cognition: one is

the traditional perspective which conceptualizes team cognition as the sum of individual

knowledge (which is linear and considered a context-independent product), and the other

is the interactionist perspective which conceptualizes team cognition as interactions

among the team members (which is nonlinear and considered a context-dependent

process). It is important to emphasize that the ITC does not ignore the traditional

approach, but, rather, builds upon it. Additionally, the choice between these two

perspectives will depend on the task that teams are subjected to (i.e., task dependency). In

25

this dissertation, the task is a simulated, highly complex and dynamic Command-and-

Control (C2) setting and, therefore, in this type of task environment, teams must interact

effectively making the ITC perspective the natural choice.

Interactionist measures. Interaction-based measures which rely on patterns of

communication and coordination and team situation awareness are important to the

measurement of team cognition. It is also important to know that team cognition that

relates to the team outcome is rooted in team interaction (i.e., communication and

coordination). This means that team interaction is team cognition and, as such, dynamical

communication and coordination patterns can be indicators to monitor team cognition

(Cooke, Gorman, Myers, & Duran, 2013). Therefore, three main measures are used to

capture team cognition:

The first one is team communication which can be considered a fundamental

component of team information processing and, in turn, cognitive processing at the team

level (Cooke, Gorman, & Kiekel, 2008; Salas, Cooke, & Rosen, 2008). Communication

can take several forms, including: non-verbal (e.g., gestures, eye contact), verbal (e.g.,

letters, notes, text message) and voice communications (Cooke, Gorman, & Kiekel,

2008). Wright and Kaber (2005) underline that even if there is a cost associated with a

“need for communication” between team members, there is a benefit when

communication is in the form of collaborative planning (p. 3).

With the availability of real-time recording and analysis of communication (which

is an indicator of team behavior), investigations into the patterns and flow of

communication have been undertaken (Cooke et al., 2008, p. 54). Cooke and Gorman

26

(2009) emphasize three characteristics of communication data that are especially useful

for measuring team cognition: (1) the data are rich, i.e., provides dynamic snapshots of

team behavior inside a given time frame, (2) communication is usually found within a

rich context, and (3) communication 'occurs in socio-technical systems naturally and in

some systems may constitute all or most of the interactions' (p.32).

In the communication analysis approach, the focus is placed on “techniques for

automating examination of the static and sequential characteristics [of] communication

content and flow” (Cooke & Gorman, 2009, p.33) to understand system interactions and

for real time processing, with some studies developing automated routines to this end

(Cooke & Gorman, 2009).

The interactionist perspective can also be applied to Team Situation Awareness

(TSA) via the Coordinated Awareness of Situation by Teams (CAST) measure (which is

the second measure). CAST (which focuses on coordination) is inspired by ecological

views while knowledge-query measures such as SAGAT (Endsley, 1995) are inspired by

the classical information processing view. Using CAST measures is advantageous for

studying TSA for two reasons: first, in contrast to using individual knowledge, it uses

team interactions which are a directly observable form of team cognition and, second, the

measures can be taken when the task is in progress and without interruption, because it is

a task-based measure (Gorman et al., 2005). According to the CAST measure, if teams

have high Situation Awareness (SA), then they will be expected to perform well. In this

case, team performance is a continuous process for a team, but CAST measures focus on

the coordination process during novel parts of the task.

27

Gorman, Cooke, and Winner (2006) highlight that, especially in highly dynamic

environments (e.g., Decentralized Command and Control) where the dynamics

themselves can also undergo change, adaptive team cognition requires continuous

coordination among team members in order to achieve a common and valued goal.

Henceforth, it is not enough to query, observe, or record performance deficiencies from a

team during conventional task performance (pp. 1315-1316). According to the CAST

measurement method, it is more realistic to test TSA within a dynamical environment

wherein teams get pushed away from common and valued goals. Particularly, using

roadblocks (which can be addressed by the CAST measurement) provide insight into the

team’s coordinated processes of perception and action (and, in turn, TSA) as they work

together to navigate a path around the roadblock (Gorman et al., 2005; Gorman et al.,

2006).

Another interactionist measure is team coordination, which is an important

correlate of team performance (Gorman, et al., 2010). Team coordination occurs when

two or more people must complete a shared and valued action by depending on each

other for some finite amount of time. (Salas et al., 1992). Team members must not only

interact effectively with each other, but also adjust their coordination in order to meet the

demand in environmental dynamics to which the team is subjected. In contrast with the

traditional approach, recent studies focus on both the dynamics of team member

interaction and environmental dynamics to which the team is subjected in order to

measure team performance via team coordination (Gorman et al., 2010; Gorman, Cooke,

Amazeen, & Fouse, 2012; Cooke et al., 2013). In this dynamic approach, real-time

28

coordination processes and their effects on individual and team performance have been

taken into account (Gorman, 2014). In accordance with this approach, there are two

components of team coordination: first, the dynamics of team member interaction and,

second, the types of environmental dynamics that the human and autonomy team

members are part of (Gorman, Amazeen, & Cooke, 2010). Accordingly, in teams, there

are internal (ongoing interactions among the team members) and external (environmental

and task conditions) constraints. An example of the former constraint is the team-

coordination processes that narrow an individual’s range of thoughts and actions (as

related to team performance). Also, in this approach, real-time awareness underlies team

coordination. Hence, from the dynamic perspective, it is real-time coordination

processes—ones that occur between individuals, not within—that form the causal basis of

team coordination (Gorman, 2014; Gorman et al., 2010).

In this dissertation, coordination at the team level and its evolvement under

different conditions and environmental dynamics is the main consideration. The detailed

analyses—for which Nonlinear Dynamical Systems (NDS) methods are used in terms of

observing how team coordination develops under dynamic environments—apply not only

to team coordination, but also team communication.

29

Summary. In the highly complex and dynamic task environment (e.g.,

Command-and-Control (C2) setting), the ITC perspective is the natural choice (i.e., task

dependency). In such an environment, teams must interact effectively under routine

conditions, but also have the ability to change their coordination patterns under novel

conditions. Teams must demonstrate somewhat stable and flexible coordination behavior

to achieve team-level goals under routine and novel task conditions. In addition to that, in

the HAT context, the autonomous agent has limited knowledge (due to limited

declarative memory) and so the static, knowledge-based approach is not applicable. In

contrast, ITC can succeed in this context because it is simultaneously informed and tested

by computational modeling: because ITC posits that teammate interaction is team

cognition, team-level models of cognitive activity should succeed insofar as they focus

on interactions. Recent work in modeling has pinpointed these interaction processes (e.g.,

language comprehension and production; situation assessment) as those most needed to

create a computational cognitive model that can stand as a fully-fledged teammate.

Nonlinear Dynamical Systems Approach

‘Dynamical’ phenomena refer to “unfolding of events in a continuing

evolutionary process” (Luenberger, 1979), whereas ‘coordination’ refers to different

kinds of and degrees of functional disordering among interacting parts and processes in

space and time (Kelso, 2003). Dynamics of coordination explain and predict how patterns

of coordination form, adapt, recover, persist, and change in physical, biological, and

social systems (Kelso, 2004). These physical, biological, and social systems contain

many heterogeneous elements which interact nonlinearly with each other and their

30

surrounding environment over time. The constituent coordinating parts that make up

these dynamical collectives can create patterns of behavior, but also adapt to those very

creations over time. Therefore, the main goal of coordination dynamics is to elaborate the

details of interaction patterns among parts, processes, and even the elements themselves

in space and time (Kelso, 2004). These dynamical systems are also nonlinear which

means that system output is not proportional to input.

Stability of nonlinear dynamical systems. Each dynamical system exists in a

state space which is an abstract construct depicting the range of behavior open to the

system and is constrained by the degrees of freedom available to the elemental

components of the system (Thelen & Smith, 1996). Where a system is located in the state

space is often determined through self-organization, which occurs when a system is

attracted to a preferred state of being out of many potential states (i.e., the system prefers

certain conditions and behaviors). These preferred states are called “attractors”, which are

determined by the interactions between the system’s components and how sensitive it is

to external conditions (Kelso, 2003). The presence of those attractors is related to both

the predictability of the system’s behavior and its response to perturbations.

Stronger attractors have more stable, but less flexible behavior. In this

dissertation, stability of team coordination is the maintenance of the team interaction

patterns over time. This particular interactions can change over time based on the degree

of variability (which is a transition process) in the coordination dynamics. Variability is

needed for changes in coordination and to achieve the right balance of stability and

flexibility of movement (Hamill, van Emmerik, Heiderscheit, & Li, 1999). Therefore, the

31

stronger attractors’ behavior has less variability, and their adaptation can be problematic

during unexpected changes in the environment. For instance, if a high-tech company is

stable and predictable, it is very difficult for it to be innovate without flexibility (which is

a dynamic and sustainable way to stay ahead) in an uncertain and rapidly changing

market environment (Reeves & Deimler, 2011). Another example of the benefits of

adaptability comes from the 2008 Summer Olympics where U.S. swimmer Michael

Phelps lost his vision (due to water in his goggles) during the race and had to adapt by

relying on his breath counts to know when to reach for the wall (Crouse, 2008). In this

situation, his motor behavior remained stable (i.e. he continued swimming) yet was

flexible in response to the loss of vision. Interestingly, Michael Phelps’s performance in

the race set a new world record in the 200 m butterfly event. Stability is also associated

with the system’s predictability. For instance, in a dynamic task environment (e.g.,

natural disaster), an emergency response team’s stability may be lost too quickly because

random perturbations corrupt the team’s dynamics over time, and therefore predictability

of the system’s behavior is lost.

The stability of the attractor can be tested by perturbing the system to see how the

system responds. If the attractor is weak, then perturbation will have stronger effect, and

the system will be disrupted more and may move toward a new attractor, but if the

attractor is stronger, then the effect of perturbation will be weak and the system will

return to its present attractor quickly. Stability in dynamical systems, then, can be

defined as a kind of robustness to perturbations (Nordham & Kelso, 2016). In this

perspective, stability is desirable, to some degree, but systems must also remain flexible

32

enough to adapt to perturbations. That is, if there is a more desirable attractor, then they

should be able to switch to that new attractor easily. If they are too stable, then they

cannot respond to changes.

When considering teams (the systems under study in this dissertation), a particular

coordination level may serve as a stable attractor for a short time in a dynamic

environment, but this attractor may change when perturbations occur. The resulting phase

shift might be observed as the team coordinates in different ways with different degrees

of stability in coordination (contrasted with their “periodic state”: the state of

coordination under initial conditions). If the new phase is not a stable solution, then the

team may come back to its periodic state. In this case, teams can demonstrate different

patterns of coordination (with different degrees of stability) to adapt and respond to

changes in the task environment. Thus, there exist a wide range of stable patterns of

coordination that emerge in order to respond to perturbations and facilitate successful

task completion (Losada, 1999; Losada & Heaphy, 2004).

Team coordination parameters (order and control parameters). In order to

capture the interaction patterns among the system components, order and control

parameters are used (Kelso & Engstrom, 2006). Haken’s synergetics (Haken, 1978),

which study self-organized (i.e., spontaneous) formation of structures in systems, focuses

on the order parameters to explain movement of the coordination which happens among

the individual components of the systems (Haken, 1989).

The point at which self-organizing systems’ behavior changes qualitatively is

called the “instability point”, and the behavior of the system is governed by order

33

parameters which describe small amounts of variation (i.e., explains behaviors of

individual parts) in the system (Haken, 2003). Therefore, the order parameter is a

collective variable (which is a geometric or mathematical representation of complex

behavior among the system components, e.g., team coordination). On the other hand, as

the control parameter is consciously varied, a threshold value is reached that causes the

coordinated behavior of the system to change in a qualitative way (Kelso, 1997; Kelso &

Engstrom, 2006). In this way, the control parameter can be used as a scaled catalyst to

change the system behavior, i.e., control parameters can be used to identify the dynamics

of the order parameter.

With that in mind, Gorman et al. (2010) developed an order parameter, called the

Kappa score (к), for team coordination dynamics (for three heterogeneous UAV team)

that captures the interactive quality of coordination. Consistent with the team cognition

perspective, this Kappa score resides at the level of the team rather than the level of the

individual; as viewed from the team mental model approach. This Kappa score (order

parameter) reflects the systems’ state changes—due to changing the control parameter—

and captures the current state of team coordination as it fluctuates over time (Gorman, et

al., 2010). This dissertation presents an examination of the order parameter—Kappa

score, and control parameter— team configuration (three participants; two participants-

one confederate team member; two participants-one synthetic team member) during

performance in the task.

34

Measuring team coordination dynamics. In previous research, team

coordination dynamics were explained by applying Nonlinear Dynamical Methods

(NDM) in the UAV environment (Gorman, et al., 2010; Gorman et al., 2012). Gorman

and colleagues (2010) explain that “systems with different material substrates can exhibit

the same dynamics”, this is known as dynamical similitude, which can be used to “guide

selection of appropriate Nonlinear Dynamical Systems (NDS) methods for a system that

has not been previously analyzed using a dynamical approach” (Gorman et al., 2010, p.

285).

In their study Gorman et al. (2010) observed how team coordination evolved

under different conditions and environmental dynamics. This evolution in coordination

was viewed as a continuous and effortful process instead of fixed and passive. Using

NDS, differences in team coordination were able to be detected instead of lost using

typical static measures like mean behavior (Gorman et al., 2010, p. 267). They were also

able to detect, in real-time, when discrepancies emerged between team and environmental

dynamics (Gorman et al., 2010, p. 286). Their analysis was comprised of two

experimental sessions in which they focused on human-human interaction (within 3

member teams) in a UAV synthetic task environment by implementing real-time NDS

methods, including analyses of Lyapunov and Hurst exponents (Gorman et al., 2010).

Their findings showed that NDS methods exposed the differences between same/intact

(i.e., those who worked together longer) and different/mixed (i.e., those who had a

shorter working history) teams: mixed teams were more stable, exhibited more controlled

task exploration, and performed better at the task than intact teams. In a follow-up study

35

using the same experimental paradigm, Gorman et al. (2012) applied discrete Recurrence

Quantification Analysis (RQA) on team communication data as an additional

measurement technique for coordination dynamics (e.g. % determinism measure). Their

findings indicate that intact teams demonstrated more stable communication behavior

than the moderately flexible behavior demonstrated by the mixed teams.

Overall, the findings from these studies indicate that a dynamical environment

requires that teams be flexible, adaptive, and responsive to novel change while

performing reliably and consistently if they hope to successful in the task. It also requires

stability, regularity and predictability so that teams can understand and trust the settings

of the novel conditions and, ultimately, overcome potential roadblocks (Farjoun, 2010;

Losada, 1999; Losada & Heaphy, 2004).

Summary. Within the coordination dynamics viewpoint, teams self-organize

from the interactions among the team members and the task environment in order to

achieve the common goal(s). Teams must interact effectively under routine conditions,

as well as under novel conditions in a dynamic task environment. However, adding a

synthetic agent as a team member may lead to different coordination patterns with

different characteristics in routine and novel conditions when compared with all-human

teams. To capture characteristics of overall system behavior, NDM can provide more

informative measures of team coordination than linear measures alone. Thus, one focus

of this dissertation is the application of NDM to coordination across human-human teams

and HATs.

36

CHAPTER 3

EXPERIMENT OVERVIEW AND HYPOTHESES

Task and Roles

In the current study, three heterogeneous and interdependent team members were

required to photograph critical target waypoints. The three roles included the 1) Data

Exploitation, Mission Planning and Communications (DEMPC/ the navigator), who

provides a dynamic flight plan with information regarding speed and altitude restrictions

of the waypoints to other team members; 2) the Air Vehicle Operator (AVO/ pilot), who

controls the Unmanned Air Vehicle (UAV) by controlling its fuel and adjusting altitude

and airspeed; and 3) the Payload Operator (PLO or photographer/sensor operator), who

takes photos of each target waypoint and adjusts camera settings (Cooke & Shope, 2004).

Therefore, the basic goal of the teams is to take photographs of ground targets during

simulated reconnaissance missions by interacting with each other through a text-based

communications system. Teams must also maintain all UAV flight and sensor systems

and observe any restrictions associated with particular waypoints. The task was carried

out over a series of five missions (each 40 minutes in length; Cooke & Shope, 2004).

In the current study, based on manipulating the pilot position, there were three

conditions (each composed of ten teams): 1) the pilot was a synthetic teammate (i.e., a

cognitively plausible ACT-R based computational model) in the synthetic condition, 2)

the pilot was a randomly assigned human participant in the control condition, and 3) the

pilot was one of the experimenters who was an expert at the pilot’s task (and used a role

specific coordination script which is depicted in Appendix A) in the experimenter

condition. Therefore, the experimenter and synthetic teammate conditions, two

37

participants per team were recruited for the navigator, and the photographer roles. The

role of pilot played by either a single trained confederate (experimenter condition) or

synthetic teammate (synthetic teammate condition). In both conditions, participants were

randomly assigned to a team and one of the roles (navigator and photographer). In the

control condition, each role was randomly assigned to one of three participants.

For all three conditions, team members communicated with each other using a

text-based interface. In the synthetic condition, because of the synthetic teammate’s

limited language capability, the navigator and the photographer needed to ensure that all

messages to the synthetic teammate were neither ambiguous nor cryptic because,

otherwise, the synthetic teammate would not understand the text-message (Demir et al.,

2015). The communication in the experimenter condition was more structured because

the expert pilot focused on structured coordination. That is, the confederate pilot by

virtue of a script asked questions when needed of the navigator and the photographer to

promote adaptive passing of information about the critical waypoints in a timely manner

(Demir, McNeese, Cooke, & Myers, 2016).

Hypotheses

This dissertation addresses the following research questions and corresponding

hypotheses for team performance and team coordination measures, each associated with

corresponding rationale.

(1) Do team coordination dynamics significantly differ across the three

conditions (synthetic versus control versus experimenter)?

38

The experimenter teams will demonstrate moderately stable coordination

dynamics because the confederate pilot (experimenter team member) used an ‘if - then’

script and acted like a synthetic teammate (but better, in terms of constructive and timely

interacting with other team members). Based on moderately stable dynamics, the

experimenter teams will also adapt to the dynamic task environment in a timely manner

(especially during perturbation), because team’s coordination behavior with some amount

of variability will help to stabilize into any patterns of control. Thus, it was hypothesized

that:

H1: the experimenter teams will have moderately stable coordination dynamics.

On the other hand, the synthetic teammate was an ACT-R based computational

model with five components. Being a programmed model, the other human teammates

were trained on how to interact with the synthetic teammate: they were given a

communication script telling them how their messages should be structured and they also

learned how the synthetic teammate interacted over time. Therefore, the team members

learned about the synthetic teammate’s limited communication capability. Taking into

account these limitations and the fact that the synthetic teammate is programmed model,

the following hypothesis is predicted that:

H2: the synthetic teams will demonstrate extremely stable team coordination

dynamics compared to the other teams (i.e., experimenter and control teams).

Finally, the control teams had a pilot who is a randomly assigned participant.

Without knowledge of the proper structure for communication (i.e., a modelling behavior

which the synthetic pilot follows, or the timely pushing and pulling of information that

39

the experimenter pilot can do), pilots in this condition will communicate more erratically.

Therefore:

H3: teams in control condition will demonstrate less stable team coordination

dynamics during the task.

(2) how are team coordination dynamics related to team performance and team

situation awareness?

The experimenter teams should possess moderately stable team coordination and

so will be able to adapt their coordination to the task environment, resulting in good

performance. The synthetic teams will have extremely stable team coordination and fail

to adapt their coordination to the dynamic task environment resulting in poor

performance. On the other hand, the control teams will have less stable team coordination

and so will be able to adapt their coordination to the task environment, resulting in better

team performance than the synthetic teams and, in some cases, performance as good as

the experimenter teams (depicted by the overlapping region in the bell-shaped curve in

Figure 2). However, the control teams’ lack of stability will result, generally, in poorer

team performance than the experimenter teams, because too much variability is unable to

stabilize the team in any pattern of control. Thus, the following general hypothesis were

proposed, and the proposed relationship between team stability and team performance is

depicted in Figure 2:

H4: the experimenter teams will be more adaptive to the task environment

resulting in better team performance than the other teams, due to balanced team

coordination (moderately stable)

40

H5: the synthetic teams will be less adaptive to the task environment resulting in

poorer performance than the control and the experimenter teams, due to

extremely stable team coordination

H6: the control teams will be overly adaptive to the task environment resulting in

performance that is generally better than the synthetic teams, but generally worse

than experimenter teams (due to less stable coordination dynamics)

Figure 2. Proposed Relationship between Team Stability and Team Performance

41

CHAPTER 4

METHODOLOGY

Participants

Overall, 30 teams (ten in each of the three conditions) completed the experiment.

Within those 30 teams, 70 participants (60 males and 10 females, Mage=23.7, SDage= 3.3)

were recruited from Arizona State University and surrounding areas, each completing a

seven-hour experiment as a team. Number of participants differed by condition: teams in

the control condition had three randomly assigned participants whereas teams in the

synthetic and experimenter conditions had two randomly assigned participants for the

navigator and photographer roles; this is because the pilot position in both of these

conditions is filled by a non-participant—the synthetic teammate in the synthetic

condition and a trained confederate in the experimenter condition. Having normal or

corrected-to-normal vision and being fluent in English were the main requirements for

participation, with which each participant being compensated $10 per hour.

Materials and Apparatus

CERTT-II test bed. The experiment was conducted in the context of the

Cognitive Engineering Research on Team Tasks Laboratory, Unmanned Aerial Vehicle -

Synthetic Task Environment (CERTT UAV-STE). The CERTT UAV-STE simulates the

teamwork aspects of ground control of a Predator UAV.

The team's task in this simulated environment is to detect and photograph target

waypoints in a timely manner while maintaining all UAV flight and sensor systems and

observing any restrictions associated with particular waypoints (Cooke & Shope, 2004).

In this experiment, the CERTT-UAV-STE software was embedded in an updated version

42

of the hardware infrastructure (CERTT-II). In order to support this experiment, CERTT-

II provides new hardware infrastructure which has varied features: (1) text chat capability

for communications between team members; and (2) eight new hardware consoles: four

consoles for up to four team members and another four consoles for two experimenters

who oversee the simulation, inject roadblocks, and make observations.

The “texting experimenter” console has two embedded computers through which

the experimenter can give ratings (for taskwork relatedness, behavior, and situational

awareness) and send text messages via chat window to the other three roles (AVO,

DEMPC, and PLO). This console can also turn on and off all the computers and software

of the other consoles via a master control window. The other console, called the “non-

texting experimenter” console, enables the experimenter to give ratings (for coordination,

taskwork relatedness, and behavior) and can also follow the participants using a camera.

Other materials. In addition to the consoles, a consent form was given to the

participants for them to sign. After that, participants received a briefing, and then a 30-

minute interactive PowerPoint training. For the hands-on training, and the task,

participants practiced interacting with the synthetic teammate using the example cheat

sheets to guide their communication with the synthetic teammate. Teamwork knowledge,

taskwork knowledge and workload measures were collected in two sessions (beginning

and at the end of the experiment).

Finally, at the end of the experiment, demographic and debriefing questions

(respectively) were given to the participants. Additionally, a “cheat sheet” and

supplemental materials for each role were displayed at the corresponding workstations.

43

Supplemental materials include role-specific rule summaries, screenshots of each

station’s displays, waypoint list for the DEMPC, and camera setting list and photo folder

comparisons of good and bad photos for the PLO). Also, the PLO and the DEMPC had a

communication cheat sheet which shows examples of how to communicate with the

synthetic teammate. Experimenters have paper checklists for set-up and data were

archived via an external hard drive.

Procedure

Overall, the procedure of the experiment can be considered in two categories:

first, the design of the UAV missions in terms of role and target and, second, the overall

experimental sessions which each participant went through.

UAV missions. In the CERT-II UAV STE task, there are three roles for team

members: pilot/ AVO, navigator/ DEMPC, and photographer/ PLO. The AVO pilots the

UAV and controls settings and systems directly related to controlling the flight path of

the UAV. The AVO flies the UAV along waypoints in accordance with a mission route

developed by the DEMPC. The DEMPC navigates and has access to data and

intelligence (i.e., an experimenter) regarding the location of and any restrictions on

critical waypoints. Using this information, the DEMPC plans a sequence of waypoints

called the mission route.

The PLO’s main objective is to take good photos of the critical waypoints (which

are pre-designated as targets). To achieve this, the PLO must monitor and control the

systems and settings relevant to the UAV sensor equipment (i.e., infrared, synthetic

aperture radar, and electro-optical photography equipment). In addition, the PLO must

44

work with the AVO to coordinate camera settings with UAV location flight settings to

ensure that the UAV is within a certain radius (e.g., five miles) of a target to take a good

photo. In all three conditions, every team was subdivided into two groups: the AVO was

isolated in one room and the DEMPC and the PLO in another. The DEMPC and the PLO

were seated in locations separated by partitions such that they did not have face-to-face

contact.

During the eight-hour experimental session each team completed five missions

with Missions 1 through 4 of comparable workload and Mission 5 of high workload (see

Table 4). Teams had to photograph 11 target waypoints for Mission 1 and Mission 3, 12

targets for Mission 2, 13 targets for Mission 4, and 20 targets for Mission 5. Target

waypoints fell into areas called Restricted Operating Zones (ROZ boxes). Mission 1 to

Mission 4 contained five ROZ boxes, whereas ROZ boxes for Mission 5 were seven.

These missions had restrictions on airspeed, altitude, and the ROZ boxes (flying within

an effective radius for entering the operating zone).

During each mission, team communication was observed by a texting and a non-

texting experimenter who checked appropriate boxes on the coordination logger in real

time. After visiting each target waypoint, both experimenters independently rated the

team’s process behaviors (i.e., adequacy of team member behaviors such as

communication) on a scale of one to five, five being the best.

45

Experimental session. When participants arrived, they read and signed an

informed consent document and were randomly assigned to one of the two team member

roles (DEMPC or PLO) in the experimenter and synthetic teammate conditions, and in

the control conditions three team members were assigned one of three roles, AVO,

DEMPC and PLO. After that, participants were received a 30-minute role-specific skills

training using an interactive PowerPoint training program which includes an audio

section. After the interactive training session, a 30-minute, hands-on practice training was

started. During the practice session, the participants were required to perform specific

tasks and the experimenters checked off participants’ demonstration of certain skills

using a basic skills checklist. After the participants had reached criterion levels of

performance on their individual tasks, they started the first mission.

After completing the first mission, a knowledge session was conducted in order to

test participants for teamwork and taskwork knowledge relevant to the UAV STE both

individually and as a team. After that, the NASA Task Load Index (TLX), which is a

subjective workload assessment (Hart & Staveland, 1988), was administered to measure

six workload components, including mental, physical, temporal demand, performance,

effort, and frustration. The participants then finished four more missions, received a

second round of knowledge and NASA TLX measures, and later on had their

demographic data (e.g., age, education, etc.) collected. Lastly, the participants debriefed

about the experiment.

46

Table 5

Experimental Session

Manipulation

For the experimenter condition, due to their familiarity with all of the roles in the

experiment, one of the experimenters served as the AVO. The experimenter focused only

on coordinating the team members, providing more structured coordination (i.e., the

AVO asked questions to other team members to ensure timely and adaptive passing of

information at target waypoints). For this purpose, the AVO used a coordination script

which indicates when some of specific information needs to be pulled and pushed (see

Appendix A). Therefore, it was assumed that this condition would have the highest

quality coordination of the three conditions.

For the AVO role in the synthetic teammate condition, there was a synthetic

teammate, which is a cognitively plausible ACT-R (Adaptive Control of Thought—

Rational) based computational model/ cognitive architecture (Anderson, 2007). The

synthetic agent had limited knowledge and expertise as compared to the experimenter. In

Sessions Procedure

1 Welcoming Consent forms.

2 Interactive Training Interactive Training PowerPoint Slides

3 Training Mission Hands on Training

4 Mission 1 Mission 1 is conducted: Total 11 targets

5 NASA TLX/ Knowledge

Measures

Session 1: Conducting taskwork and teamwork questions, and

administering the workload questions

6 Mission 2 Mission 2 is conducted: Total 12 targets

7 Mission 3 Mission 3 is conducted: Total 11 targets

8 Mission 4 Mission 4 is conducted: Total 13 targets

9 Mission 5 Mission 5 is conducted: Total 20 targets

11 NASA TLX/ Knowledge

Measures

Session 2: Conducting taskwork and teamwork questions, and

administering the workload questions

12 Demographics/ Debriefing Conducting demographic questions, and giving debriefing

13 Post Checklist

47

the control condition, the AVO was a human participant just like the other participants

(PLO and DEMPC) and without any expertise. The basic goal of the teams is to take

photographs of ground targets during simulated reconnaissance missions by interacting

with each other through a text-based communications system.

Measures

In this experiment, several measures which were also used in the previous

experiments (Cooke, Gorman, Pedersen, et al., 2007) were collected, including: team

performance measures at the mission and target levels; team process measures which

consist of process ratings, communication flow, team verbal behaviors, situation

awareness, and coordination based on the passing of information in a timely and adaptive

manner; team knowledge measures based on taskwork and teamwork knowledge; and

workload as measured with NASA Task Load Index (TLX). To test the hypotheses under

consideration in this dissertation, only a subset of the measures are examined including

team performance at the mission and target levels, situation awareness, communication

flow, and coordination measures based on passing information in a timely and adaptive

manner.

48

Team performance. A team performance score (mission level scores) was

collected for each of the five missions. The team performance score is a composite score

composed of a set of mission variables, including the time each individual spent in alarm

and warning states, the rate with which critical waypoints are acquired, and the rate with

which targets are successfully photographed (see Table 5). Each variable has associated

penalty points—weighted (in advance) to correspond with task importance—that were

deducted from the maximum score of 1000 (see Appendix B for definition of each

variable). In terms of the workload differences of each mission, no penalty was incurred

for teams photographing a smaller proportion of targets in high workload missions.

Table 6

Team Performance Score

Team Performance Score Variables Formula

Alarm Penalty: Team Alarm Duration (TAD), Mission Total

Seconds (MTS), Weight (W=393.22) (

𝑇𝐴𝐷

𝑀𝑇𝑆)

1/2

∗ 𝑊

Warning Penalty: Team Warning Duration (TWD), Mission Total

Seconds (MTS), Weight (W=112.02) (

𝑇𝑊𝐷

𝑀𝑇𝑆)

1/2

∗ 𝑊

Missed or Slow Critical Waypoints Penalty: Critical Reached

Waypoint (CRW), Mission Total Seconds (MTS), Weight

(W=318.63)

[1 − (𝐶𝑅𝑊 ∗ 60

𝑀𝑇𝑆)] ∗ 𝑊

Missed or Slow Photos Penalty: Total Good Unique Photos

(TGUP), Mission Total Seconds (MTS), Weight (W=314.96) [1 − (

𝑇𝐺𝑈𝑃 ∗ 60

𝑀𝑇𝑆)] ∗ 𝑊

Note. Adapted from “Acquisition and Retention of Team Coordination in Command-and-

Control” by N.J. Cooke, J. Gorman, H. Pedersen, J. Winner, J. Duran, A. Taylor, L.

Rowe, 2007, p. 171, Cognitive Engineering Research Institute, Mesa, Arizona.

To increase measurement sensitivity, target processing efficiency (tp) for target-

level processing also considered in this study (refined scores). These scores were

49

calculated based on 1000 points possible at each target less the number of seconds spent

in the target radius (r) and minus 200 penalty points for number of missed photos (p).

Therefore, the following formula was used for each target:

tp = 1000 – r – (p*200) (1)

Team process. Three process measures were considered in this study, including

team communication flow, team coordination, and team situation awareness.

Team communication flow: during each 40-minute mission, the chat logger

recorded each of the messages with time stamps, including, message sent time, and

message read time by receiver. In this dissertation, only the message sent time stamp

(because the reading had missing data for two teams) was used to see how team

communication patterns change across teams and time.

Team coordination comprises three key communication events at each target

waypoint, which happen in the following optimal coordination sequence (depicted in

Figure 3): (1) the Information (I) is given by the DEMPC to the AVO about the target

waypoints of the mission (i.e., altitude, speed restrictions, and effective radius), (2) the

Negotiation (N) occurs between the AVO and the PLO regarding camera settings,

airspeed, and altitude at the target, and (3) the Feedback (F) is given by the PLO to the

AVO and DEMPC about whether the photograph taken at the target is acceptable.

50

Figure 3. Team coordination sequence: Information-Negotiation-Feedback

Note. Adapted from Demir, M, McNeese, N. J., Cooke, N. J., Ball, J. T., Myers, C., &

Freiman, M. (2015). Synthetic Teammate Communication and Coordination with

Humans. Proceedings of the Human Factors and Ergonomics Society Annual Meeting,

59(1), 951–955.

As team members interact, the variability in their repeated use of this optimal

information sequence (specifically, the relationship between the timing of the three parts

of the sequence, INF) was used to compute a coordination score for each team at each

target waypoint. In this procedure, the team coordination logger—a custom-developed

software tool that is depicted in Figure 4—was used in order to record the timing of

coordination events. For each target, the non-texting experimenter will observe and

record every behavior of each role in the coordination logger.

t(I) = Information sent time

t(N) = Negotiation sent time

t(F) = Feedback sent time

51

Figure 4. Coordination logger software tool

Assuming that INF are the principal axes of the procedural model, a geometry-

based measure of coordination was created (Gorman et al., 2010). These axes are related

by a variable, Kappa (𝜅) in Figure 5, computed by normalizing the area around feedback

at every target in order to develop a distribution over the intrinsic procedural model

geometry. 𝜅 has some interesting properties. First, it has no units, because all three

constituent parts are measured in seconds and these units will cancel in the relation 𝜅.

Second, it contains two qualitatively different states: uncoordinated (𝜅 < 1) and

coordinated (𝜅 > 1) with a transition point at (𝜅 = 1) that differentiates the two

(theoretically, it could take on an infinite number of values, but these are the two states

that have a practical use). In uncoordinated teams, either ‘N’ precedes ‘I,’ or ‘F’ precedes

either ‘I’ or ‘N’ or both. When ‘N’ occurs before ‘I’, this indicates that a “backlog” of

information has formed. On the other hand, in coordinated teams, I, N, and F occur in

accordance with the procedural model, with larger values of 𝜅 indicating that the ‘I’

component is established well in advance of target approach.

52

Figure 5. The intrinsic geometry coordination score.

Note. Adapted from “Acquisition and Retention of Team Coordination in Command-and-

Control” by N.J. Cooke, J. Gorman, H. Pedersen, J. Winner, J. Duran, A. Taylor, L.

Rowe, 2007, p. 86, Cognitive Engineering Research Institute, Mesa, Arizona.

Team Situation Awareness (TSA) was measured via Coordinated Awareness of

Situation by Teams (CAST, developed in the CERTT lab; Gorman et al., 2005), and the

texting experimenter scored situation awareness through the situation awareness logger

which is depicted in Figure 6. TSA roadblocks were driven by ad hoc target waypoints

that take place within the scenario at set times within each mission. During each mission,

teams were sometimes presented with “roadblocks”, such as the introduction of a new

target waypoint. In this dissertation, completion of the roadblocks was used as a measure

of team situation awareness. The triggering mechanism for these roadblocks is based on

each team’s position relative to the waypoints in the mission; as such, the number of

waypoints triggered in each mission may vary from team to team (Jamie C. Gorman,

Cooke, & Amazeen, 2010).

53

Figure 6. Situation awareness window

54

CHAPTER 5

DATA ANALYSIS AND RESULTS

There are several types of dynamical analyses which were used in the previous

team studies (Gorman et al., 2010, Gorman et al., 2012, Gorman, Hessler, Amazeen,

Cooke, & Shope, 2012, Knight et al., 2016). In this dissertation, four different dynamical

analyses were applied to understand different characteristics of team coordination

dynamics: (1) attractor reconstruction to visualize the dynamics of the teams by using a

Kappa (к) time series at the team level, (2) stability analysis (Lyapunov exponents) for

each coordination order parameter Kappa (к) at the team level, (3) surrogate analysis to

address whether the observed dynamics were an artifact of the short к time series or

occurred randomly, and (4) Joint Recurrence Quantification Analysis (JRQA) to extract

the % Determinism (DET) measure based on team communication flow (message sent

time per minute per role) at the mission level. In this dissertation, the programming

software Matlab was used to conduct the largest Lyapunov exponent calculation. JRQA

was carried out in R version 3.2.3 (“R: The R Project for Statistical Computing,” 2015)

using the “crqa” package (Coco, Dale, Dixon, & Nash, 2015) for calculating DET

measure.

Attractor Reconstruction

Methods of attractor reconstruction were used to visualize and measure team

coordination dynamics in this task. Based on Taken's (1981) theorem, one can recover a

system’s dynamical structure (i.e. reconstruct the attractor) from only a one-dimensional

signal (in this study, this signal is the Kappa time series) and time-delayed versions of the

55

original signal. From this method, the reconstructed attractor preserves the invariant

features and dynamical structure of the original signal in the newly reconstructed phase

space. This allows the system to be evaluated in the appropriate number of dimensions in

which the behavior resides (Shockley, Santana, & Fowler, 2003).

In this dissertation, the attractor was reconstructed for each team’s Kappa time

series κ(i) via estimating two embedding parameters: the appropriate time delay (𝜏), and

the appropriate number of embedding dimensions (m) (Rosenstein, Collins, & De Luca,

1993; Abarbanel, 1996). Time delay (𝜏) parameter indicates that the original signal is

maximally different from the lagged versions. These lagged versions will then be used as

the dimensions in the phase space that the signal is unfolded onto. Therefore, we obtained

a later view of the system κ(𝑖 + 𝜏1), and another later view κ(𝑖 + 𝜏2) which are

dynamically different than the original observed Kappa series κ(i). The τ parameter was

estimated using the first local minimum of the Average Mutual Information (AMI)

function (Abarbanel, 1981; Rosenstein et al., 1993). The estimated τ is then used to

determine the appropriate m to unfold the original (and lagged) vectors onto. The

selection of m followed the “False Nearest Neighbors (FNN)” method outlined by

Rosenstein et al. (1993). This method measures the percentage of close neighboring

points in a given dimension that remain neighbors in the next highest dimension. The

selection of m must be large enough to minimize the percentage of FNN. The generally

accepted criterion for FNN to be sufficiently minimized is when the percentage of false-

nearest neighbors is below 10% (Rosenstein et al., 1993). With the appropriate m

56

determined, we can now reconstruct the phase space and observe the dynamics of the

system.

After estimating the time lag (𝜏) and embedding dimension (m) from each team’s

к series, Multivariate Analysis of Variance (MANOVA) was conducted to see how time

lag (M = 1.80, SD = 1.24) and embedding dimension (M = 2.83, SD = 0.87) differ across

the conditions (i.e., synthetic, control, and experimenter). Box’s M test (4.57) shows that

there was no significant differences between the covariance matrices (p = .27).

Therefore, the assumption of homogeneity of covariance was not violated, and Wilk’s Λ

is an appropriate test to use. According to Levene’s test, the assumption of homogeneity

of variances was not violated for both of the dependent variables (for τ: F(2, 27) = 2.31, p

= .12; for m: F(2, 27)= .24, p = .79). The findings indicate that there were no significant

differences across the conditions, F(4, 52)= .79, p = .53, Wilk’s Λ = .88.

Thus, these two parameters (τ and m) are sufficient to reconstruct the attractor

across different teams within the same task. That is, these non-significant results shows

the reliability of the used methods in attractor reconstruction for different teams

performing in the same task.

Stability Analysis

We also evaluated team coordination stability via estimating the largest Lyapunov

exponent from the reconstructed attractors. The Lyapunov exponent (λ) measures the

exponential rate of divergence of two nearby trajectories on the attractor (Rosenstein,

Collins, & De Luca, 1993; Wolf, Swift, Swinney, & Vastano, 1985): stability (λ1 < 0),

57

instability (λ1 > 0), and parallel trajectories (λ1 ≈ 0) of team coordination (Gorman et al.,

2010; Kantz & Schreiber, 2004).

For instance, for some of the high performing teams in each condition, the time

evolution trajectories as captured by attractor reconstruction are visualized in Figure 7,

and there is a clear difference between the teams in terms of coordination dynamics. The

synthetic team’s coordination was focused on a small part of phase space with less

variability and a more stable appearance (λsyn=-0.04). On the other hand, the control

(λcont= 0.02) and experimenter (λexp= 0.05) teams demonstrated more variability and less

stability in Figure 7.

Figure 7. Example Reconstructed attractors from three UAV Teams: a three-dimensional

phase space as coordinates for the three-dimensional space [к(𝑖), к(i+τ), к(i+2τ)]. These

time delays are determined by examining the average mutual information curve.

Synthetic

(λ = - 0.04)

Control

(λ = 0.02)

Experimenter

(λ = 0.05)

58

Mean Largest Lyapunov Exponents (λ) across teams in each condition (i.e.,

synthetic, control, and experimenter), is depicted in Figure 8. The null hypothesis for

Largest Lyapunov Exponents is λ0 = 0. We applied one-way Analysis of Variance

(ANOVA) to compare the means of conditions on Largest Lyapunov Exponents. The

Leven’s F test indicates that the homogeneity of variance was violated, and therefore,

adjust F ratio from Welch statistics was reported. The results (Largest Lyapunov

Exponents λ) indicate that the difference between the conditions was statistically

significant, F(2, 15.7) = 9.22, MSe = .001, p < .05.

In terms of condition level, pairwise comparisons, synthetic teams showed more

stable coordination dynamics (MSynthetic = -.02, SDSynthetic = .16) than control (MControl= .01,

SDControl = .03, p < .05) and experimenter teams (MExp= .03, SDExp. = .04, p < .05: see

Figure 8). However, the experimenter teams, which performed better, demonstrated less

stable coordination dynamics than the control teams (but not significantly so, p = .15).

Figure 8. Mean the Largest Lyapunov Exponents (λ) Across the Conditions

(Synthetic < Control < Experimenter) (vertical lines indicate Standard Error +SE)

59

Predicting Team Performance and Situation Awareness via the Largest Lyapunov

In order to predict the relationship between Lyapunov exponent and three

outcome measures (team performance score, target processing efficiency, and team

situation awareness), regression analysis was applied to each outcome separately; the

results are summarized in Table 6. All linear based Lyapunov exponents were positively

related with outcome measures, and all quadratic Lyapunov exponents were negatively

related with the outcome measures. Only the Lyapunov exponent and team situation

awareness had a statistically significant relationship: the linear relationship indicates that

every one standard deviation increase in Lyapunov exponent (holding constant other

variables) is related with a 0.77 standard deviation increase in team situation awareness.

However, this coefficient only applies when the Lyapunov exponent is equal to “0”,

because the quadratic relationship indicates that a one unit standard deviation increment

in the Lyapunov exponent (holding constant other variables) was related with a - 0.59

unit decrement in the relationship between the Lyapunov exponent and team situation

awareness.

This finding shows that, for lower values, the Lyapunov exponent had a positive

relationship with Team Situation Awareness, however, as the Lyapunov exponent

increases this relationship becomes negative. This is consistent with our proposed

inverted U-shaped model hypotheses, that is, a moderate amount of stability is associated

with the highest performance.

60

Table 7

Summary of Regression Analyses for Lyapunov Exponent Predicting Team Performance,

Team Situation Awareness, and Target Processing Efficiency

Relationship

between: B SE B β t

p

value

%

Variance

λ and Team

Performance

Linear (λ) 1209.00 800.00 0.44 1.51 0.142 0.079

Quadratic (λ2) -16480.00 15730.00 -0.30 -1.05 0.304

λ and

Situation

Awareness

Linear (λ) 17.76 6.08 0.77 2.92 0.007 0.240

Quadratic (λ2) -270.59 119.44 -0.59 -2.27 0.032

λ and Target

Processing

Efficiency

Linear (λ) 1170.00 681.90 0.49 1.72 0.097

0.098 Quadratic (λ2) -18100.00 13410.00 -0.39 -1.35 0.188

Note. “B” and “SE B” refer to unstandardized regression coefficient and its Standard

Error, respectively, while “β” refers to standardized regression coefficient.

Surrogate Analysis

Surrogate analysis is a bootstrapping method that represents a more stringent null

hypothesis for presence of significant dynamical structure than comparison to a

theoretical value alone (Theiler, Eubank, Longtin, Galdrikian, & Doyne Farmer, 1992).

Therefore, in order to address whether the observed dynamics were an artifact of a short

time series or occurred randomly, I also applied Surrogate analyses on each κ series

(Delignlères, Deschamps, Legros, & Caillou, 2003; Gorman et al., 2010). First, I

randomly shuffled surrogates of the observed κ series to compare with observed κ series,

and obtained 19 randomly shuffled surrogates for each κ series.

Each surrogate had the same marginal statistical properties (mean, standard

deviation) as its parent κ but was randomly sequenced. The results indicate that the

absolute values of the observed λ1 were significantly different than the surrogate values,

61

t(29) =2.25, p < .05 (two-tailed). This means that the observed dynamics of λ1 were not

artifacts of short or noisy κ.

Joint Recurrence Quantification Analysis (JRQA)

From recurrence quantification to joint recurrence quantification.

Synchronization is “a general process wherein more than one dynamical systems are

coupled or forced (periodically or noisy) in order to realize a collective or synchronous

behavior” (Dang, Palit, Mukherjee, Hoang, & Banerjee, 2016, p. 160). That is, when two

systems are synchronized then their recurrences are dependent on each other. Eckmann,

Kamphorst, & Ruelle (1987) originally introduced Recurrence Plots (RP) in order to

visualize complex system dynamics, i.e., the behavior of trajectories of dynamical

systems in phase space (Blasco & Carmen, 2004; Knight, Kennedy, & McComb, 2016).

After it was shown to be a strong predictor of data that captures key features of nonlinear

systems, recurrence analysis emerged as a sophisticated way to illustrate system

properties. For instance, a recurrence plot can be constructed to represent the dynamics of

a single system across a delaminated period of time; from such a plot, researchers can

develop metrics to represent properties of the system. This extension of the recurrence

plot is called Recurrence Quantification Analysis (RQA).

Later, extensions of RQA were put forward to look at more than one system and

their dynamics: a bivariate version of RQA called Cross Recurrence Plots (CRP) and its

analysis Cross Recurrence Quantification Analysis (CRQA), and, likewise, a multivariate

version of RQA called Joint Recurrence Plots (JRP) and also its analysis Joint

Recurrence Quantification Analysis (JRQA). By comparing the states of two different

62

systems, CRP reveal dependencies between the systems and show the progressions of

two different phase space trajectories. The benefits of CRP are: (1) researchers do not

have to make any assumptions beforehand about data structure and, (2) recurrent

structures between the behaviors of two individuals can be found whether in a single or

multivariate state space (Romero, Fitzpatrick, Schmidt, & Richardson, 2016).

Within the team cognition concept, several studies used RQA ( Fusaroli & Tylén,

2016; Gorman, Cooke, et al., 2012; Knight et al., 2016) and CRQA ( Fusaroli & Tylén,

2016; Strang, Funke, Russell, Dukes, & Middendorf, 2014). For instance, Gorman,

Cooke, et al. (2012) used determinism – extracted from discrete RQA – to assess team

coordination differences between teams in two conditions: one condition had changing

group membership (i.e., mixed teams), the other did not (i.e., intact teams). From this,

they found that intact teams had more determinism (less flexibility) in communication

than mixed teams. On the other hand, Strang et al. (2014) used recurrence rate and

entropy—extracted from CRQA—to assess the relationship between coupled team

members’ physiological and behavioral patterns (i.e., physio-behavioral coupling) and

team characteristics and performance.

However, CRQA cannot be used for the analysis of two physically different time

series, because of two vectors with different physical units or different phases space

dimension. In this case, one can consider the recurrences of the systems’ trajectories in

their respective phase spaces and afterwards find joint recurrences, that is, the times when

both of the systems recur simultaneously, in order to compare different systems (Marwan,

Carmen Romano, Thiel, & Kurths, 2007; Webber & Marwan, 2014).

63

The next form of analysis, then, is a mix of RQA and CRQA: Joint Recurrence

Quantification Analysis (JRQA). In this analysis, first, the recurrences of the systems are

plotted separately, then, the two separate recurrence matrices are combined to find times

of simultaneous recurrence. This analysis is especially useful for assessing

synchronization between interacting systems or assessing the systems that can jointly

influence one another (Knight et al., 2016). Within the team concept, JRQA can be

applied for studying how and why several teams differ from one another in their

dynamics. Thus, it shows the degree to which team members synchronize their activities

during their interaction via text or voice communication.

Applying JRQA on team communication flow. Joint Recurrence Quantification

Analysis (JRQA) was applied on communication flow data (i.e., sent time stamp from

each UAV mission) from which several measures were extracted from each of the

missions, including: recurrence rate, determinism, longest diagonal line, entropy,

laminarity, trapping time, and longest vertical line. From this set of variables extracted

from the JRQA, DET (measure of determinism/ or predictability of the system) was

selected as the focal variable relevant to the hypotheses. DET is defined as the ratio of

recurrence points forming diagonal lines to all recurrent points in the upper triangle

(Marwan et al., 2007).

On Joint Recurrence Plots (JRP), states of the system’s epochs of similar time

evaluation are the diagonal lines (Rizzi, Frigerio, & Iori, 2016). From this definition,

unpredictable (chaotic) processes will have no (or very short) diagonals, while

predictable (deterministic) processes will have longer diagonals and less single, isolated

64

recurrence points (Marwan et al., 2007). Thus, this measure embodies the deterministic

measure of a dynamical system (i.e., the determinism of the system), and is expressed as

the following formula (Marwan et al., 2007):

𝐷𝐸𝑇 = ∑ 𝑙𝑃(𝑙)𝑁

𝑙=𝑙𝑚𝑖𝑛

∑ 𝑙𝑃(𝑙)𝑁𝑙=1

(2)

where l is the diagonal line length considered when its value is ≥ lmin and P(l) is the

probability distribution of line lengths. This rate of determinism for a time series is a

percentage which ranges from 0% (i.e., the time series never repeats) to 100% (i.e., the

time series repeats perfectly). In this study, we will assume that the determinism rate

could take any value from the full range of percentages (Gorman et al., 2012).

In Figure 9, we give three example Joint Recurrence Plots (JRP) for three UAV

teams’ interactions for three conditions (simulated three-code sequences of length 40

minutes) for synthetic (DET = 52%), control (DET= 34%), and experimenter (DET=

52%). Each JRP was calculated based on team communication flow (each team

members’ message sent time), and they are also depicted in Fig 9 (navigator, pilot, and

photographer). In the figure, if any of these roles (navigator, pilot, and photographer)

sent a message in any minute, it was coded one, otherwise zero on y-axis (i.e., discrete

code); if no messages were sent in a given minute, no code was produced.

One team from each condition was randomly chosen for the plots, with each

mission in the figure being the selected teams’ fourth mission. The fourth mission was

purposely chosen, because the first four missions are almost identical in terms of

workload and task (as opposed to the fifth mission, which has a higher workload).

65

Therefore, we are assuming that the fourth mission has a stable communication pattern

after learning about the task over the first three missions. In the JRP, the maximum time

length for x and y axes corresponds to the mission length of 40 minutes. Sixty seconds

was used as the steady state point for quantifications in the analysis.

In each joint recurrence plot, black dots show recurrent points, otherwise the plot

is left white (i.e., none of the three team members spoke at the same time). In this case,

black spots (joint recurrence) occur at the times when all three of them act simultaneously

(either all team members sent a message or all team members were silent). For instance,

when three team members spoke at the same time (see between 10 min - 20 min in Fig 9

for experimenter team), then for the interval between 10 min – 20 min on the x-axis there

is a black dot at the corresponding y-axis interval. If three team members were silent at

the same time (see between 30 min - 35 min in Fig 9 for synthetic team), then for the

interval between 30 min – 35 min on the x-axis there is a black dot at the corresponding

y-axis interval.

In this example figure, these three joint recurrence plots have different

proportions of determinism (DET): 52% for synthetic, 34% for the control, and 47% for

the experimenter teams. In this case, the synthetic team has higher determinism than

other two conditions, and the experimenter team has slightly lower determinism rate than

the synthetic team, but higher than the control. One finding to note is that the synthetic

team’s higher determinism rate is mainly due to times when the tree team members are

silent (e.g., at between 30 to 35 minutes in Figure 9). Also, the navigator in the synthetic

team sent messages less frequently to the other team members compared to the

66

experimenter condition, even though task requirements state that this role needs to

interact with pilot. The control team, which is less deterministic (less predictable/ less

structured), shows more flexible communication pattern than other two teams. This is not

surprising given that the pilot role is assigned to a randomly selected participant (see

Figure 9).

Figure 9. Example Joint Recurrence Plots for Three UAV Teams’ Interactions in Three

Conditions - Length 40 Minutes for: Synthetic (DET: 52%), Control (DET: 34%), and

Experimenter Teams (DET: 47%).

In order to determine whether the conditions differed with respect to their DET

over time, I performed a 3 (condition) x 5 (mission) split-plot ANOVA. Mauchly’s test

indicates that the assumption of sphericity for the repeated measure (mission) was

satisfied, 𝜒2(9) = 13.4, p = .15, 𝜂2 = .82. The ANOVA results indicate that the condition

67

main effect was significant, F(2, 27) = 5.65, MSe = 67.8, p < .05, 𝜂2 = .30. The mission

main effect, F(4, 108) = .15, MSe = 63.2, p = .96, and the condition by mission interaction

effect were not significant, F(8, 108) = .69, MSe = 63.2, p = .70.

The significant condition main effect indicates that the synthetic teams (Msyn=

46.1, SDSyn = 10.1) had higher DET than both control (Mcont= 40.6, SDcont = 7.16, p < .05)

and experimenter teams (Mexp= 43.9, SDexp = 5.70, p = .20). The experimenter teams also

had higher DET than control teams (p = .053: see Figure 10). These findings indicate that

the synthetic teams demonstrated more rigid behavior than the control and the

experimenter, while the experimenter teams demonstrated more rigid than the control

condition but controlled flexibility.

Figure 10. Determinism Across the Conditions (Control < Experimenter < Synthetic)

(vertical lines indicate +SE)

68

Predicting team performance and situation awareness via determinism. In

order to predict the relationship between determinism and three outcome measures (team

performance score, target processing efficiency, and team situation awareness),

regression analysis was applied to each outcome separately. The findings indicate that

none of these relationships were statistically significant (either linear or quadratic).

However, the sign of the coefficients for quadratic term indicates that the increment of

the determinism was negatively related with the outcome measures. The summary of

findings is depicted in Table 7.

Table 8

Summary of Regression Analyses for Determinism Predicting Team Performance, Team

Situation Awareness, and Target Processing Efficiency

Relationship

between: B SE B β t

p

value

%

Variance

DET and Team

Performance

Linear (DET) 158.632 92.175 7.043 1.721 0.096

0.098 Quadratic

(DET2) -1.819 1.056 -7.047 -1.722 0.096

DET and

Situation

Awareness

Linear (DET) 1.074 0.771 5.702 1.392 0.175

0.098 Quadratic

(DET2)

-0.013 0.009 -5.877 -1.435 0.163

DET and Target

Processing

Efficiency

Linear (DET) 144.869 77.418 7.467 1.871 0.072

0.143 Quadratic

(DET2) -1.699 0.887 -7.639 -1.914 0.066

Note. “B” and “SE B” refer to unstandardized regression coefficient and its Standard

Error, respectively, while “β” refers to standardized regression coefficient.

Results Summary

Table 8 demonstrates the related findings (for this dissertation) from this

experiment. In general, the synthetic teams demonstrated extremely stable coordination

69

dynamics (i.e., more rigid) with very little flexibility. On the other hand, the experimenter

teams showed moderately stable team coordination dynamics.

In terms of team situation awareness, the synthetic teams overcame fewer

roadblocks than the other teams. In terms of team performance, synthetic teams

performed comparably to control teams, but less than experimenter teams. Also, the

synthetic teams processed targets less efficiently and they pushed and pulled the

information less than all-human teams. On the other hand, the experimenter teams

processed more targets than the other teams.

Additionally, team stability (λ) and team situation awareness related in a

nonlinear way, that is, the negative quadratic term meant that, when team stability was

low, there was a positive relationship between the two, but when team stability was high,

there was a negative relationship.

Table 9

Summary of Analysis of Variance Results

Measures Results

Stability Synthetic > Control = Experimenter

Predictability Synthetic = Experimenter > Control

Situation Awareness Experimenter = Control > Synthetic

Team Performance Experimenter > Control = Synthetic

Target Efficiency Experimenter > Control > Synthetic

70

CHAPTER 6

DISCUSSION

According to the theory of Interactive Team Cognition (ITC), team cognition is a

team process which can be measured at the team level by considering the dynamic task

environment in which the team operates (Cooke et al., 2013). Team coordination

dynamics are central to ITC and of this dissertation. This dissertation contributes two new

concepts for the examination of team cognition: (1) poorly coordinated teams with an

autonomous team member (i.e., synthetic teams), and (2) optimally coordinated teams

with a highly experienced team member (i.e., experimenter teams). These two extreme

cases were selected to show that one team member can change the coordination dynamics

of the overall team based on pushing and pulling information in a timely and constructive

manner (experimenter teams) or vice versa (synthetic teams). Thus, in order to

understand this process, this dissertation primarily focuses on team coordination

dynamics in HATs in comparison to all-human teams.

The overall findings regarding team coordination dynamics indicate that the

experimenter teams demonstrated moderate stability in their coordination in that they

adapted to the task environment during routine and novel conditions, resulting in better

performance. However, neither the synthetic teams’ extreme stability (with very little

flexibility) nor the control teams’ lack of stability (with extreme flexibility) in their

coordination resulted in task performance equaling the experimenter teams.

Synthetic teams with extreme stability did not adjust their coordination based on

the changes in the task environment, therefore, they failed to adapt to the environment

(more specifically roadblocks) and, in turn, demonstrated poor performance (especially at

71

the target level). On the other hand, the control teams with instability in their

coordination could adjust their coordination in response to changes in the task

environment. However, it is possible that with such unstable coordination, they could not

return to their routine state following the roadblocks (i.e., embedded target waypoints)

resulting in longer task times than the experimenter teams.

Similarly, this concept is also seen in the human postural system wherein a mid-

range Lyapunov value is indicative of healthy postural sway. Several studies that have

applied stability analysis during quiet standing tasks or Sit to Stand (STS) tasks have

generally shown near-zero to small positive values (Gibbons, 2016; Murata & Iwase,

1998; Yamada, 1995). Researchers have suggested that this range of Lyapunov (λ1)

values is indicative of healthy sway that is stable yet adaptive to the changing movements

in a dynamic environment. On the other hand, studies that have examined unhealthy or

injured patients (e.g. Parkinson’s disease; stroke; etc.) have shown significantly larger

Lyapunov values (e.g. λ1 > 2) (Roerdink et al., 2006). This suggests that too much chaos

(i.e., too little stability) in the system (due to injury/illness) is indicative of unstable

postural control.

Thus, team stability is desirable, to some degree, but the team must also remain

flexible enough to adapt to perturbations. That is, if there is a more desirable pattern of

team coordination, then the team should be able to switch to that new coordination

pattern easily. In this case, this desirable amount of stability in team coordination was

exemplified by the experimenter teams and is depicted in Figure 11. The team

coordination of each of the three conditions is also described in Table 9.

72

Figure 11. The Predicted Relationship between Team Stability and Team Performance for

Each Condition(left); Observed Example Coordination Dynamics for Each

Condition(right)

Table 10

Team Coordination Dynamics across the Conditions and Their Implications on Team

Performance and Situation Awareness

Condition Overall Results Description

Synthetic Extremely stable team coordination dynamics, and less adaptive to

roadblocks resulting in poor performance

Control Less stable team coordination dynamics, and more adaptive to roadblocks

resulting in better performance

Experimenter Moderately stable team coordination dynamics and more adaptive to

roadblocks resulting in high levels of performance

Contributions of This Research

The theoretical contribution of this study is a model depicting an inverted “U”-

shaped relationship between team stability and team performance. This model was

developed by observing the interactions among the team members and with their task

73

environments during the UAV task. Therefore, this model may only be applicable to this

type of action-oriented command-and-control task environment. Our findings also

indicate a curvilinear relationship between team stability and team performance. From

this, we know that a balance exists in terms of stability of team coordination dynamics:

having extremely stable or unstable team coordination hinders team performance in a

cognitively demanding task environment. Based on the inverted “U”-shaped model

(Figure 11), as all-human teams (i.e., control) learn, their coordination should become

more stable and approach optimal coordination and performance. In this case, if one of

the team members is highly trained in terms of team process behaviors (i.e., interaction

with team members in a timely manner), then it is possible that the control teams can

approach the optimum point on the inverted “U-shaped” curve. In contrast, synthetic

teams need to develop flexibility to also approach the optimal point. That is, if HATs are

to function optimally, then any autonomous agent added to the team must be able to

interact with its team members in timely and constructive manner.

Another theoretical contribution of this study is the introduction of an

autonomous agent as a team member in the UAV task environment and observing team

coordination dynamics in HATs in comparison with all-human teams. This study reveals

teamwork deficiencies of HATs by comparing them with all-human teams. The findings

based on HATs indicate that their extremely stable coordination must be addressed by

building a mechanism within them to increase their ability to adapt unexpected changes.

The methodological contribution of this study is the application of Joint

Recurrence Quantification Analysis (JRQA) to team communication data. One of the

74

approaches for investigating team interaction patterns and concomitant change over time

involves looking at communication flow using JRQA that quantifies how many

recurrences (and their length) are present by phase space trajectory in a dynamical

system. JRQA can be applied to understand how and why several teams differ from one

another in their dynamics. Accordingly, it shows the degree to which team members

synchronize their activities during their interaction via text or voice communication.

Thus, JRQA measures whole system which makes it different than Cross-Recurrence

Quantification Analysis (CRQA-considers only two parts of the system)

Based on the experimenter teams’ overall success in this task, the practical

contribution of this dissertation is showing the potential of the synthetic teammate to

deliver team training if it could be developed to equal an ‘experienced’ pilot within the

UAV-STE. Such a synthetic teammate could quicken the team coordination training

process by efficiently pushing and pulling information among the team members.

Overall these contributions indicate that the impact of coordination quality on team

performance and this quality of coordination can be helpful to build mechanisms in HAT

for effective teamwork and to develop the synthetic agent as a trainer.

Limitations

There are several limitations in this study that are related to the use of text chat

communications, a relatively small sample of teams, and using only limited roadblocks.

In this experiment, team members communicated via text-chat, because the synthetic

team member did not have voice communication capabilities. In this study, text-chat is

another limitation which is related with the short time series length and low statistical power.

The team members spent more time communicating with each other because texting and

75

reading take more time than using voice. Therefore, this resulted in fewer instances of

communication and, thusly, short time series lengths. Applying dynamical analysis on short-

time series cause some problems such as the results can be bias. Using voice communication

results in a richer time series. Thus, what needs to be done is a through comparison of text vs

voice coordination.

Statistical power was also relatively low because only a limited number of teams

could be included (in this experiment 30 teams with five missions) due to the expensive

nature of running UAV-STE teams. Although increasing the sample size is prohibitive, one

possible solution is to reduce error variance in the outcome measures by including covariates

in the analysis—such as physiological measures (e.g., respiratory rate and heart rate) and

neural activity (e.g., electroencephalogram - EEG)—or by blocking on some baseline

covariates and randomizing participants within those blocks; either approach would increase

statistical power.

Using only one type of roadblock (i.e., ad hoc target waypoints) was another

limitation in this study that was a side effect of the limited capabilities of the synthetic

teammate to deal with change. Injecting roadblocks during the task is important to test team

coordination dynamics to see how adaptive a team is to unexpected changes, how

coordination dynamics switch to a different state, and how they return to a routine state. In

this study, applying only one type of roadblock limited opportunity to observe changes in

team coordination dynamics. Therefore, applying different types of roadblocks in future work

will help to more clearly to understand how team coordination dynamics changes across

different levels of roadblocks (e.g., user display failure, UAV failure).

76

In this dissertation, team cognition was observed via team process measures (i.e.,

communication and coordination). However, in order to observe team cognition as a whole,

at least some of the knowledge-based measures must also be considered. Therefore, moving

to a complementary approach which combines interactionist measures (e.g., dynamic

measures) and traditional measures (e.g., mental models) will give a complete picture of team

cognition in cognitively-demanding task environment.

Future Work

Future work should be directed at exploring the phase transition of team

coordination dynamics in HAT by applying different types and levels of perturbations

(e.g., automation-related failures, autonomy-related failures). To capture such transitions

in the system, we will create and apply a specific coordination metric to predict team

performance and express the quality of coordination based on dynamics of resilience and

adaptation. This will also help to extend the inverted U-shaped model. In this case, a

pattern of team coordination that adapts to changes in the team or task environment and

bounces back to a stable, but flexible coordination pattern following an unexpected

disruption could be deemed qualitatively superior compared to one that does not or that

does so more slowly. While extending the model and its related coordination metrics,

four phases can be considered: before perturbation (pre-failure), perturbation (failure),

adaptation, and resilience. These phases will be the context for exploring the changing

dynamics of the interaction process and we will analyze the dynamics by graphing the

data using various windows sizes (in seconds). The following example (Figure 12) shows

those phases based on team coordination during one mission (2400 seconds). In this

experiment, the four phases were: pre-failure - when the team was informed about the

77

roadblock by the experimenter, failure – when the team was dealing with the roadblock,

adaptation - when the team completed roadblock, and finally, resilience - when team

came back to its routine state.

Figure 12. Results of the Real-Time Dynamical Analysis of Team Communication

Stability in the UAV Simulator for One Mission

In conclusion, the impact of coordination quality on team performance is discussed

within the HAT context. Although the current synthetic teammate—serving as an Air Vehicle

Operator (AVO)—provides an example of poor quality coordination, in terms of team

performance, synthetic teams performed comparably to control teams. This result means

that future development of the synthetic teams should focus on their coordination

dynamics, specifically, their extreme stability in coordination. If such improvements

could be made, then synthetic teams would be more adaptive and resilient in their task

and in their context.

78

REFERENCES

Abarbanel, H. (1996). Analysis of Observed Chaotic Data (1st edition). Berlin;

Heidelberg u.a.: Springer.

Anderson, J. R. (2007). How can the human mind occur in the physical universe?

Oxford; New York: Oxford University Press.

Ball, J., Myers, C., Heiberg, A., Cooke, N. J., Matessa, M., Freiman, M., & Rodgers, S.

(2010). The synthetic teammate project. Computational and Mathematical

Organization Theory, 16(3), 271–299. https://doi.org/10.1007/s10588-010-9065-3

Bolstad, C. A., & Endsley, M. R. (2003). Measuring Shared and Team Situation

Awareness in the Army’s Future Objective Force. Proceedings of the Human

Factors and Ergonomics Society Annual Meeting, 47(3), 369–373.

https://doi.org/10.1177/154193120304700325

Brannick, M. T., Prince, A., Prince, C., & Salas, E. (1995). The Measurement of Team

Process. Human Factors: The Journal of the Human Factors and Ergonomics

Society, 37(3), 641–651. https://doi.org/10.1518/001872095779049372

Cannon-Bowers, J., Salas, E., & Converse, S. (1993). Shared mental models in expert

team decision making. In N. J. Castellan (Ed.), Individual and Group Decision

Making: Current Issues (pp. 221–246). Psychology Press.

Christoffersen, K., & Woods, D. D. (2002). 1. How to make automated systems team

players. Advances in Human Performance and Cognitive Engineering Research,

2, 1–12. https://doi.org/10.1016/S1479-3601(02)02003-9

Coco, M.I., Dale, R., Dixon, J.D., & Nash, J. (2015). crqa: Cross-Recurrence

Quantification Analysis for Categorical and Continuous Time-Series (Version

1.0.6). Retrieved from https://cran.r-project.org/web/packages/crqa/index.html

Cooke, N. J., Demir, M., & McNeese, N. J. (2016). Synthetic Teammates as Team

Players: Coordination of Human and Synthetic Teammates (Technical Report No.

N000141110844). Mesa, Arizona: Cognitive Engineering Research Institute.

Cooke, N. J., & Gorman, J. C. (2009). Interaction-Based Measures of Cognitive Systems.

Journal of Cognitive Engineering and Decision Making, 3(1), 27–46.

https://doi.org/10.1518/155534309X433302

Cooke, N. J., Gorman, J. C., Duran, J. L., & Taylor, A. R. (2007). Team cognition in

experienced command-and-control teams. Journal of Experimental Psychology:

Applied, 13(3), 146–157.

79

Cooke, N. J., Gorman, J. C., & Kiekel, P. A. (2008). Communication as Team-Level

Cognitive Processing. In M. P. Letsky, N. W. Warner, S. M. Fiore, & C. A. P.

Smith (Eds.), Macrocognition in Teams : Theories and Methodologies (pp. 51–

64). Abingdon, Oxon, GBR: Ashgate Publishing Group.

Cooke, N. J., Gorman, J. C., Myers, C. W., & Duran, J. L. (2013). Interactive Team

Cognition. Cognitive Science, 37(2), 255–285. https://doi.org/10.1111/cogs.12009

Cooke, N. J., Gorman, J. C., & Rowe, L. J. (2008). An Ecological Perspective on Team

Cognition. In Eduardo Salas, G. F. Goodwin, & C. S. Burke (Eds.), Team

Effectiveness In Complex Organizations: Cross-Disciplinary Perspectives and

Approaches (pp. 157–182). New York: Routledge.

Cooke, N. J., Gorman, J., Pedersen, H., Winner, J., Duran, J., Taylor, A., … Rowe, L.

(2007). Acquisition and Retention of Team Coordination in Command-and-

Control. Retrieved from http://www.dtic.mil/docs/citations/ADA475567

Cooke, N. J., Kiekel, P. A., Salas, E., Stout, R., Bowers, C., & Cannon-Bowers, J. (2003).

Measuring team knowledge: A window to the cognitive underpinnings of team

performance. Group Dynamics: Theory, Research, and Practice, 7(3), 179–199.

https://doi.org/10.1037/1089-2699.7.3.179

Cooke, N. J., Salas, E., Cannon-Bowers, J. A., & Stout, R. J. (2000). Measuring Team

Knowledge. Human Factors: The Journal of the Human Factors and Ergonomics

Society, 42(1), 151–173. https://doi.org/10.1518/001872000779656561

Cooke, N. J., Salas, E., Kiekel, P. A., & Bell, B. (2003). Advances in Measuring Team

Cognition. Retrieved from http://www.certt.com/publications/Team%20Cognition

%20Cooke,%20Salas,%20Kiekel,%20Bell.pdf

Cooke, N. J., & Shope, S. M. (2004). Designing a Synthetic Task Environment. In L. R.

E. Schiflett, E. Salas, & M. D. Coovert (Eds.), Scaled Worlds: Development,

Validation, and Application (pp. 263–278). Surrey, England: Ashgate Publishing.

Cooke, N. J., Stout, R., & Salas, E. (1998). Exploring Measures of Team Knowledge.

Proceedings of the Human Factors and Ergonomics Society Annual Meeting,

42(3), 215–219. https://doi.org/10.1177/154193129804200307

Cox, M. T. (2013). Goal-Driven Autonomy and Question-Based Problem Recognition. In

Poster Collection. Palo Alto, CA: Cognitive Systems Foundation. Retrieved from

http://mcox.org/

80

Crouse, K. (2008, August 16). Phelps’s Epic Journey Ends in Perfection. The New York

Times. Retrieved from http://www.nytimes.com/2008/08/17/sports/olympics/

17swim.html

Cuevas, H. M., Fiore, S. M., Caldwell, B. S., & Strater, L. (2007). Augmenting Team

Cognition in Human-Automation Teams Performing in Complex Operational

Environments. Aviation, Space, and Environmental Medicine, 78(5), B63–B70.

Dang, T. S., Palit, S. K., Mukherjee, S., Hoang, T. M., & Banerjee, S. (2016).

Complexity and synchronization in stochastic chaotic systems. The European

Physical Journal Special Topics, 225(1), 159–170.

https://doi.org/10.1140/epjst/e2016-02616-9

Dekker, S. W. A., & Woods, D. D. (2002). MABA-MABA or Abracadabra? Progress on

Human–Automation Co-ordination. Cognition, Technology & Work, 4(4), 240–

244. https://doi.org/10.1007/s101110200022

Delignlères, D., Deschamps, T., Legros, A., & Caillou, N. (2003). A Methodological

Note on Nonlinear Time Series Analysis: Is the Open-and Closed-Loop Model of

Collins and De Luca (1993) a Statistical Artifact? Journal of Motor Behavior,

35(1), 86–96. https://doi.org/10.1080/00222890309602124

Demir, M., & Cooke, N. J. (2014). Human Teaming Changes Driven by Expectations of

a Synthetic Teammate. Proceedings of the Human Factors and Ergonomics

Society Annual Meeting, 58(1), 16–20.

https://doi.org/10.1177/1541931214581004

Demir, M., McNeese, N. J., & Cooke, N. J. (2016). Team communication behaviors of

the human-automation teaming. In 2016 IEEE International Multi-Disciplinary

Conference on Cognitive Methods in Situation Awareness and Decision Support

(CogSIMA) (pp. 28–34). https://doi.org/10.1109/COGSIMA.2016.7497782

Demir, M., McNeese, N. J., Cooke, N. J., & Myers, C. (2016). The Synthetic Teammate

as a Team Player in Command-and-Control Teams. Proceedings of the Human

Factors and Ergonomics Society Annual Meeting, 60(1), 116–116.

https://doi.org/10.1177/1541931213601026

Demir, M., McNeese, N. J., & Cooke, N. J. (2016). Team situation awareness within the

context of human-autonomy teaming. Cognitive Systems Research.

https://doi.org/10.1016/j.cogsys.2016.11.003

Demir, M., McNeese, N. J., Cooke, N. J., Ball, J. T., Myers, C., & Freiman, M. (2015).

Synthetic Teammate Communication and Coordination with Humans.

81

Proceedings of the Human Factors and Ergonomics Society Annual Meeting,

59(1), 951–955. https://doi.org/10.1177/1541931215591275

Donmez, B., Nehme, C., & Cummings, M. L. (2010). Modeling Workload Impact in

Multiple Unmanned Vehicle Supervisory Control. IEEE Transactions on Systems,

Man and Cybernetics, Part A: Systems and Humans, 40(6), 1180–1190.

https://doi.org/10.1109/TSMCA.2010.2046731

Durso, F. T., & Gronlund, S. D. (1999). Situation awareness. In Handbook of applied

cognition (pp. 283–314). New York, NY, US: John Wiley & Sons Ltd.

Eckmann, J. P., Kamphorst, S. O., & Ruelle, D. (1987). Recurrence Plots of Dynamical

Systems. EPL (Europhysics Letters), 4(9), 973. https://doi.org/10.1209/0295-

5075/4/9/004

Endsley, M. R., & Kaber, D. B. (1999). Level of automation effects on performance,

situation awareness and workload in a dynamic control task. Ergonomics, 42(3),

462–492. https://doi.org/10.1080/001401399185595

Endsley, M. R. (1988). Design and Evaluation for Situation Awareness Enhancement.

Proceedings of the Human Factors and Ergonomics Society Annual Meeting,

32(2), 97–101. https://doi.org/10.1177/154193128803200221

Endsley, M. R. (1995). Measurement of Situation Awareness in Dynamic Systems.

Human Factors: The Journal of the Human Factors and Ergonomics Society,

37(1), 65–84. https://doi.org/10.1518/001872095779049499

Endsley, M. R. (2015). Autonomous Horizons: System Autonomy in the Air Force - A

Path to the Future (Autonomous Horizons No. AF/ST TR 15-01). Washington

DC: Department of the Air Force Headquarters of the Air Force. Retrieved from

http://www.af.mil/Portals/1/documents/SECAF/AutonomousHorizons.pdf?timest

amp=1435068339702

Farjoun, M. (2010). Beyond Dusalism: Stability and Change as a Duality. The Academy

of Management Review, 35(2), 202–225.

Fracker, M. L. (1989). Attention Allocation in Situation Awareness. Proceedings of the

Human Factors and Ergonomics Society Annual Meeting, 33(20), 1396–1400.

https://doi.org/10.1177/154193128903302007

Fusaroli, R., & Tylén, K. (2016). Investigating Conversational Dynamics: Interactive

Alignment, Interpersonal Synergy, and Collective Task Performance. Cognitive

Science, 40(1), 145–171. https://doi.org/10.1111/cogs.12251

82

Gibbons, C. (2016). A Nonlinear Analysis of Movement Variability: Stability in a Sit to a

Stand. (Master's Thesis). Arizona State University, Tempe, Arizona. Retrieved

from http://hdl.handle.net/2286/R.I.38515

Gorman, J. C. (2006). Team coordination dynamics in cognitively demanding

environments. (Doctoral Dissertation). New Mexico State University, United

States -- New Mexico.

Gorman, J. C. (2014). Team Coordination and Dynamics: Two Central Issues. Current

Directions in Psychological Science, 23(5), 355–360.

https://doi.org/10.1177/0963721414545215

Gorman, J. C., Amazeen, P. G., & Cooke, N. J. (2010). Team coordination dynamics.

Nonlinear Dynamics, Psychology, and Life Sciences, 14(3), 265–289.

Gorman, J. C., Cooke, N. J., & Amazeen, P. G. (2010). Training Adaptive Teams.

Human Factors: The Journal of the Human Factors and Ergonomics Society,

52(2), 295–307. https://doi.org/10.1177/0018720810371689

Gorman, J. C., Cooke, N. J., Amazeen, P. G., & Fouse, S. (2012). Measuring Patterns in

Team Interaction Sequences Using a Discrete Recurrence Approach. Human

Factors: The Journal of the Human Factors and Ergonomics Society, 54(4), 503–

517. https://doi.org/10.1177/0018720811426140

Gorman, J. C., Cooke, N. J., Pedersen, H. K., Winner, J., Andrews, D., & Amazeen, P. G.

(2006). Changes in Team Composition after a Break: Building Adapative

Command-and-Control Teams. Proceedings of the Human Factors and

Ergonomics Society Annual Meeting, 50(3), 487–491.

https://doi.org/10.1177/154193120605000358

Gorman, J. C., Cooke, N. J., Pederson, H. K., Olena, O. C., & DeJoode, J. A. (2005).

Coordinated Awareness of Situation by Teams (CAST): Measuring Team

Situation Awareness of a Communication Glitch. Proceedings of the Human

Factors and Ergonomics Society Annual Meeting, 49(3), 274–277.

https://doi.org/10.1177/154193120504900313

Gorman, J. C., Cooke, N. J., & Winner, J. L. (2006). Measuring team situation awareness

in decentralized command and control environments. Ergonomics, 49(12–13),

1312–1325. https://doi.org/10.1080/00140130600612788

Guastello, S. J., & Guastello, D. D. (1998). Origins of coordination and team

effectiveness: A perspective from game theory and nonlinear dynamics. Journal

of Applied Psychology, 83(3), 423–437.

https://doi.org/http://dx.doi.org.ezproxy1.lib.asu.edu/10.1037/0021-9010.83.3.423

83

Gorman, J. C., Hessler, E. E., Amazeen, P. G., Cooke, N. J., & Shope, S. M. (2012).

Dynamical analysis in real time: detecting perturbations to team communication.

Ergonomics, 55(8), 825–839.

Hackman, J. R. (1987). The Design of Work Teams. In J. W. Lorsch (Ed.), Handbook of

Organizational Behavior (pp. 315–342). Englewood Cliffs, NJ: Prentice-Hall.

Haken, H. (1978). Synergetics: An introduction : nonequilibrium phase transitions and

self-organization in physics, chemistry, and biology (Enlarged 2nd edition).

Berlin ; New York: Springer-Verlag.

Haken, H. (1989). Synergetics: an overview. Reports on Progress in Physics, 52(5), 515.

https://doi.org/10.1088/0034-4885/52/5/001

Haken, H. (2003). Intelligent behavior: a synergetic view. In The Dynamical Systems

Approach to Cognition (Vol. Volume 10, pp. 3–16). WORLD SCIENTIFIC.

https://doi.org/10.1142/9789812564399_0001

Hamill, J., van Emmerik, R. E. A., Heiderscheit, B. C., & Li, L. (1999). A dynamical

systems approach to lower extremity running injuries. Clinical Biomechanics,

14(5), 297–308. https://doi.org/10.1016/S0268-0033(98)90092-4

Hart, S. G., & Staveland, L. E. (1988). Development of NASA-TLX (Task Load Index):

Results of empirical and theoretical research. In P. A. Hancock & N. Mashkati

(Eds.), Human Mental Workload. Amsterdam: North Holland Press.

Kantz, H., & Schreiber, T. (2004). Nonlinear Time Series Analysis (2 edition).

Cambridge, UK ; New York: Cambridge University Press.

Kelso, J. A. S. (1997). Dynamic Patterns: The Self-organization of Brain and Behavior.

MIT Press.

Kelso, J. A. S. (2003). Cognitive Coordination Dynamics. In J.-P. Dauwalder & W.

Tschacher (Eds.), Dynamical Systems Approach to Cognition, The: Concepts and

Empirical Paradigms Based on Self-Organization, Embodiment, and

Coordination Dynamics (pp. 45–67). River Edge, NJ: World Scientific Publishing

Company.

Kelso, J. A. S. (2004). Coordination Dynamics: Issues and Trends. (V. K. Jirsa & S.

Kelso, Eds.) (2004 edition). Berlin ; New York: Springer.

Kelso, J. A. S., & Engstrom, D. A. (2006). The Complementary Nature. Cambridge,

Massachusetts: The MIT Press.

84

Klein, G., Woods, D. D., Bradshaw, J. M., Hoffman, R. R., & Feltovich, P. J. (2004). Ten

Challenges for Making Automation a “Team Player” in Joint Human-Agent

Activity. IEEE Intelligent Systems, 19(6), 91–95.

https://doi.org/10.1109/MIS.2004.74

Knight, A. P., Kennedy, D. M., & McComb, S. A. (2016). Using recurrence analysis to

examine group dynamics. Group Dynamics: Theory, Research, and Practice,

20(3), 223–241. https://doi.org/ 10.1037/gdn0000046

Krogmann, U. (1999). From Automation to Autonomy-Trends Towards Autonomous

Combat Systems (Unclassified No. RTO MP-44). France: NATO Science and

Technology Organization. Retrieved from

http://www.dtic.mil/dtic/tr/fulltext/u2/p010300.pdf

Langan-Fox, J., Canty, J. M., & Sankey, M. J. (2009). Human–automation teams and

adaptable control for future air traffic management. International Journal of

Industrial Ergonomics, 39(5), 894–903.

https://doi.org/10.1016/j.ergon.2009.04.002

Langan-Fox, J., Code, S., & Langfield-Smith, K. (2000). Team Mental Models:

Techniques, Methods, and Analytic Approaches. Human Factors: The Journal of

the Human Factors and Ergonomics Society, 42(2), 242–271.

https://doi.org/10.1518/001872000779656534

Losada, M. (1999). The complex dynamics of high performance teams. Mathematical

and Computer Modelling, 30(9–10), 179–192. https://doi.org/10.1016/S0895-

7177(99)00189-2

Losada, Marcial, & Heaphy, E. (2004). The Role of Positivity and Connectivity in the

Performance of Business Teams: A Nonlinear Dynamics Model. American

Behavioral Scientist, 47(6), 740–765. https://doi.org/10.1177/0002764203260208

Luenberger, D. G. (1979). Introduction to Dynamic Systems: Theory, Models, and

Applications. Wiley.

MacMillan, J., Entin, E. E., & Serfaty, D. (2004). Communication overhead: The hidden

cost of team cognition. In Eduardo Salas & S. M. Fiore (Eds.), Team cognition:

Understanding the factors that drive process and performance. (pp. 61–82).

Washington, DC, US: American Psychological Association. Retrieved from

http://content.apa.org/books/10690-004

85

Marks, M. A., Mathieu, J. E., & Zaccaro, S. J. (2001). A Temporally Based Framework

and Taxonomy of Team Processes. Academy of Management Review, 26(3), 356–

376. https://doi.org/10.5465/AMR.2001.4845785

Marwan, N., Carmen Romano, M., Thiel, M., & Kurths, J. (2007). Recurrence plots for

the analysis of complex systems. Physics Reports, 438(5–6), 237–329.

https://doi.org/10.1016/j.physrep.2006.11.001

Mathieu, J. E., Heffner, T. S., Goodwin, G. F., Salas, E., & Cannon-Bowers, J. A. (2000).

The influence of shared mental models on team process and performance. Journal

of Applied Psychology, 85(2), 273–283. https://doi.org/10.1037/0021-

9010.85.2.273

McNeese, N. J., Demir, M., Cooke, N. J., & Myers, C. W. (2017). Teaming with a

Synthetic Teammate: Insights into Human-Autonomy Teaming. The Journal of the

Human Factors and Ergonomics Society. Retrieved from Manuscript submitted

for publication

Mohammed, S., Ferzandi, L., & Hamilton, K. (2010). Metaphor No More: A 15-Year

Review of the Team Mental Model Construct. Journal of Management, 36(4),

876–910. https://doi.org/10.1177/0149206309356804

Mohammed, S., & Dumville, B. C. (2001). Team mental models in a team knowledge

framework: expanding theory and measurement across disciplinary boundaries.

Journal of Organizational Behavior, 22(2), 89–106.

https://doi.org/10.1002/job.86

Murata, A., & Iwase, H. (1998). Chaotic analysis of body sway. In Proceedings of the

20th Annual International Conference of the IEEE Engineering in Medicine and

Biology Society. Vol.20 Biomedical Engineering Towards the Year 2000 and

Beyond (Cat. No.98CH36286) (Vol. 3, pp. 1557–1560 vol.3).

https://doi.org/10.1109/IEMBS.1998.747186

Nordham, C. A., & Kelso, J. A. S. (2016). The Nature of Interpersonal Coordination:

Why do people coordinate with others? In P. Passos, K. Davids, & J. Y. Chow

(Eds.), Interpersonal Coordination and Performance in Social Systems (First

Edition, pp. 32–52). New York, NY: Taylor & Francis Group.

Pina, P. E., Cummings, M. L., Crandall, J. W., & Penna, M. D. (2008). Identifying

Generalizable Metric Classes to Evaluate Human-Robot Teams. In Proceedings of

Metrics for Human-Robot Interaction Workshop (pp. 13–20). Amsterdam.

Retrieved from http://web.mit.edu/aeroastro/labs/halab/papers/Pina-

Generalizable%20metrics-HRI2008.pdf

86

Priest, H. A., & Stader, S. (2012). A Framework for Developing Synthetic Agents as

Pedagogical Teammates: Applying What We Already Know. Proceedings of the

Human Factors and Ergonomics Society Annual Meeting, 56(1), 2552–2556.

https://doi.org/10.1177/1071181312561530

R: The R Project for Statistical Computing. (2015). Retrieved August 24, 2016, from

https://www.r-project.org/

Reeves, M., & Deimler, M. (2011, July 1). Adaptability: The New Competitive

Advantage. Retrieved March 31, 2017, from https://hbr.org/2011/07/adaptability-

the-new-competitive-advantage

Rizzi, M., Frigerio, F., & Iori, V. (2016). The Early Phases of Epileptogenesis Induced by

Status Epilepticus Are Characterized by Persistent Dynamical Regime of

Intermittency Type. In C. L. W. Jr, C. Ioana, & N. Marwan (Eds.), Recurrence

Plots and Their Quantifications: Expanding Horizons (pp. 185–208). Springer

International Publishing. https://doi.org/10.1007/978-3-319-29922-8_10

Roerdink, M., De Haart, M., Daffertshofer, A., Donker, S. F., Geurts, A. C. H., & Beek,

P. J. (2006). Dynamical structure of center-of-pressure trajectories in patients

recovering from stroke. Experimental Brain Research, 174(2), 256–269.

https://doi.org/10.1007/s00221-006-0441-7

Rodgers, S., Myers, C., Ball, J., & Freiman, M. (2011). The Situation Model in the

Synthetic Teammate Project. In The 20th Annual Conference on Behavior

Representation in Modeling and Simulation. Retrieved from

http://palm.mindmodeling.org/synwiki/images/c/cf/BRIMS_2011_Situation_Mod

el_Final_Submission.pdf

Romero, V., Fitzpatrick, P., Schmidt, R. C., & Richardson, M. J. (2016). Using Cross-

Recurrence Quantification Analysis to Understand Social Motor

CoordinationMotor Coordination in Children with Autism Spectrum

DisorderAutism Spectrum Disorder. In C. L. W. Jr, C. Ioana, & N. Marwan

(Eds.), Recurrence Plots and Their Quantifications: Expanding Horizons (pp.

227–240). Springer International Publishing. https://doi.org/10.1007/978-3-319-

29922-8_12

Rosenstein, M. T., Collins, J. J., & De Luca, C. J. (1993). A practical method for

calculating largest Lyapunov exponents from small data sets. Physica D:

Nonlinear Phenomena, 65(1), 117–134. https://doi.org/10.1016/0167-

2789(93)90009-P

Salas, E., Cooke, N. J., & Rosen, M. A. (2008). On Teams, Teamwork, and Team

Performance: Discoveries and Developments. Human Factors: The Journal of the

87

Human Factors and Ergonomics Society, 50(3), 540–547.

https://doi.org/10.1518/001872008X288457

Salas, E., Dickinson, T. L., Converse, S. A., & Tannenbaum, S. I. (1992). Toward an

understanding of team performance and training: Robert W. Swezey, Eduardo

Salas: Books. In R. W. Swezey & E. Salas (Eds.), Teams: Their Training and

Performance (pp. 3–29). Norwood, N.J.: Ablex Pub. Corp.

Salas, E., Prince, C., Baker, D. P., & Shrestha, L. (1995). Situation Awareness in Team

Performance: Implications for Measurement and Training. Human Factors: The

Journal of the Human Factors and Ergonomics Society, 37(1), 123–136.

https://doi.org/10.1518/001872095779049525

Schooley, L. C., Zeigler, B. P., Cellier, F. E., & Wang, F. Y. (1993). High-autonomy

control of space resource processing plants. IEEE Control Systems, 13(3), 29–39.

https://doi.org/10.1109/37.214942

Schulte, A., Donath, D., & Lange, D. S. (2016). Design Patterns for Human-Cognitive

Agent Teaming. In D. Harris (Ed.), Engineering Psychology and Cognitive

Ergonomics (pp. 231–243). Springer International Publishing.

https://doi.org/10.1007/978-3-319-40030-3_24

Semin, G. R. (2007). Grounding communication: Synchrony. In A. W. Kruglanski & E.

T. Higgins (Eds.), Social psychology: Handbook of basic principles, 2nd ed (pp.

630–649). New York, NY, US: Guilford Press.

Shockley, K., Santana, M.-V., & Fowler, C. A. (2003). Mutual interpersonal postural

constraints are involved in cooperative conversation. Journal of Experimental

Psychology: Human Perception and Performance, 29(2), 326–332.

https://doi.org/http://dx.doi.org.ezproxy1.lib.asu.edu/10.1037/0096-1523.29.2.326

Strang, A. J., Funke, G. J., Russell, S. M., Dukes, A. W., & Middendorf, M. S. (2014).

Physio-behavioral coupling in a cooperative team task: Contributors and relations.

Journal of Experimental Psychology: Human Perception and Performance, 40(1),

145–158. https://doi.org/http://dx.doi.org.ezproxy1.lib.asu.edu/10.1037/a0033125

Sycara, K., & Lewis, M. (2004). Integrating intelligent agents into human teams. In E.

Salas & S. M. Fiore (Eds.), Team cognition: Understanding the factors that drive

process and performance (pp. 203–231). Washington, DC, US: American

Psychological Association.

Takens, F. (1981). Detecting strange attractors in turbulence. In D. A. Rand & L. S.

Young (Eds.), Dynamical Systems and Turbulence, Lecture Notes in Mathematics

(Vol. 898, pp. 230–242). Springer-Verlag. Retrieved from

88

http://izt.ciens.ucv.ve/ecologia/Archivos/ECOLOGIA_DE%20_POBLACIONES

_Hasta%202004/ECOL_POBLAC_Hasta%202004_(O-Z)/Takens%201981.pdf

Theiler, J., Eubank, S., Longtin, A., Galdrikian, B., & Doyne Farmer, J. (1992). Testing

for nonlinearity in time series: the method of surrogate data. Physica D:

Nonlinear Phenomena, 58(1), 77–94. https://doi.org/10.1016/0167-

2789(92)90102-S

Thelen, E., & Smith, L. B. (1996). A Dynamic Systems Approach to the Development of

Cognition and Action (Revised ed. edition). Cambridge, Mass.: A Bradford Book.

Webber, C. L., & Marwan, N. (Eds.). (2014). Recurrence Quantification Analysis:

Theory and Best Practices (2015 edition). New York: Springer.

Wijngaards, N., Kempen, M., Smit, A., & Nieuwenhuis, K. (2006). Towards Sustained

Team Effectiveness. In O. Boissier, J. Padget, V. Dignum, G. Lindemann, E.

Matson, S. Ossowski, … J. Vázquez-Salceda (Eds.), Coordination,

Organizations, Institutions, and Norms in Multi-Agent Systems (pp. 35–47).

Springer Berlin Heidelberg. Retrieved from

http://link.springer.com.ezproxy1.lib.asu.edu/chapter/10.1007/11775331_3

Wolf, A., Swift, J. B., Swinney, H. L., & Vastano, J. A. (1985). Determining Lyapunov

exponents from a time series. Physica D: Nonlinear Phenomena, 16(3), 285–317.

https://doi.org/10.1016/0167-2789(85)90011-9

Yamada, N. (1995). Chaotic swaying of the upright posture. Human Movement Science,

14(6), 711–726. https://doi.org/10.1016/0167-9457(95)00032-1

89

APPENDIX A

COORDINATION SCRIPT

90

Table

A1

Coord

inat

ion s

crip

t

91

APPENDIX B

VARIABLES FOR TEAM PERFORMANCE SCORE

92

Team Performance Score Variables:

Team Alarm Duration (TAD): All the alarm durations (in seconds) during the

mission, including: airspeed, altitude, fuel, hazards, lost, route request, flaps,

gears, lens, temperature, battery, and film.

Team Warning Duration (TWD): All the warning durations (in seconds) during

the mission, including: airspeed, altitude, fuel, hazards, lost, route request, flaps,

gears, lens, temperature, battery, and film.

Mission Total Second (MTS): Mission total time, 2400 seconds

Team Warning Duration (TWD): All warning durations (in seconds) during the

mission, including: airspeed, altitude, fuel, hazards, lost, route request, flaps,

gears, lens, temperature, and film.

Critical Reached Waypoints (CRW): Number of critical waypoints actually

visited during the mission.

Total Good Unique Photos (TGUP): Number of the photos taken by PLO during

the mission.