Effect of Auditory Peripheral Displays Unmanned …Effect of Auditory Peripheral Displays On...

Effect of Auditory Peripheral Displays On Unmanned Aerial Vehicle Operator Performance

by

Hudson D. Graham

B.S. Systems Engineering

United States Air Force Academy, 2006

Submitted to the Engineering Systems Division in partial fulfillment of the requirements for the degree of

Master of Science in Engineering Systems

at the Massachusetts Institute of Technology

June 2008

© 2008 Massachusetts Institute of Technology. All rights reserved.

Signature of Author: ___________________________________________________________

Engineering Systems Division May 9, 2008

Certified by: __________________________________________________________________

Mary Cummings Associate Professor of Engineering Systems

Thesis Supervisor

Accepted by: ___________________________________________________________________ Richard Larson

Professor of Engineering Systems Chairman, Engineering Systems Division Education Division

Report Documentation Page Form ApprovedOMB No. 0704-0188

Public reporting burden for the collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering andmaintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information,including suggestions for reducing this burden, to Washington Headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, ArlingtonVA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to a penalty for failing to comply with a collection of information if itdoes not display a currently valid OMB control number.

1. REPORT DATE JUN 2008 2. REPORT TYPE

3. DATES COVERED 00-00-2008 to 00-00-2008

4. TITLE AND SUBTITLE Effect of Auditory Peripheral Displays On Unmanned Aerial VehicleOperator Performance

5a. CONTRACT NUMBER

5b. GRANT NUMBER

5c. PROGRAM ELEMENT NUMBER

6. AUTHOR(S) 5d. PROJECT NUMBER

5e. TASK NUMBER

5f. WORK UNIT NUMBER

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) United States Air Force Academy,US Air Force Academy,CO,80840

8. PERFORMING ORGANIZATIONREPORT NUMBER

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR’S ACRONYM(S)

11. SPONSOR/MONITOR’S REPORT NUMBER(S)

12. DISTRIBUTION/AVAILABILITY STATEMENT Approved for public release; distribution unlimited

13. SUPPLEMENTARY NOTES

14. ABSTRACT With advanced autonomy, Unmanned Aerial Vehicle (UAV) operations will likely be conducted by singleoperators controlling multiple UAVs. As operator attention is divided across multiple supervisory tasks,there is a need to support the operator?s awareness of the state of the tasks for safe and effective taskmanagement. This research explores enhancing audio cues of UAV interfaces for this futuristic control ofmultiple UAVs by a single operator. This thesis specifically assesses the value of continuous and discreteaudio cues as indicators of course‐deviations or late‐arrivals to targets for UAV missionswith single and multiple UAVs. In particular, this thesis addresses two questions: (1) when compared withdiscrete audio, does continuous audio better aid human supervision of UAV operations, and (2) is theeffectiveness of the discrete or continuous audio support dependent on operator workload? An experimentwas carried out on the Multiple Autonomous Unmanned Vehicle Experiment (MAUVE) test bed with 44military participants. Specifically, two continuous audio alerts were mapped to two human supervisorytasks within MAUVE. These continuous alerts were tested against single beep discrete alerts. The resultsshow that the use of the continuous audio alerts enhances a single operator?s performance in monitoringsingle and multiple, semi‐autonomous vehicles. The results also emphasize the necessity to properlyintegrate the continuous audio with other auditory alarms and visual representations in a display, as it ispossible for discrete audio alerts to be masked by continuous audio, leaving operators reliant on the visualaspects of the display.

15. SUBJECT TERMS

16. SECURITY CLASSIFICATION OF: 17. LIMITATION OF ABSTRACT Same as

Report (SAR)

18. NUMBEROF PAGES

105

19a. NAME OFRESPONSIBLE PERSON

a. REPORT unclassified

b. ABSTRACT unclassified

c. THIS PAGE unclassified

2

The views expressed in this article are those of the author and do not reflect the official

policy or position of the United States Air Force, Department of Defense, or the U.S.

Government.

Effect of Auditory Peripheral Displays On Unmanned Aerial Vehicle Operator Performance

by

Hudson D. Graham

Submitted to the Engineering Systems Division on May 9th, 2008, in partial fulfillment of the requirements for the degree of

Master of Science in Engineering Systems

Abstract

With advanced autonomy, Unmanned Aerial Vehicle (UAV) operations will likely be conducted by single operators controlling multiple UAVs. As operator attention is divided across multiple supervisory tasks, there is a need to support the operator’s awareness of the state of the tasks for safe and effective task management. This research explores enhancing audio cues of UAV interfaces for this futuristic control of multiple UAVs by a single operator. This thesis specifically assesses the value of continuous and discrete audio cues as indicators of course‐deviations or late‐arrivals to targets for UAV missions with single and multiple UAVs. In particular, this thesis addresses two questions: (1) when compared with discrete audio, does continuous audio better aid human supervision of UAV operations, and (2) is the effectiveness of the discrete or continuous audio support dependent on operator workload?

An experiment was carried out on the Multiple Autonomous Unmanned Vehicle

Experiment (MAUVE) test bed with 44 military participants. Specifically, two continuous audio alerts were mapped to two human supervisory tasks within MAUVE. These continuous alerts were tested against single beep discrete alerts. The results show that the use of the continuous audio alerts enhances a single operator’s performance in monitoring single and multiple, semi‐autonomous vehicles. The results also emphasize the necessity to properly integrate the continuous audio with other auditory alarms and visual representations in a display, as it is possible for discrete audio alerts to be masked by continuous audio, leaving operators reliant on the visual aspects of the display.

Thesis Supervisor: Mary Cummings Title: Associate Professor of Engineering Systems

3

Acknowledgements

I owe many people a thank you for the successful completion of this thesis. First, thank you to my research and academic advisor, Missy Cummings. Being

a part of your lab is the best first Air Force assignment I could have had. Having you as my advisor has been a true honor. From you, I have learned not just academics, but professional skills that I will use throughout my career as an officer.

Thank you also to Mark Ashdown and Birsen Donmez. I am grateful for your feedback throughout the writing and countless revisions of this thesis.

To Adam Fouse, Jonathan Pfautz, Ryan Kilgore, and Charles River Analytics, it was a privilege to work with you on this Army funded project. Adam, I could not have conducted experimentation without your countless hours of working to get the audio displays integrated into MAUVE. Thank you.

Thank you to David Silvia and Bill D’Angelo for your thoughtful comments and feedback throughout my research.

To the undergraduates, Brian Malley and Teresa Pontillo, who helped me setup and run the experiment, thank you for your time and energy.

Thank you to Stephen Jimenez and the ROTC detachments for helping me recruit participants for my experiment.

Fellow HALiens past and present: Jim, Amy, Yves, Carl, Sylvain, Geoff, Anna, and many others, thank you for sharing your lives with me and making my time here a great experience. It has been a joy to get to know each of you, and I look forward to keeping up with you in the years to come.

Thank you to Mom and Dad for your constant support throughout my time at MIT. I am blessed to call you my parents and praise God each time I think of you.

To my beautiful bride, a highlight of these two years at MIT has been God making you a part of my life. Thank you for your daily support and encouragement.

Most importantly, I thank my Lord and Savior. It is by His grace that I wake each morning, and by His grace that I have been afforded the opportunity to serve at MIT (Ephesians 2:8‐9). I thank Him for His faithfulness to me (John 3:16).

4

Table of Contents

Abstract ..........................................................................................................................................3 Acknowledgements .....................................................................................................................4 Table of Contents.........................................................................................................................5 List of Figures ...............................................................................................................................7 List of Tables.................................................................................................................................9 Nomenclature..............................................................................................................................10 1. Introduction ............................................................................................................................11 1.1. Problem Statement.......................................................................................................... 14 1.2. Thesis Organization........................................................................................................ 14

2. Background .............................................................................................................................17 2.1. UAS Evolution............................................................................................................ 17

2.1.1. The Current Push..............................................................................................................................17 2.1.2. The Human in the System ...............................................................................................................19 2.1.3. The UAV Ground Control Station..................................................................................................20

2.2. Attention Theories ..................................................................................................... 21 2.2.1. Selective, Divided, and Sustained Attention.................................................................................21 2.2.2. Attention Resource Theories ...........................................................................................................22 2.2.3. Multiple Resource Theory ...............................................................................................................23

2.3. Audio Research .......................................................................................................... 25 2.3.1. Supervisory Control with Multi‐Modal Displays ........................................................................26 2.3.2. Continuous Audio ............................................................................................................................27 2.3.3. Supervisory Control Sonifications..................................................................................................29

2.4. Research Hypotheses................................................................................................. 31 3. Simulator and Interface Design ..........................................................................................33 3.1. Multiple Aerial Unmanned Vehicle Experiment (MAUVE) Test Bed .................... 33 3.2. Four Auditory‐alerts....................................................................................................... 38 3.3. Sensimetrics HDiSP ........................................................................................................ 39 3.4. Technical Description of Auditory‐alerts .................................................................... 39

4. Methods ...................................................................................................................................43 4.1. Experimental Questions................................................................................................. 43 4.2. Experimental Apparatus................................................................................................ 44 4.3. Experimental Design ...................................................................................................... 46

4.3.1. Independent Variables..........................................................................................................................46 4.3.2. Dependent Variables.............................................................................................................................48

4.4. Participants ...................................................................................................................... 50 4.5. Testing Procedures.......................................................................................................... 51 4.6. Data Collection ................................................................................................................ 52

5

5. Results ......................................................................................................................................53 5.1. Course‐deviation Errors of Omission .......................................................................... 53 5.2. Course‐deviation Reaction Times................................................................................. 53 5.3. Late‐arrival Errors of Omission .................................................................................... 56 5.4. Late‐arrival Reaction Times........................................................................................... 56 5.5. Workload.......................................................................................................................... 56

5.5.1. Secondary Task Assessment ................................................................................................................56 5.5.2. Subjective Assessment ..........................................................................................................................57

5.6. Post‐Experiment Subjective Responses ....................................................................... 58 5.7. Summary of Experimental Findings ............................................................................ 60

6. Discussion ...............................................................................................................................63 6.1. Multiple Resource Theory ............................................................................................. 63 6.2. Mapping Audio Alerts to Intuitive Triggers............................................................... 64 6.3. Change Blindness............................................................................................................ 65 6.4. Masking ............................................................................................................................ 66 6.5. Workload.......................................................................................................................... 66

7. Conclusion...............................................................................................................................69 7.1. Findings............................................................................................................................ 69

7.1.1. Value Added by Continuous Audio...................................................................................................69 7.1.2. The Impact of Workload.......................................................................................................................70

7.2. Integration Issues ............................................................................................................ 71 7.3. Cost‐Benefit Analysis ..................................................................................................... 72 7.4. Limitations and Future Work........................................................................................ 74

Appendix A: Audio Alert Guidelines....................................................................................77 Appendix B: Scenario Events...................................................................................................79 Appendix C: Participant Consent Form.................................................................................81 Appendix D: Demographics Survey ......................................................................................85 Appendix E: MAUVE‐MITUS Tutorial .................................................................................87 Appendix F: Post‐Experiment Survey....................................................................................93 Appendix G: GLM Analysis: SPSS OUTPUT ......................................................................95 References..................................................................................................................................101

6

List of Figures

Figure 2‐1: Possible UAV Missions (Nehme et al., 2007). .................................................... 18 Figure 2‐2: Transition to Multiple‐UAV Supervision. .......................................................... 19 Figure 2‐3: Attentional Resource Pools (Wickens & Hollands, 2000)................................. 24 Figure 3‐1: Multiple Aerial Unmanned Vehicle Experiment (MAUVE) Test Bed. .......... 34 Figure 3‐2: UAV Interaction Control Panel ............................................................................ 34 Figure 3‐3: Late‐arrival Illustration. ........................................................................................ 36 Figure 3‐4: Decision Support Visualization (DSV)................................................................ 36 Figure 3‐5: Course‐deviation Illustration. .............................................................................. 37 Figure 3‐6: Four Auditory‐alerts.............................................................................................. 38 Figure 3‐7: Sensimetrics Headset Display (HDiSP). ............................................................. 39 Figure 4‐1: Multi‐Modal Workstation (MMWS) (Osga et al., 2002).................................... 44 Figure 5‐1: Course‐deviation Reaction Times Treatment Means Plot................................ 54 Figure 5‐2: Post Hoc Analysis Course‐deviation Reaction Times Treatment Means Plot.

................................................................................................................................ 55 Figure 5‐3: Transformed (Natural Log) Late‐arrival Reaction Times Treatment Means

Plot......................................................................................................................... 57 Figure B‐1: Major Events of Single‐UAV and Multiple‐UAV Test Scenarios. .................. 79

7

List of Tables

Table 1‐1: Auditory Versus Visual Presentations (Deatherage, 1972; Sorkin, 1987). ....... 13 Table 3‐1: UAV Color‐Coded Flight Phases........................................................................... 35 Table 4‐1: Experimental Conditions ........................................................................................ 47 Table 5‐1: Post Hoc Experimental Conditions. ...................................................................... 55 Table B‐1: Comparison of Single‐UAV and Multiple‐UAV Scenarios. .............................. 80 Table G‐1: Course‐deviation Reaction Times (4 audio conditions) Within‐Subjects

Contrasts............................................................................................................... 95 Table G‐2: Course‐deviation Reaction Times (4 audio conditions) Between‐Subjects

Effects. ................................................................................................................... 95 Table G‐3: Course‐deviation Reaction Times (4 audio conditions) Tukey Test

Comparisons. ....................................................................................................... 96 Table G‐4: Course‐deviation Reaction Times (2 audio alerts) Within‐Subjects Contrast.96 Table G‐5: Course‐deviation Reaction Times (2 audio alerts) Between‐Subjects Effects. 97 Table G‐6: Transformed (natural log) Late‐arrival Reaction Times Within‐Subjects

Contrasts............................................................................................................... 97 Table G‐7: Transformed (natural log) Late‐arrival Reaction Times Between‐Subjects

Effects. ................................................................................................................... 97 Table G‐8: Transformed (natural log) Late‐arrival Reaction Times Tukey Test

Comparisons. ....................................................................................................... 98 Table G‐9: Transformed (natural log) Late‐arrival Reaction Times (with

BothCont/LateCont/BothThresh Combined against DevCont) Within‐Subjects Contrasts. .............................................................................................. 98

Table G‐10: Transformed (natural log) Late‐arrival Reaction Times (with BothCont/LateCont/BothThresh Combined against DevCont) Between‐Subjects Effects..................................................................................................... 99

Table G‐11: NASA TLX Scores Within‐Subjects Contrasts. ................................................. 99 Table G‐12: NASA TLX Between‐Subjects Effects................................................................. 99 Table G‐13: Missed Radio Calls Within‐Subjects Contrasts. ............................................. 100 Table G‐14: Missed Radio Calls Between‐Subjects Effects................................................. 100

9

Nomenclature

ANOVA Analysis of Variance

COCOM Combatant Commander

DOD Department of Defense

DSV Decision Support Visualization

HDiSP Headset Display

MALE Minimum Altitude, Long Endurance

MAUVE Multiple Aerial Unmanned Vehicle Experiment

MIT Massachusetts Institute of Technology

MMWS Multi‐Modal Workstation

NASA National Aeronautics and Space Administration

NATO North Atlantic Treaty Organization

RTB Return to Base

TLX Task Load Index

TOT Time on Target

UAS Unmanned Aerial System

UAV Unmanned Aerial Vehicle

VTUAV Vertical‐take‐off‐and‐landing Tactical Unmanned Aerial Vehicle

3D Three‐dimensional

10

1. Introduction

The use of Unmanned Aerial Vehicles (UAVs) is growing in the military, federal,

and civil domains. Worldwide, in the next 8 years UAVs will be a 15.7 billion dollar

industry, where in the United States alone the plan is to have 9,420 mini UAVs, 775

tactical UAVs, 570 minimum altitude, long endurance (MALE) UAVs and 60 Global

Hawks (Tsach, Peled, Penn, Keshales, & Guedj, 2007). In fall 2006, over 40 countries

were operating UAVs and 80 types of UAVs were in existence, with the United States

operating 3,000 UAVs and the North Atlantic Treaty Organization (NATO) operating

3,600 UAVs (Culbertson, 2006). UAVs are not just in demand on the battlefield; other

government departments seeking the use of UAVs include the Department of

Homeland Security and the U.S. Coast Guard for law enforcement and border patrol

(DOD, 2007). UAVs are also sought after in support of humanitarian relief efforts, such

as with flood regions in the United States. In Missouri in May 2007, Air Force Predator

UAVs were on standby to offer assistance in the flooding recovery efforts (Arana‐

Barradas, 2007).

In his spring 2008 speech to Air Force officers at Air University, Defense

Secretary Robert M. Gates pointed to the fact that within the Department of Defense

(DOD) there has been a 25‐fold increase in UAV operations since 2001. He then went

on to say that this increase is not enough. To support the troops in Afghanistan and

Iraq, work must be done to further integrate UAVs into the force and operations (Gates,

2008).

One of the efforts to further integrate UAVs into the force is to maximize the

human‐to‐UAV ratio as a “force multiplier.” In fall 2006, General William T. Hobbins,

in command of the Allied Air Component Command and European Command Air

Component, publicly stated that part of the solution to the growing UAV demand will

be to have a single operator controlling multiple UAVs (Culbertson, 2006), because a

11

single operator is more efficient for cost and operational tempo (Barbato, Feitshans,

Williams, & Hughes, 2003; Tsach et al., 2007).

To achieve force multiplication, humans will need to perform a supervisory role

as vehicles become more autonomous, instead of attending to low‐level tasks like

manually flying the aircraft. One way to help humans efficiently and safely execute this

supervisory task is to maximize the use of each sensory channel to convey information.

In aviation, pilots use visual and auditory senses when flying an aircraft. A unique

aspect of UAV supervision, which occurs remotely, is that if the operator requires

natural signals that occur in manned aviation, the signals must be synthetically created.

UAV displays are predominantly visual displays and are typically mounted in mobile

trailers, truck beds, or backpacks. With advancing technology, there are new,

unexplored ways to provide display information across multiple sensory channels.

When there is an overload of information, utilizing different and multiple

sensory channels can effectively increase the amount of information processed

(Wickens, Lee, Liu, & Becker, 2004). When using multiple sensory channels (or modes),

certain forms of information are better conveyed through an audio presentation, while

others are better conveyed through a visual presentation (Deatherage, 1972; Sorkin,

1987). Table 1‐1 provides a list of known benefits for the visual and audio channel

presentations. A notable benefit for auditory presentation is its effectiveness at

representing simple events occurring in time and requiring immediate action, as

opposed to complex events occurring at a location in space and not requiring

immediate action. Further, because of its omnipresence, audio is usually the preferred

sense for presenting a warning (Simpson & Williams, 1980; Sorkin, 1987). Thus, the

audio channel is effective in warning of potential problems.

12

Table 1‐1: Auditory Versus Visual Presentations (Deatherage, 1972; Sorkin, 1987).

Use auditory presentation if: Use visual presentation if: 1. The message is simple. 1. The message is complex. 2. The message is short. 2. The message is long. 3. The message will not be referred to later. 3. The message will be referred to later. 4. The message deals with events in time. 4. The message deals with location in space. 5. The message calls for immediate action. 5. The message does not call for immediate action. 6. The visual system of the person is overburdened.

6. The auditory system of the person is overburdened.

7. The receiving location is too bright or dark. 7. The receiving location is too noisy. 8. The person's job requires him to move about continually.

8. The person's job allows him to remain in one position.

In addition to information presentation modality, another concern is what

specific information is conveyed. When placed in a supervisory control role, a human

operator may have to respond to an interruption to a primary task of monitoring. To

respond, the operator needs an understanding of what has occurred in the system,

either as an output of an action or as a result of an unseen change (Scott, Mercier,

Cummings, & Wang, 2006; St.John, Smallman, & Manes, 2005). A potential way to

support operator understanding of a monitored task’s state is to continuously present

information so that the operator can immediately determine a task’s current state, as

well as projected future states.

The objective of this research is to explore ways to combine audio displays with

visual displays to support supervisory tasks. In particular, this research focuses on

comparing continuous to discrete audio displays to understand the effects of a constant

versus discrete presentation of information. Discrete audio displays play an alert once

for about a second when a monitored task exceeds limits. In contrast, continuous audio

displays are audio displays that always indicate some system state. Chapter 2,

Background, will further frame the context of the research of this thesis.

13

1.1. Problem Statement

The primary questions addressed through this research are:

1. When compared with discrete audio, does continuous audio better aid human

supervision of UAV operations?

2. Is the effectiveness of the discrete or continuous audio support dependent on

operator workload?

Three steps are followed in this thesis to address the research questions: (1) a

multiple‐UAV simulator, the Multiple Aerial Unmanned Vehicle Experiment (MAUVE)

test bed, is selected, (2) audio displays are developed and integrated into this multiple‐

UAV simulator, and (3) human operator performance is tested in the MAUVE

environment to compare continuous and discrete audio displays.

1.2. Thesis Organization

This thesis is organized into seven chapters:

• Chapter 1, Introduction, provides the motivation of this research, research

questions, and the research objectives of this thesis.

• Chapter 2, Background, reviews the current Unmanned Aerial System (UAS)

environment and the needs of the human operator supervising UAS operations.

The chapter then frames the research of this thesis in terms of meeting the needs

of human operators supervising UAS operations.

• Chapter 3, Simulator and Interface Design, presents details of MAUVE and the

associated tasks operators supervise on MAUVE. The chapter then defines the

four audio alerts tested in this experiment with a technical description of the

alerts and their functions.

14

• Chapter 4, Method, presents the research questions of the experiment, the

experimental apparatus, the experimental task, the participants, the independent

and dependent variables, and the data collection.

• Chapter 5, Results, discusses the statistical analysis results. The chapter also

provides a description of the participants’ post‐test subjective feedback.

• Chapter 6, Discussion, synthesizes the applicable lessons that can be extrapolated

from the experimental results.

• Chapter 7, Conclusion, reviews the answers to the research questions, discusses

continuous audio integration in the real world, and suggests areas for future

research.

15

2. Background

This chapter highlights the current shift toward unmanned systems and the

needs this shift generates for controlling these unmanned systems. It presents attention

theories linked to possible solutions for aiding human operators in supervising

unmanned systems. Further, this continuous audio research is framed in the context of

previous research in audio display support of supervisory tasks. Finally, a discussion

of the research hypotheses and the groundings of these hypotheses closes the chapter.

2.1. UAS Evolution

2.1.1. The Current Push

Senior military officials see the need for integrating more UAV operations in

support of the troops on the ground in Iraq and Afghanistan (USAF, 2007; Whitney,

2007). Four star Combatant Commanders (COCOMs) highly desire unmanned systems

for the many roles these systems can play (Sullivan, 2008; DOD, 2007). Major

advantages of unmanned systems are that they are cheaper and have more endurance

than manned aircraft (Barbato et al., 2003; Gates, 2008). Many of the integral roles that

COCOMs may envision for UAVs are encapsulated in recent work by Nehme, Crandall,

and Cummings (2007), which includes a taxonomy of all the current and potential roles

that UAVs can perform with today’s technology (Figure 2‐1).

Unmanned vehicles are changing how warfighting is conducted, and they are

vital to providing intelligence and reconnaissance for troops on the ground. As of

October 2006, the DOD has used hand‐launched UAVs to fly over 400,000 hours of

support missions for Operation Enduring Freedom and Operation Iraqi Freedom (DOD,

2007). The Navy used UAVs in the first Gulf War, with Pioneer UAVs performing

17

Figure 2‐1: Possible UAV Missions (Nehme et al., 2007).

reconnaissance missions (Banks, 2000). Currently, the Navy is developing a vertical‐

take‐off‐and‐landing tactical UAV (VTUAV), designated the MQ‐8B Fire Scout. The

Navy plans to have Fire Scout operational and throughout the fleet in fiscal year 2009

(Schroeder, 2008). The Army has 400 UAVs in theater for the Iraq and Afghanistan

wars (Sullivan, 2008). Currently, the Army has 785 UAVs, comprised of Raven,

Shadow, Hunter, and Warrior systems, with a proposed increase to 4,755 UAVs by 2023

(Sullivan, 2008). The Air Force, too, is transitioning toward the use of unmanned

vehicle technology. Since 2001, the Air Force has reduced fighter inventory by 152

aircraft, while simultaneously increasing UAS platforms by 113, with intentions of

continuing this over the next several years (Randolph, 2007).

There is a need for automation and technology to better support human

involvement in managing these unmanned systems. From January to October 2007, the

number of hours for mission sorties doubled for UASs in the Air Force, creating a

manning crisis. Near the end of 2007, the Air Force shifted 120 pilots out of the manned

cockpit to the ground control stations for UASs (Staff, 2008). A technological solution

proposed by many DOD senior leaders and researchers to alleviate this manning crisis

is to move from a team of operators controlling a UAV to having a single operator

18

controlling multiple UAVs (Barbato et al., 2003; Culbertson, 2006; Tsach et al., 2007).

Figure 2‐2 illustrates this transition.

Team of people flying

one UAVOne person supervising

mission of multiple UAVS

Team of people flying

one UAVOne person supervising

mission of multiple UAVS

Figure 2‐2: Transition to Multiple‐UAV Supervision.

2.1.2. The Human in the System

The two terms UAV and UAS may appear interchangeable. However, there is a

key distinction between them. The Office of the Secretary of Defense has issued the

following definition:

Unmanned Vehicle. A powered vehicle that does not carry a human operator,

can be operated autonomously or remotely, can be expendable or recoverable,

and can carry a lethal or nonlethal payload…Unmanned vehicles are the primary

component of unmanned systems (DOD, 2007).

The key point is that the UAV is a sub‐component that helps comprise the overall UAS.

Another key sub‐component to the UAS is the human operator controlling UAV

operations through a ground control station.

The term unmanned does not imply removing the human completely because a

human is still involved in unmanned operations, even in the most autonomous

situation. The human’s role is not eliminated just because the human is no longer co‐

located with the mission (DOD, 2007). Even as autonomy progresses, the need for

19

human reasoning and judgment will never be replaced within the UAS (Economist,

2007). Furthermore, the DOD has stated that humans are still necessary “to interpret

sensor information, monitor systems, diagnose problems, coordinate mission time lines,

manage consumables and other resources, authorize the use of weapons or other

mission activities, and maintain system components” (DOD, 2007). Thus any UAS must

be designed to support the role of the human operator in establishing the system’s goal,

supervising the system to guide it toward the goal, and ensuring mission success

(Barbato et al., 2003; DOD, 2007). The human operator’s supervisory role is essential to

UAS operations. This thesis explores areas in which new technology can support the

single, human operator in supervising the operations of multiple UAVs.

2.1.3. The UAV Ground Control Station

To support the supervision of UAS operations, an understanding of the current

UAV ground control stations is needed. Today, the Air Force and Army operate most

UAV missions from trailers or the backs of trucks. Most operations rely primarily on

visual displays. In the Air Force, the solution for providing additional missing

information to UAS operators has been to add more visual displays. However, humans

have a limited capacity of resources to process information, so adding more visual

displays will only further overload operators (Wickens et al., 2004). One solution

explored by this research to help humans process more information is adding audio

displays that can draw upon more attentional resources without adding significantly to

the workload.

Operating in field conditions, UAS ground control environments can be very

noisy and aurally cumbersome, with generators running in the background and

required communication between inter and intra team members. This is significant to

note when considering auditory displays. Any implementation of an audio alert in

20

these types of field conditions will have other audio alerts competing for attention and

only certain available frequencies over which alerts can be generated and heard.

2.2. Attention Theories

The processes by which humans acquire and aggregate information for proper

decision making and taking action rely upon attention resources (Parasuraman,

Sheridan, & Wickens, 2000). Understanding these processes is important in the design

of UAV supervisory control because various types and amounts of attention are needed

for the multiple tasks performed by the UAVs. This section discusses different attention

types and the allocation of attention resources that are used in human cognitive

processing.

2.2.1. Selective, Divided, and Sustained Attention

The primary task for an operator overseeing multiple‐UAV operations will be to

simultaneously monitor particular tasks across multiple UAVs. This will require the

UAV operator to time‐share between the tasks to maintain continual awareness and

respond at the appropriate time to each task. Further, the missions will occur over

prolonged periods of time, during which the human operator will monitor for abnormal

conditions and also provide high‐level directives to the semi‐autonomous UAV

operations. Therefore, it is assumed that in the supervisory control role, UAV operators

will primarily perform their role with selective, divided, and sustained attention.

Selective attention occurs when an operator must “monitor several sources of

information to determine whether a particular event has occurred” (Sanders &

McCormick, 1993). An example of selective attention in multiple‐UAV operations is

when a sensor operator is watching multiple UAV video feeds to locate a particular

enemy vehicle.

21

Divided attention occurs when “two or more separate tasks must be performed

simultaneously, and attention must be paid to both” (Sanders & McCormick, 1993).

Continuing with the same example, the sensor operator may be forced to rely on

divided attention if he is required to keep track of a vehicle he has located, as well as

continuing to monitor the other video inputs for potential enemy vehicles that require

some action. This type of attention requires a form of time‐sharing, where the operator

splits his time between tasks.

Sustained attention occurs when an operator “sustains attention over prolonged

periods of time, without rest, in order to detect infrequently occurring signals”

(Parasuraman, Warm, & Dember, 1987; Sanders & McCormick, 1993). This is generally

the bulk of a UAV surveillance mission. The sensor operator in the previous example

will spend a majority of his time simply watching for a change in the videos that would

indicate an enemy vehicle.

2.2.2. Attention Resource Theories

To maintain a high level of performance, an operator must manage his

attentional resources. A scarcity of resources exists, and this limited pool of resources is

what is drawn upon by the various attention types (Wickens et al., 2004). Three

hypotheses proposed in the literature (Hirst, 1986) to explain attention resources are as

follows:

1. One central resource – a single resource from which all tasks draw.

2. Multiple resources – with some tasks drawing from some resource

pools and not others. For example, verbal tasks draw from a verbal

resource pool and not a visual resource pool.

3. Both a multiple resources pool as well as a central resource pool.

22

In general, these theories assume a scarcity of resources. The single resource

theory holds that there is a single pool for attention resources that is drawn upon for all

the mental processing (Kahneman, 1973; Moray, 1967). It proposes that humans have

limited cognitive resources in this single pool and that when some of these resources

are allocated, alternative tasks can only use what remains unallocated. There are some

problems with the single resource theory. First, when a task is actually performed on a

single processing code (e.g., verbal, spatial) or modality (e.g., visual, auditory), more

attention resources appear to be required than when the task is performed using

multiple processing codes or modalities. Second, within some groupings of tasks, if the

difficulty of one task increases, there appears to be no effect on the other tasks.

However, in groupings with similar tasks, when the difficulty of one task increases, it

seems to affect the performance on the other tasks (Sanders & McCormick, 1993;

Wickens & Hollands, 2000). There appears to be interference and failed time‐sharing

within certain task groupings, but not others. These problems indicate that perhaps

there are separate attentional resource pools for the different processing codes or

modalities.

2.2.3. Multiple Resource Theory

As opposed to the single resource theory, the multiple resource theory proposes

that there are multiple independent pools from which attentional resources are drawn.

As a result, there can be increased efficiency when time‐sharing between multiple tasks

(Sanders & McCormick, 1993). Interference, discussed in the single resource theory

above, is a result of multiple tasks requiring resources from the same pool. Multiple

resource theory propounds that information can be processed concurrently by dividing

presentation over various channels so that the information is processed simultaneously

by different pools of attentional resources (Wickens et al., 2004).

23

Multiple resource theory holds that the effect one task has on another depends

on the type of tasks. An example is when someone tries rubbing his stomach while

patting his head. This is difficult because he has run out of attentional resources, while

speaking and rubbing one’s stomach is completely possible because there are ample

attentional resources. Wickens and Hollands (2000), in their model in Figure 2‐3,

explains why. Their model provides four dimensions to classify which resource pools

attentional resources are coming from: stages, input modalities, processing codes, and

responses. The example of patting one’s head and rubbing one’s stomach is an

illustration of two manual responses, which require resources from the same pool.

While rubbing one’s stomach and speaking, respectively, are illustrations of manual

and vocal responses. Because they are different response types, they have different

attentional pools of resources to draw upon.

Figure 2‐3: Attentional Resource Pools (Wickens & Hollands, 2000).

The focus of this thesis is on using aural and visual input modalities to aid in the

multiple‐UAV supervision task. Input modalities are the channels over which

information may be presented to an individual. Research has shown that if a task’s

information is presented in dual coding, on differing modalities, such as the auditory

and visual sensory channels, time‐sharing is more efficient (Miller, 1991). In contrast, if

24

tasks are redundantly coded, on the same modality, such as the auditory and auditory

or visual and visual sensory channels, interference occurs (Sanders & McCormick,

1993). Research by Streeter, Vitello, and Wonsiewicz (1985) showed human operators

can better process aural commands for driving, while the drivers are focusing their

visual attention on the road.

Previous research has shown that dual coding, i.e., displaying information on

two separate modalities is beneficial (Miller, 1991; Streeter et al., 1985). For example, it

has been found that using both visual and audio multimedia in unison is a useful way

to present information (Mayer, 1999). Similarly, it should be beneficial to dual code

information in both the visual and audio modalities to aid operators in monitoring UAV

operations. The premise of this research is that by dividing the monitoring of tasks over

multiple modalities, the operator will better be able to integrate all attentional resource

pools toward efficiently controlling multiple tasks. In fact, this is not a new idea. In a

meta‐analysis of 43 previous studies (Burke et al., 2006), researchers concluded that in

UAV operations dual coding information on the visual and auditory or visual and

tactile modalities increases performance, while potentially decreasing workload. One

study in the UAV domain compared a visual display with a visual display

supplemented with an audio alert and found enhanced performance with the latter.

This same study also showed that when going from supervising one UAV to

supervising two UAVs, operator performance decreased, regardless of the display

(Dixon, Wickens, & Chang, 2005). This thesis explores continuous audio alerts which

have not yet been combined with visual displays for supporting UAV operators.

2.3. Audio Research

This thesis focuses on the use of continuous audio alerts, in particular,

sonifications, in a multi‐modal display to help an operator supervise multiple tasks

25

across multiple unmanned aerial vehicles. The existing audio literature is approached

from three perspectives. First, audio research for supervisory control with multi‐modal

displays is reviewed to help frame what types of supplemental audio displays have not

been researched for application with UAV supervisory tasks. Second, a form of audio

that has not been extensively researched in its ability to aid supervisory control is

continuous audio. For this reason, a review of applicable continuous audio research is

completed. Third, a form of continuous audio that portrays relations in information

and information changes is sonifications. In closing this section, a review of key

supervisory control sonifications research is accomplished.

2.3.1. Supervisory Control with Multi‐Modal Displays

A significant body of audio research applicable for supervisory control of

multiple unmanned vehicles focuses on spatial audio (Nehme & Cummings, 2006).

Spatial audio is 3‐dimensional (3D) audio in which audio is presented so that specific

signals come from specific locations in the 360‐degree range around someone’s head.

An example is an audio alert for one UAV presented over a headset directly in front of

the operator, while alerts for another UAV are connected to signals directly behind the

operator.

The spatial audio research applicable to supporting operators performing

supervisory tasks has primarily focused on sound localization (Martin, McAnally, &

Senova, 2001; Wightman & Kistler, 1989), detection and localization of visual targets

(Bolia, DʹAngelo, & McKinley, 1999; Bronkhorst, Veltman, & vanBreda, 1996; Flanagan,

McAnally, Martin, Meehan, & Oldfield, 1998; Nelson et al., 1998), collision avoidance

(Begault & Pittman, 1996), and navigation (Moroney et al., 1999). In the aviation

domain, this research has primarily been applied in the manned cockpit; because of the

supervisory nature of these tasks, this research is applicable in the unmanned cockpit as

well. In particular, spatial audio has been shown to help reduce search times and

26

produce faster response times for target acquisition (Parker, Smith, Stephan, Martin, &

McAnally, 2004) and alert and warning detection (Begault & Pittman, 1996). The

literature suggests that spatial audio helps mitigate visual change blindness effects,

which occur when an operator is so focused visually on one task that the operator’s

focused attention causes another visual warning to be missed (Nehme & Cummings,

2006). Outside of spatial audio, little other audio research has been done in the

supervisory domain of multiple UAVs. In combing the audio literature, little applicable

research is found where ambient noise headphones and associated filtering techniques

are used to support a supervisory task in a multi‐modal display (Nehme & Cummings,

2006). A form of this audio is continuous audio, which is explored next.

2.3.2. Continuous Audio

Continuous audio alerts play continually to map information on the state of a

task. The contrast to continuous audio is discrete audio, which only plays at a discrete

point in time when a task’s state changes. Discrete audio can be used to aurally

interrupt an operator to notify him of a problem. Continuous audio is essentially a

peripheral monitoring system, which allows the operator to constantly monitor for

abnormal situations; in this case, an operator is using divided attention to time‐share

their focus between performing or monitoring multiple tasks, with the audio as a cue

guiding when to shift focus.

Most audio alert research has focused on discrete alerts in the forms of a “beep”

and “ding” (Brewster, Wright, & Edwards, 1994). However, a limited number of formal

human studies have investigated continuous audio alerts (Pacey & MacGregor, 2001).

The continuous audio research that has been completed has focused on using

continuous audio and discrete audio alerts to enhance a visual display for a monitoring

task. Research has shown the addition of continuous audio with a visual representation

enhances operator performance (Crease & Brewster, 1998; Pacey & MacGregor, 2001).

27

The following paragraphs will discuss the details of these experiments that have been

run with continuous audio.

One of the first experiments run with human participants was by Crease and

Brewster (1998). Participants completed a primary task of typing poems, while

monitoring the downloading of files as a secondary task. Participants had either a

visual progression display of the downloading or a visual and auditory progression

display. The results of the experiment showed that participants with the continuous

audio display were significantly quicker at responding than the participants with only

the visual display. This is an example of how displaying the information on multiple

modalities enhanced operator performance for a monitoring task.

In a similar experiment, Pacey and MacGregor (2001) compared a baseline visual

display with the visual display plus one of three different audio displays: discrete

chimes, continuous wind, and a continuous audio progress bar. The primary difference

between the audio displays was the amount of information presented by the display.

The discrete chimes presented two of the task’s properties, i.e., when a task began and

when the task ended, with separate discrete chimes that played when an event began

and when an event ended. The continuous wind was a low wind noise that played

continually from when a task began until it ended. It added a third piece of

information; as long as the wind was playing, the operator knew that the task was still

occurring. The continuous audio progress bar presented six of the task’s properties,

such as scope, initiation, progress, heartbeat, remainder, and completion, through

continuous audio created by different arrangements of 12 different notes; this audio

played continually throughout an event. While the results were mixed dependent upon

what performance metrics were considered, it is clear that adding the audio alert to the

visual display significantly enhanced the participants’ ability to complete the primary

and secondary tasks. Pacey and MacGregor’s findings showed that when simply

considering the reaction times for catching the downloads ending, the audio progress

28

bar was significantly better than the chimes, but that the chimes were significantly

better than the wind or visual‐only display. However, as indicated by the primary task

metrics, typing speed and accuracy, participants using the chimes performed

significantly better than the auditory progress bar or visual‐only display. The

conclusion, therefore, is that the chimes provided the best results of enhancing the

ability to perform the secondary task of catching download endings, without

hampering the primary task of typing poetry.

With Pacey and MacGregor’s mixed results on the effects of discrete versus

continuous audio support on both a secondary and primary task, this thesis will further

explore whether monitoring tasks in supervisory control is better supported by a typical

discrete audio alert or a by a continuous audio alert. Further, this thesis will explore

using both discrete audio and continuous audio in a single audio condition to support

multiple supervisory tasks. Because of the limited research available on continuous

versus discrete audio, the results of this thesis will be an important contribution to the

audio alert research.

2.3.3. Supervisory Control Sonifications

Continuous audio is defined above as audio alerts that play continually to

provide continual information on a task’s state. Sonifications are a form of continuous

audio, in which data and relations in data are mapped to relations in the audio display

(Kramer, 1994). An example of a sonification that Tannen (1998) gives is sonar on a

submarine that indicates the distance of a torpedo. As the torpedo gets closer, the sonar

beeps become more intense on a continual scale. The intensity of the beeps provides

relational data of how far the torpedo is from the submarine. Sonifications are a subset

of continuous audio. The torpedo example is sonification in particular, because the

intensity of beeping noise provides the relational data of how far the torpedo is from

the submarine.

29

Research has shown sonifications to be beneficial in medical and assistive

technologies, scientific visualizations, engineering analysis, emergency alerts, and

aircraft cockpits (Barrass & Kramer, 1999). One example in the medical domain is a

study by Loeb and Fitch (2002) that compared three types of anesthesiology displays:

visual only, auditory only, and a combined visual and auditory display. The task for

participants was to monitor if any of six vital signs exceeded normal limits. For the

auditory displays, participants received independent streams of audio for the vital

signs. For the heartbeat, the audio was a low‐pitched, repetitive, thudding to resemble

what one may envision the heart sounding like. For the respiratory system, a higher‐

pitched amplitude‐modulated signal was used. For the remaining four vitals,

variations of modulation and filtering were used to generate the audio. The

participants with the combined visual and audio display had the quickest reaction

times, but the visual‐only display produced the most accurate reactions.

In a similar anesthesiology study, Watson and Sanderson (2004) asked

participants to divide their attention between monitoring for an abnormal event and

performing some other primary task. Thus, monitoring was a secondary task. In

particular, they tested various modalities in supporting anesthesiologists and non‐

anesthesiologists in monitoring a respiratory rate as the secondary task. There were

three displays tested: sonifications only, visual only, and combined sonifications and

visual. The sonifications were respiratory sonifications in which the breathing rate was

mapped to a pure tone with musical notes on the third interval, for inhalation and

exhalation. Watson and Sanderson (2004) results illustrated that regardless of the

display modality, the performance was the same for the secondary task of monitoring

the breathing, but that for the time‐shared primary task, participants performed best

with the combined display. These results show the benefit of the combined display in

helping participants time‐share. However, because only sonifications were used as the

audio support in these experiments, there is no comparison between sonifications and

30

discrete audio. This comparison is what this thesis will explore. Further, these results

are limited in that they only tested the effect of the displays on an anesthesiologist

monitoring a single patient. This thesis will analyze the benefits of sonifications while

monitoring multiple tasks (UAVs) as opposed to a single task.

2.4. Research Hypotheses

Having framed the research of this thesis, two hypotheses are proposed. The

first hypothesis is that in comparison with a discrete audio scheme, continuous audio,

in the form of a sonification, improves operator performance for monitoring sustained

and divided attention tasks in a time‐intensive dynamic environment. Continuous

audio reduces the uncertainty of a future state by allowing operators to monitor trends

in a changing state over a secondary modality. Further, by placing this information

over the audio channel, there is immediate access to the information. Immediate

information access is important to performance because humans tend to re‐check

signals repeatedly because of limited human memory (Moray, 1981).

The second hypothesis is that the benefits of continuous audio will be more

beneficial for operators controlling multiple vehicles rather than a single vehicle, since

the workload under the multiple case is significantly higher. Because the monitoring of

multiple vehicles increases workload, more attentional resources are required from

supervisors. The addition of the audio displays to the visual display provides an

additional pool from which to draw attentional resources for processing some of the

information for managing the UAVs. Furthermore, the continuous audio allows for

more information to be presented over the audio sensory channel than the discrete

audio because it provides a continual status. Because it provides more information than

the discrete audio, the expectation is that in a high workload situation, where the

attentional resources are fully utilized, the additional continuous audio will allow

31

operators to draw more attentional resources from the audio resource pool, which will

better support their performance.

To test these hypotheses, a multiple‐UAV simulator experiment was conducted,

which will be discussed in more detail in Chapters 3 and 4.

32

3. Simulator and Interface Design

The Multiple Aerial Unmanned Vehicle Experiment (MAUVE) test bed was

developed in previous research to allow human operators to perform supervisory

control tasks as they monitor the progression of multiple UAVs arming and firing on

targets, performing battle damage assessment, and avoiding threat areas (Mitchell,

2005). The following sections will present how MAUVE was customized for this

experiment and the form of the audio displays that were created and integrated into the

MAUVE interface for this experiment.

3.1. Multiple Aerial Unmanned Vehicle Experiment (MAUVE) Test Bed

MAUVE is a two screen interface that allows the simulation of any number of

UAVs conducting strike operations in hostile environments. MAUVE, for this

experiment, allowed the operator to arm and fire on targets with various priorities,

while monitoring each UAV’s flight of path, event timeline, and radio traffic.

The MAUVE test bed provided a map for geo‐spatial tracking of UAVs on their

preset flight paths, along with an interaction control panel for making UAV control

inputs (Left Display, Figure 3‐1). Control inputs included arming and firing a UAV to

destroy targets, directing specific path changes, and acknowledging radio calls. For

each control input, operators simply clicked one or two buttons on the control panel

(Figure 3‐2).

A timeline for each UAV, a scheduling decision support visualization (DSV) for

each UAV (which will be discussed in detail in a subsequent section), and chat

interfaces were also provided (Right Display, Figure 3‐1). This timeline display helped

operators ensure UAVs would be on time to targets by monitoring when UAVs would

arrive at targets and in what phase of the mission they were currently located. In

33

Map Display Timeline Display

Figure 3‐1: Multiple Aerial Unmanned Vehicle Experiment (MAUVE) Test Bed.

Figure 3‐2: UAV Interaction Control Panel

34

Table 3‐1: UAV Color‐Coded Flight Phases

UAV Action Color

Enroute Gray

Loitering Blue

Arming Payload Yellow

Firing Payload Orange

Return to Base Green

particular, the timeline used the color‐coding provided in Table 3‐1 to represent the

phase of flight.

The general task for MAUVE operators for this experiment was to monitor one

or more UAVs on a preplanned mission involving several waypoints and targets. The

primary task of the operator was to ensure weapons were dropped on the correct

targets during the specified windows of opportunity. However, unexpected head winds

or crosswinds could cause UAVs to slow their speed or drift off course, requiring the

operator to take corrective action for projected target late‐arrivals or course‐deviations.

Late‐arrivals were a higher priority than course‐deviations. In addressing late‐arrivals,

the priority of the target (high, medium, or low) also had to be addressed, because a

high priority target was more important than a medium priority target, and so on. In

addition, operators also had to ensure the UAVs would return to base by a specified

time.

Figure 3‐3 provides the visual representation for a late‐arrival. A black rectangle

on the timeline showed when the UAV was going to arrive at the labeled target, and it

changed from black to red when the UAV was going to be late. This could also be seen

when the black rectangle moved past the orange window of opportunity. On the

timeline below, the red rectangle labeled “T‐1L” indicates that the UAV will be late to

35

target T‐1L. In contrast, the black rectangle labeled “T‐2M” indicates that the UAV will

be on time to target T‐2M.

Figure 3‐3: Late‐arrival Illustration.

The decision support visualization (DSV) was an additional visual tool used

when a UAV was late to a target. The DSV was designed to illustrate the current state

of a UAV and the projected future what‐if states of a UAV. The DSV was located at the

right edge of each UAV’s timeline in the Timeline Display (Figure 3‐1). For example,

the DSV in Figure 3‐4 illustrates that the UAV is late to a high priority target by having

a dark gray rectangle with an H in the rectangle’s center in the late‐arrivals section of

the DSV. After the operator checks the timeline and sees that the UAV is going to be

late to T‐7H, he highlights T‐7H on the DSV, posing a what‐if. The DSV now shows the

projected future state in the projection section. If target T‐7H’s time on target is

delayed, the UAV will still be late to a high priority target. The light gray rectangle

with the H within the rectangle of Figure 3‐4 illustrates that the UAV will still be late to

a high priority target.

H

H

Figure 3‐4: Decision Support Visualization (DSV)

The green line in Figure 3‐5 indicates the plotted and desired flight path.

Anytime the UAV completely departed from the green course line, the UAV was

36

considered to have deviated. During the simulation, the UAV constantly fluctuated

back and forth on the flight path, but participants were trained to respond when the

UAV actually was no longer touching the course line. Below are images illustrating

UAV 3 on course, almost off course, and off course.

On Course Almost Off Course Off Course

Figure 3‐5: Course‐deviation Illustration.

Two tasks were selected in MAUVE for continuous audio (sonification)

representation. Both were secondary monitoring tasks, and both were also intertwined

with the primary task in a way that the secondary task had to be completed for the

primary task, firing on the various targets on the flight path, to be successfully

achieved.

The first secondary task was monitoring for late‐arrivals to targets. Each target

was either a high, medium, or low priority target. On time arrivals and late‐arrivals to

high, medium, or low priority targets were four discrete events to which the continuous

audio had to be mapped. Continuous audio could be used to represent the state of

UAVs in that the audio would play continually, indicating which of the four states a

UAV was in at any point in time. In particular, for this setup the continuous audio was

modulated so that it would convey the priority of the target to which one was late. For

example, as will be discussed later, there were modulated increases in the audio to

indicate a high priority target in contrast to a low priority target. That is what made

this a sonification.

Monitoring for course‐deviations was the other event to which continuous audio

was mapped. This was an ideal continuous event in that the UAV could start on course

37

but then on a continual scale progress away from the course line. Again, this changing

continuous audio was mapped to the rate of change with which a UAV traveled off

course, making the audio a sonification.

3.2. Four Auditory‐alerts

Four auditory alarms were created to test the effect of discrete and continuous

alarms in helping operators monitor for late‐arrivals and course‐deviations in MAUVE.

Presented in Figure 3‐6, these four auditory‐alerts were 1) an oscillating course‐

deviation alert, 2) a modulated late‐arrival alert, 3) a threshold course‐deviation alert,

and 4) a threshold late‐arrival alert.

Figure 3‐6: Four Auditory‐alerts.

The oscillating and modulated alerts were continuous alarms; they provided

continual audio information about the state of a particular variable within the

simulator. Instead of indicating a clear distinct threshold, the oscillating and

modulating alerts gradually increased as the severity of either the late‐arrival or course‐

deviation condition increased. The threshold alerts were designed to be discrete

38

alarms; they emitted a single discrete audio alert when some precondition was met.

These four auditory alerts were presented to MAUVE operators through the HDiSP

(Headset Display), which will be discussed further in the next section.

3.3. Sensimetrics HDiSP

All audio alerts were presented equally in both ears through the Sensimetrics

HDiSP1. Pictured in Figure 3‐7, the HDiSP is an over‐the‐head headset with sealed

around‐the‐ear ear cups. The HDiSP provides ambient noise attenuation ranging from

35 to 40 dB for the spectrum between 500 and 8 kHz. The headset has integrated digital

signal processors, which produce the signals used in this experiment based on

generative audio filters and processing of the ambient signal, which was received from

microphones mounted on the headband. The next section will provide the technical

description of how each alert was displayed.

Figure 3‐7: Sensimetrics Headset Display (HDiSP).

3.4. Technical Description of Auditory‐alerts

The auditory‐alerts created for this experiment were generated through

collaborative work between MIT’s Humans and Automation Laboratory and Charles

1 Contact Dr. Thomas Edward von Wiegand, at [email protected], for further information on the Sensimetrics HDiSP.

39

mailto:[email protected]

River Analytics, Inc. The primary consideration in designing the audio alerts was to

create audio that would capture the human operator’s attention and effectively convey

the necessary information to the operator. To design to these parameters, there were a

few guidelines followed Deatherage (1972) and Sanders and McCormick (1993)).

Appendix A has a compilation of the design principles. Further, each audio alert was

designed to always alert to the worst condition. Therefore, in the multiple‐UAV

condition, where multiple UAVs could be having course deviations, the audio alert

played for whichever UAV currently was furthest off course.

The oscillating course‐deviation alert represented both the existence and severity

of the UAV course‐deviations. The alert consisted of comb filters that were applied to a

mix of pink noise and the ambient signal. Using pink noise means that filters equalized

the energy of sound at each octave level, resulting in a constant level presentation of the

audio. The mix ranged from 0.2 pink noise for low deviation to 0.9 pink noise for high

deviation. The comb filters had a base delay of 0.2 ms, with a 50 percent mix of the base

signal and a feed forward delay. The delay values were then oscillated to create a

periodic signal. Because this was a continuous audio scheme, the oscillating course‐

deviation alert played continually to provide an auditory image of UAV path position.

As a UAV drifted further off course, the frequency of oscillation of the comb filter delay

decreased from 17 Hz to 4.5 Hz, and the depth of oscillation increased from 0.2 ms to 0.7

ms, on a continual scale.

The threshold course‐deviation alert consisted of a single beep with a

fundamental frequency of 1000 Hz. The beep lasted around 8 ms and played whenever

a UAV moved completely away from the UAV’s plotted course line.

The modulated late‐arrival alert consisted of discrete harmonic signals

continuously playing to indicate a projected late‐arrival at a target. These sounds

played continually until the operator addressed the issue with the corrective action of

choosing to delay the UAV, skip a target, or the issue disappeared because the UAV

40

automatically continued past the target when the UAV was going to be late to the next

target. The harmonic signals were composed of five Formant filters that were applied to

a mix of pink noise and the ambient signal. During the condition of no late‐arrivals, a

baseline audio condition was generated with two filters set to 261.6 Hz, two filters set at

329.4 Hz, and one filter set at 392.0 Hz (a major C triad with extra balance in the root

and third). If the UAV was late to a low priority target, a signal was generated with two

filters set to 261.6 Hz, two filters set at 311.1 Hz, and one filter set at 392.0 Hz (a minor C

triad with extra balance in the root and third). If the UAV was late to a medium priority

target, a signal was generated with three filters set to 293.6 Hz and two filters set at

415.3 Hz (a tritone interval with a D root). If the UAV was predicted to be late to a high

priority target, a signal was generated with three filters set to 369.9 Hz and two filters

set at 523.25 Hz (a tritone interval with an F# root). As the priority increased, the pink

noise mix also increased, from 0.25 for the baseline, to 0.7, 0.8, and 1.0 for the three

priority levels.

Differing from the threshold course‐deviation alert, the threshold late‐arrival

alert consisted of a single beep with a fundamental frequency of 415 Hz, instead of 1000

Hz. In addition, the beep lasted around 18 ms, instead of 10 ms, and played whenever a

UAV was projected to be late to any target. A UAV was determined late to a target

when the UAV had exceeded the UAV’s window of opportunity to destroy the target

and was not going to be able to get over the target in time to fire on it during the

allotted time window in which the UAV was required to be at the target.

41

43

4. Methods

After integrating the audio alerts discussed in Chapter 3, Simulator and Interface

Design, into MAUVE, an experiment was conducted to compare continuous audio and

discrete audio alerts in aiding single operator awareness in controlling multiple

unmanned vehicles. This chapter will review the experimental questions, the setup of

the experiment, and the design of the experiment for evaluating the research questions.

4.1. Experimental Questions

The objective of this experiment was to determine whether continuous audio

helps maximize the information conveyed to UAV operators more efficiently than

discrete alarms, which are typically used in current ground control stations. In

addition, the impact of continuous versus discrete alerting on operators moving from

control of single to multiple UAVs was also a research question.

To explore these objectives, two null hypotheses were tested in this research:

1. For various combinations of discrete and continuous alerts, there is no difference

in the operator’s performance.

2. There is no interaction between the discrete and continuous alert combinations

and whether the operator is controlling one or multiple UAVs.

The alternative hypothesis for the first null hypothesis, therefore, was that there is a

difference, and for the second null hypothesis, that there is an interaction.

4.2. Experimental Apparatus

The experiment was administered in a testing room with a background ambient

noise level of ~64dB on a C‐scale. MAUVE was run on a multi‐modal workstation

(MMWS) (Osga, VanOrden, Campbell, Kellmeyer, & Lulue, 2002). Pictured in Figure

4‐1, the MMWS is a four‐screen computer display.

Figure 4‐1: Multi‐Modal Workstation (MMWS) (Osga et al., 2002).

The three screens across the top were 21 inches and set to display at 1280 x 1024

pixels, 16 bit color resolution, and the 15 inch bottom center screen was set at 1024 x 768

pixels, 32 bit color resolution. The computer used to run the simulator was a Dell

Optiplex GX280 with a Pentium 4 processor and an Appian Jeronimo Pro 4‐Port

graphics card. Participants controlled the simulator through a generic corded computer

mouse. Throughout the experiment, the top left display showed the mission objectives

and priorities, and the bottom center one displayed the color‐coding for MAUVE (Table

3‐1). The top center display contained the left MAUVE map and interaction display, and

the top right display included the right MAUVE timeline and decision support display

(Figure 3‐1).

44

For this experiment, the operator was given the following as the primary

objective for each mission: Make sure the UAV(s) maximize the number of targets

engaged as well as arrive back at the base safely.

Further, supervision of each of the UAV(s) for each mission was broken down

into the following prioritized sub‐tasks, from highest priority to lowest:

1. Return to base (RTB) within the time limit for the mission (this limit was clearly

marked).

2. Comply with recovery rules for course‐deviations.

3. Comply with recovery rules for target late‐arrivals.

4. Destroy all targets before their time on target (TOT) window ends..

5. Avoid taking damage from enemies by avoiding all threat areas.

6. Acknowledge all “Push” radio calls.

Participants were trained to follow these priorities, and the objectives and

priorities above were displayed to them throughout the experiment. Within the

priorities, recovery rules of how to recover the UAV from a course‐deviation or late‐

arrival were also displayed. The instructions on how to comply with recovery rules for

course‐deviations were to click “Reset Navigation” on the control panel (Figure 3‐2).

This theoretically reset the onboard navigation in the UAV and caused that UAV to fly

back to the planned flight path.

The instructions on how to comply with recovery rules for target late‐arrivals

were as follows:

• Low Priority – Click “Skip Target.”

• Medium Priority – Click “Skip Target” or employ the decision support

visualization (DSV).

• High Priority – Employ DSV before requesting delay or clicking “Skip Target.”

45

For low priority targets, clicking the “Skip Target” button on the control panel (Figure

3‐2) would cause the UAV to proceed immediately to the next target. For medium and

high priority targets, the option was given to use the DSV. As discussed in Chapter 3,

Simulator and Interface Design, the DSV (Figure 3‐4) is a tool that illustrates the effect that

delaying the time on target for one late target has on the remaining targets for that

UAV’s flight plan. With the assistance of the information presented in the DSV,

participants then decided whether they should click the “Skip Target” button or the

“Request Delay” button on the DSV. If they clicked “Request Delay,” participants then

had to wait to see if a delay was granted, and if the delay was granted, the UAV was

then no longer late to the target and would be able to destroy the target. For a high

priority target, the immediate response was to employ the DSV.

4.3. Experimental Design

4.3.1. Independent Variables

The experiment was a 4x2 fixed factor repeated measures model, with two

independent variables: the audio condition (a between‐subjects treatment), and the

number of vehicles under control (a repeated within‐subjects factor).

The four levels of the audio condition factor were combinations of the four

auditory‐alerts described earlier in Figure 3‐6 in Chapter 3, Simulator and Interface

Design. Every participant was exposed to two of the alerts shown in Figure 3‐6.

Participants were presented one audio alert from the course‐deviation columns, either

the oscillating course‐deviation alert or the threshold course‐deviation alert, and one

audio alert from the late‐arrival column, either the modulated late‐arrival alert or the

threshold late‐arrival alert. The four audio conditions were the following: (1) the

threshold audio condition for both the late‐arrivals and course‐deviations (BothThresh),

46

(2) the continuous oscillating course‐deviation audio condition, with threshold alert for

the late‐arrivals (DevCont), (3) the continuous modulated late‐arrival audio condition,

with a threshold alert for course‐deviations (LateCont), and (4) the both continuous

audio condition, which consisted of the oscillating course‐deviation alert and the

modulated late‐arrival alert (BothCont).

The second independent variable, the number of vehicles under control, had two

levels: single‐UAV and multiple‐UAV. In the single level, the participant supervised

only one UAV, while in the multiple level, the participant supervised four UAVs. The

framework of this experiment is represented in Table 4‐1.

The experiment was counterbalanced; a random half of the participants

completed the single‐UAV scenario first, and the other half finished the multiple‐UAV

scenario first. Additionally, the participants were randomly assigned to the four audio

schemes to avoid confounding effects.

Table 4‐1: Experimental Conditions

Scenario (Repeated)

Single-UAV Multiple-UAV

BothThresh Threshold Course-deviation,

Threshold Late-arrival Participants 1-9 Participants 1-9

DevCont Oscillating Course-deviation,

Threshold Late-arrival Participants 10-19 Participants 10-19

LateCont Threshold Course-deviation,

Modulated Late-arrival Participants 20-29 Participants 20-29

Audio Scheme

(Between)

BothCont Oscillating Course-deviation,

Modulated Late-arrival Participants 30-39 Participants 30-39

47

4.3.2. Dependent Variables

Dependent variables included the number of missed course‐deviations, the

reaction time to correct course‐deviations, the number of missed late‐arrivals, the

reaction time to correct projected late‐arrivals, the number of missed radio calls, and the

NASA TLX scores. These are discussed in detail below.

4.3.2.1. Course Deviation Errors of Omission

In each test scenario, participants were expected to respond to four triggered

course‐deviations. See Appendix B for a timeline of the events for both the single and

multiple‐UAV scenarios. Again, as illustrated in Figure 3‐5, a course‐deviation was

defined as a UAV no longer appearing to follow its planned flight path. Anytime the

participant failed to respond to one of four triggered course‐deviations, an error of

omission was counted. This simply meant that the participant failed to recognize and

address one of the four cued course‐deviations before the UAV self‐corrected. In the

event that a course‐deviation was missed, this data point was treated as a missing data

point for calculating the course‐deviation reaction times.

4.3.2.2. Course‐deviation Reaction Times

The reaction time for a course‐deviation event was the time taken by the

participant to make a corrective input after one of the four course‐deviations had been

triggered. The course‐deviation reaction time used for data analysis was an average of

the reaction times for the course‐deviations that the participant responded to per test

scenario.

4.3.2.3. Late‐arrival Errors of Omission

Four late‐arrivals occurred in each test scenario. Illustrated in Figure 3‐3, late‐

arrivals were caused when a UAV slowed down because of unforeseen headwinds and

was no longer able to reach a target in time to complete the firing mission. Errors of

48

omission for late‐arrivals occurred when the participant failed to respond to one of the

four triggered late‐arrivals. This simply meant that the participant failed to recognize

and address the late‐arrival before the UAV automatically moved past the target. In the

event that a late‐arrival was missed, this data point was treated as a missing data point

for calculating the late‐arrival reaction

4.3.2.4. Late‐arrival Reaction Times

The late‐arrival reaction time was the time taken by the participant to make a

corrective action after one of the four late‐arrivals was indicated. The late‐arrival

reaction time used for data analysis was an average of the reaction times for the late‐

arrivals that the participant responded to per test scenario.

4.3.2.5. Secondary Workload Assessment (Radio Calls Missed)

As a secondary workload task, the number of missed radio calls was an

indication of the operator’s level of mental workload. The count of missed radio calls

measured spare mental capacity. Participants were instructed to monitor a recording of

continual air traffic radio chatter for the word “Push.” The word “Push” occurred 62

times in a 30 minute session, with an average of 27 seconds between each “Push” radio

call. To acknowledge the radio call, participants clicked an “Acknowledge Push”

button on the display (Figure 3‐2).

4.3.2.6. Subjective Workload Assessment (NASA TLX Score)

The National Aeronautics and Space Administration (NASA) Task Load Index

(TLX) gathered participants’ subjective assessment on a scale of 1 to 20 of mental

demand, physical demand, temporal demand, effort, performance, and frustration. A

participant was then asked to rate these six dimensions against each other to determine

their importance in the participant’s workload (Hart & Staveland, 1988). Since there was

49

no physical demand in this experiment, each participant was told to rate the physical

demand so that it was zeroed out of the score.

4.4. Participants

Given an a priori power analysis for a power of 0.80, the estimated minimum

number of needed participants was determined to be 23, and 44 were recruited. Each

participant was paid $10 an hour to participate. The experiment took between 2.5 and 3

hours to complete. Participants’ ages ranged from 20 years to 42 years, with an average

age of 26 years and standard deviation of 6 years. There were 3 Navy midshipmen, 1

Army reservist specialist, 1 Air Force staff sergeant, and 39 officers from the

Army/Navy/Air Force. Overall, the personnel tested had a combined experience of over

250 years of active duty military service, with each member having an average of 5.8

years of active duty service. Five pilots contributed as test participants, and most of the

28 junior officers tested will be future military pilots. Overall there were 7 female and

37 male participants.

For the data analysis, 5 participants were dropped because of problematic data.

The first two were omitted because of the failure of the test proctor to administer proper

training. The third was dropped because of a 3 day interruption between the single‐

UAV and multiple‐UAV test sessions. The fourth participant was not used because his

secondary task data (i.e., number of push call responses) was an outlier of more than

3.29 standard deviations from the mean for both the single‐UAV and multiple‐UAV

scenarios. The fifth participant dropped was an active duty Air Force lieutenant colonel

with 4 years experience as a maintenance officer and an additional 4 years of flight line

time as a pilot and flight test engineer. He reported having been diagnosed with

tinnitus and loss of high frequency tone in one ear. During the audio training, he

reported not being able to hear the differentiation between a low and high course‐

50

deviation with the oscillating course‐deviation alert. His hearing test results indicated

age‐induced hearing loss.

4.5. Testing Procedures

Each participant experienced three separate consecutive phases in completing

the experiment. Each had a 60 to 70 minute training session, followed by a 70 to 80

minute test session and then a 10 minute post‐test survey. For the training session, each

participant completed a consent form and demographics survey, received standardized

training from a PowerPoint® tutorial, a hearing test, audio training for the specific

audio test condition, and a thirty minute practice test session with a trial run of the

NASA TLX. If required, a participant could do an additional 10 minute practice session

to ensure understanding of the proper MAUVE control actions. Copies of the

participant consent form, demographics survey, tutorial, and post‐test survey are in

Appendices C, D, E, and F, respectively.

The hearing test was designed to identify whether the participant suffered any

hearing loss (either temporary or permanent). For the hearing test, participants listened

to a CD of test tones from 40dBSPL down to 16dBSPL, with 5 frequencies tested (500

Hz, 1 kHz, 2 kHz, 4 kHz, and 8 kHz). Participants acknowledged how many beeps

were heard in each ear and each frequency. The range tested ensured participants could

hear to within 3dB of the noise floor of the headset in the normal lab conditions. Based

on the method of descending limits, the test was designed so that hearing 5 beeps or

more at each frequency showed adequate hearing (Snodgrass, 1975; Wundt, 1902). If

fewer beeps were heard at 8 kHz than at 1 kHz, the participant exhibited high

frequency loss (Cooper & Owen, 1976; Humes, Joellenbeck, & Durch, 2006). Age‐

related permanent loss showed up most at high frequencies (i.e. 8 kHz congestion and

other losses show up at lower frequencies and across range) (Walden, Prosek, &

51

Worthington, 1975). The hearing test was generated by Dr. Thomas Edward von

Wiegand of Sensimetrics Corporations, in conjunction with the HDiSP.

The audio training gave each participant exposure to all of the audio alerts each

participant would hear. Participants were presented with their respective late‐arrival

and course‐deviation alerts, as well as the radio chatter. After introducing each of these

individually, they were played to provide examples of real test scenarios. After a

demonstration of the audio condition, participants were encouraged to play with the

audio demonstration software until they were comfortable with recognizing each of the

alerts.

After the training was completed, each participant then completed both the

single‐UAV scenario and the multiple‐UAV scenario. At the conclusion of every test

session, MAUVE generated a data log, with all the previously discussed dependent

variables. Additionally, participants completed a NASA TLX following each test

session. At the conclusion of the test session, each participant completed a post‐test

survey, was paid, and thanked for his time and involvement.

4.6. Data Collection

For the collection of the data, all pertinent action sequences in MAUVE, such as

the reaction times to correct course‐deviations and late‐arrivals and the radio call

acknowledgements of the “Push” radio calls, were recorded in a MAUVE data file for

each scenario. In addition, an Excel® file with the NASA TLX score was generated and

saved for each scenario, and a Camtasia® video recording screen capture of the two

MAUVE displays was saved.

52

53

5. Results

For statistical analysis, repeated measures analysis of variances (ANOVAs) were

conducted for each dependent variable. The audio scheme factor was a between‐

subjects factor for the four audio conditions, and the scenario was a repeated within‐

subjects factor of single‐UAV versus multiple‐UAV levels. Excluding the late‐arrival

reaction times and error of omission counts which were non‐parametric, all of the

remaining variables analyzed met normality and homogeneity assumptions. While α

was set to 0.05, p values between 0.05 and 0.10 were considered marginally significant.

See Appendix G for further details on the statistical analysis tests.

5.1. Course‐deviation Errors of Omission

Out of the 156 course‐deviations presented to the 39 participants, 9 course‐

deviations were missed (5.8%). Of the missed course‐deviations, approximately half (4

course‐deviations) were missed by participants in the LateCont audio condition. The

data was not normal, and the non‐parametric tests (Kruskal‐Wallis) showed no

significant differences between the different audio conditions.

5.2. Course‐deviation Reaction Times

Figure 5‐1 shows the means for the course‐deviation reaction times across the

four audio conditions (BothThresh, DevCont, LateCont, and BothCont), with standard

error bars. The omnibus test shows there was a statistically significant difference in the

performance of participants based on the audio scheme (F(3,35)=2.878, p=.05), and a

marginally significant difference due to the scenario (F(1,35)=3.215, p=.08). Interaction

was not significant. After the omnibus testing, a Tukey post‐hoc comparison showed

that there was one primary difference across the audio scheme. The significant

difference was between the BothCont and BothThresh audio conditions for the single‐

UAV scenario (p=0.02). There was not a significant difference between any of the other

points.

7

8

9

10

11

12

13

14


Mea

n Co

urse

-dev

iatio

n R

eact

ion

Tim

es

(sec

onds

)

BothCont LateCont DevCont BothThresh

Figure 5‐1: Course‐deviation Reaction Times Treatment Means Plot.

In Figure 5‐1 there was an apparent division in the data based on which audio

alert was present for the course‐deviation. As part of the post‐hoc analysis based on the

data clustering in Figure 5‐1, with the BothCont and DevCont appearing in one group

and the LateCont and BothThresh in another group, the model was reconfigured as a

2x2 fixed factor repeated measures model with audio scheme and scenario as the

independent variables, respectively (Table 5‐1). Instead of the four audio conditions,

the factor levels were collapsed into two levels: those participants exposed to the

continuous oscillating course‐deviation alert (participants in the BothCont or DevCont

audio conditions) and those exposed to the threshold course‐deviation alert

(participants in the LateCont and BothThresh audio conditions). The audio scheme

54

remained a between‐subjects factor, while the scenario remained a repeated within‐

subjects factor.

Table 5‐1: Post Hoc Experimental Conditions.

Scenario (Repeated)


Continuous Oscillating Course-deviation Alert (BothCont or DevCont)

Participants 1-20 Participants 1-20 Audio Scheme

(Between) Threshold

Course-deviation Alert (LateCont or BothThresh)

Participants 21-39 Participants 21-39

As seen in Figure 5‐2, the analysis confirms that the participants with the

continuous oscillating alert condition performed significantly better than those with the

threshold course‐deviation alert condition (F(1,37) = 8.874, p=.01).

7

8

9

10

11

12

13

Mea

n Co

urse

-dev

iatio

n Re

actio

n Ti

mes

(sec

onds

)

Continuous Oscillating Course-Deviation AlertThreshold Course-Deviation Alert


Figure 5‐2: Post Hoc Analysis Course‐deviation Reaction Times Treatment Means Plot.

55

5.3. Late‐arrival Errors of Omission

All 39 participants responded to all of the 4 late‐arrivals presented to each of

them, thus there were no late‐arrival errors of omission.

5.4. Late‐arrival Reaction Times

The original late‐arrival reaction times were not normally distributed, thus a

natural log transformation was performed. The omnibus results showed a significant

difference in the performance of participants between the four audio conditions

(F(3,35)=3.345, p=.03) and a significant difference due to the scenario (F(1,35)=20.737,

p<.001). Interaction was not significant.

As seen in Figure 5‐3, there was an apparent split in the data. A post‐hoc Tukey

comparison revealed there was a significant or marginally significant difference

between the data for the DevCont audio condition and the other three audio conditions:

BothThresh audio condition (p=.05), LateCont audio condition (p=.08), and BothCont

audio condition (p=.07). These post‐hoc comparisons confirmed the apparent

differences within the treatment means plot in Figure 5‐3.

5.5. Workload

To measure workload, the performance on a secondary task was monitored, and

scores for subjective workload were recorded. These are detailed in the following

sections.

5.5.1. Secondary Task Assessment

For the number of missed radio calls, there was no significant difference between

the audio conditions or across the scenarios. According to this secondary workload

56

0

0.5

1

1.5

2

2.5

3

3.5


Mea

n Tr

ansf

orm

ed (n

atur

al lo

g) L

ate-

arriv

al R

eact

ion

Tim

es (s

econ

ds)

BothCont LateCont DevCont BothThresh

Figure 5‐3: Transformed (Natural Log) Late‐arrival Reaction Times Treatment Means Plot.

measurement, the participants were equally saturated, regardless of their assigned

audio condition or whether they supervised a single or multiple UAVs. This was not a

surprise; as discussed before and illustrated in Appendix B, the scenarios were designed

with an equal number of tasks.

5.5.2. Subjective Assessment

Similar to the secondary workload measurement, the subjective measurement of

the NASA TLX scores was not significantly different across audio conditions or

scenarios. The non‐significant results across the scenario also were not a surprise,

because as illustrated in Appendix B, the single‐UAV and multiple‐UAV scenarios were

designed to have the same number and placement of events. In the multiple‐UAV

scenario, the number of events and same point in time of the events was just split over

four UAVs instead of all being on one UAV’s flight path.

57

5.6. Post‐Experiment Subjective Responses

Subjective responses regarding the usefulness of the audio cues, whether discrete

or continuous, were overwhelmingly positive. In reviewing the participants’ comments,

some valuable insights were gained.

The overarching feeling toward the discrete threshold alerts was that while they

were better than nothing, the discrete threshold alerts could have been improved.

Many felt that these audio alerts were nice to have as they “made [the participant] pay

attention” and were “a good back‐up for the visual cue.” The majority commented the

audio alerts worked well in conjunction with the visual display to better enhance their

awareness. One participant reported the audio alert “was especially helpful because

noticing course‐deviations visually was difficult.”

A key concern raised about the threshold audio alerts was that the alerts were

“hard to distinguish from background noise.” The feeling was that the beeps blended

into the background too easily. Threshold audio alerts were masked by things such as

the general radio chatter or other alerts in the mission such as the beep that alerted the

operator of new threats to avoid. One reported he found the threshold audio alert to be

“too short and too discrete to call my attention. I usually noticed the late‐arrivals

visually.” Others offered possible solutions. One said the audio alert “should be more

annoying and should last longer.” A common feeling was that if the audio alert was

“longer and more prominent,” the audio would be more effective as a supplemental

tool to the visual alert. In fact, over half of the participants exposed to one of the

discrete alerts asked for alerts with “more than just a simple beep” and wished for a

more repetitive audio alert that would “last,” so operators would have to address the

alert.

Another concern raised in the threshold audio alert feedback was that the audio

alerts were “difficult to use effectively with four UAVs.” Some suggested, “It might

58

have been more helpful if it referred to the selected UAV.” The results of this research

support this observation. Participants needed audio support that would point them to

the problem UAV, not just the fact that there was a problem with one of the UAVs.

The continuous audio alerts were generally appreciated as situational enhancers

and attention managers, but some felt the continuous audio was potentially fatiguing.

Although generally positive, participants did report the continuous modulated late‐

arrival alert “focused attention to late UAVs.” Like the discrete threshold alerts,

continuous audio was “audio backup” to the visual display and “alerted [participants]

to the impending late‐arrivals more quickly.” With continuous audio, participants

learned to use the continuous nature to supplement their regular visual scan, with one

recounting, “I mostly looked at the left screen” and “the audio was the cue to look at the

right screen.” Though one reported that the continuous modulated late‐arrival alert

“helped with many UAVs” and another that it was a “good alert of a late‐arrival,”

participants found it difficult to distinguish the priority of the target based on the audio

alone.

While the modulated late‐arrival alert was touted as an enhancement tool, it was

noted by participants that the tones “provoked a stress response so that [the participant]

felt more anxious.” An Air Force Predator pilot noted that the modulated late‐arrival

alert, “while effective at alerting to issues, was extremely annoying,” and he “would not

subject [himself] to that background noise for hours at a time day after day.”

Similar to the modulated late‐arrival alert, the continuous oscillating course‐

deviation alert was lauded as “useful as a backup secondary cue.” A negative comment

about this continuous audio alert was that the audio alert could “contribute greatly to

fatigue” over a long mission. Ten percent of the participants exposed to the continuous

oscillating course‐deviation suggested that this additional alert be left off throughout

the bulk of the mission and only have the additional alert on during particular parts of

the mission. This way the effectiveness of the tool could be used when the operator

59

needed an additional cue. At the same time, operators could be spared the annoyance

of continuous audio input over a long duration mission, when they were not

continually at a peak stress and work level.

Interestingly, all of the negative comments regarding the oscillating course‐

deviation alert came from the participants who were exposed to both the oscillating

course‐deviation alert and the modulated late‐arrival alert (BothCont), instead of the

oscillating course‐deviation alert and the threshold late‐arrival alert (DevCont). Twenty

percent of those exposed to both were annoyed by the oscillating alert. Regardless, the

majority felt the continuous audio alerts helped, but some time may be needed for

operators to get used to them. As one operator stated, “Effective as they were, I just

couldn’t remember which was which.”

5.7. Summary of Experimental Findings

Key findings of the data analysis were found in the post‐hoc analyses discussed

previously in this section. When the course‐deviation reaction times were collapsed,

those with the continuous oscillating alert condition were shown to have performed

significantly better than those with the threshold course‐deviation alert condition.

Further, though the count of error of omissions for the course‐deviations was not

significant, it should be noted that almost half of all of the missed course‐deviations

occurred under the LateCont (comprised of threshold course‐deviation and modulated

late‐arrival alerts) condition.

Another key finding was that for the late‐arrival reaction times, the only

condition that had a significant difference in performance from the other conditions was

DevCont (comprised of oscillating course‐deviation and threshold late‐arrival alerts).

The DevCont condition was shown to have significantly poorer performance than the

three other audio conditions: BothCont (comprised of oscillating course‐deviation and

60

modulated late‐arrival alerts), LateCont (comprised of threshold course‐deviation and

modulated late‐arrival alerts), and BothThresh (comprised of threshold course‐

deviation and threshold late‐arrival alerts).

The subjective feedback results showed an overall preference for audio aids to

supplement visual events, but there was annoyance reported for some of the continuous

audio presentations. Many felt it would be fatiguing to run this form of audio

continually throughout the long missions that some UAV operators supervise. Overall

though, there was a request for more than a discrete, single beep alert.

61

63

6. Discussion

This chapter discusses the findings of the results presented in Chapter 5, Results.

The discussion that follows presents these findings in relation to multiple resource

theory, mapping of audio alerts, change blindness, and masking. It then closes with an

explanation of the causes of the workload results.

6.1. Multiple Resource Theory

The anesthesiology studies discussed in Chapter 2 showed that sonifications

successfully supplemented the visual modality (Watson & Sanderson, 2004). They did

not replace the visual representation on the screen, but they enhanced the visual

display. Similarly, within this experiment, as operators were performing tasks or even

monitoring the course‐deviation visually, operators relied on the aural channel to

correct course‐deviations quickly, which was reported in their post‐test subjective

feedback. This illustrates that participants were integrating the audio input to confirm

and guide their visual scan to more readily identify problems.

The multiple‐UAV scenario’s reaction times were expected to be longer than the

single‐UAV scenario’s; however, there appeared to be almost no differences in course‐

deviation reaction times between the single‐UAV and multiple‐UAV scenarios when

the discrete threshold course‐deviation audio alert was used. In comparison, the

scenarios using the continuous oscillating course‐deviation audio alert showed the

expected trend of the multiple‐UAV scenario reaction times being longer than the

single‐UAV scenario. This difference, between the discrete and continuous audio

conditions, is likely a result of the fact that in the discrete audio condition, operators

had already determined the problem visually and then waited for aural confirmation

before addressing the problem. Conversely, in the continuous audio condition, the

audio first and more readily alerted the operator that there was a problem. The

operator then addressed the problem by visually determining the UAV that the

problem had occurred on, causing the longer reaction time for the multiple‐UAV

scenarios.

The continuous audio provided constant auditory input that participants relied

upon to more quickly confirm their interpretation of the visual displays. Those exposed

to the discrete audio had to wait until the threshold at which the audio was

programmed to alert operators was met, before receiving any auditory confirmation of

what they may have already perceived visually. The results of this experiment, for the

course deviation reaction times, illustrate that operators had the resources to integrate

the continuous audio with their visual perception, allowing them to out‐perform those

with the discrete audio. Thus, continuous audio better utilized the audio channel’s pool

of resources.

6.2. Mapping Audio Alerts to Intuitive Triggers

While this experiment showed the continuous audio helped in monitoring course

deviations, it also demonstrated a case where continuous audio did not aid an operator

in a supervisory role. For the late‐arrival reaction times, there was no significant benefit

gained by having a continuous audio alert rather than a discrete audio alert for late

arrivals. The primary reason there was no performance enhancement with the

continuous modulated late‐arrival alert was that the late‐arrivals were discrete events (a

UAV was either late or not late). In relation to discrete audio, the continuous audio did

not provide any extra, useful information in the late arrival condition. The audio

represented one of four discrete states, rather than a continuous parameter, so it could

not convey progression towards a new state, and thus the benefit of continuous audio

was not fully exploited.

64

Conversely, the positive results of the continuous oscillating course‐deviation

alert were attributable to its intuitive mapping to the continually changing state of the

UAV course deviation. As the UAV moved off course, the continuous audio

progressively changed, and the participant was then given useful information in the

sonification that allowed him to choose his own threshold for course deviations, as

discussed above in the multiple resource theory discussion.

6.3. Change Blindness

In Chapter 2, Background, the point was raised that some forms of audio have

been shown to help mitigate visual change blindness that may result from relying on

only a visual presentation (Nehme & Cummings, 2006). Because there was no baseline

condition, with only visual displays and no audio alerts, there is no empirical evidence

that the discrete or continuous audio displays helped mitigate visual change blindness.

It is noteworthy, however, that no late‐arrivals and an insignificant number of course‐

deviations were missed by all the participants. The indication is that the combination of

the visual and audio displays did, in fact, alert the participants to problem situations.

In terms of error rates in responding to both course‐deviation and late‐arrival

events, only a very small percentage of course‐deviation alerts were missed, and all of

the late‐arrivals were recognized by the participants. This slight difference was most

likely caused by the visual alert representation in that the late‐arrival visual alert was

easily seen by the physical displacement and color change of the target icon (Figure 3‐3).

In contrast, whether a UAV was truly off course was more visually ambiguous, as is

seen in Figure 3‐5. This was exacerbated by the simulation, which caused the UAV to

automatically resume its correct course if a participant did not recognize and correct an

off‐course error within thirty seconds. This illustrates the importance of dual coding

65

alerts on the visual and aural channels, which is one way to combat change blindness

and was generally effective in this experiment.

6.4. Masking

The subjective feedback of the participants suggested that they felt the discrete

threshold alerts were masked by other clutter and noise in the radio chatter, which is

why the discrete threshold alerts may not have performed as well as the continuous

ones. In general, the results showed that almost half of the missed course‐deviations

occurred in the audio condition, where the continuous modulated late‐arrival audio

alert was present in a scenario with the discrete threshold course‐deviation audio alert.

Further, the results showed that late‐arrival recognition was no different across

conditions, except for a significant decrease in performance when the continuous

oscillating course‐deviation alert was present along with the discrete threshold late‐

arrival audio alert. Both of these statistical results are examples of how masking can

result in degraded performance. Therefore, a continuous audio alert should be

integrated with the other alerts so that masking of less prominent discrete or even

continuous alerts does not occur.

6.5. Workload

The workload measurements (subjective responses and secondary task

performance) showed no significant difference between the single‐UAV and multiple‐

UAV scenarios. The operators reportedly perceived no difference in the workload,

which was not a surprise. As discussed before and presented in Appendix B, the two

scenarios were designed with the same number and types of events. The only

difference was that for the multiple‐UAV scenario, the events were separated over four

66

UAVs instead of one UAV. The intent was to provide the same number of tasks, and

thus the same amount of work, in each scenario. Because there was an identical set of

events for much of the experiment, participants likely recognized no difference in their

efforts between the two scenarios. This was supported by the similar performance on

the secondary task for the single‐UAV and multiple‐UAV scenarios.

Although the workload measurements were not significant, the performance

metrics were different between the scenarios. The late‐arrival reaction times were

significantly different, and the course‐deviation reaction times were marginally

different. In both cases, there were shorter reaction times for the single‐UAV scenario

than the multiple‐UAV scenario. This performance difference is likely the result of

participants having to divide their attention while monitoring four UAVs for the

multiple‐UAV scenario. The impact of divided attention for multiple UAV control will

be discussed further in the findings section of Chapter 7, Conclusion.

67

69

7. Conclusion

In this concluding chapter, the primary research questions are answered,

followed by a discussion of workarounds to potential integration issues and areas for

future research.

7.1. Findings

The primary questions addressed through this research are as follows:

1. When compared with discrete audio, does continuous audio better aid human

supervision of UAV operations?

2. Is the effectiveness of the discrete or continuous audio support dependent on

operator workload?

7.1.1. Value Added by Continuous Audio

From the experiment results for the single and multiple‐UAV scenarios, it

appears that the continuous oscillating course‐deviation audio alert helped participants

respond more quickly to the task of recovering from UAV course‐deviations. The

continuous modulated late‐arrival audio alert, in contrast, did not help or hinder the

participants in responding to late‐arrivals, primarily due to a confound in alert

mapping.

These results highlight the importance of context in alert design. When the

continuous oscillating course‐deviation audio alert was used with the discrete threshold

late‐arrival audio alert, reaction times to addressing late‐arrivals were significantly

worse than all other cases, due to masking. In contrast, when used with the continuous

modulated late‐arrival audio alert, the late‐arrival reaction times did not suffer, and the

course‐deviation reaction times improved.

Thus, continuous audio alerts helped in UAV control, but they must be added in

light of all the other audio input occurring in the display. Caution must be taken so that

the added benefit of a continuous audio alert is not lost because of a lack of proper

system integration. Further, continuous audio must only be applied in cases where it

maps to the occurrences of the event that is being monitored. If it is a discrete event,

like late‐arrivals, there may not be a gain in using continuous audio in place of discrete

audio.

7.1.2. The Impact of Workload

The results indicate a decrease in performance when supervising multiple UAVs

instead of a single UAV, regardless of the audio condition. The performance

degradation in the multiple‐UAV condition is not a surprise. Task load stress means

that even with a constant signal rate, performance will decline if more information

inflow must be used (Conrad, 1985). This means that even with the same number of

tasks, if the tasks are distributed over multiple vehicles, the performance will decrease.

The effects of load stress have been illustrated in air traffic control (Cummings &

Tsonis, 2005) and UAV studies (Cummings & Mitchell, 2008; Cummings, Nehme, &

Crandall, 2007; Dixon et al., 2005). Given the results of previous research, there is no

reason to assume the results of this study would have been any different. However, the

use of continuous audio displays was meant to alleviate some of this increased

workload, and participants recognized this in the subjective responses, noting that

though the audio alerts were “very helpful” when focusing on other tasks, it was

“harder to comprehend with four UAVs than one.”

On the whole, though, participants exposed to continuous audio outperformed

participants exposed to discrete audio in both the single‐UAV and multiple‐UAV

70

scenarios. Previous research controlling two UAVs showed that an audio alert

improves performance over a baseline condition of no audio but that generally,

performance degrades when the number of UAVs under control increases (Dixon et al.,

2005). The results of this thesis mirror this previous research, except that it was shown

that continuous audio, when used appropriately, can actually mitigate the negative

impacts of increased workload due to increased numbers of UAVs under control.

The conclusion to these research questions is that with correct application,

continuous audio is more helpful than discrete audio in supporting the supervision of

multiple tasks over multiple vehicles. While continuous audio may be a performance

enhancer, it is also important to assess the research and development, acquisition, and

maintenance costs associated with fielding this new headphone technology.

Quantification of these various factors will determine whether such technology should

ultimately be employed, which will be discussed next.

7.2. Integration Issues

A shortcoming in the use of headphones for the presentation of the continuous

audio is the isolation of the operator from outside communication (Tannen, 1998). In

operational integration with military personnel, operators will often only wear one half

of the headset because wearing the headset on both ears isolates them from inter‐team

interaction. It is important to make sure that any integration of a headset does not

interfere with critical work environment constraints, especially those in a team setting.

Furthermore, though continuous audio can promote objective performance,

some participants in the experiment noted an annoyance with long term exposure to

these forms of continuous audio. One solution in integration is to limit exposure to the

audio and only use it when needed. For example, rather than play continuous audio

throughout an entire four‐hour mission, the continuous audio tool could be active only

71

during the portion of the mission when workload is heavy, or when one of the

peripheral tasks reaches a cautionary state that may require the human operator’s

intervention. Thus, this kind of adaptive display could either add or remove audio aids

as required by the workload situation.

7.3. Cost‐Benefit Analysis

While the results show that continuous audio displays improved operators’

performance, the question is to what extent? In addition, in comparison to this benefit,

what is the cost to the DOD to actually acquire these displays?

To consider quantitative benefits of the continuous audio displays, the reaction

times to course‐deviations and late‐arrivals are reviewed. The results show that with

the continuous audio, participants were on average 2 seconds or 6 seconds faster at

responding to course‐deviations or late‐arrivals, respectively, than the participants with

the discrete audio display. Thus, these experimental results show a performance

enhancement of a 31 percent decrease in reaction times for monitoring tasks with the

aid of continuous audio displays instead of the aid of discrete audio displays. In

aviation this time interval can be critical, particularly in UAS operations where

operators deal with time lags in the control of remote vehicles. Linking continuous

audio to support operators in monitoring time critical events could be very beneficial.

In terms of the actual hardware, the HDiSP is still a new technology and is the

only headset of its kind. The HDiSP used in this experiment was a prototype version

provided by Sensimetrics. To date, Sensimetrics is only providing HDiSP for custom

orders. Two HDiSPs, along with the software, have been sold at a cost of 3,750 dollars

per headset (T. E. von Wiegand, personal communication, April 2, 2008). Sensimetrics

estimates that if they were mass producing the HDiSP, the headset would sell between

1,000 and 2,000 dollars per headset (T. E. von Wiegand, personal communication, April

72

2, 2008). In contrast, the best aviation headsets cost about 300 dollars (SkyGeek, 2008).

Overall the price increase of purchasing the HDiSP instead of aviation headsets would

be about 1,200 dollars (400 percent), if the HDiSP was mass produced at 1,500 dollars

per headset.

Rough performance and cost increases can be calculated, but these numbers

mean nothing unless placed in some context. For instance, in the motivation section of

Chapter 1, Introduction, the statistic was cited that over the next 8 years, 15.7 billion

dollars will be spent on UAVs in the United States, yielding 11,000 UAVs in operation

(Tsach et al., 2007). Equipping each UAV with an HDiSP for single operator

supervision would result in a cost of 13 million dollars, which is less than 0.1 percent of

all the money projected to be spent on UAVs in the United States in the next 8 years.

While there is a cost to integrating a new continuous audio system, there are also

research and development costs for continuous audio displays and HDiSP technology.

Further, there would likely be additional maintenance costs for the HDiSP, which is a

more complex hardware device that may not be as sturdy in field conditions as older,

more robust aviation headphones. Dependent upon implementation, if there were

significant maintenance issues, there would be social costs as well. Operators might

lose trust or grow frustrated with the constant maintenance and then stop using the

device.

The results of the experiment in this thesis illustrate continuous audio displays

provide a performance enhancement. When compared with participants’ performance

with discrete audio displays, the continuous audio displays are shown to decrease

reaction times by 31 percent. An initial cost benefit analysis shows that the cost of

implementing such a device is minimal, but future research is needed to do a full

analysis of economic, operational, and social costs of implementing these new audio

displays into UAS interfaces.

73

7.4. Limitations and Future Work

• A limitation of this research is that it assumes a certain level of autonomy in the

unmanned vehicles, such as the vehicles flying themselves from waypoint to

waypoint, with the operator performing a distinct payload mission. The research

presented here focuses on the human operator acting in a higher supervisory

role.

• The tasks represented by the continuous audio alerts were not primary tasks, but

secondary tasks that required occasional operator input. The results may have

been different had the continuous audio alerts been linked directly to the

primary task.

• Within the MAUVE simulator, there is a rapid onset of change for the course‐

deviation and late‐arrival events. Future research could be done on a simulator

that allows for a more gradual onset of change to further explore how the rate of

increase for the audio intensity affects the speed and confidence of response by

the participants. In particular, how the onset of change for the continuous audio

helps alert the participant to the problem is an area of future research.

• While continuous audio alerts have been proven beneficial in comparison to

discrete audio alerts, further research could be completed to investigate the

different patterns of sonifications to see which is best for helping operators

supervise multiple tasks on multiple vehicles.

• Another future study could compare the benefits of continuous audio alerts with

spatial audio and test the integration of continuous audio into a spatial audio

presentation. The point of this future study would be to test the value added

when continuous audio is used with or in addition to spatial audio.

• These results suggest that perhaps a better implementation of continuous audio

alerts is to use them only during high workload situations, so as to minimize any

74

annoyance factor, while maximizing the objective benefits of the tool. Future

research could test ways to build the sonifications into adaptive audio displays

that change the amount and type of audio output based on the workload

presented by the system to the operator.

• Another extension of this research would be investigating how haptic cueing,

either as a replacement or an addition to the continuous audio alerts, would

affect performance.

75

Appendix A: Audio Alert Guidelines

According to Deatherage (1972) and Sanders and McCormick (1993), the

following guidelines should be considered in designing to meet the physical parameters

of the human ear and hearing:

• Use sounds in the 200 to 5000Hz range, in particular the 500 to 3000Hz because

this middle range is the most sensitive region for human hearing.

• To avoid masking in noise, use signal frequencies different from the noises’ most

intense frequencies.

• To capture attention, use modulated sounds of intermittent beeps repeating one

to eight beeps per second or warbling sounds that vary between 1 to 3 times per

second, because these sounds rarely occur naturally and will capture operator

attention.

• If representing different conditions, different warning signals should be

discriminable from each other, and moderate‐intensity signals should be used.

77

Appendix B: Scenario Events

Appendix B shows that the single‐UAV and multiple‐UAV scenarios were

designed with the same number of events. The only difference was that the multiple‐

UAV scenario has the events divided over four UAVs, while the single‐UAV scenario

has them all occurring with one UAV. The timeline of events for the two scenarios

(Figure B‐1) illustrates this.

Figure B‐1: Major Events of Single‐UAV and Multiple‐UAV Test Scenarios.

79

Table B‐1 shows side‐by‐side images of the timeline display and map display for

the single‐UAV and multiple‐UAV scenarios. Again, as with the timeline in Figure B‐1,

these displays show that in both scenarios the operator has the same number of tasks to

complete, and the only difference between the two scenarios is that for the multiple

scenario, the tasks are divided over four UAVs instead of just one UAV.

Table B‐1: Comparison of Single‐UAV and Multiple‐UAV Scenarios.

Single‐UAV Scenario Multiple‐UAV Scenario

Tim

elin

e D

ispl

ay

Map

Dis

play

80

81

Appendix C: Participant Consent Form

CONSENT TO PARTICIPATE IN NON-BIOMEDICAL RESEARCH

Developing Decision Support for Supervisory Control of Multiple Unmanned Vehicles

You are asked to participate in a research study conducted by Professor Mary Cummings Ph.D, from the Aeronautics and Astronautics Department at the Massachusetts Institute of Technology (M.I.T.). You were selected as a possible participant in this study because the expected population this research will influence is expected to contain men and women between the ages of 18 and 50 with an interest in using computers. You should read the information below, and ask questions about anything you do not understand, before deciding whether or not to participate. • PARTICIPATION AND WITHDRAWAL Your participation in this study is completely voluntary and you are free to choose whether to be in it or not. If you choose to be in this study, you may subsequently withdraw from it at any time without penalty or consequences of any kind. The investigator may withdraw you from this research if circumstances arise which warrant doing so. • PURPOSE OF THE STUDY The study is designed to evaluate how decision support tools or recommendations, both audio and visual, assist an operator supervising multiple simultaneous dynamic tasks, and how decision support assistance and effectiveness changes as workload increases. In measuring the effectiveness of decision support, an operator’s performance and situation awareness are used as metrics. Situation awareness is generally defined as the perception of the elements in the environment, the comprehension of the current situation, and the projection of future status of the related system. • PROCEDURES If you volunteer to participate in this study, we would ask you to do the following things:

• Attend a training and practice session to learn a video game-like software program that will have you supervising and interacting with multiple unmanned aerial vehicles (estimated time 0.75 hours).

• Practice on the program will be performed until an adequate level of performance is achieved, which will be determined by your demonstrating basic proficiency in monitoring the vehicles, redirecting them as necessary, executing commands such as firing and arming of payload at appropriate times, using decision support visualizations and/or recommendations to mitigate timeline problems, and responding to radio calls by clicking an acknowledge button on the software interface (estimated time 0.75 hours).

• Execute two thirty minute trials consisting of the same tasks as above (1 hour) • Attend a debriefing to determine your subjective responses and opinion of the software

(10 minutes). • Testing will take place in MIT building 37, room 301. • Total time: 2-3 hours, depending on skill level.

• POTENTIAL RISKS AND DISCOMFORTS There are no anticipated physical or psychological risks in this study. • POTENTIAL BENEFITS While there is no immediate foreseeable benefit to you as a participant in this study, your efforts will provide critical insight into the human cognitive capabilities and limitations for people who are expected to supervise multiple complex tasks at once, and how decision support tools can support their task management. • PAYMENT FOR PARTICIPATION You will be paid $10/hr to participate in this study which will be paid upon completion of your debrief. Should you elect to withdraw in the middle of the study, you will be compensated for the hours you spent in the study. • CONFIDENTIALITY Any information that is obtained in connection with this study and that can be identified with you will remain confidential and will be disclosed only with your permission or as required by law. You will be assigned a subject number which will be used on all related documents to include databases, summaries of results, etc. Only one master list of subject names and numbers will exist that will remain only in the custody of Professor Cummings. • IDENTIFICATION OF INVESTIGATORS If you have any questions or concerns about the research, please feel free to contact the Principal Investigator, Mary L. Cummings, at (617) 252-1512, e-mail, [email protected], and her address is 77 Massachusetts Avenue, Room 33-305, Cambridge, MA 02139. The student investigators are Hudson D. Graham (719-238-1713, email: [email protected]), and Amy Brzezinski (617-276-6708, [email protected]). • EMERGENCY CARE AND COMPENSATION FOR INJURY “In the unlikely event of physical injury resulting from participation in this research you may receive medical treatment from the M.I.T. Medical Department, including emergency treatment and follow-up care as needed. Your insurance carrier may be billed for the cost of such treatment. M.I.T. does not provide any other form of compensation for injury. Moreover, in either providing or making such medical care available it does not imply the injury is the fault of

82



the investigator. Further information may be obtained by calling the MIT Insurance and Legal Affairs Office at 1-617-253-2822.” • RIGHTS OF RESEARCH SUBJECTS You are not waiving any legal claims, rights or remedies because of your participation in this research study. If you feel you have been treated unfairly, or you have questions regarding your rights as a research subject, you may contact the Chairman of the Committee on the Use of Humans as Experimental Subjects, M.I.T., Room E25-143b, 77 Massachusetts Ave, Cambridge, MA 02139, phone 1-617-253-6787. SIGNATURE OF RESEARCH SUBJECT OR LEGAL REPRESENTATIVE I understand the procedures described above. My questions have been answered to my satisfaction, and I agree to participate in this study. I have been given a copy of this form. ________________________________________ Name of Subject ________________________________________ ______________ Signature of Subject Date

SIGNATURE OF INVESTIGATOR In my judgment the subject is voluntarily and knowingly giving informed consent and possesses the legal capacity to give informed consent to participate in this research study. ________________________________________ ______________ Signature of Investigator Date

83

85

Appendix D: Demographics Survey

MAUVE‐MITUS Demographic Survey

1. Age: ____________________

2. Gender: □ Male □ Female

3. Occupation: ___________________

If student:

a. Class Standing: □ Undergraduate □ Graduate b. Major: ____________________

If currently or formerly part of any country’s armed forces:

a. Country/State: ____________________ b. Status: □ Active Duty □ Reserve □ Retired c. Service: □ Army □ Navy □ Air Force □ Other ____________________ d. Rank: ____________________ e. Years of Service: ____________________ f. Did you ever serve in high noise environments? □ Yes □ No

If yes, please explain what the duties were, how long the shifts were, and how many times you

served these shifts?

________________________________________________________________________

________________________________________________________________________

4. Do you have experience with remotely piloted vehicles (land, sea, air)? □ Yes

□ No

If yes:

a. Vehicle type(s)/class(es): ____________________________________________________________________________

b. Number of hours: ____________________

5. Do you have experience with radios such as those used for communication in flying? □ Yes

□ No

If yes:

Please explain:

________________________________________________________________________________

________________________________________________________________________________

6. Have you been to a music concert in the last Month?

□ Yes

□ No

If yes:

a. Concert type:_______________________________________________________________________ b. When: ____________________

7. Do you have any hearing loss?

□ Yes

□ No

If yes:

Please explain:

________________________________________________________________________________

________________________________________________________________________________

8. How often do you play video games?

□ Never

□ Less than 1 hour per week

□ Between 1 and 4 hours per week

□ Between 1 and 2 hours per day

□ More than 2 hours per day

9. Are you color blind?

□ Yes

□ No

If yes:

Which type of color blindness (if known): ___________________________________________

86

Appendix E: MAUVE‐MITUS Tutorial

Amy Brzezinski Amy Brzezinski –– MIT Humans and Automation LabMIT Humans and Automation Lab

Multi-Aerial Unmanned Vehicle Experiment (MAUVE)

TUTORIAL

Spring 2007Spring 2007

Hudson GrahamHudson Graham

IntroductionIntroduction

Welcome!

This tutorial is designed to give you some background on the Multi-Aerial Unmanned Vehicle Experiment (MAUVE) interface before you arrive on testing day. Please take the time to look over the following slides and come prepared with questions. Before testing you will be thoroughly trained on the actual interface, but being exposed to it beforehand will be invaluable in speeding up this process.

Thank you in advance for your participation!

Experiment Overview Experiment Overview

In this experiment, you are an unmanned aerial vehicle (UAV) operator that is responsible for supervising 1 to 4 UAVs collectively tasked with destroying a set of time-sensitive targets in a suppression of enemy air defenses mission. The area contains enemy threats capable of firing on your UAVs.

The UAVs are highly autonomous, and therefore only require high level mission execution from you. The UAVs launch with a pre-determined mission plan, so initial target assignments and routes have already been completed for you. Your job will be to monitor their progress, re-plan aspects of the mission in reaction to unexpected events, and in some cases manually execute mission critical actions such as arming and firing of payloads.

The interface we have developed for this experiment is called the Multi-Aerial Unmanned Vehicle Experiment (MAUVE) and will be referred to by this name from here out.

ObjectivesObjectives

Your primary objective in this mission is:

To make sure the UAV(s) maximize the number of targets engaged as well as arriving back to the base safely.

Supervision of the UAVs can be broken down into the following prioritized sub-tasks, from highest priority to lowest:

1. Return to base (RTB) within the time limit for the mission (this limit will be clearly marked).

2. Comply with recovery rules for course deviations.

3. Comply with recovery rules for target late arrivals.

4. Destroy all targets before their time on target (TOT) window ends.

5. Avoid taking damage from enemies by avoiding all threat areas.

6. Acknowledge all “Push” radio calls.

These sets of objectives will often conflict with one another. In these cases, you must perform the actions that have the highest priority first.

Your performance will be judged based on how well you follow the above priorities.

Audio AlertsAudio Alerts

To help you meet your objectives you will receive auditory signals for both course deviations and late arrivals. Both are induced by unanticipated high winds along the planned flight path.

Course deviations are when a UAV is blown off of the planned path. It is significantly deviated when you visually see that the UAV has left the course line. Deviations may occur over targets as well.

Late arrivals are when the UAV has hit stronger than anticipated head winds and slows down. As a result it will now be late to the next target.

Your test proctor will provide further training as to what these auditory signals sound like during the test day training.

Other auditory sounds to be familiar with are: (Note all three are the same because all are related to a new message in your message box.)

For new messages in your message boxes

For pop-up threats For when your UAV is being fired upon

while flying through a threat area.

Color CodingColor Coding

Throughout the displays you’re about to see, the following color coding is used to indicate each of the 5 possible actions a UAV can perform in MAUVE:

GreenReturn to Base

OrangeFiring Payload

YellowArming Payload

BlueLoitering

GrayEnroute

ColorUAV Action

87

Displays Displays –– Overview Overview

During the experiment, you will see two side-by-side displays that contain the following major elements:

• Left Display− Mission Time− Map Display− Mission Execution

• Right Display− Unmanned Aerial Vehicle (UAV) Status− Decision Support− Chat Box− UAV Health & Status Updates

The following slides will show these displays in detail and explain how to use them properly.

Left Display Left Display –– Overview Overview

The three major screen elements on

the left display are:

1Mission

Execution

2Mission

Time

3Map

Display

33

22

11

Right Display Right Display –– OverviewOverview

The four major screen elements

on the right display are:

1UAV Status

2Decision Support

3UAV Health &

Status Updates

4Chat Box

11 22

3344

Left Display Left Display –– Detail Detail

The following slides detail all of the elements contained on the left display, in this order:

• Map Display

• Mission Execution

• Mission Time

Map Display Map Display –– Detail Detail –– 22

Naming Conventions• UAVs

− Numbered 1-4• Targets

− T-XXP where XX = target number and P = priority− Priority may be High (H), Medium (M), or Low (L)− Examples:

T-1H – Target 1 a high priority targetT-12M – Target 12 a medium priority targetT-23L – Target 23 a low priority target

• Waypoints (WP)− WP-XY where X = UAV# the waypoint is associated with and Y = waypoint

letter− Examples: WP-1A, WP-2C

• Threats/Hazards− H-XXX where XXX = threat number− Example: H-001, H-012

88

Mission Time Mission Time –– Detail Detail

The mission time display element shows the following:

1. Absolute Time

− These clocks are on the left side of the “Mission Time” title

− Two formats

Current Time: Current mission time

End Time: End time of the current mission

2. Relative Time

− These clocks are on the right side of the “Mission Time” title

− Two formats

Elapsed Time: How long has elapsed since the start of the scenario

Remaining Time: How long remains until the mission is over

Example Mission Time Display Element

Mission Execution Mission Execution –– Detail Detail –– 1 1

Each UAV has its own mission execution bar.

To bring it up on the left display, click anywhere on the desired UAV’s status window on the right display OR on the UAV icon itself on the left display.

Light green highlighting around the UAV’s status bar and its current mission plan on the map display tell you which UAV/route is currently selected

In the display below UAV 4 is highlighted so this is the mission execution bar on the left side of the left display.

Mission Execution Mission Execution –– Detail Detail –– 22

Mission Execution Functions

1. Arm Payload

− This button is only enabled if the UAV is selected while directly on top of a target, and within the arming or firing windows

2. Fire Payload

− This button is only enabled if the UAV is selected while directly on top of a target, armed, and within the firing window for that particular target

3. Skip Target

− This button is used if you decide to skip a target because you are going to be late to it. It causes the UAV to skip the next target/waypoint and move to the next waypoint/target

11 22

33

Mission Execution Mission Execution –– Detail Detail –– 3 3

Mission Execution Functions4. Reset Navigation

− Click anytime you have a significant course deviation and want to return to the planned course; it will return the UAV to the next plotted waypoint/target.

− Causes the UAV to be inactive while resetting; you may not be able to arm or fire with that UAV when its navigation system is resetting.

5. Radio Monitoring

− Radio chat from the Boston ground control will be playing and you must click the “Acknowledge PUSH” each time “push” is called on the radio chat. Hint: Push calls usually come in pairs (but not always); the tower and an aircrew or vice versa.

6. Target Assignment Queue

− You will not be using this portion of the display.

44

55

66

Right Display Right Display –– Detail Detail

The following slides detail all of the elements contained on the right display, in this order:

• UAV Status

• Health & Status Updates

• Decision Support

• Chat Box

A Reminder of How it All Fits Together A Reminder of How it All Fits Together –– Right DisplayRight Display

The four major screen elements

on the right display are:

1UAV Status

2Decision Support

3UAV Health &

Status Updates

4Chat Box

11 22

3344

89

UAV Status UAV Status –– Detail Detail –– 11

The UAV status display shows the following real-time information for each UAV:• Status / Current Action

−This is written out as well as represented by the color of the UAV icon to the right

−For example: a blue UAV would be loitering. This means it has arrived over the target but is there before the arming and firing window so is in a holding pattern waiting for these windows.

• Current Target Name• Position in Latitude & Longitude

−You will not need this in the scenario. −Given in degrees, minutes, and seconds

• Altitude−This is a static number that is not used

in the scenario in any way

(continued on next slide)

Example UAV Status Display ElementExample UAV Status Display Element

UAV Status UAV Status –– Detail Detail –– 22

The UAV status display shows the following real-time information for each UAV:

• Course

−You will not need this during the scenario.

−0˚ indicates due north; increases in a clockwise manner

• Speed

−The UAVs are set to travel at a constant speed of 200 kts.

−If there is significant head wind the UAV may slow down. This can precipitate late target arrivals

• Payload Ready

−This reflects whether the UAV has a payload ready for the current target

−Will say “Yes” if the UAV is armed, “No” if not

Example UAV Status Display ElementExample UAV Status Display Element

UAV Health & Status Updates UAV Health & Status Updates –– Detail Detail

The Health & Status Updates box contains messages from specific UAVs intended to inform the operator. Messages are color coded as follows:

• Red = UAV Health messages

− UAV is under fire from a threat

− Again, a standard audio alert will play when you receive red messages.

• Bold Black = UAV Status messages, action required

− UAV is available to arm or fire

• Black = UAV Status messages, no action required

− UAV has completed arming or firing

Example Health & Status Updates Window

Decision Support Decision Support –– Remember the Color CodingRemember the Color Coding

Color coding is an important element of the decision support, so take a look at it again!

GreenReturn to Base

OrangeFiring Payload

YellowArming Payload

BlueLoitering

GrayEnroute

ColorUAV Action

Decision Support Decision Support –– Detail Detail –– 1 1

The active level of decision support contains a visual representation of what and when targets are approaching through a relative timeline and projective decision support display for each UAV.

• The arming and firing windows cannot by changed solely at the will of the the operator. i.e. Operators may request time on target (TOT) delay requests, but must get approval before the arming and firing window will be moved back (if approved).

Example Active Decision Support Window

Decision Support Decision Support –– Detail Detail –– 2 2

Arming and firing elements are color coded in the same way as corresponding UAV actions. For each target the following information is represented:

1. Arming Window = Yellow

− 10 seconds long and takes approximately 3-7 seconds to arm.

− Payload for the relevant target may be armed, but not fired during this time

− Always occurs immediately before the firing window

2. TOT/Firing Window = Orange

− 20 seconds long and takes approximately 3-7 seconds to fire.

− Payload must be fired at the relevant target during this time

− In addition to the arming window, a payload may also be armed during this time

− Target name is printed vertically in the center of the window, priority is printed above the window.

90

Decision Support Decision Support –– Detail Detail –– 33

Mission planning information reflects when UAVs will reach imporMission planning information reflects when UAVs will reach important points tant points in the scenario, such as:in the scenario, such as:

1. Waypoints/Loiterpoints/Base = Black Triangles

− Names are printed above the relevant UAV’s timeline

2. UAV Arrival at Targets = Black Rectangles

− Names are printed below the relevant UAV’s timeline

− Note that each target name will appear twice on the timeline, once for when the UAV will arrive at that target and once at thecenter of that target’s firing window

3. Late UAV Arrival = Red Rectangles

− Black Rectangle turns red and moves past the target to when the UAV will arrive.


The active level of decision support aids the user by identifying possible late target arrivals and areas of potential high workload on the timeline.

• A late target arrival is defined as when a UAV will arrive to a target after its scheduled time on target

• Corrective actions for late arrivals are based on the priority of the target you are projected to be late to. For:

− Low priority targets, skip them by clicking “Skip Target” on the navigation screen.

− Medium priority targets, either skip them or use the decision support visualization (DSV) to possibly request a delay. Remember your priorities of wanting to hit all the targets. See the next four slides for an explanation of how to use the DSV (shown below).

− High priority targets, use the DSV and then decide whether to request a TOT delay or to skip the target by clicking “Skip Target”

Note: The corrective actions above should only be taken when a late arrival is projected by the red rectangle on the screen and your audio.

H


The decision support visualization (DSV) helps the user manage the schedule by showing timeline issues and projecting “what if”conditions of the effects on the timeline based upon user decisions.

• Each UAV’s DSV is uses emergent features to show problems that currently exist or that may exist if a TOT delay is given.

• No issues (late target arrivals) are indicated by no rectangles being displayed.

− The picture below shows the DSV for when there is no late arrivals. It depicts this by having no rectangles above the line on the left side of the display in the “Late Arrival”section.

Note: The DSV will be inactive except for when you are going to be late to a medium or high priority target.


• Late arrivals are represented by a rectangle occurring on the DSV above the line on the left side of the display in the “Late Arrivals” section. It will also be highlighted yellow as it is below.

− The target’s priority is indicated within the rectangle and by the rectangle’s size. The higher the priority of the target that the UAV will be late for the taller the rectangle will be.


• Below the center line is for the “what if” condition; after the user selects a target that they might request a TOT delay for itwill show the projected future late arrivals for that UAV below the centerline on the left side of the display in the “Projection”section.

− The example below shows that if a TOT delay request is granted for target T-16H, the UAV will then be late to a Low Priority target.

− The Probability in the bottom right of this display shows the likelihood of a TOT delay being granted. The further in advance a delay is requested the higher the likelihood of it being granted. Do not request a delay again if your first request for a delay on that target is denied.

H


Each UAV possesses a DSV display to help the user understand the potential effects of decisions

• A list of all the mission targets on the timeline appears to the right of each UAV’s DSV display.

• In using the display to the right for the top UAV:

− The user is considering requesting a delay for T-7H, a high priority target. However, it shows they will now be late to another high priority target even with this delay. This is where you, as the user, will have to make a value call. Do you request a delay or skip the target by clicking “Skip Target?”Most of the time you will not want to request a delay if you know it is going to create another delay.

• In using the display to the right for the second UAV:

− The user is considering requesting a delay for T-16H, a high priority target. This will result in the UAV being late to a low priority target. So in this case you, as the user, would want to request this delay because it means you can hit the high priority target for the trade-off of now missing a low priority target.

H

H

91

Chat Box Chat Box –– Detail Detail

The chat box contains a time history of all human interactions. These interactions are color coded as follows:

• Red = Intelligence updates

− Again, a standard audio alert will play when you receive red messages.

• Black = Message to/from Base, no response required

− Messages that inform you, but do not require a specific response

The chat box is purely informational. It will provide you with updates, but you will not input anything in the chat box.

Example Message History Window

ConclusionConclusion

You are now ready to proceed to hands-on training with the MAUVE interface. Remember to bring any questions you have to the

experimenter on testing day!

Predator B

Global Hawk

92

93

Appendix F: Post‐Experiment Survey

MAUVE Post-Test Feedback

1. How did the audio cues help or hinder you in managing late-arrivals?

________________________________________________________________________________________

________________________________________________________________________________________

________________________________________________________________________________________

________________________________________________________________________________________

________________________________________________________________________________________

2. How did the audio cues help or hinder you in managing course-deviations?

________________________________________________________________________________________

________________________________________________________________________________________

________________________________________________________________________________________

________________________________________________________________________________________

________________________________________________________________________________________

3. How would you change the audio cues for the late-arrivals to have them better aid you during your mission?

________________________________________________________________________________________

________________________________________________________________________________________

________________________________________________________________________________________

________________________________________________________________________________________

4. How would you change the audio cues for the course-deviations to have them better aid you during your mission?

________________________________________________________________________________________

________________________________________________________________________________________

________________________________________________________________________________________

________________________________________________________________________________________

________________________________________________________________________________________

5. In general how would you change all the audio to help you with the control during your mission?

________________________________________________________________________________________

________________________________________________________________________________________

________________________________________________________________________________________

________________________________________________________________________________________

________________________________________________________________________________________

________________________________________________________________________________________

6. Please express any other comments you may like to share:

________________________________________________________________________________________

________________________________________________________________________________________

________________________________________________________________________________________

________________________________________________________________________________________

________________________________________________________________________________________

94

Appendix G: GLM Analysis: SPSS OUTPUT

Included in this appendix are the key SPSS outputs for the data analysis.

G.1. Course‐deviation Reaction Times (for 4 audio conditions)

Met normality and homogeneity assumptions.

Table G‐1: Course‐deviation Reaction Times (4 audio conditions) Within‐Subjects Contrasts.

Source Scenario

Type III Sum

of Squares Df Mean Square F Sig.

Scenario Linear 14.029 1 14.029 3.215 .082

scenario *

Audio_Scheme Linear 10.780 3 3.593 .824 .490

Error(scenario) Linear 152.711 35 4.363

Table G‐2: Course‐deviation Reaction Times (4 audio conditions) Between‐Subjects Effects.

Source

Type III Sum


Intercept 8998.235 1 8998.235 1089.907 .000

Audio_Scheme 71.289 3 23.763 2.878 .050

Error 288.959 35 8.256

95

Table G‐3: Course‐deviation Reaction Times (4 audio conditions) Tukey Test Comparisons.

(I)

Audio_Scheme (J) Audio_Scheme

Mean

Differen

ce (I-J)

Std.

Error Sig. 95% Confidence Interval

Lower Bound Upper Bound

BothCont LateCont -1.6292 .90862 .294 -4.0796 .8213

DevCont .3667 .90862 .977 -2.0838 2.8171

BothThresh -1.7949 .93352 .237 -4.3125 .7227

LateCont BothCont 1.6292 .90862 .294 -.8213 4.0796

DevCont 1.9958 .90862 .144 -.4546 4.4463

BothThresh -.1657 .93352 .998 -2.6834 2.3519

DevCont BothCont -.3667 .90862 .977 -2.8171 2.0838

LateCont -1.9958 .90862 .144 -4.4463 .4546

BothThresh -2.1616 .93352 .114 -4.6792 .3560

BothThresh BothCont 1.7949 .93352 .237 -.7227 4.3125

LateCont .1657 .93352 .998 -2.3519 2.6834

DevCont 2.1616 .93352 .114 -.3560 4.6792

G.2. Course‐deviation Reaction Times (for 2 audio alerts)


Table G‐4: Course‐deviation Reaction Times (2 audio alerts) Within‐Subjects Contrast.

Source Scenario

Type III Sum


Scenario Linear 14.534 1 14.534 3.311 .077

Scenario *



96

Table G‐5: Course‐deviation Reaction Times (2 audio alerts) Between‐Subjects Effects.

Source

Type III Sum


Intercept 9007.397 1 9007.397 1146.991 .000

Audio_Scheme 69.684 1 69.684 8.874 .005

Error 290.564 37 7.853

G.3. Transformed (natural log) Late‐arrival Reaction Times

Met normality and homogeneity assumptions after original data was transformed with

a natural log transformation.

Table G‐6: Transformed (natural log) Late‐arrival Reaction Times Within‐Subjects Contrasts.

Source Scenario

Type III Sum


Scenario Linear 6.513 1 6.513 20.737 .000

scenario *

Audio_Scheme Linear 1.619 3 .540 1.719 .181

Error(scenario) Linear 10.992 35 .314

Table G‐7: Transformed (natural log) Late‐arrival Reaction Times Between‐Subjects Effects.

Source

Type III Sum


Intercept 244.837 1 244.837 305.851 .000

Audio_Scheme 8.028 3 2.676 3.343 .030

Error 28.018 35 .801

97

Table G‐8: Transformed (natural log) Late‐arrival Reaction Times Tukey Test Comparisons.

(I) Audio_Scheme (J) Audio_Scheme

Mean

Difference

(I-J)

Std.

Error Sig. 95% Confidence Interval

Lower Bound Upper Bound

BothCont LateCont .0092 .28293 1.000 -.7539 .7722

DevCont -.7099 .28293 .076 -1.4729 .0532

BothThresh .0633 .29069 .996 -.7206 .8473

LateCont BothCont -.0092 .28293 1.000 -.7722 .7539

DevCont -.7191 .28293 .071 -1.4821 .0440

BothThresh .0542 .29069 .998 -.7298 .8381

DevCont BothCont .7099 .28293 .076 -.0532 1.4729

LateCont .7191 .28293 .071 -.0440 1.4821

BothThresh .7732 .29069 .054 -.0107 1.5572

BothThresh BothCont -.0633 .29069 .996 -.8473 .7206

LateCont -.0542 .29069 .998 -.8381 .7298

DevCont -.7732 .29069 .054 -1.5572 .0107

G.4. Transformed (natural log) Late‐arrival Reaction Times (with

BothCont/LateCont/BothThresh Combined against DevCont)

Met normality and homogeneity assumptions after original data was transformed with

a natural log transformation.

Table G‐9: Transformed (natural log) Late‐arrival Reaction Times (with

BothCont/LateCont/BothThresh Combined against DevCont) Within‐Subjects Contrasts.

Source Scenario

Type III Sum


Scenario Linear 7.476 1 7.476 23.730 .000

scenario *

Audio_Scheme Linear .954 1 .954 3.029 .090

Error(scenario) Linear 11.657 37 .315

98

Table G‐10: Transformed (natural log) Late‐arrival Reaction Times (with

BothCont/LateCont/BothThresh Combined against DevCont) Between‐Subjects Effects.

Source

Type III Sum


Intercept 228.000 1 228.000 300.624 .000

Audio_Scheme 7.984 1 7.984 10.528 .002

Error 28.062 37 .758

G.5. NASA TLX Scores


Table G‐11: NASA TLX Scores Within‐Subjects Contrasts.

Source Scenario

Type III Sum


Scenario Linear 3.364 1 3.364 .058 .811

Scenario *

Audio_Scheme Linear 215.197 3 71.732 1.234 .312


Table G‐12: NASA TLX Between‐Subjects Effects.

Source

Type III Sum


Intercept 96127.322 1 96127.322 188.194 .000

Audio_Scheme 1244.548 3 414.849 .812 .496

Error 17877.551 35 510.787

99

G.6. Missed Radio Calls


Table G‐13: Missed Radio Calls Within‐Subjects Contrasts.

Source Scenario

Type III Sum


Scenario Linear 34.307 1 34.307 1.411 .243

scenario *



Table G‐14: Missed Radio Calls Between‐Subjects Effects.

Source

Type III Sum


Intercept 7966.465 1 7966.465 92.043 .000

Audio_Scheme 210.033 3 70.011 .809 .498

Error 3029.300 35 86.551

100

101

References

Arana‐Barradas, L. A. (2007). Predators ready to aid Missouri flood victims [Electronic Version]. Air Force Link, from http://www.af.mil/news/story.asp?storyID=123053041

Banks, R. L. (2000). The Integration of Unmanned Aerial Vehicles into the Function of Counterair. Air University, Maxwell Air Force Base, Alabama.

Barbato, G., Feitshans, G., Williams, R., & Hughes, T. (2003). Unmanned Combat Air Vehicle Control & Displays for Suppression of Enemy Air Defenses. Paper presented at the 12th International Symposium on Aviation Psychology, Dayton, Ohio.

Barrass, S., & Kramer, G. (1999). Using sonification. Multimedia Systems, 7, 23‐31. Begault, D. R., & Pittman, M. T. (1996). Three‐dimensional audio versus head‐down

traffic alert and collision avoidance system displays. International Journal of Aviation Psychology, 6(1), 79‐93.

Bolia, R. S., DʹAngelo, W. R., & McKinley, R. L. (1999). Aurally aided visual search in three‐dimensional space. Human Factors, 41(4), 664‐669.

Brewster, S. A., Wright, P. C., & Edwards, A. D. N. (1994). A Detailed Investigation into the Effectiveness of Earcons. In Auditory Display. Reading, Massachusetts: Addison‐Wesley Publishing Company.

Bronkhorst, A. W., Veltman, J. A., & vanBreda, L. (1996). Application of a three‐dimensional auditory display in a flight task. Human Factors, 38(1), 23‐33.

Burke, J. L., Prewett, M. S., Gray, A. A., Yang, L., Stilson, F. R. B., Coovert, M. D., et al. (2006). Comparing the Effects of Visual‐Auditory and Visual‐Tactile Feedback on User Performance: A Meta‐analysis. Paper presented at the 8th International Conference on Multimodal Interfaces, Banff, Alberta, Canada.

Conrad, R. (1985). Information processing rates in the elderly. Psychological Bulletin, 98, 67‐83.

Cooper, J. C., & Owen, J. H. (1976). Audiologic profile of noise‐induced hearing loss. Archives of Otolaryngology, 102(3), 148‐150.

Crease, M., & Brewster, S. A. (1998). Making Progress With Sounds‐‐The Design & Evaluation of an Audio Progress Bar. Paper presented at the ICAD 98, University of Glasgow, United Kingdom.

Culbertson, E. (2006). COMUSAFE: unmanned aircraft key to future decision superiority [Electronic Version]. Air Force Link, from http://www.af.mil/news/story.asp?storyID=123029520

Cummings, M. L., & Mitchell, P. J. (2008). Predicting Controller Capacity in Remote Supervision of Multiple Unmanned Vehicles. IEEE Systems, Man, and Cybernetics, Part A Systems and Humans, 38(2).

http://www.af.mil/news/story.asp?storyID=123053041


Cummings, M. L., Nehme, C. E., & Crandall, J. (2007). Predicting operator capacity for supervisory control of multiple UAVs. In J. S. Chahl, L. C. Jain, A. Mizutani & M. Sato‐Ilic (Eds.), Innovations in intelligent machines: Studies in computational intelligence (First Ed., Vol. 70). Australia: Springer.

Cummings, M. L., & Tsonis, C. G. (2005). Deconstructing Complexity in Air Traffic Control. Paper presented at the HFES 2005: 49th Annual Meeting of the Human Factors and Ergonomics Society, Orlando, Florida.

Deatherage, B. H. (1972). Auditory and other sensory forms of information processing. In H. P. VanCott & R. G. Kinkade (Eds.), Human engineering guide to equipment design (pp. 124). Washington, D.C.: American Institutes for Research.

Dixon, S., Wickens, C. D., & Chang, D. (2005). Mission control of multiple unmanned aerial vehicles: A workload analysis. Human Factors, 47, 479‐487.

DOD. (2007). Unmanned Systems Roadmap (2007‐2032). Office of the Secretary of Defense, Washington, D.C.

Economist. (2007). Unmanned and dangerous [Electronic Version]. The Economist, from http://www.economist.com/printedition/displaystory.cfm?story_id=10202603

Flanagan, P., McAnally, K., Martin, R., Meehan, J., & Oldfield, S. (1998). Aurally & visually guided visual search in a virtual environment. Human Factors, 40(3), 461‐468.

Gates, R. M. (2008). Speech at Air University. Unpublished manuscript, Maxwell Air Force Base, Alabama.

Hart, S., & Staveland, L. (1988). Development of the NASA‐TLX: Results of empirical and theoretical research. In P. A. Hancock & N. Meshkati (Eds.), Human mental workload (First Ed., pp. 139‐183). Amsterdam: North Holland.

Hirst, W. (1986). The psychology of attention. In J. LeDoux & W. Hirst (Eds.), Mind and brain (pp. 105‐141). New York: Cambridge University Prep.

Humes, L. E., Joellenbeck, L. M., & Durch, J. S. (Eds.). (2006). Noise and Military Service Implications for Hearing Loss and Tinnitus. Washington, D.C.: Institute of Medicine of the National Academies.

Kahneman, D. (1973). Attention and effort. Englewood Cliffs, New Jersey: Prentice‐Hall. Kramer, G. (1994). Some organizing principles for representing data with sound. In

Auditory Display (pp. 1‐77). Reading, Massachusetts: Addison‐Wesley Publishing Company.

Loeb, R. G., & Fitch, W. T. (2002). A Laboratory Evaluation of an Auditory Display Designed to Enhance Intraoperative Monitoring. Anesthesia and Analgesia, 94, 362‐368.

Martin, R. L., McAnally, K. I., & Senova, M. A. (2001). Free‐field equivalent localization of virtual audio. Journal of Audio Engineering Society, 49, 14‐22.

Mayer, R. E. (1999). Instructional Technology. In F. Durso (Ed.), Handbook of Applied Cognition (pp. 551‐570). Chichester, U.K.: John Wiley.

102

http://www.economist.com/printedition/displaystory.cfm?story_id=10202603

Miller, J. (1991). Channel interaction and the redundant‐targets effect in bimodal divided attention. Journal of Experimental Psychology: Human Perception and Performance, 17(1), 160‐169.

Mitchell, P. J. (2005). Mitigation of Human Supervisory Control Wait Times through Automation Strategies. Massachusetts Institute of Technology, Cambridge, MA.

Moray, N. (1967). Where is capacity limited? A survey and a model. Acta Psychologica, 27, 84‐92.

Moray, N. (1981). The role of attention in the detection of errors and the diagnosis of errors in man‐machine systems. In J. Rasmussen & W. Rouse (Eds.), Human detection and diagnosis of system failures. New York: Plenum.

Moroney, B. W., Nelson, W. T., Hettinger, L. J., Warm, J. S., Dember, W. N., Stoffregen, T. A., et al. (1999). An Evaluation of Unisensory and Multisensory Adaptive Flight Path Navigation Displays: An Initial Investigation. Paper presented at the Human Factors and Ergonomics Society 43rd Annual Meeting, Houston, Texas.

Nehme, C. E., Crandall, J. W., & Cummings, M. L. (2007). An Operator Function Taxonomy for Unmanned Aerial Vehicle Missions. Paper presented at the 12th International Command and Control Research and Technology Symposium, Newport, Rhode Island.

Nehme, C. E., & Cummings, M. L. (2006). Audio Decision Support for Supervisory Control of Unmanned Vehicles (HAL2006‐06). Cambridge, Massachusetts: Humans and Automation Laboratory.

Nelson, W. T., Hettinger, L. J., Cunningham, J. A., Brickman, B. J., Haas, M. W., & McKinley, R. L. (1998). Effects of localized auditory information on visual performance using a helmet‐mounted display. Air Force Research Laboratory, Wright‐Patterson Air Force Base, 40(3), 452‐460.

Osga, G., VanOrden, K., Campbell, N., Kellmeyer, D., & Lulue, D. (2002). Design and Evaluation of Warfighter Task Support Methods in a Multi‐Modal Watchstation (Tech. Report 1874). San Diego, California: Space & Naval Warfare Center.

Pacey, M., & MacGregor, C. (2001). Auditory Cues for Monitoring a Background Process A Comparative Evaluation. Paper presented at the Human Computer Interaction: INTERACT ʹ01, Tokyo, Japan.

Parasuraman, R., Sheridan, T. B., & Wickens, C. D. (2000). A Model for Types and Levels of Human Interaction with Automation. IEEE Transaction on Systems, Man, and Cybernetics ‐ Part A: Systems and Humans, 30, 286‐297.

Parasuraman, R., Warm, J., & Dember, W. (1987). Overview paper: Vigilance: Taxonomy and utility. In L. Mark, J. Warm & R. Huston (Eds.), Ergonomics and human factors: Recent research. New York: Springer Verlag.

Parker, S. P. A., Smith, S. E., Stephan, K. L., Martin, R. L., & McAnally, K. I. (2004). Effect of supplementing head‐down displays with 3‐D audio during visual target acquisition. International Journal of Aviation Psychology, 14(3), 277‐295.

103

Randolph, M. (2007). Changes on the horizon for Air Force pilots [Electronic Version]. Air Force Link, from http://www.af.mil/news/story.asp?storyID=123054831

Sanders, M. S., & McCormick, E. J. (1993). Human Factors In Engineering and Design (Seventh Ed.). New York: McGraw‐Hill, Inc.

Schroeder, S. (2008). Enhance Fire Scout Makes Flight Debut [Electronic Version]. Navy.mil, from http://www.news.navy.mil/search/display.asp?story_id=27145

Scott, S. D., Mercier, S., Cummings, M. L., & Wang, E. (2006, October 16‐20). Assisting Interruption Recovery in Supervisory Control of Multiple UAVs. Paper presented at the HFES 2006: 50th Annual Meeting of the Human Factors and Ergonomics Society, San Francisco, California.

Simpson, C., & Williams, D. H. (1980). Response time effects of alerting tone and semantic context for synthesized voice cockpit warnings. Human Factors, 22, 319‐330.

SkyGeek. (2008). David Clark Headsets [Electronic Version]. Skygeek.com, from http://www.skygeek.com/h20‐10.html

Snodgrass, J. G. (1975). Psychophysics. In B. Scharf (Ed.), Experimental Sensory Psychology. Glenview, Illinois: Scott Foresman & Co.

Sorkin, R. D. (1987). Design of auditory and tactile displays. In G. Salvendy (Ed.), Handbook of human factors (pp. 549‐576). New York: Wiley.

St.John, M., Smallman, H. S., & Manes, D. I. (2005). Recovery from Interruptions to a Dynamic Monitoring Task: the Beguiling Utility of Instant Replay. Paper presented at the HFES 2005: 49th Annual Meeting of the Human Factors and Ergonomics Society, Orlando, Florida.

Streeter, L. A., Vitello, D., & Wonsiewicz, S. A. (1985). How to tell people where to go: Comparing navigational aids. International Journal on Man‐Machine Studies, 22, 549‐562.

Staff. (2008). Rise of the Machines: UAV Use Soars [Electronic Version]. Military.com: Today in the Military, from http://www.military.com/NewsContent/0,13319,159220,00.html

Sullivan, G. R. (2008). U.S. Army Aviation: Balancing Current and Future Demands, Torchbearer National Security Report. Arlington, Virginia: Association of the United States Army.

Tannen, R. S. (1998). Breaking the Sound Barrier: Designing Auditory Displays for Global Usability. Paper presented at the 4th Conference on Human Factors & the Web, Basking Ridge, New Jersey.

Tsach, S., Peled, A., Penn, D., Keshales, B., & Guedj, R. (2007). Development Trends for Next Generation UAV Systems. Paper presented at the AIAA 2007 Conference and Exhibit, Rohnert Park, California.

USAF. (2007). Air Force chief of staff initiates MQ‐1 Predator plus‐up [Electronic Version]. Air Force Link, from http://www.af.mil/news/story.asp?id=123060692

104


http://www.news.navy.mil/search/display.asp?story_id=27145

http://www.skygeek.com/h20-10.html

http://www.military.com/NewsContent/0,13319,159220,00.html

http://www.af.mil/news/story.asp?id=123060692

105

Walden, B. E., Prosek, R. A., & Worthington, D. W. (1975). The Prevalence of Hearing Loss within Selected U.S. Army Branches. Washington, D.C.: Walter Reed Army Medical Center.

Watson, M., & Sanderson, P. (2004). Sonification Supports Eyes‐Free Respiratory Monitoring and Task Time‐Sharing. Human Factors, 46(3), 497‐517.

Whitney, R. (2007). Air Force stands up first unmanned aircraft systems wing [Electronic Version]. Space War: Your World at War, from http://www.spacewar.com/reports/Air_Force_Stands_Up_First_Unmanned_Aircraft_Systems_Wing_999.html

Wickens, C. D., & Hollands, J. G. (2000). Engineering Psychology and Human Performance (Third Ed.). Upper Saddle River, New Jersey 07458: Prentice Hall.

Wickens, C. D., Lee, J. D., Liu, Y., & Becker, S. E. G. (2004). An Introduction to Human Factors Engineering (Second Ed.). Upper Saddle River, New Jersey 07458: Pearson Prentice Hall.

Wightman, F. L., & Kistler, D. J. (1989). Headphone simulation of free‐field listening. I: Stimulus synthesis. Journal of the Acoustical Society of America, 85(2), 858‐867.

Wundt, W. M. (1902). Principles of Physiological Psychology. London: Swan Sonnenschein & Co. Ltd.

http://www.spacewar.com/reports/Air_Force_Stands_Up_First_Unmanned_Aircraft_Systems_Wing_999.html

http://www.spacewar.com/reports/Air_Force_Stands_Up_First_Unmanned_Aircraft_Systems_Wing_999.html

Effect of Auditory Peripheral Displays Unmanned …Effect of Auditory Peripheral Displays On...

Documents

Transcript of Effect of Auditory Peripheral Displays Unmanned …Effect of Auditory Peripheral Displays On...