Capstone Project EAD 861 1 Running Head: Capstone Project ...
AJUDY ERAU Capstone Project
-
Upload
keith-craine -
Category
Documents
-
view
1.355 -
download
3
Transcript of AJUDY ERAU Capstone Project
AN ANALYSIS OF UNMANNED SYSTEMS CONTROL STATIONS AND AIRCREW TRAINING AS IT PERTAINS TO SAFETY OF FLIGHT
by
Aaron David Judy
A Graduate Capstone Project Submitted to the Extended Campus
in partial fulfillment of the Requirements of Degree of Master of Science in Aeronautics
Embry-Riddle Aeronautical University Extended Campus
Patuxent River Center March 2008
AN ANALYSIS OF UNMANNED SYSTEMS CONTROL STATIONS AND AIRCREW TRAINING AS IT PERTAINS TO SAFETY OF FLIGHT
by
Aaron David Judy
This Graduate Capstone Project was prepared under the direction of the candidate’s Project Review Committee Member,
Mr. Duane Mallicoat, Adjunct Associate Professor, Extended Campus, and the candidates Project Review Committee Chair,
Dr. C. J. Schumaker, Associate Professor, Extended Campus, and has been approved by the Project Review Committee. It was submitted
to the Extended Campus in partial fulfillment of the requirements for the degree of
Master of Science
PROJECT REVIEW COMMITTEE:
______________________________ Mr. Duane Maillicoat, Ed.D
Committee Member
______________________________ Dr. C. J. Schumaker, Ph.D
Committee Chair
ii
ABSTRACT
Researcher: Aaron David Judy
Title: An Analysis of Unmanned Systems Control Stations and Aircrew Training as it Pertains to Safety of Flight Institution: Embry-Riddle Aeronautical University Degree: Master of Science Aeronautics Year: 2008
While Unmanned Aerial Vehicles (UAVs) have been known around the world since the early
twentieth century, technology hindered application to present day situations. Over the past
ten years with technology advances in materials and microprocessors these UAVs have
exponential applications important to agriculture, law enforcement, air carriers, military,
homeland security, and private use. However, issues associated with Unmanned Control
Station (UCS) design and aircrew training is impacting UAV safety of flight. UAVs truly
became a part of our aviation sector during the Afghanistan war shortly after 9/11. They were
not only providing sensor capability but also began to release weapons. This has created a
paradigm shift in aviation and is reason to believe UAVs are here to provide a valuable asset
to the military and civilian sectors alike. The purpose of this study was to analyze the effects
control station design and aircrew training has upon UAV safety of flight. For the purpose of
this research, a survey was conducted on engineers, scientists, or technicians, unmanned
pilots, and manned pilots to address this problem. A literature review of current research and
the respondent’s answers were carefully evaluated and statistically analyzed to determine the
effect upon UAV safety of flight and the need for commonality.
iii
TABLE OF CONTENTS
Page
PROJECT REVIEW COMMITTEE ii
ABSTRACT iii
LIST OF TABLES vi
LIST OF FIGURES vii
Chapter
I INTRODUCTION 1
Background of the Problem 1
The Statement of the Problem 2
The Statement of the Sub-problems 2
The Study Questions 2
Limitations 2
Assumptions 2
Importance of the Study 3
Researcher’s Work Setting and Role 3
Definition of Terms 4
Summary 5
II REVIEW OF LITERATURE AND RESEARCH 7
SEARCH STRATEGY 7
UNMANNED SYSTEMS 7
Unmanned Control Stations 10
iv
Unmanned Aircrew Training 14
Summary 15
III RESEARCH METHOD 17
The Statement of the Problem 17
The Statement of the Sub-problems 17
The Study Questions 17
Research Model 17
Survey Population and Sample Size 18
Specific Treatment of the Data 18
Data Needed 18
Data Location 18
Securing the Data 18
Treatment of the Data 19
Data Analysis 19
IV RESULTS AND DISCUSSION 20
VI CONCLUSIONS AND RECOMMENDATIONS 30
REFERENCES 31
APPENDIXES 33
A DATA COLLECTION DEVICE 33
v
LIST OF TABLES
Table Page
1 UAV Levels of Control 5
2 Human Factor Issues for Hunter Accidents 8
3 Question 3 Chi Square Test 21
4 Question 4 Chi Square Test 23
5 Question 9 Chi Square Test 24
6 Question 5 Chi Square Test 26
7 Question 6 Chi Square Test 27
8 Question 8 Chi Square Test 29
vi
LIST OF FIGURES
Figure Page
1 U.S. Army Hunter UAV 6 2 U.S. Air Force Predator UAV 7 3 Air Force Predator Accident Factors 8 4 Predator UCS 10 5 Survey Question 3 20 6 Survey Question 4 22 7 Survey Question 9 24 8 Survey Question 5 25 9 Survey Question 6 26 10 Survey Question 7 28 11 Survey Question 8 28
vii
viii
1
CHAPTER I
INTRODUCTION
Background of the Problem
Aircrew training and safety of flight have been a concern for aircraft since Orville and
Wilbur Wright first flew the Wright Flyer in 1903 (Sossong, 2006). Many Class A mishaps
have occurred due to inadequate aircrew training and safety of flight issues (Sossong, 2006).
This has been a known issue with manned aircraft and is currently a serious concern with
unmanned aircraft. Unmanned aircraft have been quickly fielded without addressing the need
for commonality and is beginning to hinder advanced development.
Unmanned Aerial Vehicles (UAVs) consist of a variety of vehicles both fixed wing
and rotary wing ranging from micro-size to as big as a Boeing 737 civilian transport
(Sossong, 2006). UAV operators use a variety of different Unmanned Control Station(s)
(UCS) to operate UAVs. Sometimes more than one UCS is used and the layout can differ
depending on what particular type of UAV is being operated and the flight environment. This
is a solid reason why UAV uncommon systems can limit common training and impact safety
of flight (Sossong, 2006).
Poor aircrew training and a non-standard Human Machine Interface (HMI) have a
direct impact on UAV safety of flight. Williams stated, “Estimates of the percentage of
accidents that implicate human error range from 70% to 80%” (Williams, 2004, p. 1). “In
addition, over the past 40 years, the percentage of accidents attributable to human error has
increased relative to those attributable to equipment failures” (Williams, 2004, p. 1).
Therefore, there is a strong need to develop a common UCS and develop a UAV aircrew
training method to solve poor aircrew training and the safety of flight implication.
2
The Statement of the Problem
UAVs consist of a variety of uncommon control station functions and inadequate
aircrew training which can impact UAV safety of flight.
The Statement of the Sub-problems
1. The first sub-problem. The first sub-problem is to determine the effect on how
uncommon control stations impacts UAV safety of flight.
2. The second sub-problem. The second sub-problem is to determine how inadequate
aircrew training impacts UAV safety of flight.
Study Questions
1. The first study question. The first study question is to determine how and if
uncommon control stations impact UAV operations.
2. The second study question. The second study question is to determine if aircrew
training impacts UAV safety of flight.
Limitations
Since many civil and government organizations have yet to determine the
qualifications of UAV operators or pilots, there will be a limited amount of qualified
candidates to research. However, government and civil organizations have used manned
pilots to fill this gap. This will allow the study to investigate how pilots are experiencing
uncommon control stations and inadequate aircrew training. This study should be completed
again once civil and government organizations establish solid guidelines for UAV pilots.
Assumptions
UAVs are continually increasing in size, number, and complexity. Therefore, this
study will assume the problem and results are applicable to all types of UAV systems. There
3
will also continue to be an interest in UAVs as alternatives to manned aircraft. These two
assumptions will allow the researcher to come to conclusions and recommendations worth
carrying over into future research or studies.
The Importance of the Study
While Unmanned Aerial Vehicles (UAVs) have been known around the world since
the early twentieth century, technology hindered application to present day situations. Over
the past ten years with technology advances in materials and microprocessors these UAVs
have exponential applications important to agriculture, law enforcement, air carriers, military,
homeland security, and private use. However, many UAV systems have not been thoroughly
designed neither analyzed before being presented in a real world situation with real world
operators (humans). Many UAV operators will easily say, “There is nothing Unmanned about
an Unmanned Aerial Vehicle” (Sossong, 2006). Human operators have experienced
catastrophic failures due to poor aircrew training and uncommon systems. There are many
important crew resources to consider when operating a UAV. Some of these factors include
following checklists, workload, situational awareness, fatigue, training and autonomy
(Sossong, 2006). UAVs are the next frontier in aviation similar to jet and space flight were in
the late 1950’s and early 60’s. However, if UAVs are going to provide the next leap in
aviation, the concerns with inadequate aircrew training and uncommon control station design
need to be researched and analyzed.
Researcher’s Work Setting and Role
The researcher currently has numerous experiences in UAV engineering design,
testing, and flight operations. The researcher holds a B.S. in Aerospace Engineering and a
B.S. in Mechanical Engineering. Finally, the researcher is considered a subject matter expert
4
in UAV flight test engineering at the Naval Air Warfare Center Aircraft Division, Patuxent
River, Maryland.
Definition of Terms
Autonomous. Describes a self-contained system that carries out programs or performs
tasks without outside control by acquiring, processing and acting on information (Adams,
2006).
Class A Mishap. A mishap (accident or incident) that results in loss of aircraft or
damage of one million dollars or greater, or loss of life.
External Pilot (EP). A UAV pilot external to the UCS operation, normally a type of
Radio Control pilot who manipulates the UAVs control surfaces.
Federal Aviation Administration (FAA). The federal agency responsible for the safety
of civilian aviation and enforcement of Federal Aviation Regulations.
Human Machine Interface (HMI). Where the human and the machine meet as well as
interact for a given task. Interaction can include touch, sight, sound, heat transference or any
other physical or cognitive function (Adams, 2006).
Internal Pilot (IP). A UAV pilot who controls the UAV with the UCS functions.
Mission Commander (MC). The UAV operator responsible for ensuring critical
mission functions is being prioritized within the aircraft’s flight limits and environment.
National Transportation Safety Board (NTSB). An independent federal agency
responsible for investigating civil aviation accidents (NTSB, 2007).
North Atlantic Treaty Organization (NATO). An alliance of 26 countries from North
America and Europe committed to fulfilling the goals of the North Atlantic Treaty signed on 4
April 1949 (NATO, 2007).
5
Naval Air Warfare Center Aircraft Division (NAWCAD). A Department of Navy
branch responsible for aircraft and aircraft system test and engineering.
Remotely Piloted. A pilot who controls an aircraft outside the physical domain of the
aircraft.
Standardization Agreement 4586 (STANAG). A NATO standardization agreement
that addresses the interoperability of UAVs through the UCS (Kirschbaum, 2002).
Unmanned Control Station (UCS). Provides the UAV operator with the functionality
to conduct all phases of a UAV mission. Also provides a high resolution, computer
generated, graphical user capability that enables a qualified UAV operator the ability to
control different types of UAVs and payloads (Kirschbaum, 2002).
Unmanned Aerial Vehicle (UAV). A powered, aerial vehicle that does not carry a
human operator uses aerodynamic forces to provide vehicle lift, can fly autonomously or be
piloted remotely, can be expendable or recoverable, and can carry a lethal or nonlethal
payload (Kirschbaum, 2002).
Summary
The sections above provide a background on why this research is an important one to
study. UAVs are not much different than manned aircraft when it comes to operations and
aviation rigor. The aviation community is becoming more familiar with these types of
systems and how they are impacting our aviation sector. Many organizations realize the
importance of aircrew training, common control, and the impact of safety of flight.
Unmanned aircraft obviously do not carry humans, but this does not mean dollars, resources,
and assets are worth loosing over poor implementation of aviation standards.
6
There have been many research topics written about the safety of UAVs. The
Department of Defense has pioneered the way for unmanned systems, but many of these
systems are still very immature. The following section will address how the problem
statement above ties into recently conducted research in the area of unmanned systems. The
review of literature and research will also provide more evidence towards the importance of
this study.
7
CHAPTER II
REVIEW OF LITERATURE AND RESEARCH
SEARCH STRATEGY
When researching this topic, many libraries, books, periodicals, magazines,
newspapers, and the World Wide Web were used. Key words consisted of UAV aircrew
training, human factors issues with unmanned control stations, and looking up mishap or
accidents that had occurred because of poor control station design and inadequate aircrew
training. The Embry-Riddle Aeronautical University Hunt Library Voyager and periodical
database was used to pull recent studies and results of these types of issues. Also, course
materials from the Embry-Riddle Aeronautical University Master of Aeronautical Science
program were used to provide an academic basis for conducting and analyzing the research.
The following pages and headings provide the results of this search strategy.
UNMANNED SYSTEMS
There are 5 levels of control associated with operating UAVs. Each UAV is different
in size and mission and therefore identified via a level of control. The levels of control shown
on the following page in Table 1 are based on the North Atlantic Treaty Organization
(NATO) Standardization Agreement (STANAG) 4586.
8
Table 1.
UAV Levels of Control
Level Explanation
I Indirect receipt/transmission
of UAV related payload data.
II Direct receipt of data where “direct”
covers reception of the UAV payload
communication with the UAV.
III Control and monitoring of the
UAV payload in addition to
direct receipt of data.
IV Control and monitoring of the
UAV, less launch and
recovery.
V Control and monitoring of the
UAV (Level IV) plus launch and
recovery functions.
Note. From Kirshbaum, A. (2002). NATO UAV Systems Interoperability. Unmanned Systems, 20-22. Reprinted with permission.
The most complex level of control is level V. This level not only includes controlling and
monitoring the UAV, but also launching and recovering the UAV. UAVs can be categorized
into two types: autonomous or remotely piloted. Autonomous UAVs perform missions under
control of software and a human operator controls remotely piloted UAVs
9
(Sossong, 2006). Present day UAVs are mostly a combination of both autonomous and
remotely piloted. Two specific examples are the Army’s Hunter UAV (Figure 1) and the Air
Force’s Predator UAV (Figure 2).
Figure 1. U.S. Army Hunter UAV. Note. Retrieved May 29, 2007, from http://media.primezone.com/noc/gallery/display?o=189&pkgid=2361&max=9&start=0.
Figure 2. U.S. Air Force Predator UAV. Note. Retrieved May 29, 2007, from http://www.globalsecurity.org/intell/systems/images/rq-1_011130-f-3869j-006.jpg.
10
Unmanned Control Stations
Control station design is highly dependent upon implementing the appropriate
fundamentals of human factors. Most human factor subject matter experts will agree that the
more human interaction with the machine the more likely the cause of error. The U.S.
Army’s Hunter UAV has many examples of these types of human implications. Almost 47%
of human factors related Hunter accidents occurred during External Pilot (EP) landings
(Williams, 2004). An additional 20% of the accidents occurred during takeoff (Williams,
2004). Most of the problem lies with the pilots’ perception when landing the UAV. When
the aircraft is approaching, the control inputs to maneuver the aircraft are opposite what they
would be if the aircraft were moving away from the pilot.
Display design is an issue if the right information is not being displayed at the appropriate
time or refresh rate. UAVs normally have an installed payload such as a day or night looking
camera to identify targets of interest. “Overall mission success is normally determined by the
crewmembers’ efficiency in communicating target location” (Draper, Geiselman, Lu, Roe, &
Haas, 2000, ¶ 3). Therefore, the display information is an integral part of the UAV mission.
The Internal Pilot (IP) flying the UAV from a display, keyboard, and mouse, or in some cases
a joystick, rudder pedals, and throttle controls normally gets situational awareness from the
display and voice communications with the EP and Mission Commander (MC). For example,
the Hunter UAV had an incident where a crewmember toggled off the autopilot feature, and
the information was not properly displayed. The autopilot is normally engaged when the IP is
flying the UAV, but in this case it was not and thus contributed to an accident (Williams,
2004). There are many crew errors that have contributed to this type of accident, such as
11
following the wrong procedures or in some cases the lack of procedures. Table 2 shows the
breakdown of specific Hunter human factor errors.
Table 2
Human Factor Issues for Hunter Accidents.
Issue Number Percent
Pilot-In-Command 1 7%
Alerts and Alarms 2 13%
Display Design 1 7%
External Pilot Landing Error 7 47%
External Pilot Takeoff Error 3 20%
Procedural Error 3 20%
Note. From “A Summary of Unmanned Aircraft Accident/Incident Data” by Williams, 2004, p. 5.
The U.S. Air Force Predator is also a commonly flown UAV; however the Predator is
flown all within the Unmanned Control Station (UCS) with a joystick, keyboard, rudder
pedals, and throttle controls. Also, a forward looking camera attached to the UAV provides a
3-degree field of view. Predator human factor incidents were accounted at 67% as seen in
Figure 3, much higher than Hunter.
12
Figure 3. Air Force Predator Accident Factors. Note. From “A Summary of Unmanned Aircraft Accident/Incident Data” by Williams, 2004, p. 10. The Predator has experienced many aircrew error related accidents. One error occurred
when the pilot activated a program that erased the aircraft memory. One extreme example
was the way the function keys were assigned on the keyboard. A particular order of pressing
function keys that controlled the lights on Predator was almost the same order as cutting off
the engine.
A recent example with the U.S. Customs Border Patrol Predator on April 26, 2006 shows
the concern and implications these systems can have on safety of flight. The flight was a
routine flight conducting border surveillance around the Arizona-Mexico border near
Nogales, Arizona. The flight was operating in visual meteorological conditions and an
instrument flight rules plan had been filed. The flight originated from Libby Army Airfield
(HFU), Sierra Vista, Arizona. The flight was flown from HFU by the UCS. The UCS is
configured for two operators, a pilot who controls the vehicle, pilot payload operator (PPO-1)
and a pilot payload operator (PPO-2). The controls are identical for both PPO-1 and PPO-2
(National Transportation Safety Board (NTSB), 2006). However, when control is being
13
performed from PPO-1, the controls at PPO-2 are used to control the payload or camera.
Figure 4 below shows the Predator UCS layout.
Figure 4. Predator UCS. Note. From “Unmanned Aerial Vehicles for Maritime Patrol” by Hopcroft, Burchat, & Vince, 2006, p. 6. The pilot (PPO-1) during flight reported a console "lock-up", prompting him to switch
control of the UAV to PPO-2. Checklist procedures state that prior to switching control
between the two consoles, the pilot must match the control positions at the new console as
was configured at the previous console. The PPO-2 operator stated that he did not perform
this task. Therefore, the stop/feather control in PPO-2 was in the fuel cutoff position when the
switch from PPO-1 to PPO-2 occurred (NTSB, 2006).
PPO-1 stated that after he switched control to PPO-2 the UAV was not maintaining
altitude but did not know why. Therefore he decided to shut down the UCS so the UAV
would enter its lost link procedure. The lost link procedure tells the UAV to climb to 15,000
feet mean sea level and fly a loiter pattern until link can be established (NTSB, 2006).
However, with no engine power the UAV continued to descend and ultimately crashed about
1,000 feet from nearby homes. A USA Today writer Alan Levin recently published an article
14
about the Predator crash and interviewed nearby witnesses stating, "It glided as close as 100
feet above two homes before striking the ground" (Levin, 2006, ¶5). The NTSB has only
issued a preliminary report (NTSB ID: CHI06MA121) and has yet to issue a probable cause
or any recommendations. This incident is an excellent example of how standards, human
factors, aircrew training all can affect UAV safety of flight.
Unmanned Aircrew Training
There have been two issues effecting safety of flight as seen in the examples above,
control station design and aircrew training. The human factors and Ergonomics Society
conducted an experiment on crew simulations for UAV applications and found training was a
consistent issue. Sixteen operators were studied and during the simulation indicated flight
control problems were related to poor training (Barnes and Matz, 1998). Therefore, if
unmanned pilots are not properly trained, the implication can be detrimental to flight
operations and mission success.
Currently both IP and EP UAV pilots have been mostly taken from the manned pilot
pool. There are benefits to this approach, but there are also consequences. Conventional
manned pilots are use to stick, rudder, and throttle controls. Manned pilots also do not
experience the out of the cockpit feeling resulting in a different human stimulation. Also, as
mentioned previously, manned pilots become very confused when the UAV is approaching
them in a landing pattern since the controls then become reversed. The Air Force is currently
looking at ways to better train UAV pilots, which is probably a combination of some stick
time, but more UAV flight time, to keep the operator from having to re-learn a new way of
flying.
15
The solid goal of training should be to improve mission effectiveness by minimizing
crew preventable errors, maximizing crew coordination, and optimizing risk management.
(Department of Navy, 2006). This goal should not only be applied to manned operations, but
also applied and enforced with unmanned operations. Normally, the outcome of emergencies
is dependent on the quality of teamwork the crew tries and utilizes to avoid conflicts.
Therefore, the success or failure of a team is dependent on well-laid down rules for the
activity (Sharma & Chakravarti, 2005). All of the principles of communication, situational
awareness, problem solving, decision making, judgment, leadership-followership, stress, and
interpersonal skills are also vital to effective UAV Crew Resource Management (CRM).
Formal aircrew training has helped integrate teams and more importantly help limit
and avoid fatal errors. The goal of a UAV training program should be similar to the goals of
manned operations. These goals should include, understanding the UAV operator
environment, identifying important teamwork roles to safely execute missions, and enforcing
standard operating procedures. Therefore, as explained above, there is a definitive need for a
dedicated UAV training program.
Summary
The U.S. Army Hunter and the U.S. Air Force Predator were only two examples given
on how control station design and aircrew training can impact safety of flight. There are
many different categories of UAVs; some much smaller than the Predator or Hunter, but the
same principle applies. Government or civilian organizations have yet to determine the
qualifications for a UAV pilot, aircrew training, or commonality between systems. It is
important to note, that these issues are trying to be addressed with the recently instated U.S.
Air Force UAV training school at Creech Air Force Base in Nevada. The U.S. Army has also
16
begun to setup training programs for the Hunter at Fort Huachuca, Arizona. However, other
communities have yet to determine if the U.S. Air Force or U.S. Army will establish these
guidelines or other military organizations will have an impact. Whoever establishes these
guidelines rather military or maybe the Federal Aviation Administration, the need for them is
now. Therefore, the results of this study will provide quantitative and qualitative data to help
determine an appropriate path. The study results will be extremely helpful to both policy
makers, maintainers, program offices, and more importantly the warfighter.
There are many study questions or hypotheses that could be derived from the research
shown above. However, for this study the area of importance is uncommon control station
design and inadequate aircrew training. When conducting the review of relevant literature,
these two topics continuously were brought up in many different areas of UAV research.
Therefore, the following section will address how these two issues will be researched and
analyzed.
17
CHAPTER III
RESEARCH METHOD
The Statement of the Problem
UAVs consist of a variety of uncommon control station functions and inadequate
aircrew training which can impact UAV safety of flight.
The Statement of the Sub-problems
1. The first sub-problem. The first sub-problem is to determine the effect on how
uncommon control stations impacts UAV safety of flight.
2. The second sub-problem. The second sub-problem is to determine how inadequate
aircrew training impacts UAV safety of flight.
Study Questions
1. The first study question. The first study question is to determine how and if
uncommon control stations impact UAV operations.
2. The second study question. The second study question is to ask and determine if
aircrew training impacts UAV safety of flight.
Research Model
The research model consisted of a straight forward quantitative descriptive method of
analyzing the first sub-problem and the second sub-problem. Both sub-problems data was
generated by a web survey found in Appendix A. The objective was to answer the study
questions by surveying UAV systems subject matter experts. A quantitative statistical
analysis of the results was then compounded to support the researchers’ conclusions and
recommendations.
18
Survey Population and Sample Size
The survey population consisted of three categories: Engineers, scientists, or
Technicians; unmanned pilots/operators, and manned pilots. Since there are very few
qualified unmanned pilots it is difficult to determine the total population size of this category.
Engineers, scientists, or technicians and manned pilots exceed 5,000 in population. However,
since fewer unmanned pilots are part of the population, this decreased the total population
size. Therefore to get an accurate estimate of the total population, 25 engineers, scientists, or
technicians, 25 unmanned pilots/operators, and 25 manned pilots made a total population size
of 75. Since the survey population is less than 100, the entire 75 was surveyed.
Specific Treatment of the Data
Data Needed.
The data needed to resolve the problem were the perceptions and opinions from the
three categories on how control station design and inadequate aircrew training can affect
UAV safety of flight. All of the data was carefully collected and analyzed as detailed below.
Once all data was collected it was presented in a sequential order and can be found in Chapter
IV.
Data Location.
The location of the data was contained in the responses of the web survey and the
literature review as a guide to collect the data.
Securing the Data.
Securing the data was accomplished by collecting responses from a web survey that
was given to technical individuals, unmanned pilots, and manned pilots. The web survey was
placed on the Internet and an email was sent out to the sampled individuals. The email
19
contained a web link that would direct their web browser to the survey. The questions
consisted of rating scales, agreement scales, multiple choice, and text open ended.
Treatment of the Data.
A total of 75 surveys were distributed within the community to gather a representative
pool of individuals. Each question was be identified by a number and then follow by a series
of selectable responses. To ensure confidence in the validity of the instrument tool, a pilot
survey was used. The pilot survey was administered to Senior Engineers and Chief Pilots.
The responses of the pilot survey was also tested through debriefings to ensure the survey was
producing the results needed by the researcher to conduct the study and answered both sub-
problem one and sub-problem two.
Data Analysis.
The results of the web survey were analyzed using quantitative statistical techniques.
The statistical technique used was the Pearson’s chi square test. By using this statistical
method, the study was then able to either accept or reject the hypothesis. This method also
determined in what degree the study questions answered the problem as well as determined
the distribution of the responses.
20
CHAPTER IV
RESULTS AND DISCUSSION
Sub-problem one was to determine the effect on how uncommon control stations
impacts UAV safety of flight. To solve this problem 75 surveys, included in Appendix A,
were distributed as described in Chapter III. A total of 34 surveys were completed through
Survey Monkey’s online program and analyzed. Questions 3, 4, and 9 were designed to
answer this sub-problem.
Question 3 asked the respondents to choose which UAV control functions you believe
should be associated with appropriating a UAV. Figure 5 shows the percentage of the
responses.
Figure 5. Survey Question 3. Of those that responded, 79.4% stated display, 76.5% mouse 64.7% keyboard and 52.9%
joystick. Question 3 was subjected to the chi square test, however the expected frequency
was less than 5 due to the small number of responses for throttle control and rudder pedals.
Therefore, these two categories were collapsed into joystick as well as combining the manned
pilots and unmanned pilots. Table 3 shows the calculated results. The results indicated the
21
calculated value is 5.887 and the critical value is 7.815. Since 5.887 is less than 7.815 the null
hypothesis could not be rejected.
Table 3
Question 3 Chi Square Test
Function Engineer Unmanned/Manned Pilot Total
Display 19 (27.5%) 8 (19.0%) 27
Keyboard 16 (23.2%) 7 (16.7%) 23
Mouse 18 (26.0%) 8 (19.0%) 26
Joystick 16 (23.2%) 19 (45.2%) 35
Total 69 (62.2%) 42 (37.8%) 111
chi square = 5.887; df = 3; p > .05
Question 4 asked the respondents to choose what flight critical information should be
constantly viewed by the UAV pilot or operator. Figure 6 shows the percentage of those that
responded.
22
Figure 6. Question 4.
Of those that responded, 90.9% stated location, 81.8% altitude, 81.8% airspeed, 75.8%
heading, 75.8% command and control strength, 48.5% fuel quantity, 39.4% engine
temperature, and 21.2% outside air temperature. Question 4 was also subjected to the chi
square test, but to get frequencies of less than 5, the manned and unmanned pilots were
combined. Table 4 shows the calculated results. The calculated value was found to be 1.492
and the determined critical value 14.067. The response to this question shows that detailed
performance parameters such as engine temperature and outside air temperature are not as
important as say location. Therefore, one can assume UAV dynamic parameters may not
need to be as detailed as manned dynamic parameters. This would therefore allow for a more
software driven architecture and thus decrease some workload.
23
Table 4
Question 4 Chi Square Test
Function Engineer Unmanned/Manned Pilot Total
Altitude 14(15.6%) 13 (15.3%) 27
Heading 14 (15.6%) 13 (15.3%) 27
Airspeed 14 (15.6%) 13 (15.3%) 27
Location 17 (18.9%) 13 (15.3%) 30
Fuel Quantity 9 (10.0%) 9 (10.6%) 18
Outside Air Temp 3 (3.3%) 4 (4.7%) 7
Engine Temp 5 (5.6%) 8 (9.4%) 13
C2 14 (15.6%) 12 (14.1%) 26
Total 90 (51.4%) 85 (48.6%) 175
chi square = 1.492; df = 7; p > .05
Question 9 asked the respondents to choose why they believe operator errors occurred.
Of those that responded, 61.8% cited inadequate aircrew training, 58.8% poor control station
design, and 44.1% poor engineering design. Aircrew training and control station design are
tied very close together. Therefore the response to this question was not out of line with the
research conducted in Chapter III. Both aircrew training and control station design play a
major role in safety of flight. Figure 7 shows the responses to question 9.
24
Figure 7. Question 9.
The response for “other” was recorded as the following: errors occur because
operators are human; crew rest, fatigue, focus; loss of situational awareness; electromagnetic
interference; lack of standards and risk management. Question 9 was also analyzed using chi
square and the results can be found in Table 5. Due to the low number of responses for
unmanned and manned pilots both of these columns had to be combined so a frequency would
be greater than 5. The chi square calculated value was found to be 0.87 and a critical value of
5.991. Therefore, from this analysis we can conclude both control station design and aircrew
training play a role and impact UAV safety of flight.
Table 5
Question 9 Chi Square Test
Function Engineer Unmanned/Manned Pilot Total
Poor Control Station Design 13 (40.6%) 9 (37.5%) 22
Inadequate Aircrew Training 13 (40.6%) 8 (33.3%) 21
Poor Engineering Design 6 (18.8%) 7 (29.2%) 13
Total 32 (57.1%) 24 (42.9%) 56
chi square = 0.87; df = 2; p > .05
25
Sub problem two was to determine how inadequate aircrew training impacts UAV
safety of flight. To solve this problem 75 surveys, included in Appendix A, were distributed
as described in Chapter III. A total of 34 surveys were completed through Survey Monkey’s
online program and analyzed. Questions 5, 6, 7, and 8 were designed to answer this sub-
problem. Question 5 asked the respondents how qualified do you feel you are to perform
UAV operations. Figure 8 below shows the responses to question 5.
Figure 8. Survey Question 5. Initially looking at the responses to question 5, it is evident that most of the
respondents do not feel very qualified to perform UAV operations. Statistical analysis similar
to how sub-problem one was analyzed was used to determine if and how aircrew training
plays a role in UAV safety of flight. Table 6 shows the calculations. To allow the expected
frequencies to be less than 5, manned and unmanned pilots were combined, not qualified /
somewhat qualified, and very qualified / extremely qualified. This provided a 2 x 2 chi
square analysis and then Yates’ correction was applied to the analysis since the frequency was
still less than 5.
26
Table 6
Question 5 Chi Square Test
Function Engineer Unmanned/Manned Pilot Total
Not Qualified 18 (90.0%) 7 (50.0%) 25
Somewhat Qualified 2 (10.0%) 7 (50.0%) 9
Total 20 (58.8%) 14 (41.2%) 34
chi square = 4.871; df = 1; p < .05
The calculated chi square value was found to be 4.871 with a corresponding critical
value of 3.841. Therefore, since the calculated value is greater than the critical value the null
hypothesis can be rejected. This shows that most of UAV operators feel that they are
somewhat qualified. Question 6 takes another step and asked the respondents if they have
ever been formally trained on how to operate a UAV. Figure 9 below shows the responses.
Figure 9. Survey Question 6. As with question 5, question 6 was also subjected to the chi square test to determine if
the hypothesis could be accepted or rejected. Table 7 shows the calculated results. Manned /
27
unmanned were combined as well as extremely qualified and very qualified. This was done
to so the frequency would not be less than 5.
Table 7
Question 6 Chi Square Test
Function Engineer Unmanned/Manned Pilot Total
Not Trained 12 (60%) 3 (21.4%) 15
Somewhat Trained 8 (40%) 11 (78.6%) 19
Total 20 (58.8%) 14 (41.2%) 34
chi square = 3.528; df = 1; p > .05
The chi square calculated value was found to be 3.528 and the corresponding critical
value was found to be 3.841. Therefore, since the calculated value is greater than the critical
value the null hypothesis cannot be rejected. When looking at the analysis of question 5 and
question 6, one can assume that the number of respondents who feel somewhat qualified have
also been somewhat qualified. To ensure a greater increase in UAV safety of flight, these two
groups need to be enhanced to very qualified and very trained.
Questions 7 and 8 are also related in trying to determine if the respondents were
involved in a UAV operating error and if the respondents feel if training was a cause of this
error. Figure 10 shows the response to question 7 and an overwhelming number of
respondents said No to this question, however some did say Yes. Therefore, question 8 shows
that the respondents do feel that insufficient training and experience were a cause of this error.
Figure 11 shows the responses to question 8.
28
Figure 10. Survey Question 7. Figure 11. Survey Question 8. To get a better understanding of question 8, the chi square test was subjected to the
response. The calculated results are found in Table 8. Not all of the respondents were
required to answer this question unless the respondents answered yes to question 7. The other
responses were the following: the person starting the software was unaware after the software
started, preset parameters were changed and needed to be updated; the operators did not have
sufficient manuals for conducting flight operations. The analysis in Table 8 shows the
calculated value to be 0.96 with a critical value of 3.841. Since 0.96 is less than 2.706 we
cannot reject the null hypothesis. This shows that insufficient training was the biggest factor
in aircrew error.
29
Table 8
Question 8 Chi Square Test
Function Engineer Unmanned/Manned Total
Yes 1 (100%) 4 (80%) 5
No 0 (0%) 1 (20%) 1
Total 1 (16.7%) 5 (83.3%) 6
chi square = 0.96 ; df = 1; p > .05
30
CHAPTER VI
CONCLUSIONS AND RECOMMENDATIONS
Chapter IV detailed a straight forward approach to analyzing the collected data. The
survey questions were designed to address sub-problem one and sub-problem two. All the
data was analyzed using the chi square test and as can be seen in Chapter IV was in
accordance with the research conducted in Chapter II. It is evident that both UAV control
stations and UAV aircrew training play an important role in impacting UAV safety of flight.
It can be concluded that desire for both engineers and pilots is to have a UAV system
that is more reliable on software than on conventional manned aircraft controls. This was
proved by the responses to question 3 and 4. Questions 5 and 6 proved that aircrew training is
not currently efficient to reduce UAV operating errors. This analysis was backed up by
asking the respondents question 7 and 8.
A more complete analysis should be conducted within two or three years to gather if
this trend has changed. As discussed in Chapter II, the Department of Defense and other
federal agencies are trying to standardize control station requirements and qualifications for
unmanned aircrews. A larger sample size would also be desirable to gain a better
understanding of how unmanned pilots are handling currently fielded unmanned systems.
The need for unmanned systems is increasing both in the military and commercial
markets. It is imperative UAV safety of flight is addressed as more systems enter the aviation
industry. Unmanned systems are a great asset and lessons learned from manned aircraft
should be included within these studies, designs, standards and implementations.
31
REFERENCES
Adams, C. (2006). Human-machine interface. Retrieved June 2, 2007, from
http://ergonomics.about.com/od/glossary/g/man_machine.htm
Barnes, M., & Matz, M. (1998). Crew simulations for unmanned aerial vehicle (UAV)
applications: Sustained effects, shift factors, interface issues, and crew size. Santa
Monica,CA: Human Factors and Ergonomics Society.
Department of Navy (2006). Crew resource management program: OPNAVINST 1542.7C.
Retrieved February 2, 2007, from http://www.safetycenter.navy.mil
Draper, M., Geiselman, E., Lu, L., Roe. M., & Haas, M. (2000). Display concepts supporting
crew communication of target location in unmanned air vehicles. Santa Monica,CA:
Human Factors and Ergonomics Society.
Hopcroft, R., Burchat, E., & Vince, J. (2006). Unmanned aerial vehicles for maritime patrol.
Australia: Defence Science and Technology Organization.
National Transportation Safety Board. (2006). General atomics predator B preliminary report
NTSB ID: CHI06MA121. Retrieved September 12, 2006, from http://www.ntsb.gov/
North Atlantic Treaty Organization. (2007). About NATO. Retrieved May, 22, 2007, from
http://www.nato.int/issues/faq/index.html#A1
Kirshbaum, A. (2002, March). NATO UAV systems interoperability. Unmanned Systems, 20,
20-22.
Levin, A. (2006, August 6). Crash stirs debate on drone safety. Retrieved September 5, 2006,
From http://www.usatoday.com/tech/news/surveillance/2005-08-06-drone-
Safey_x.htm
32
Sharma, S., Chakravarti, D. (2006). UAV operations: An analysis of incidents and accidents
With human factors and crew resource management perspective. Aerospace Med
49(1) 2005.
Sossong, P. (2006). Crew resource management in unmanned aircraft. Naval Safety Center.
Retrieved May 1, 2007, from http://safetycenter.navy.mil/aviation/articles/CRM_
UAV.htm
Williams, K. (2004). A summary of unmanned aircraft accident/incident data: Human
factors implications. Oklahoma: Federal Aviation Administration.
33
APPENDIX A
DATA COLLECTION DEVICE
34
35
36
37