Helmet-Mounted Display Technology for EVA Training in NASA ...
Transcript of Helmet-Mounted Display Technology for EVA Training in NASA ...
50th International Conference on Environmental Systems ICES-2021-276 12-15 July 2021
U.S. Government Funded Work
Helmet-Mounted Display Technology for EVA Training in
NASA’s Neutral Buoyancy Lab
Janine R. Moses1
KBR, 2400 NASA Parkway, Houston, TX 77058 and University of California-Davis, Davis, CA 95616
James R. Stoffel2
KBR, 2400 NASA Parkway, Houston, TX 77058 and
Ruby Z. Houchens3
KBR, 2400 NASA Parkway, Houston, TX 77058 and University of California-Davis, Davis, CA 95616
Jocelyn T. Dunn, Ph.D.4
KBR, 2400 NASA Parkway, Houston, TX 77058
Stephen K. Robinson, Ph.D.5
University of California-Davis, Davis, CA 95616
and
Andrew F.J. Abercromby, Ph.D.6
NASA Johnson Space Center, 2101 NASA Parkway, Houston, TX 77058
The Human/Robotic/Vehicle Integration and Performance (HRVIP) Lab at University of
California, Davis is collaborating with NASA’s Johnson Space Center (JSC) to design and test
an extravehicular activity (EVA) spacesuit helmet-mounted display (HMD) to enhance
astronaut situational awareness during underwater EVA training. An EVA HMD will enable
astronauts to monitor and react to real-time information including physiological biometrics,
spacesuit status, environmental factors, task procedures, and navigation aids. To meet
operational EVA challenges, HRVIP Lab is partnering with the JSC’s Human Physiology,
Performance, Protection, and Operations (H-3PO) Lab to create HMD prototypes and test
them during underwater EVA training in JSC’s Neutral Buoyancy Laboratory (NBL). Two
HMD mounting styles were designed and tested in the NBL. The swing arm HMD mount holds
the display a short distance in front of the helmet to allow focusing on text-based real-time
data. The surface HMD mount, positioned in the astronaut’s peripheral vision on the helmet
visor, displays flashing colors as a minimal distraction alert to the user to check system status.
NBL testing of the HMD prototype during 2020 has resulted in the following findings: (1)
users (astronauts) found the real-time biofeedback and EVA parameters useful and readable;
(2) there was minimal physical conflict between the HMD hardware on the spacesuit and EVA
training operations; (3) the peripheral visual cues from HMD’s visor surface mount were
effective only in certain scenarios; and (4) voice control enabled astronauts to use HMD
autonomously, but also requires system improvements for increased reliability. This HMD is
a test bed for evaluating data visualizations and interfaces for potential future flight
1 EVA Informatics Intern, KBR, 2400 NASA Parkway, Houston, TX 77058 and graduate student, UC Davis, Davis,
CA 95616 2 EVA Human Performance & Analog Engineer, KBR, 2400 NASA Parkway, Houston, TX 77058 3 EVA Informatics Intern, KBR, 2400 NASA Parkway, Houston, TX 77058 and undergraduate student, UC Davis,
Davis, CA 95616 4 Human Performance Engineer, KBR, 2400 NASA Parkway, Houston, TX 77058 5 Director, UC Davis Center for Spaceflight Research; Professor, UC Davis, Davis, CA 95616 6 H-3PO Laboratory Lead, NASA Johnson Space Center, 2101 NASA Parkway, Houston, TX 77058
International Conference on Environmental Systems
2
informatics. Feedback from these HMD evaluations will inform future heads-up displays, both
for EVA training and for the next-generation spacesuit.
Nomenclature
API = Application Programming Interface
ARGOS = Active Response Gravity Offload System
ASR = Automatic Speech Recognition
AWS = Amazon Web Services
BTU = British Thermal Unit
CAD = Computer Aided Design
COTS = Commercial Off-the-Shelf
CWS = Caution and Warning System
DAQ = Data Acquisition
ECS = Environmental Control System
EMU = Extravehicular Activity Mobility Unit
EVA = Extravehicular Activity
H-3PO = Human Physiology, Performance, Protection, and Operations
HMD = Helmet-Mounted Display
HRVIP = Human/Robotics/Vehicle Integration and Performance
HUD = Heads-Up Display
IRIS = Intelligent Response and Interaction System
ISS = International Space Station
JSC = Johnson Space Center
NASA = National Aeronautics and Space Administration
NBL = Neutral Buoyance Laboratory
OLED = Organic Light-Emitting Diode
PET = Phase-Elapsed Time
PGT = Pistol Grip Tool
PIL = Python Imaging Library
PLSS = Portable Life Support System
PoE = Power over Ethernet
TRL = Technology Readiness Level
xEMU = Exploration Extravehicular Activity Mobility Unit
I. Introduction
As human space exploration extends to deep space, communication delays increase. To supplement the limited
communication with Mission Control Center during exploration missions, extravehicular activities (EVAs) will
require new technology to enable crew autonomy. Heads-up displays (HUDs) aim to enhance astronaut situational
awareness, providing crew with real-time information about their EVA to help inform crucial decision-making1. As
an incremental step in beginning to utilize HUD technology for EVA training, the NASA Johnson Space Center’s
Human Physiology, Performance, Protection, and Operations (H-3PO) Laboratory has developed a sensor system to
enable real-time biofeedback for astronauts during EVA training in JSC’s Neutral Buoyancy Laboratory (NBL).
Specifically, this metabolic rate data collection system provides an approximation of crewmember physical workload
and spacesuit consumables usage. With this EVA informatics project, the H-3PO laboratory aims to enhance astronaut
training by tracking differences in EVA task designs and techniques to identify areas to improve and enable
efficiencies in EVA performance. In addition, H-3PO laboratory personnel plan to utilize this project as a test bed for
evaluating data visualizations and interfaces for potential future flight informatics systems.
This EVA Informatics project – Helmet-Mounted Display (HMD) – is a joint collaboration between University of
California, Davis’ Human/Robotics/Vehicle Integration and Performance (HRVIP) Laboratory and the H-3PO
International Conference on Environmental Systems
3
Laboratory. HMD is a low-cost and low-tech prototype that was
developed and tested in 2020 for EVA training in the NBL (see Figure
1).
During this testing, astronauts had access to real-time biofeedback
data for the first time during NBL training. Another benefit of the project
is that user feedback from HMD is helping to inform the development of
more advanced HUDs for the Exploration Extravehicular Mobility Unit
(xEMU) and provides a test bed in a high-fidelity operational
environment for other informatics display concepts.
For this testing, HMD served two distinct applications. First, it
presented readable information, such as real-time physiological data.
Second, HMD presented a visual peripheral signal (e.g., flashing light)
on the display, as a precursor to a caution and warning system (CWS)
that would serve as a minimal-distraction alert to the EVA astronaut to
check system status. This paper describes the HMD hardware assembly
(Section II) and software systems (Section III) and their integration with
NBL testing (Section IV). The paper also outlines results and feedback
from subjects/crewmembers (Section V), with a focus on the
effectiveness and utility of HMD during underwater EVA training.
II. Hardware
The HMD hardware is low-tech, low-cost, and quick turn-around
as this project was developed and tested in less than a year. Extensive
subsystem testing was performed on each waterproofed piece of
hardware and each electronic component. The fully assembled system
then underwent testing, both before and after it was integrated with the
extravehicular mobility unit (EMU). This section outlines the major
components of HMD, with a focus on the elements with which the user
interacts: the display screen and mounting hardware.
Raspberry Pi & DAQ Box: A Raspberry Pi Model 3 B+ and a Power
over Ethernet (PoE) adapter are housed in an aluminum alloy
container, the data acquisition (DAQ) Box. The DAQ Box was
purchased commercial-off-the-shelf (COTS) and modified in-house to
be waterproof for more than 6 hours at 40-foot depth. It was secured
with Velcro in an empty cavity in the EMU’s portable life support
system (PLSS).
Cable Harness: The cable harness, designed and built in-house,
connects the display screen to the Raspberry Pi, and the Raspberry Pi
to its power and network source. Because Bluetooth and WiFi do not transmit underwater at long distances, HMD’s
Raspberry Pi is hardwired both for network connectivity (Ethernet) and power (via power over Ethernet). The
power/Ethernet cable exits the Raspberry Pi DAQ Box and is routed through the PLSS, through the umbilical in the
NBL (see Figure 2), and is connected to an Ethernet port on the
pool deck. The cable for the display screen is routed through the
PLSS to the helmet where the mounting hardware is attached.
Display Screen2: The display screen from Adafruit Industries
is a 1-inch × 1-inch (2.54-cm × 2.54-cm) OLED, 16-bit color
screen with 128 × 128 resolution. It was waterproofed in-house
with a transparent silicone elastomer material, see Figure 3. The
display screen is positioned 6-8 inches from the user’s eyes
(depending on where their head sits inside the EMU helmet).
This eye-to-display distance is slightly shorter than the distance
required for a naked human eye to resolve an image (9-10
inches). Thus, large text size is readable on HMD, but it is more difficult for the user to resolve detailed images or
small text size, especially if they use focal lenses in the EMU helmet. Typically, at water depth, colors are lost because
Figure 1. HMD on subject in NBL. The
HMD swing arm is mounted on the right
side of the subject’s EMU helmet visor.
Figure 3. Potted display screen in swing arm.
Figure 2. Umbilical from EMU to pool
deck power/Ethernet source. Courtesy
of NASA.
International Conference on Environmental Systems
4
the light wavelengths from the surface are absorbed. However, because HMD’s OLED is its own light source, just
inches away from the viewer (compared to light traveling from the surface), the colors of the OLED screen are not
compromised. The EMU helmet visor bubble does not distort the display and the water columnates the light from the
screen, making HMD slightly more clear and crisp when viewed underwater.
Mounting Hardware – Swing Arm Mount: The Swing Arm Mount Assembly, in Figures 4 and 5, was designed
and built in house. It integrates the display screen into a modular and reconfigurable design, and is mounted to the
helmet visor with suction cups. The swing arm assembly can be positioned in four distinct positions on either side of
the EMU helmet, in the upper and lower quadrants of the visor (+/- 30 degrees). The overall design incorporates a
noninvasive, impact resistant, and visor-friendly configuration with no impact or change to the existing EMU system.
A hazard analysis was conducted with spacesuit engineers, and it was concluded that the scratch risk to the helmet
bubble is minimal. The swing arm assembly was created in a Computer Aided Design (CAD) program and
manufactured via Additive Manufacturing, also known as 3D printing. The swing arm assembly is made of tough
polylactic acid (PLA) filament material, ideal for printing lightweight and strong functional prototypes. The swing
arm mount was assembled with corrosion resistant fasteners to survive the underwater environment. To enable this
mount to conform to the curvature of the visor, the swing arm includes three lightweight nylon swivel joints (spherical
bearings) and strong suction cups. This provided a flexible platform for location, position, orientation, and safety for
early development.
Mounting Hardware – Surface Mount: The surface mount assembly as shown in Figure 6, also was designed
and built in house. It integrates the same display screen into a simple chassis, and contains suction cups with swivel
joints. The surface mount assembly was designed to the same overall requirements, manufacturing, and safety
standards. It can be integrated on either side of the EMU helmet. For this phase of the study, the surface mount
assembly was integrated approximately at the visor’s midline and as far aft as possible, see Figure 7.
Figure 4. HMD swing arm mount assembly.
Figure 5. HMD swing arm on
EMU visor.
Figure 7. HMD surface mount on
EMU visor. Partially obstructed
by the helmet light.
Figure 6. HMD surface mount assembly.
International Conference on Environmental Systems
5
III. Software
A. Data Architecture and Metabolic Rate Data Collection at the NBL
At the NBL, an umbilical connects the Class III-W EMU spacesuit to the NBL’s environmental control system
(ECS) system that provides power, data, and breathing gas to the spacesuit. An earlier project in 2019 integrated a
metabolic rate data collection system with the ECS panel on the pool deck at the NBL. This project has provided the
foundation for the data architecture and data security for the HMD project. The metabolic rate data collection system
is a Raspberry Pi-based data acquisition system with a touchscreen application for starting and stopping data collection
(independent from HMD). This metabolic rate data collection system measures the flow rate of the breathing gas
supply to the suit and the concentration of carbon dioxide in the breathing gas returned to the ECS panel. These two
measures are combined to determine the volume of carbon dioxide that the suited crewmember is exhaling to estimate
metabolic rate or energy expenditure during the time course of the EVA training events in the NBL. This data is
processed and stored in a Cloud-based data architecture that enables remote access to these data via a custom-built
web application. With HMD, real-time metabolic rate data also can now be accessed by the crewmember as well.
The Raspberry Pi is the “brains” of HMD. It contains an SD card that hosts the Python code running HMD.
Commands are sent to the Amazon Web Services (AWS) GovCloud via an application programming interface (API),
either using voice control or remote manual control. The HMD scripts utilize open source software from Adafruit and
from the Python Imaging Library (PIL). Figure 8 shows a sequential concept of operations for controlling HMD during
NBL operations, from the crewmember verbalizing a voice command to the display changing modes accordingly:
1. Crewmember utters wake word.
2. Voice control software transcribes command and picks out key words or phrases.
a. Alternatively: the HMD team remotely and manually sends specific command words/phrases.
3. Command is sent via API to the GovCloud.
4. Raspberry Pi extracts the command from the GovCloud and interprets it.
5. If the command includes a request for metabolic rate data, then that data is extracted from the GovCloud
and processed (i.e., time-averaged) on the Raspberry Pi.
Figure 8. Data flow during NBL test. The green arrows represent data flow of commands to control HMD.
The blue arrows represent data flow for real-time metabolic rate.
International Conference on Environmental Systems
6
6. The Raspberry Pi sends a signal to the display to change modes, turn on/off, etc.
B. Voice Control
Voice control enables subjects/crewmembers to interact with HMD autonomously. HMD utilizes a voice control
system called Intelligent Response and Interaction System (IRIS) which is currently being developed and tested for a
helmet display project for the xEMU. Audio feed was stripped from the NBL communications/video system and fed
to designated computers that host IRIS.
IRIS is made up of several key components:
1. Wake word: the wake word for the first subject, EV1, was “grapefruit” and the wake word for the second
subject, EV2, was “blueberry.” These were chosen because they are multisyllabic, distinct, and not likely to
be uttered during typical NBL operations. A beep indicates that IRIS recognized the wake word and is
beginning to transcribe speech.
2. Speech-to-text: Google’s Automatic Speech Recognition (ASR) transcribes the speech that is heard after
the wake word is uttered.
3. Natural language understanding: Google’s Dialogflow picks out phrases/words from a predetermined list
(i.e. “phased elapsed time”, “elapsed time”, or “PET”; “metabolic rate” or “met rate”).
4. Interfaces: external API through which to track IRIS in real-time.
IRIS handles both conversational and terse interactions, as long as the key words/phrases are uttered. For example,
a crewmember can say either “grapefruit, what is my phase-elapsed time?” or “grapefruit, PET.” Once a command is
sent, the HMD python script decodes the meaning of the command and directs the display screen to change display
modes accordingly. This was the first time that voice control was utilized with NBL communication loops. As
described in results (Section V), future work will be required to improve the audio integration and investigate why
Figure 9. HMD display modes. From left to right: welcome screen, metabolic rate (numerical),
metabolic rate (bar graph), phase-elapsed time (PET), timer #1, static image (EV locations), static
image (ISS translations), flashing blue square, flashing yellow circle.
International Conference on Environmental Systems
7
some errors occurred during testing. During HMD testing, HMD personnel closely monitored the voice control system
and occasionally had to interfere if there was a missed transcription. The team would reset IRIS or manually send
command payloads to the display via the API, in place of voice commands.
C. Display Modes
Figure 9 shows the HMD display modes. Note that the metabolic rate and PET are real-time data. In the future,
HUDs may be used as navigational aids, so two of the display modes (the static images) mimic navigation settings.
IV. Underwater Testing
Prior to integrated underwater testing, extensive
subsystem testing was performed to ensure the
hardware/cable harness was waterproof and the
software/voice control system functioned as expected. The
NBL is a high-fidelity EVA training analog, mimicking
microgravity and housing a 1:1 scale mockup of the
International Space Station (ISS). HMD was integrated with
EVA training in this environment to better understand the
advantages of utilizing HMD as an EVA training aid. The
following section walks through a typical HMD test day in
sequential order, focusing on the choreography between
HMD and typical NBL operations.
Pre-Dive Brief: Before the day of the NBL run, the HMD
team briefs the subjects about the upcoming test and the
subjects choose their swing arm location (upper left/lower
left/upper right/lower right).
HMD Donning: The morning of the NBL test, the HMD
team sets up and tests the hardware and cable harness before
the subject dons the EMU. After suit donning, when the
helmet is secure, the HMD team secures the suction cups of
the swing arm onto the visor in the predetermined position. The subject verifies that there are no major conflicts
between HMD and visibility or movement.
Display Checks: Throughout the first half of the NBL run (over the course of 3-4 hours), the HMD team sits in the
test conductor room. During designated periods, the HMD team walks the subject through a HMD check by telling
them specific voice control commands to repeat. Meanwhile, the HMD team monitors the voice control system and
the Raspberry Pi (via remote access) to ensure that the commands are being sent and interpreted correctly. The subjects
also are encouraged to utilize HMD independently to check real-time data such as metabolic rate or phase-elapsed
time. Figure 10 shows a crewmember
checking the PET right after egressing the
airlock in the beginning of the NBL run.
Mounting Hardware Reconfiguration:
Approximately half-way through the NBL
run, two HMD team members dive in the NBL
to reconfigure the hardware. They detach the
swing arm suction cups from the visor,
remove the display from the swing arm, place
the display in the surface mount, and attach
the surface mount suction cups to the visor.
This process takes about 5 minutes. Figure 11
shows a HMD diver performing the
reconfiguration.
Peripheral Signal Tests: After
reconfiguration, the HMD test team (back in
the test conductor room) manually sends a
signal to HMD to display flashing
Figure 11. Mounting hardware reconfiguration. HMD diver,
Janine Moses, attaching the surface mount to the EMU visor.
Figure 10. HMD check during NBL testing. A
crewmember checks the PET during airlock egress at
the beginning of the NBL training run (reflection of
display through helmet is also visible).
International Conference on Environmental Systems
8
colors/shapes. Without prior knowledge of when the signaling will take place, the subjects are asked to report when
they notice the flashing display. Each time a signal is sent to HMD, the response time is recorded: the time from when
the visual signal first appeared on the display to when the subject/crewmember reported seeing the flashing colors.
HMD Doffing: At the end of the NBL run when the subjects are out of the water, HMD is detached from the visor
so that the helmet can be removed. The rest of the hardware and cable harness is also removed from the EMU for post-
test checkouts and maintenance.
Post-Test Debrief & Survey: At the end of the day, the HMD team debriefs with the subjects/crewmembers and
administers an electronic survey (See Section V. Results).
V. Results
This section reports feedback from the users about their experiences with HMD during EVA training in the NBL
and about the HMD hardware itself. Feedback from HMD evaluations will inform future HUDs, both for EVA training
and for the next generation spacesuit.
During November and December 2020, HMD was tested during 5 NBL runs, each with 2 subjects/crewmembers
(10 users total). The first 2 tests included EVA training personnel as the suited subjects, and the following 3 tests had
current NASA astronauts as test subjects. HMD performance feedback was captured during the NBL tests by recording
users’ verbal comments and after the NBL test through an electronic survey3.
A. HMD Positioning, Visibility, and Physical Conflicts
Users did not report any visibility conflicts during NBL Ops; they had an
unobstructed view of their chest-mounted workstations, even with HMD. If
subjects used a Fresnel lens in the helmet bubble, they were advised to mount
HMD opposite the lens to avoid visibility conflict (although none of the
individuals in this subject pool used a lens). Tests showed minimal physical
conflicts with HMD in the NBL, except for during airlock ingress/egress.
During these operations, HMD was accidentally bumped or knocked off on
three occasions during airlock ops because both crewmembers were in a
confined space with one crewmember’s feet near the other’s helmet.
Before the underwater runs, each test subject was given the choice of
desired position for the swing arm mount (lower left/upper left/lower
right/upper right). Table 1 indicates the position chosen by the
subjects/crewmembers, and Figure 12 shows examples of positioning. When HMD was reconfigured, the surface
mount was placed on the same
side (left/right) of the visor
where the swing arm was
originally placed. Subjects
considered a variety of factors
when choosing the swing arm
mount’s position:
• Position in the suit: some
subjects tend to ride low
in the suit during NBL
training so they chose the
upper left/right positions
for better visibility of the
display and of their
workstation.
• Cuff checklist location:
Some subjects chose to
place the display on the
same side as the cuff checklist so that they could check both simultaneously, while others chose to place the
display on the opposite side of the cuff checklist to minimize any visibility interference.
• Access to PGT/other tools: Some crewmembers chose to place the display on the opposite side of their tools
(mounted to their waist). For example, if the pistol grip tool (PGT) was mounted on the right side of the EMU
Table 1. Chosen HMD Positions
Position # Subjects
Lower Left 5
Upper Right 3
Upper Left 1
Lower Right 1
a) Lower Left Position b) Upper Right Position
Figure 12. HMD swing arms mounted on EMU visors.
International Conference on Environmental Systems
9
tool mount, the subject chose to mount HMD on the left side to reduce visibility interference when accessing
the PGT.
• Dominant eye/field of view: Some subjects chose to place the swing arm on the opposite side from their
dominant eye so as to not obstruct their primary view. Others chose to place it on the same side as their dominant
eye so it would be easier to check when sweeping their field of view.
• Other helmet equipment: Subjects ensured that, if HMD was positioned in one of the lower quadrants, the
Valsalva device (inside the helmet to block one’s nose and equalize pressure) did not interfere with visibility.
B. User Feedback
This section focuses on (1) the usefulness of HMD content specifically for EVA training in the NBL and (2) the
visibility/readability of the content itself. Table 24 shows the 2 scales subjects used to rate each display mode (listed
in Section III: Software).
Note that this part of the
survey was only
administered to the test
subjects for NBL Tests 2-5,
so there are 8 responses. The
highest rated display modes
were the real-time metabolic
rate and phase-elapsed time
(PET). Figure 14 shows 7 of
8 subjects/crewmembers rated the metabolic rate mode as “probably useful” or “certainly useful”. Figure 14
summarizes that all 8 subjects/crewmembers thought that access to phase-elapsed time was “probably useful” or
“certainly useful.” Subjects thought the metabolic rate display mode was useful because it provided a quantified
measure through which they could compare their relative exertion levels for various tasks during EVA training.
Subjects thought the PET display mode was useful because it provided high-level insight into timeline progress. Most
subjects thought that the metabolic rate and PET data were visible, but 4 subjects reported that the labels on the display
for “Met Rate” and “PET” were too small or lacked color contrast (see Figure 9: Display Modes). Users did not report
eye fatigue, but most did report that it was hard to focus on the smaller text/numbers, especially when switching from
far-field focus (workstation) to near-field focus (display screen). This is expected because HMD does not address the
optical challenges of a near-eye display, beyond offering a displaced swing-arm mount to position the display away
from the subjects’ eyes.
During their NBL underwater runs, half (5 of 10) of subjects independently checked HMD for their metabolic rate
or PET outside of the designated HMD tests. Those subjects appreciated HMD’s access to operationally useful data,
and utilized the displayed information to make decisions and increase their overall situational awareness. For example,
one crewmember who felt discomfort in their EMU was debating about continuing with the NBL training run or
pausing the run, returning to the pool deck to readjust the suit, and then resuming the run. This crewmember
Figure 13. Metabolic rate usefulness for EVA training and visibility ratings. Most subjects rated the metabolic
rate (numerical) display mode as “probably useful” or “certainly useful” and as “acceptable visibility” or
“excellent visibility”.
0
1
3
4
0
1
2
3
4
5
Certainly notuseful
probably notuseful
probablyuseful
certainlyuseful
NU
MB
ER O
F C
REW
USEFULNESS RATING
Met Rate (Numerical) Usefulness Scale
0
1
2
5
0
1
2
3
4
5
6
Very poorvisibility
Poor visibility Acceptablevisibility
Excellentvisibility
NU
MB
ER O
F C
RE
W
VISIBILITY RATING
Met Rate (Numerical) Visibility Scale
Table 2. Survey Scale. Subjects/crewmembers rated each display mode with scores
for (a) Usefulness (left) and (b) Visibility (right).
International Conference on Environmental Systems
10
independently checked HMD for the phase-elapsed time, determined that it was still early in the NBL training run,
and chose to have their suit readjusted on the pool deck before proceeding.
C. Voice Control Feedback
The audio recordings of the subjects/crewmembers during the NBL runs were analyzed to determine the scenarios
under which conflict arose between the users’ HMD voice control and any verbal communications with the other EVs
or test conductor/personnel. The consensus was that voice control significantly enhances HMD use, interaction, and
autonomy. As such, the voice control system must continue to be improved for it to be integrated with nominal
operations, because it was the greatest source of frustration for users. There were a few times when a
subject/crewmember wanted to interact with HMD but needed to wait until a conversation between other test personnel
were complete before uttering voice commands. The following list summarizes the feedback received from the users
and from other test personnel:
Pros for the automatic voice interpretation software:
• Wake words (“blueberry” and “grapefruit”) were easy to memorize.
• The “beep” sound to acknowledge that the wake word was recognized by the software was helpful.
• Good at interpreting garbled speech, including English-as-a-second-language accents.
Cons for the automatic voice interpretation software:
• Required a lot of technical support in development stages (during earlier NBL tests).
• Not always consistent with picking up wake word (due to NBL audio quality).
• Sometimes “grapefruit” sounded like “break-break” which is used to completely halt an NBL test.
D. Peripheral Signaling
After HMD reconfiguration, the surface mount, positioned in the subject’s peripheral vision, displayed flashing
colors at the discretion of the HMD test team. Each time a signal of flashing colors was shown on HMD, the response
time was recorded from when the visual signal first appeared on the display to when the subject reported seeing the
flashing colors. Subjects most often noticed the visual signal (flashing colors) within 5 seconds under these scenarios:
• Their head was turned in a direction such that the HMD surface mount was in their visual field of view.
• They were translating along a truss section of the ISS mockup, also with the HMD surface mount in view.
• They were waiting for instructions from the test conductor, being repositioned by divers, or engaged in
other passive activities.
When the subjects were engaged in a more complex task or the display was not in their field of view, they often
did not notice the flashing signal. Any future caution and warning system based on peripheral signaling would
therefore require an audio alert to supplement (or replace) the visual cue. Also, during periods of significant chatter
over the NBL audio communications, a subject often had to wait for a lull in the conversation before reporting that
they had noticed the HMD signal, which resulted in inaccurately long recorded response times.
Figure 14. Phase-elapsed time (PET) usefulness for EVA training and visibility ratings. Most subjects rated
the PET display mode as “probably useful” or “certainly useful” and as “acceptable visibility” or “excellent
visibility.”
0 0
1
7
0
1
2
3
4
5
6
7
8
Certainly notuseful
probably notuseful
probablyuseful
certainlyuseful
NU
MB
ER O
F C
RE
W
USEFULNESS RATING
PET Usefulness Scale
0 0
3
5
0
1
2
3
4
5
6
Very poorvisibility
Poor visibility Acceptablevisibility
Excellentvisibility
NU
MB
ER O
F C
REW
VISIBILITY RATING
PET Visibility Scale
International Conference on Environmental Systems
11
E. HMD Hardware Challenges
There were a number of challenges that arose during NBL operations related specifically to underwater testing.
Most of these challenges did not significantly affect the user’s experience, but they offer room for improvement during
future iterations of HMD technology. Examples include:
Potted display: The display and swing arm were jostled during airlock ingress and egress operations. On a few
occasions, the potting on the display tore when it was jostled, risking electronics exposure to water.
Suction cups: If the display was knocked by the subject/crewmember, a diver, or a mockup, the suction cups would
slide around, occasionally becoming detached from the visor.
Cable harness: If the display was mounted on the right side of the visor, there was 6-10 inches of excess cable that
needed to be secured in the PLSS. This was never a snag hazard, but it often required the HMD team to spend several
minutes managing the cables during HMD donning (on the pool deck) and reconfiguration (in the pool).
DAQ Box: The aluminum alloy container for housing the computing unit developed external chalky stains after
each 6-hour use in the chlorinated pool.
VI. Future Work
Work on this project over the course of the past year has elevated the technology readiness level (TRL) of this
device, from a concept to a fully operational HMD. However, this technology remains in its early phases of
development, and it is important that additional HMD capabilities are iteratively developed and tested. The results
from the initial HMD testing demonstrate that EVA informatics is a useful and important technology to study and
improve. In the next set of HMD evaluations, the team plans to inquire not just if each display mode was useful during
EVA training, but specifically why the visual content from a display mode was useful and how the information on the
display was utilized. The future work planned for HMD is outlined in the following subsections.
A. Upgrades for Additional NBL Testing
The next iteration of HMD will incorporate hardware upgrades specifically tailored to eliminate or mitigate the
hardware challenges outlined in Section V. In addition, in place of the visor-surface mount to test response to a
peripheral signal, HMD may utilize the swing arm mount for peripheral signaling. This will ensure that the flashing
light signal is visible to the
subjects/crewmembers regardless of the
task complexity with which they are
engaged. HMD software also will be
expanded to include more display modes,
such as custom text inputs for tool settings
and additional real-time biofeedback data.
The IRIS voice control system will be
improved to pick up the wake words with
greater consistency, a task that will require
additional integration with the NBL audio
communication system.
B. Upgrades for Use in Different
Spacesuits and Environments
Depending on the specific interests of
the EVA community, HMD may be
adapted to other spacesuits (see Figure 15),
such as the Mark III, Z-2, Z-2.5, and
xEMU, or to hard-hat surface-supplied
diving in the NBL. For example, the swing
arm mounting hardware may be redesigned
to accommodate the different shape of the
xEMU helmet bubble. The HMD technology and integration with NBL operations can be shifted to accommodate
these newer spacesuits, including during lunar (1/6-G) operations in the NBL.
HMD can be adapted to other training environments, such as the JSC Rock Yard. In the near-term, H-3PO
personnel also plan to utilize HMD for cognitive testing in the active response gravity offload system (ARGOS)
a) Hard-hat (Kirby Morgan 97) b) Z-2.5 Spacesuit
for Lunar EVA training
Figure 15. Lunar Ops in NBL. HMD can be modified to be used
during Lunar EVA training at the bottom of the NBL.
International Conference on Environmental Systems
12
environment. Slight hardware or software upgrades may be necessary, but the overhead required to adapt HMD to
these environments will be significantly lower than at the NBL.
C. Upgrades for UC Davis Research
UC Davis’s HRVIP Lab has an elevated interested in pairing the use of this technology with motion-capture
software to gain a comprehensive understanding of pre-existing EVA training motion patterns and how the use of
HMD may enhance those patterns. HMD may be tested in a partial-G flight environment as we are considering HMD
implementation not only for EVA training in the NBL, but for EVAs on future space exploration missions.
VII. Conclusions
The HMD project provides an important first step in developing EVA Informatics to enhance astronaut autonomy.
Subjects/crewmembers provided high ratings for the usefulness and readability of real-time biofeedback and EVA-
related data, even utilizing HMD to make real-time decisions based off PET readings. HMD offers insight into the
benefits of real-time access to EVA Informatics with biofeedback, in addition to how EVA Informatics could be
integrated into NBL training on a daily basis. Even though the voice control system was the primary source of
frustration for users, when it worked successfully, it enabled greater autonomy. HMD is still a work in progress: it is
a low-tech and low-cost prototype that demonstrated the effectiveness and importance of EVA Informatics to enhance
training in a high fidelity analog environment. Results from HMD testing and lessons learned from the development
and test experience will help to:
1. Inform the next iteration of HMD for EVA training in the NBL.
2. Provide a test bed for low-cost and minimal impact testing of other HUDs in the NBL.
3. Inform the adaptation of HMD for use in other analog environments (JSC Rock Yard and ARGOS).
4. Inform the development of more advanced HUDs for the xEMU and other future spacesuits.
Even in its early stages, HMD proved to be effective and informative. HMD offers a glimpse into the benefits of using
biofeedback and other EVA parameters to enhance astronaut autonomy during EVA training.
Acknowledgments
This ongoing research is supported by the NASA JSC H-3PO Laboratory and by the NASA SSERVI Research
Institute: Radiation Effects on Volatiles and Exploration of Asteroids and Lunar Surfaces (REVEALS). Thank you to
Austin Alexander for software development and support and to the rest of the H-3PO team that implemented the
metabolic rate data collection system at the NBL. Thank you to the IRIS Voice Control Team, Aly Shehata and
William Baker, for providing the voice control software and support, and for the great teamwork and integration with
HMD. Thank you to Ryan Amick and Jerri Stephenson for the human factors-related input and guidance in creating
the surveys. Thank you to the NBL team: the test support personnel, test directors, divers, umbilical technicians,
spacesuit lab technicians, spacesuit engineers, flight leads, and NBL leadership for making HMD testing possible.
Thank you to the suited subjects and crewmembers for taking the time to learn about HMD and provide details about
their experience using HMD.
References 1. Carr, C. E., Schwartz, S. J., Rosenberg, I., “A Wearable Computer for Support of Astronaut Extravehicular
Activity,” Proceedings of the 6th Int’l Symposium on Wearable Computers, 0-7695-1816-8/02, IEEE, 2002.
2. Neighbors, H., Salazar, G., Steele, G., Sarma, K. R., Schmidt, J., Wiggs, J., Mejias-Rolon, Y., “OLED
Technology Evaluation for Space Applications,” AIAA Space, Pasadena, CA, 2015.
3. Mitra, P., “Human Systems Integration of an Extravehicular Activity Space Suit Augmented Reality Display
System,” Masters Thesis, Aerospace Engineering Dept., Mississippi State University, Starkville, MS, 2018.
4. Pauwels, R., Seynaeve, L., Bosmans, H., Bogaerts, R., & Jacobs, R., “Technical versus diagnostic image
quality in dental CBCT imaging,” European Congress of Radiology, C-2391, European Society of Radiology,
Vienna, Austria, 2013.