STEREO 3-3 AND NON-STEREO PRESENTATIONS OF A COMPUTER-GENEZATSD PICTORIAL PRIMARY FLIGHT DISPLAY WITH PATHWAY AUGMENTATI3N

Program Coordinator and Human Factors Researcher NASA Langlsy Research Center

Hampton, Virginia

and

Lucille critter id en*^

Simulation Engineer Research Triangle Institute

Research Triangle Park, North Carolina

Abstract -- Stereo 3-D was researched as a means to present cockpit displays which enhance a pilot's situational awareness while maintaining a desirable level of mental workload. The initial study at the NASA Langley Research Center used two different pathways-in-the-sky to augment a computer-generated pictorial primary flight display. One pathway resembled the outline of signposts, while the other pathway resembled a monorail. That display was configured for a curved approach to a landing such as could be lJSed in a Microwave Landing System (MLS) approach. It could also be used for military transports which would have to fly a precision curved pathway. Each trial was initialized with the pilot on the desired flight path. After 2 seconds, he suddenly was shifted to one of eight flight path offsets. The pilot was then required to make the initial pitch and/or roll input to correct back to the nominal flight path. As soon as the input was made, the trial was over. No input was required for control trials with no flight path offset. Pilots responded statistically significantly faster when the display was presented in the stereo version than when it was presented in the non- stereo version.

Introduction

With the advent of digital avionics, the use of computer-generated flight displays has become increasingly prevalent in both commercial and

*Lt. Col., LUSAF, detailed to the NASA Langley Research Center

**Co-located at the NASA Langley Research Center

military cockpits. Those flight displays, however, typically are renditions of the electro-mechanical displays that are being replaced. But there is great potential for display enhancements such as pictorial, real- world displays that would enhance situation awareness for the crew. This potential will continue to increase with technology advances in graphics generation equipment (1 ) . The flexibility inherent in computer-generated graphics may help provide an advanced airborne system capability which can lead to increased energy efficient terminal area operations, increased airport and runway capacity, reduced weather dependence with safety, and reduced conmunity noise through the use of improved flight procedures. The introduction of liquid crystal-based stereoscopic display technology offers the potential for further enhancement of situation awareness in cockpit flight displays. A recent technology review rated liquid crystal shuttered glasses as having the greatest near-term capability to present stereo 3-D images in a cockpit environment (2). Further enhancements such as the ability to locate the liquid crystal shutter on a screen in front of the monitor should improve pilot acceptance of the system.

In a recent study, the effectiveness of stereo 3-D cues was demonstrated with three-axis tracking tasks. The general task was analogous to an air traffic controller's task. Subjects were able to perform with less error when they used a stereoscopic display than with any of the other display configurations ( 3 ) .

Thlr paper b dcel8md m work of the U.S. Government m d b no1 subject lo copyrllht protcetlon In the Unlted State#.

The thrust ~f the present study is to buiid on the prior works of references 2 and 4 t;, exploit stereopsiv cueing i:i new flight display concepts. This paper describes the dev2lopment of a stereo 3-D pictorial primary flight display used in a flight simulation eriviroriment as well as preliminary results of an initial research evaluation of the concept in a transport aircraft flying a "curved approach" task. The purpose of this research is to investigate the applicability of stereo 3-D displays for aerospace crew stations, in order to meet the anticipated needs of the 2000-2020 time frame.

3isplay Features -- All of tne display features were evaluated both in a stereo 3-D and in a non-stereo version. The main features of the display were as follows: an ownship symbol, two pathways-in-the- sky (i.e., monorail or "5ignposts1'), a reference ship, a ground grid around the runway, a pitch ladder, and digital readouts for altitude, heading, and airspeed. The digital readouts displayed the irlstantaneous values for the ownship and the desired pre-programmed flight path. The ownship remained fixed relative to the display frame (i.e., an llinside-outll display was represented).

A detailed description of the procedure to generate the stereo 3-D effect is in reference 4. Figure 1 represents the hardware configuration.

(3) (b) (c)

Fig. 2 Evolution of ownship symbol.

Pathways

One pathway, referred to as ''sigr~posts ,I1 looked like the outline of a box, with a vertical line from the box's lower horizontal, perpendicular to the ground (see figure 3). The nominal

Fig. 1 Hardware to generate stereo display. flight path was an imaginary line through the center of the boxes. As the pilot got closer to touchdown, the boxes got smaller to indicate

Ownship Symbol there was a smaller allowable margin of deviation from the nominal flight path. This

Figure 2 shows the evolution of the ownship pathway is a derivative of the perspective symbol. The original configuration, figure 2a, tunnel display which originally was developed presented the pilot with two problems. First, for a helicopter-flown tunnel-in-the-sky it was impossible to perceptually fuse the application (5).

right- and left-eye viewpoints to form the 3-D image. Ori the other hand, tnis fusion problem was surprising because in the case of the signpost pathway, also made of single, straight lines, there was no problem with its visual fusion. The ownship symbol appeared to be closer to the pilot than the plane of the screen; attempts to achieve visual fusion by changi:lg its position along the z-axis were not successful. An additional problem was that the ownship symbol tended to "get lost" in the display. The signpost symbol was constructed of perpendicular horizontal and vertical lines; the same was true of the ownship symbol. Therefore, there were many instances in which the ownship symbol would overlay the signposts and could not be perceiv~d. Color, hue, intensity, and saturation were varied in an urisuccessful attempt to increase the visibility of the ownship symbol. Therefore, the shape of the owriship symbol was modified.

In order to increase the pilot's ability to perceive t h ~ ownship symbol, the center slanted lines were drawn as shown in figure 2b. Although the ability to perceive the symbol was greatly increased, there was still the problem of inability to visually fuse the stereo 3-D image.

The symbol of figure 2c was originally constructed in an attempt to further enhance the pilot's ability to perceive the ownship symbol. Not only was the denser symbol more r3eadily seen, but also a serendipitous benefit was that the symbol now visually fused. At this time there is no explanation for this fusion phenomenon.

3 g . 3 S i g n p d s t pathway.

The o t h e r pathway, a m o n o r a i l , was based on 3

s e r i e s of p a r s l l ? l l l i n e s i ihich a r e p e r p e n d i c u l a r t~ t h e ground. There was one l i n e w'lich connected t h e !;ops of t h e p a r a l l e l l i n e s ( i . e . , t h e d e s i r e d f l i g h t p a t h ) and a second l i n e which connected t h e bottoms of t h e p a r a l l e l l i n e s i n t h e ground plarie ( s e e f i g u r e 4 ) . No c u e s were provided f o r t h e a l l c ~ w a b l e d e v i . s t i o n s from t h e nominal f l i g h t p a t h .

F ig . 4 Monorail pathway

l a r g e r o r s m a l l e r , r e s p e c t i v e l y . The r e f e r e n c e a i r c r a f t symbol a l s o augmented t h e d i g i t a l r e a d o u t s f o r a i r s p e e d by changing c o l o r a s a f u n c t i o n of t h e p i l o t s a i r s p e e d r e l a t i v e t o t h e d e s i r e d a i r s p e e d . When t h e p i l o t was ? l y i n g a t t h e p rope r a i r s p e e d , t h e r e f e r e n c e a i r c r a f t was wh i t e . A r e d r e f e r e n c e a i r c r a f t i n d i c a t e d t h e p i l o t was f l y i n g t o o f a s t , w h i l e a g r e e n r e f e r e n c e a i r c r a f t i n d i c a t e d t h e p i l o t was f l y i rig t o o s low.

T e s t Ma t r ix and Procedure

The i n i t i a l e v a l x ~ a t i o n o f t h e s t e r e o 3-D d i s p l a y u s i n g a 7 i i o t in- the- loop was 3 s t u d y of r e c o v e r y from f l i g h t - p a t h o f f s e t . E i g h t A i r Fo rce p i l o t s s e r v e d a s s u b j e c t s . A l l t h e p i l o t s x e r e q u a l i f i e d i r l mu l t i - eng ine j e t a i r c r a f t , bu t had never s e e n d i s p l a y s s i m i l a r t o t h e ones used i n t h i s expe r imen t . Each p i l o t was i n i t i a l i z e d on t h e nominal f l i g h t pa th . A f t e r 2 s econds , he suddenly was o f f s e t and was r e q u i r e d t o make t h e i r i i t i a l s t i c k i n p u t t o r e j o i n t h e nominal TI-ight p a t h . Each t r i a l t e rmina ted wi th t h e i a i t i a l s t i c k i n p u t .

V i sua l Evoked P o t e n t i s l s (VEPs) a r e p h y s i o i o g i c a l changes i n e l e c t r o e n c e p h a l o g r a p h (EEG) r e c o r d i n g s which r e s u l t from v i s u a l s t i m u l a t i o n ( 6 ) . I n t h i s s t u d y , VEPs were t r i g g e r e d from t h e sudden s w i t c h t o t h e f l i g h t p a t h o f f s e t . I n a d d i t i o n , r e a c t i o n t i m e s , r e s p o n s e a c c u r a c y , and a p r o j e c t e d workload e s t i m a t e a l s o were r e c o r d e d . The S u b j e c t i v e Workload Assessment Technique (SWAT) was used f o r t h e workload e s t i m a t e ( 7 ) and (8). A t e s t f o r s t e r e o s c o p i c a c u i t y was a d m i n i s t e r e d p r i o r t o d a t a c o l l e c t i o n .

I n a d d i t i o n t a u s i n g s t e r e o 3-D o r [ ion-s tereo c u e s a s an independent v a r i a b l e , t h e i n c l u s i o n o r e x c l u s i o n of t h e r e f e r e n c e s h i p was t h e second independent v a r i a b l e . The t h i r d i l d e p e n d e n t v a r i a b l e was t h e pathway. The p i l o t was i n i t i a l i z e d on t h e approach i n one of t h e f o i l o w i n g t h r e e p o s i t i o n s : ( 1 ) b e f o r e t h e f i n a l t u r n t o t h e runway, ( 2 ) i n tRe middle of t h e f i n a l t u r n , o r ( 3 ) a f t e r t h e f i n a l t u r n . A t e ach of t h e p o i n t s of i n i t i a l i z a t i o n a l o n g t h e t u r n , t h e r e were e i g h t d i f f e r e n t f l i g h t p a t h o f f s e t s a s w e l l a s a c o n t r o l c o n d i t i o n i n which t h e r e was no f l i g h t p a t h o f f s e t ( s e e f i g u r e 5 ) . T h i s r e s u l t e d i n a 2 x 2 ~ 2 ~ 3 ~ 9 f a c t o r f a c t o r i a l expe r imen t . Each p i l o t responded t o each of t h e 216 t r i a l s d e f i n e d by t h e unique combina t ions of a l l f a c t o r s .

I n a d d i t i o n t o t h e two v i s u a l pa thways , t h e r e was a r e f e r e n c e a i r c r a f t symbol. When t h e p i l o t was f l y i n g t h e p r e s c r i b e d c o u r s e , t h e r e was a 6-second s e p a r a t i o n between t h e r e f e r e n c e a i r c r a f t ; symbol arid t h e p i l o t ' s a i r c r a f t symbol. When t h e p i l o t g o t t o o c l o s e o r dropped beh ind , t h e r e f e r e n c e a i r c r a f t appea red t o g e t

'iigh- Left

Yigh- High- Center Right

Center- Nominal Plight Path Center- Left (no offset) Right

Low- Left

Low- Low- Center Right

Fig. 5 Schematic flight path offsets from nominal flight path.

Results and Discussion

Data analysis has been corn?lnted only for the reaction time data. The anslysis was accomplished usi~g a repeated measures, within subjects model from the SMDP/PC Ststistical Software (Program 2V) (3). All subject error terms were po~led.

The summary ANOVA for reaction t i ~ e iwludes only those factors which were statistically significant at the -05 level or better. It appears that 2ilots responded faster with the stereo display, with the signpost pat3way, and with the presence of the reference ship. The point of initialization along the path, as well as the particular offset, were significant as were several i?teractioris. (see tables 1 and 2).

Stereopsis 1,1337 16.11 .000 Pathway 1,1337 91.02 .OOO Reference 1,1337 9.49 .002 Curve 2,1337 36.55 .000 Offset 7,1337 22.82 .000 Stereopsis x Pathway 1,1337 8.15 .005 Curve x Offset 14,1337 1.80 .034

Table 1 Summary ANOVA - Reaction Time.

Stereopsis

Pathway

Reference

Curve

Off set

Stereopsis x Path

Curve x Offset

Stereo Nori-Stereo

Signpost Moriorai L

Uith Reference Without Ref ererice

Before Turn In Middle of Turn After Turn

High-Left (HL) Yigh-Center (HC) High-Right (HR) Centzr-Lef t (EL) Center-Right (CR) Low-Left (LL) Low-Center (LC) Low-3ight (LR)

Stereo-Signpost Stereo-Monorail Non-Stereo Signpost 670.27 Non-Stereo Monorail '778.58

Before Turn HL HC HR CC CR LL LC L R

In Middle of Turn HL HC H R CC CR L L LC LR

After Turn HL HC HR CC C R LL LC LR

Table 2 Significantly Different Reaction Time Yeans.

The stereopsis by path interaction is particularly noteworthy because it is relevant both to designers of non-stereo and stereo 3-D

displays (see figure 6 ) . The pathway effects in reaction time were much greater when a non- stereo display was used as opposed to a stereo 3-D version of the same display. Although those results might be a function of only the particular displays used in this study, they do caution the designers of non-stereo pathways-in- the-sky. It appears that the format of the pathway might be more critical in a non-stereo environment than in a stereo 3-D environment. The interaction also can be viewed as demonstrating that stereopsis is less effective with some displays than with others.

Examinatiori of the course by offset interaction revealed the apparent increase in confusion (i.e., loss of situation awareness) detectable in the purely lateral pat11 offsets when they were introduced in the middle of the curve (i.e., in the turn). The reaction tiae for the left-center and the right-center offsets were much faster for both the before- and after-the- turn cases.

., Non-Stereo

Monorail

Signpost

PATHS

Stereo

Fig. 6 Stereopsis by pathway interaction.

Analysis of the remaining dependent variables is in progress. The relationships between objective performance data, subjective evaluations, and brain wave recordings will be fully explored.

Conclusions

The initial NASA study using stereo 3-D cues in a head-down transport flight simulation environment was conducted. Stereo 3-D cues were used to represent the T1real-worldn visual environment. This study did not exploit the maximum potential of stereo 3-D cues; it was the first step in researching the possible benefits of the technology for transport aircraft flight displays.

stereo 3-3 display was presented to them than when a non-stereo display was used. The i~iteraction between the use of stereo and the type of pathway-in-the-sky symbology indicated that the choice of pathway is even more critical when the display is in a non-stereo mode than when there is a stereo 3-D presentation. Another interpretation is that the stereo 3-D cues do not greatly enhance a well designed display of that type. That doos not preclude the possibility that stereo 3-D cues might be highly effective for tasks other than the otie used in this study or for different applications of the stereo 3-D effect.

Additional research needs to focus on flying tasks that require maintaining longitudinal spacing (i.e., along the z-axis) to place a higher pre-nium on 3-D cueing. Other uses of stereo 3-D cues also are possible. For example, some of the perceiv~d display clutter might be alleviated by locating some of the symbology at different planes along the z-axis. Another potential use would be to use the z-axis to present new or alerting information to the pilot. With a well-designed pictorial display in the x/y-axis, using the z-axis for high priority information could be an effective way to communicate to the pilot.

The success of display designers in developing innovative applicatians of stereopsis cueing to further enhance situation awareness, to relieve information clutter, and to provide a medium for implementing alerting functions will determine its place in future cockpits.

References

J. J. Hatf ield, "Crew Station Technology ," Tutorial at 5th Digital Avionics Systems Conference, p. 88: Oct., 1983.

T. Turner arid R. Hellbaum, "A 3-D Pictorial Cockpit Display,'? SID 86 Digest: 1986.

W. S. Kim, S. R. Ellis, M. E. Tyler, and B. Hannaford, "Quantitative Evaluation of Perspective and Stereoscopic 9isplays in Three-Axis Manual Tracking Tasks," IZEE Transactions on Systems, Man, and Cybernetics, Vol. SMC-17, No. 1: Jan./Feb., 1987.

T. L. Turner, R. Suresh, and R. F. Hellbaum, "An Experimental Stereoscopic Cockpit Display System,I1 Third Annual Electronic Imaging 86 International Electronic Imaging Exposition & Conference: Nov., 1986.

A. J. Grunwald, J. B. Robertson, and J. J. Hatfield, "Evaluation of a Computer- Generated Perspective Tunnel Display for Flight Path Following," NASA-TP-1736, NASA Langley Research Center, Hampton, VA.

Nonetheless, pilots were able to respond more quickiv to correct a flight path offset when a

G. L. Zacharias, "Identification of Visual Evoked Response Parameters Sensitive to Pilot Mental State," NASA Contract Report 4140, Langley Research Center Contract NAS1-17816: Apr., 1988.

G. B. Reid, C. A. Shingledecker, A. Clark, and F. T. Eggemeier, "Application of Conjoint Measurement to Workload," Proceedings of the Human Factors Society- 25th Annual Meeting, Santa Monica, CA, Human Factors Society: 1981a.

G. B. Reid, C. A. Shingledecker, and F. T. Eggemeier, "Development of Multidimensional Subjective Measures of Workload," Proceedings of the International Conference on Cybernetics and Society, IEEE Systems, Man and Cybernetics Society: 19a1 b .

BMDP Statistical Software, University of California Press: 1985.

Acknowledgements

authors would like to express their appreciation to several individuals whose efforts made this research project possible. Mr. Dean E. Nold and Mr. M. J. Neubauer created the initial flight path dlsplay. Mr. Timothy L. Turner developed the stereopsis software, and Mr. Harold H. Lane developed the application software.