Training Effectiveness of Flight Simulators

with Outside Visual Cues

Miguel Freitas da Silva Mendes

miguelfsmendes@tecnico.ulisboa.pt

Instituto Superior Tecnico, Lisboa, Portugal

September 2016

Abstract

This paper reports a training experiment performed in the SIMONA Research Simulator at Delft

University of Technology in which the effectiveness of outside visual cues during the development of

multimodal control skills was evaluated. Twenty fully task-naive participants were divided in two

experimental groups and performed a skill-based compensatory roll tracking task. Both groups were

trained in a fixed-base setting, but one group had additional out-of-the-window visual cues. After

training, subjects were transferred to a motion-base condition where pure roll motion cues were provided.

The development of skills in both groups throughout the experiment was studied and compared utilizing

multiple derived variables. Specifically, to allow a direct evaluation of the effectiveness of outside visual

cues as a source of roll feedback, the human operator multimodal structure was the same for both

feedback sources, outside visual scene and physical motion. During training the group in which outside

visual cues were available attained better tracking performance levels. However, no significant transfer

of these superior control skills to the motion condition occurred, suggesting that training with outside

visual cues might not be sufficiently effective for a subsequent transfer to motion settings.

Keywords: Flight Simulator, Cybernetics, Outside Visual Cues, Manual Control, Transfer of Training

1. Introduction

In order to control and steer any vehicle, whether itis a bicycle, a car or an aircraft, humans depend ontheir sensory systems to perceive the surroundingreality and thereby gather information relevant forcontrol. Visual and motion stimuli are the mostrelevant control inputs and control proficiency isattained by training responses to what the visualand vestibular systems perceive [1]. In case of anaircraft, pilots’ learning process usually includes asimulator-based phase in which simulators replicateflight reality for pilots to develop their control skills.The control skills acquired by a pilot in his simula-tor training are brought into use when transferredto a real-world setting as flying an actual aircraft.Understanding what changes in pilots’ responsesto their sensory systems inputs throughout theirlearning process, together with why and how thosechanges occur, will result in improved flight simu-lators, pilot training, and ultimately pilot skills.A cybernetic approach to understand these

changes consists on transfer-of-training experi-ments, in which the transfer of control behavioracquired in a training condition (e.g., a flight sim-ulator) to the evaluation setting (e.g., a real air-craft) is investigated and directly assessed. How-ever, given the impracticability in performing trans-fer to real settings, the majority of the studies per-formed are in fact quasi-transfer-of-training exper-

iments, where the evaluation setting is not true re-ality but a more realistic simulation environment[2]. Multiple transfer-of-training experiments wereperformed to understand what are the effects of dif-ferent types of simulator cues on humans’ learningof control behavior and how these cues affect skilltransfer. Most of them focused on the training effec-tiveness of motion cues, having found that motionfeedback is required for effective simulator-basedtraining of manual control skills [3]. This happensbecause motion feedback strongly influences humanoperators’ behavior, especially when the controlleddynamics require lead equalization [4].

Recent studies with compensatory tracking taskshave shown that peripheral visual cues are utilizedby human operators to support a human feedbackcontrol organization similar to the one observed intasks with physical motion cues [5]. It was proventhat the presence of a strong outside visual sceneprovides lead information on the controlled dynam-ics in a similar way as achieved by the physical mo-tion feedback, though not as effectively [6]. If thesefindings are taken into consideration from a per-spective of simulator-based training, similarities inthe way human operators deal with both sensory in-puts suggest that outside visual cues might be usedfor initial simulator-based training, as they mightcreate and establish a feedback channel in the hu-man operator without the need of actual physical

1

motion cues. At this point, such transfer has neverbeen studied explicitly and it is thus hypothesizedthat the feedback channel created by the existenceof a peripheral visual scene is effective in easing thedeveloping of manual control skills in a motion con-dition.

The goal of the research described in this paperis to discover to which extent visual cues are ef-fective in developing multimodal control skills dur-ing simulator-based pilot training. To achieve thisgoal, a quasi-transfer-of-training experiment wasconducted in the SIMONA Research Simulator atDelft University of Technology and it is hereby de-scribed and analyzed.

This paper is structured as follows. First, themethods, the organization of the experiment, andthe hypotheses are described in Section 2. Section3 contains the results of the experiment. A discus-sion follows in Section 4 and the paper ends withconclusions in Section 5.

2. Methods

In this section, the methods followed in this quasi-transfer-of-training experiment are given, explain-ing how the experiment was prepared, conductedand analyzed. The hypotheses to be tested withthe experimental results are formulated in the endof this section.

2.1. Control Task

This human-in-the-loop training experiment con-sidered a compensatory roll-axis tracking task,schematically represented in Fig. 2. The humancontroller was asked to follow a target roll angle,specified by the tracking signal ft, as accuratelyas possible, while simultaneously rejecting distur-bances on the controlled system Hc, which were in-duced by the disturbance signal fd. This distur-bance signal was summed to the human operator’sinput, u, and directly affected the controlled dy-namics. In order to identify and model the multi-channel human operator response, characterized byHpe

and Hpφ, the disturbance signal fd and the tar-

get signal fd were independent sum-of-sines signals[7]. Given the quasi-linear human operator modelused, the control input had contributions from theerror response, ue, the roll response, uφ, and a rem-nant n accounting for nonlinear behavior and mea-surement noise.

In the experiment (see Fig. 1(a)), the human op-erator perceived the dynamics being controlled witha compensatory display which resembled a basicPrimary Flight Display (PFD), shown in Fig. 1(b).This compensatory display showed the deviation e

between the current aircraft roll angle, φ, and thetarget roll angle, ft. The out-of-the-window periph-eral visual cues used in this experiment were basedon various compensatory roll-axis tracking exper-

iments [6] and consisted on two vertically movingcheckerboard panels, providing a strong roll motionsensation without giving reference of roll-attitude.In the moving base condition pure roll motion wasprovided, thus without any washout or lateral spe-cific force compensation.

(a) Simulator Cockpit.

e

(b) Central Display.

Figure 1: Simulator cockpit and the central display.

The controlled dynamics Hc were the dynamicsused in Ref. [8] and presented in Eq. (1), multipliedby a gain of 5. These dynamics correspond to a mid-size twin-engine commercial transport aircraft witha gross weight of 185,800 lbs, linearized in a flightcondition close to the stall point, at an altitude of41,000 ft and an airspeed of 150 kts. An obviousremark regarding the transfer function presented inEq. (1) is that the roll dynamics of the aircraft areunstable in this flight condition. These dynamicsapproximate a single integrator

(

1s

)

at low frequen-

cies (< 0.76 rad/s) and a double integrator(

1s2

)

atfrequencies higher than 0.76 rad/s, due to the effectof the pole at s = -0.76 rad/s.

Hc(s) = 3.91

(

s2 + 0.22s+ 0.59)

(s+ 0.76) (s− 0.02) (s2 + 0.11s+ 0.64)(1)

The human operator control behavior was mod-eled in this compensatory tracking task using aquasi-linear model [9]. As shown in Fig. 2, theoutput of the human operator, the control inputsignal u, is the sum of a linear response and a rem-nant signal n. The linear response has two contribu-tions, Hpe

and Hpφ, which respectively model the

response to the roll tracking error (available fromthe PFD) and the response to the roll feedback(available via the out-of-the-window cues or simula-tor’s motion) [10]. The remnant signal accounts formeasurement noise and nonlinearities which are notdescribed by the linear response functions. Deter-mining the form of the transfer functions Hpe

andHpφ

and the evolution of their parameters through-out the progress of a training experiment has provento be of great help in understanding and quantify-ing the learning process of both visual and motioncues by the initially task-naive participants [8].

2

Human Operator

ft+

-

e

Error Response

Hpe

Roll Response

Hpφ

φ

ue

n

uφ

-

+ + u

fd

++

Controlled Dynamics

Hc

φδ

Figure 2: Schematic representation of the roll tracking task.

2.2. Experiment SetupThe experiment was divided in two phases, hence-forth referred to as training and evaluation. Dur-ing the training phase, the task-naive participantswere trained in the roll tracking task previously de-scribed until their level of task performance stabi-lized. They were subsequently transferred to theevaluation phase where the same roll tracking taskwas performed with different cues being provided.Participants were divided in two experimental

groups. The first group, henceforth referred to asGroup NV, was trained with only the PFD, thuswithout neither out-of-the-window visuals or mo-tion cues (NV, NM), and they were transferred toevaluation conditions where they had access to thePFD and motion feedback (NV, M). The secondgroup, henceforth referred to as Group V, wastrained without motion but with the PFD and theout-of-the-window visual cues (V, NM), and it wastransferred to the same evaluation configuration asthe Group NV where the PFD and motion cues wereavailable, without peripheral visual cues (NV, M).Therefore, the only difference between the two ex-perimental groups was the presence of out-of-the-window visual cues in the training phase.Each phase of the experiment consisted on a fixed

number of 100 tracking runs, therefore each sub-ject performed 200 runs in total. The 95-secondruns were performed in eight sessions of 25 runseach. The eight sessions were performed in fourconsecutive days, therefore two sessions on each day,with a 20 minutes break between sessions (subjectsleft the simulator between sessions). This experi-mental configuration allowed convergence of man-ual control skills in both experimental phases, witha consolidation of the acquired control skills in be-tween days happening outside the simulator, an ef-fect known as offline learning [11], and it also re-spected the optimum retention time between train-ing sessions of 24 hours [12].The experiment was performed during five weeks

on four consecutive working days of each week. Twosubjects performed their two daily sessions in themorning and two subjects performed their two dailysessions in the afternoon, meaning a total of foursubjects performed the experiment in each week.To guarantee the balancing between subjects in thegroups, every subject in one week was placed inthe same group. Therefore, two weeks had sub-jects from Group NV, two weeks had subjects fromGroup V, and the fifth week had subjects from bothgroups.In the end of each tracking run, the researcher

informed the subject of their score in that run (thescore being the root mean square value of the track-ing error signal) and asked if they were ready forthe following run. In case of an affirmative answer,the next run would start. Otherwise, some secondscould be taken as a small break in between runs toassure subjects’ concentration levels were high andas constant throughout the experiment as possible.The quasi-transfer-of-training experiment was

performed in the SIMONA Research Simulator(SRS) at the Aerospace Engineering Faculty atDelft University of Technology. Both SRS motionand outside visual systems were used, depending onthe phase of the experiment. The SRS motion sys-tem is a hexapod with hydraulic actuators, provid-ing a six degrees-of-freedom hydraulic motion sys-tem which reproduces the aircraft’s motion with atime delay of 30 ms [13]. Given that the task per-formed was a pure roll tracking task, only SIMONAroll rotation was utilized. The SRS workspace interms of roll rotation is ±25.9◦, and in this exper-iment no roll motion filter was used, thus the con-trolled roll attitude φ was given one-to-one, with-out washout filtering. The visual system of theSRS consisted on two projectors generating the twocheckerboard panels on the left and right windowviews of the simulator cockpit. The visual systemdelay is approximately 30 ms and all displays wereran at a 60 Hz refresh rate [13].

3

2.3. Participants

To perform this training experiment, subjects couldneither have any type of previous piloting experi-ence nor have participated in earlier tracking ex-periments. Another requirement the subjects hadto comply with was to be right-handed. An initialgroup of twenty fully task-naive subjects, who gavetheir written consent to participate in this study,performed the experiment. A total of ten subjectswere included in each group, considering people be-tween 18 and 23 years old, with nine different na-tionalities and three females (two placed in GroupNV and one in Group V). Two more subjects weretested in the experiment set, but due to perfor-mance inconsistencies were omitted from the finaldata set.

2.4. Human Operator Modeling

To understand how human operators acquire con-trol skills throughout their learning process, theircontrol behavior was modeled and identified in eachtracking run using a multimodal quasi-linear opera-tor model. Namely, the defining parameters of boththe error response transfer functionHpe

and the rollfeedback response transfer function Hpφ

were de-termined using identification and optimization al-gorithms. The models used for Hpe

and Hpφwere

successfully used in earlier studies [14, 15].

The considered model for the human operator er-ror response Hpe

is given by:

Hpe(s) = Ke (Tleads+ 1) e−τesHnm(s) (2)

With Hnm being the neuromuscular dynamicsmodeled by:

Hnm(s) =ω2nm

s2 + 2ζnmωnms+ ω2nm

(3)

For this roll tracking task with the consideredcontrolled element dynamics, the error responsemodel included a gain Ke, a lead equalizationterm Tlead, a human operator response delay τe,and the neuromuscular dynamics Hnm, modeled asa second-order mass-spring-damper system with aneuromuscular frequency ωnm and a neuromuscu-lar damping ratio ζnm. The considered structureof Hpe

is explained by the fact that human op-erators needed to generate lead, because the con-trolled dynamics approximated a double integratorin the frequency range where the human operatorcrossover frequency was expected to be for compen-satory tracking (1 - 5 rad/s) [8].

The human operator roll responseHpφis modeled

by:

Hpφ(s) = sKφe

−τφsHnm(s) (4)

The roll response included a pure derivative termand an equalization gain Kφ, modeling human op-erator limitations with a roll response delay τφ andthe neuromuscular system, modeled as in Hpe

. It isconvenient to mention that Hpφ

characterized thesum of multiple and separate feedback channels,namely the ones related with motion feedback, i.e.,angular accelerations detected by the semicircularcanals, linear accelerations detected by the otoliths,and motion cues from the somatosensory system[8]. The same human operator model was usedfor experimental conditions in which either motionor out-of-the-window cues were available. This al-lowed a direct assessment of how well out-of-the-window visual cues can replace motion cues.

The multi-channel pilot model defined in Eqs. (2)to (4) contained seven free parameters (Ke, Tlead,τe, Kφ, τφ, ωnm, and ζnm) which were estimatedusing maximum likelihood time-domain parameterestimation techniques, described in Ref. [7], on thecollected experimental data (the time-domain sig-nals e, φ, and u). Obviously, in the training phaseof Group NV, only Hpe

was fitted, as no out-of-the-window visual or motion cues were available.Firstly, ten repetitions of a genetic algorithm opti-mization were performed in order to obtain ten ini-tial rough estimates of the parameters, which werethe starting point of a Gauss-Newton optimizationalgorithm, yielding ten estimates for the set of pa-rameters. The estimate yielding the lowest value ofthe likelihood function was selected as the one de-scribing the control activity of the human operatorin that run. If the lowest likelihood solution failedto satisfy the physical restrictions inherent to themodel (neuromuscular frequency between 0 and 30rad/s and neuromuscular damping ratio between 0and 1), another solution from the set of ten Gauss-Newton estimates was considered. If none of theGauss-Newton estimates was in the domain of themodel parameters, the genetic algorithm solutionholding the lowest likelihood was considered as theidentified model of that run, with the validity ofthis lower likelihood solution being carefully ana-lyzed. Should this model describe the human op-erator control behavior with an unacceptable lowquality, the respective run would be omitted fromthe final data set. This procedure was applied tothe 200 runs in the training and evaluation phasesof each of the twenty subjects who performed theexperiment, and from the set of 4000 tracking runsthat compose the experiment, three were omittedfrom the final data set.

In every tracking run, the identified model Vari-ance Accounted For (VAF) was calculated as a mea-sure of the human operator model accuracy in de-scribing the measured control signal. It is an usualpractice in cybernetic studies to average the time

4

signals over a certain number of runs, to attenuatenoise in the measured data and thus improving themodel quality [14]. However, given the need to eval-uate acquisition of control skills throughout individ-ual runs, averaging was not an option and there-fore slightly lower VAF values were obtained due tohigher noise levels. Nevertheless, runs with abnor-mally low model VAFs (lower than 40 %) were con-sidered as identification outliers and excluded fromthe final data set considered, so that they would notinfluence the group average results shown in Sec-tion III. A total of 56 runs were excluded from thefinal data set due to this reason, which is a reducedpercentage of excluded runs (1.4 %) for a trainingexperiment with task-naive participants [3].

2.5. Hypotheses

Based on a number of previous tracking experi-ments where the effects of both out-of-the-windowand motion cues were studied, together with ear-lier quasi-transfer-of-training studies, the followinghypotheses were formulated for this experiment:

H1: Training causes an improvement in perfor-mance and task proficiency in both experimentalgroups. Clear effects of training were expected tooccur in both experimental groups during the train-ing phase, as seen in a number of previous trainingexperiments (Refs. [3, 8] and [16]), which are visi-ble in improved performance (lower σ2

e), increasedcontrol activity (higher σ2

u), and higher crossoverfrequencies and phase margins. In the human oper-ator modeling results, it was expected to see adjust-ments in parameters that are known to be relatedto improved performance (increased Ke, Kφ, lowerhuman operator delays).

H2: The presence of peripheral visual cues intraining of control skills provides a feedback channelof the controlled dynamics output. For the grouptrained with visual conditions (Group V), previousstudies (Refs. [1] and [6]) suggest that visual cuesavailable in training provide a feedback channel forthe roll angle and this was expected to be visiblein better performance (lower σ2

e) and in the humanoperator parameters describing the response to rollangle feedback. The roll gain Kφ and the roll delayτφ were expected to be different from zero in thetraining phase of Group V.

H3: In the evaluation phase, the presence of mo-tion allows reaching better task performance lev-els. The addition of motion cues in a tracking taskallows reaching better levels of task performance[3, 8, 17]. This effect was expected to be mainly visi-ble in performance metrics with lower σ2

e and higherσ2u. In the human operator parameters, higher gains

(Ke and Kφ), lower delays (τe and τφ), and espe-cially lower values of Tlead were expected as a con-sequence of the lead information provided by the

motion feedback.

H4: Adaption to motion conditions is faster forsubjects who were trained with out-of-the-window vi-sual cues. The final level of task proficiency wasexpected to be reached earlier by subjects in GroupV, meaning less hours of training would be neededin a flight simulator with motion conditions, asthe transfer of control skills to a motion conditionwas expected to be more effective for subjects whotrained with the presence of out-of-the-window vi-suals. This is supported by earlier findings thatout-of-the-window visuals function to a certain ex-tent as a motion feedback channel [5]. This wouldbe visible in higher learning rates in the evaluationcondition for subjects in Group V and better per-formances of this group immediately after transfer.

3. Results

In the Figures of this section, data from Group NVis shown in blue and data from Group V in red.In plots where data evolution is shown over 200runs, a black vertical line indicates the transfer af-ter run 100. Learning curves are fitted to the datawhen Pearson’s correlation coefficient is higher than0.5, with the Pearson’s correlation coefficients beingshown in the plot’s legend with the following orga-nization: ρ = [ρtraining, ρevaluation]. Gray error barsare plotted indicating the 95% confidence intervalsof mean data for plots showing the evolution withthe runs.

3.1. Tracking Performance

Tracking performance was measured with the vari-ance of the roll error, i.e., the error presented to thehuman operator on the PFD. The lower the value ofσ2e , the better the performance. Figure 3(a) shows

the average variance of the tracking error per ex-periment run, together with fitted learning curvesand the 95% confidence intervals of the mean data.Furthermore, to evaluate the performance improve-ment throughout the experiment, a decompositionin components of tracking error variance was made,separating the contributions from the disturbanceforcing function, the target forcing function, andthe remnant noise [18]. Results are shown, respec-tively, in Figs. 3(b), 3(c), and 3(d). The parametersof the fitted learning curves in Fig. 3 are presentedon the left side of Tables 1 to 4.

Some important conclusions can be drawn whenlooking at the tracking error results presented inFig. 3. In the training phase, the performance levelof Group V was always better than the performancelevel reached by Group NV in every component ofthe tracking error, suggesting that the presence ofout-of-the-window visual cues indeed improved hu-man operator performance, as expected from pre-vious studies. In Fig. 3(a), it can be seen thatGroup NV showed a steeper improvement in per-

5

Average of group NV

Average of group V

Fit Group NV, ρ = [0.83, 0.93]

Fit Group V, ρ = [0.81, 0.92]

Evaluation RunsTraining Runs

σ2 e,deg2

0 25 50 75 100 125 150 175 200

0.5

1

1.5

2

2.5

(a) Total tracking error variance.

Average of group NV

Average of group V

Fit Group NV, ρ = [0.74, 0.93]

Fit Group V, ρ = [0.83, 0.95]

Evaluation RunsTraining Runs

σ2 ed,deg2

0 25 50 75 100 125 150 175 200

0.2

0.4

0.6

0.8

1

(b) Disturbance tracking error variance.

Average of group NV

Average of group V

Fit Group NV, ρ = [0.78, 0.84]

Fit Group V, ρ = [0.81, 0.75]

Evaluation RunsTraining Runs

σ2 et,deg2

0 25 50 75 100 125 150 175 200

0

0.2

0.4

0.6

0.8

(c) Target tracking error variance.

Average of group NV

Average of group V

Fit Group NV, ρ = [0.81, 0.85]

Fit Group V, ρ = [0.68, 0.76]

Evaluation RunsTraining Runs

σ2 en,deg

2

0 25 50 75 100 125 150 175 200

0

0.5

1

1.5

(d) Remnant tracking error variance.

Figure 3: Measured tracking error variance and the disturbance, target, and remnant contributions.

Table 1: Learning curve parameters and statistical analysis for total tracking error.

σ2e

Learning Curve Parametersσ2e

Statistical SignificanceTraining Phase Evaluation Phase Training Transfer Evaluation

p0, deg2 pa, deg

2 F (×10−2) p0,deg2 pa, deg

2 F (×10−2) Ws Sig. Ws Sig. Ws Sig.Group NV 2.94 1.32 5.42 1.04 0.51 2.72 Group NV 55 ∗∗ 51 ∗ 55 ∗∗

Group V 2.26 1.09 1.38 0.93 0.51 3.05 Group V 55 ∗∗ 55 ∗∗ 55 ∗∗

Table 2: Learning curve parameters and statistical analysis for disturbance tracking error.

σ2ed

Learning Curve Parametersσ2ed

Statistical SignificanceTraining Phase Evaluation Phase Training Transfer Evaluation

p0, deg2 pa, deg

2 F (×10−2) p0,deg2 pa, deg

2 F (×10−2) Ws Sig. Ws Sig. Ws Sig.Group NV 0.89 0.57 2.15 0.45 0.23 2.79 Group NV 55 ∗∗ 51 ∗∗ 54 ∗∗

Group V 0.86 0.54 2.17 0.44 0.21 2.38 Group V 55 ∗∗ 55 ∗∗ 55 ∗∗

Table 3: Learning curve parameters and statistical analysis for target tracking error.

σ2et

Learning Curve Parametersσ2et

Statistical SignificanceTraining Phase Evaluation Phase Training Transfer Evaluation

p0, deg2 pa, deg

2 F (×10−2) p0,deg2 pa, deg

2 F (×10−2) Ws Sig. Ws Sig. Ws Sig.Group NV 0.66 0.33 4.94 0.28 0.15 3.12 Group NV 53 ∗∗ 49 ∗ 55 ∗∗

Group V 0.66 0.31 6.49 0.26 0.16 4.63 Group V 54 ∗∗ 41 — 55 ∗∗

Table 4: Learning curve parameters and statistical analysis for remnant tracking error.

σ2en

Learning Curve Parametersσ2en

Statistical SignificanceTraining Phase Evaluation Phase Training Transfer Evaluation

p0, deg2 pa, deg

2 F (×10−2) p0,deg2 pa, deg

2 F (×10−2) Ws Sig. Ws Sig. Ws Sig.Group NV 1.13 0.31 7.35 0.26 0.08 1.41 Group NV 55 ∗∗ 30 — 54 ∗∗

Group V 0.57 0.15 2.33 0.19 0.11 3.73 Group V 55 ∗∗ 45 — 47 ∗

Legend:∗∗ indicates highly significant (p < 0.01) statistical differences between compared samples.∗ indicates significant (0.01 ≤ p < 0.05) statistical differences between compared samples.— indicates no significant (p ≥ 0.05) statistical differences between compared samples.

6

formance during the training phase, whereas thelearning curve for Group V was smoother, mainlybecause of the contribution of the remnant errorseen in Fig. 3(d). This is also seen in the learn-ing rates in Table 1, as F in the training phase washigher for Group NV. However, while Group NVstabilized their performance approximately at σ2

e =1.32 deg2 around run 60, Group V kept a steady im-provement throughout the 100 training runs, reach-ing σ2

e = 1.09 deg2 at the end of the training phase.This suggests it took longer to adapt to the periph-eral visual scene but the information it providedallowed better performance, even in the first runswhere a clear difference between groups was alreadyvisible. Statistical data shown in Tables 1-4 for thetraining phase confirm the significant improvementsin performance. It is also important to notice thatthe contribution of the disturbance forcing functionwas larger than the contribution of the target forc-ing function, as seen in a comparison of Figs. 3(b)and (c), given that the experiment had mainly adisturbance-rejection character.

Upon transfer, both groups showed a similar evo-lution in tracking performance, with an instanta-neous decrease in total error variance of 0.3 deg2 inGroup NV and 0.2 deg2 in Group V.

For the evaluation phase, it can be seen that, asreported by numerous previous studies [3, 6], mo-tion cues were more effective and allowed for no-table improvements in performance levels of sub-jects in both groups. Both groups ended in thesame level of asymptotic task proficiency in termsof performance (total error of 0.51 deg2), showinga convergence of control skills, a necessary premiseto validate the results of any training experiment.The confidence intervals for the evaluation phasewere much smaller in amplitude and the Pearson’scoefficients of the learning curves were also higher inthis phase, which suggest lower subject variabilityas the experiment approached its end and subjectsbecame more proficient in the task. Group NV hada lower evaluation learning rate in the total error,target error, and remnant error than Group V whichmeans Group V was faster in learning how to usemotion cues. The improvement in performance forboth groups during the evaluation phase was vali-dated by the statistical analysis.

3.2. Control Activity

Figure 4 shows the evolution of the human operatorcontrol input throughout the 200 runs performed,in a similar way as it was done for the trackingperformance.

Looking at Fig. 4, it can be seen that the im-provement in performance made by both groups inthe training phase was reached with approximatelyconstant control activity throughout the runs, as

Average of group NV

Average of group V

Fit Group NV, ρ = [—, 0.72]

Fit Group V, ρ = [—, 0.93]

Evaluation RunsTraining Runs

σ2 u,deg

2

0 25 50 75 100 125 150 175 200

2

4

6

8

10

12

Figure 4: Average control input variance.

no significant difference exists. Learning curveswere not fitted in the training phase control datagiven the low Pearson’s correlation coefficient. Af-ter the transfer of conditions, Group NV imme-diately showed a great increase in control power,whereas Group V showed a smaller increase. Forthe evaluation phase, the increase in control activ-ity because of motion was expected [3, 6]. GroupNV registered much stronger inputs than GroupV, in spite of an unusually high spread of values.Notwithstanding, in the last runs, a leveling in thepower of the control signal was visible, approachingthe levels of Group V. This behavior was not por-trayed by the learning curve, which translated in alower Pearson’s coefficient.

3.3. Human Operator Model Parameters

In this section, the estimated parameters of the er-ror response, the roll response and the neuromus-cular system are presented. The average parameterestimation results do not show a learning trend inthe training phase, and therefore learning curves arenot shown for this phase.

Considering the human operator error parame-ters in the training phase, it is clear that boththe gain, the lead time constant and the delay didnot show any consistent trend, remaining approx-imately constant throughout the 100 runs. Thisis consistent with the results of Ref. [8], where asimilar task with the same controlled dynamics wasperformed. These evolutions are also consistent towhat was found in control input metrics, in Fig. 4.Similarities between both groups indicate that pe-ripheral visual cues do not affect the response inthis channel. For the evaluation phase, the pres-ence of motion induced a significant increase in theerror gain for both groups, whereas the lead timeconstant decreased, as expected given the lead in-formation motion provides [17, 19]. Comparing thelearning curve shape of both groups, it is visiblethat Group V had lower learning rates in Kφ andTlead suggesting less transfer of skills for Group V.

With respect to the neuromuscular system pa-rameters, no effects of learning were observed in

7

Average of group NV

Average of group V

Fit Group NV, ρ = [—, 0.90]

Fit Group V, ρ = [—, 0.92]

Evaluation RunsTraining Runs

Ke,—

0 25 50 75 100 125 150 175 200

0

0.5

1

1.5

(a) Error gain.

Average of group NV

Average of group V

Fit Group NV, ρ = [—, 0.91]

Fit Group V, ρ = [—, 0.93]

Evaluation RunsTraining Runs

TLead,s

0 25 50 75 100 125 150 175 200

0

0.5

1

1.5

2

(b) Error lead time constant.

Average of group NV

Average of group V

Evaluation RunsTraining Runs

τe,s

0 25 50 75 100 125 150 175 200

0.1

0.2

0.3

0.4

0.5

0.6

(c) Error Delay.

Average of group NV

Average of group V

Evaluation RunsTraining Runsω

nm,ra

d/s

0 25 50 75 100 125 150 175 200

5

10

15

20

25

(d) Neuromuscular frequency.

Average of group NV

Average of group V

Evaluation RunsTraining Runs

ζnm,—

0 25 50 75 100 125 150 175 200

0

0.2

0.4

0.6

0.8

1

(e) Neuromuscular damping ratio.

Average of group NV

Average of group V

Fit Group NV, ρ = [—, 0.95]

Fit Group V, ρ = [—, 0.97]

Evaluation RunsTraining Runs

Kφ,—

0 25 50 75 100 125 150 175 200

−0.2

0

0.2

0.4

0.6

0.8

1

(f) Roll Gain.

Figure 5: Human operator model parameters and roll channel variance fraction.

the neuromuscular frequency, which was higher forGroup V in training and it increased after trans-fer. This was an expected effect of experimentalconditions with motion and can be seen as thehuman arm getting stiffer in motion conditions,corresponding to the contraction of the arm andhand muscle [14]. Neuromuscular damping ratio de-creased throughout the experiment, which was alsoexpected and it is a sign of task proficiency becausewith decreasing damping ratios phase lag is slightlylower in the frequencies where the human opera-tor is actively controlling (frequencies around thecrossover frequency).

Finally, for the roll gain, it is visible that theuse of the roll feedback channel was much smallerin training than in evaluation, suggesting that out-of-the-window visual cues were not as effective inproviding a roll feedback channel as motion. It was

seen though a positive evolution in roll gain duringtraining, which was initially zero and in the endof training was slightly greater than zero, but thiseffect is not significant enough to state an effectivedifference introduced by the presence of the outsidevisual scene.

4. Discussion

Considering the results from previous training ex-periments [3, 16, 20], it was expected to see cleareffects of skill development in the training phase inboth groups (Hypothesis H1). The progress seenin the first 100 runs showed a positive evolution interms of task performance, with a decrease in track-ing error variance due to a consistent reduction ofdisturbance-rejection, target-tracking and remnanterror variances. The decrease in remnant error vari-ance means the initial task-naive participants in-creased their linearity, which is a clear training ef-

8

fect. In terms of control activity, no clear tendencyexisted during training, which was consistent withthe human operator model parameters whose aver-age estimates were approximately constant for thefirst 100 runs. Therefore, training causes an im-provement in task proficiency but not necessarilyin terms of human control dynamics, whose param-eters remained constant throughout training. How-ever, these parameters describe the human controlbehavior progressively better as the human opera-tor linearity increases with the number of runs.

Based on results from earlier studies, such asRef. [5], which investigated the effect of out-of-the-window visual cues on tracking task performanceand human control behavior, it was hypothesizedthat subjects in Group V would develop duringtraining a roll feedback channel similar as the onecreated when motion cues are available (Hypoth-esis H2). Analyzing the results obtained in thisexperiment, out-of-the-window visuals helped sub-jects performing the control task, as Group V had alower tracking error variance in the training phase.However, the average estimates of the motion gainKφ were close to zero throughout training, meaningno strong roll feedback channel was used. This re-duced effect of a roll feedback channel created withperipheral visual cues is not entirely consistent withprevious studies on the effect of out-of-the-windowvisual cues. A reason for this to happen mightbe due to the fact that the roll stimulus providedby the checkerboards was weaker than the yaw vi-sual stimulus provided in Ref. [5]. Furthermore, inRef. [5], participants were not task-naive but expe-rienced subjects who logically attain better perfor-mance easier. Another cause might be the differentdynamics controlled, as the unstable roll dynamicsused here require a control strategy with a strongerneed for lead equalization.

When transferring to motion conditions, bothgroups were seen to achieve better performance us-ing stronger control activity (Hypothesis 3), follow-ing what was observed in previous studies, as inRefs. [6] and [8]. A clear evolution in the humanmodel parameters was also seen, with higher er-ror and roll gains and lower lead time constants,as a consequence of the lead information motionfeedback provides. Therefore, motion significantlyhelped human operators performing this controltracking task, confirming Hypothesis 3. Great dif-ferences were, however, found when groups werecompared, with motion and visual gains beinghigher for Group NV, together with lower lead con-stants. This is explained with the significant differ-ences in control activity levels between the groupsin the evaluation phase, as Group NV adopted a sig-nificantly stronger control activity. Stronger inputsmean the dynamics are being more excited and thus

better perceived by the human operator. Therefore,stronger control inputs increase the benefits frommotion feedback.

As a consequence of the absence of a roll feed-back channel with visual cues in the training phase,the benefit of training with visual cues was alsonot verified when transferring to motion (Hypoth-esis H4). On one side, supporting this hypothesis,lower tracking errors and higher learning rates intracking error data were indeed found for subjectsin Group V when compared to subjects in GroupNV, but on the other side this tendency was notconfirmed by the human operator model parame-ters, where higher learning rates were in fact foundfor Group NV in the evaluation phase.

Finally, looking at the evaluation phase of bothgroups, the fact that a great improvement wasachieved with respect to training suggests an in-effective training setup. The control skills learnedin a fixed-base environment showed limited directtransfer to the moving-base condition, which hadbeen described in previous experiments for the casewhere no visual scene was provided. [3, 16, 20] Thedata collected in this experiment supports the con-clusion that manual control skills developed duringtraining with a peripheral visual scene also do notpositively transfer to a motion condition. While pe-ripheral visual cues are beneficial in terms of per-formance and simulator realism, they seem to noteffectively create the feedback channel motion uti-lizes and therefore they do not replace motion as acue in an initial phase of simulator-based training.

5. Conclusion

A quasi-transfer-of-training experiment was per-formed to evaluate to which extent out-of-the-window visual cues are effective as an initial settingof simulator-based training, under the hypothesisthat such cues ease the developing of manual con-trol skills with physical motion. This easiness wouldbe created because peripheral visual cues would de-velop a feedback channel similar to the one motionis known to introduce in human operator controlstrategy. Looking at the results from the experi-ment, training with peripheral visual cues causedlower tracking errors which can be seen as a clearbenefit from this source of stimuli. However, whenanalyzing parameters describing human operatorcontrol behavior, a response to a roll feedback pro-vided by peripheral visual cues was barely existent,which indicates that developing control skills witha peripheral visual scene does not hold any strongbenefit when transferring to a motion condition.Physical motion is ultimately the most relevant cuein simulator-based training of piloting skills.

9

References

[1] R.J.A.W. Hosman and J.C. van der Vaart. Ef-fects of vestibular and visual motion percep-tion on task performance. Acta Psychologica,48(13):271 – 287, 1981.

[2] Gavan Lintern, Stanley N. Roscoe, andJonathan E. Sivier. Display Principles, ControlDynamics, and Environmental Factors in PilotTraining and Transfer. Human Factors: TheJournal of the Human Factors and ErgonomicsSociety, 32(3):299–317, June 1990.

[3] Daan M. Pool, Gertjan A. Harder, and Mari-nus M. van Paassen. Effects of Simulator Mo-tion Feedback on Training of Skill-Based Con-trol Behavior. Journal of Guidance, Control,and Dynamics, 39(4):889–902, 2016.

[4] Peter R. Grant, Bonnie Yam, Ruud J. A. W.Hosman, and Jeffery Allyn Schroeder. Effect ofSimulator Motion on Pilot Behavior and Per-ception. Journal of Aircraft, November 2006.

[5] Henk P. M. Peterse, Daan M. Pool, Marinus M.van Paassen, and Max Mulder. Interactions ofOutside Visual Cues and Motion Cueing Set-tings in Yaw Tracking. In Proceedings of theAIAA Modeling and Simulation TechnologiesConference, Washington, D.C., June 2016.

[6] Daan M. Pool, Max Mulder, Marinus M. vanPaassen, and J. C. van der Vaart. Effects ofPeripheral Visual and Physical Motion Cuesin Roll-Axis Tracking Tasks. Journal of Guid-ance, Control, and Dynamics, 31(6):1608–1622, 2008.

[7] Peter M. T. Zaal, Daan M. Pool, Q. P. Chu,Marinus M. van Paassen, Max Mulder, andJan A. Mulder. Modeling Human MultimodalPerception and Control Using Genetic Maxi-mum Likelihood Estimation. Journal of Guid-ance, Control, and Dynamics, 2009.

[8] Peter M. T. Zaal, Alexandru Popovici, andMelinda A. Zavala. Effects of False Tilt Cueson the Training of Manual Roll Control Skills.In Proceedings of the AIAA Modeling and Sim-ulation Technologies Conference, Kissimmee(FL), number AIAA-2015-0655, 2015.

[9] Duane T McRuer, Dunstan Graham, ES Kren-del, and William Reisener Jr. Human pilot dy-namics in compensatory systems. Air ForceFlight Dynamics Laboratory, 1965.

[10] William H. Levison and Andrew M. Junker. AModel for the Pilot’s use of Motion Cues inRoll-Axis Tracking Tasks. Technical report,Bolt Beranek and Newman Inc., April 1977.

[11] Edwin M Robertson, Alvaro Pascual-Leone,and Daniel Z Press. Awareness modifies theskill-learning benefits of sleep. Current Biol-ogy, 14(3):208–212, 2004.

[12] Shailesh S. Kantak and Carolee J. Winstein.Learning-performance distinction and memoryprocesses for motor skills. Behavioural BrainResearch, 228(1):219–231, 2012.

[13] Olaf Stroosma, Marinus M. van Paassen, MaxMulder, and Ferdinand N. Postema. Measur-ing Time Delays in Simulator Displays. In Pro-ceedings of the AIAA Modeling and SimulationTechnologies Conference and Exhibit, HiltonHead (SC), number AIAA-2007-6562, 2007.

[14] Peter M. T. Zaal, Daan M. Pool, Jaapde Bruin, Max Mulder, and Marinus M. vanPaassen. Use of Pitch and Heave Motion Cuesin a Pitch Control Task. Journal of Guidance,Control, and Dynamics, 32(2):366–377, 2009.

[15] Daan M. Pool, Peter M. T. Zaal, Marinus M.van Paassen, and Max Mulder. Effects ofHeave Washout Settings in Aircraft Pitch Dis-turbance Rejection. Journal of Guidance, Con-trol, and Dynamics, 33(1):29–41, 2010.

[16] Joost C. F. de Winter, Dimitra Dodou, andMax Mulder. Training Effectiveness of WholeBody Flight Simulator Motion: A Comprehen-sive Meta-Analysis. The International Journalof Aviation Psychology, 22(2):164–183, 2012.

[17] Richard S Shirley and Laurence R Young.Motion cues in man-vehicle control effects ofroll-motion cues on human operator’s behav-ior in compensatory systems with disturbanceinputs. IEEE Transactions on Man-MachineSystems, 9(4):121–128, 1968.

[18] Henry R. Jex, R. E. Magdaleno, and An-drew M. Junker. Roll Tracking Effects ofG-vector Tilt and Various Types of MotionWashout. In Proceedings of the FourteenthAnnual Conference on Manual Control, pages463–502, 1978.

[19] Ronald A. Hess. Model for Human Use of Mo-tion Cues in Vehicular Control. Journal ofGuidance, Control, and Dynamics, 1990.

[20] Daan M. Pool, Gertjan A. Harder, Herman J.Damveld, Marinus M. van Paassen, and MaxMulder. Evaluating Simulator-Based Trainingof Skill-Based Control Behavior using Multi-modal Operator Models. In Proceedings of the2014 IEEE International Conference on Sys-tems, Man, and Cybernetics, October 2014.

10