Post on 28-Apr-2018
A Simulator Study Evaluating the Efficacy of Group-View Displays in Nuclear Control Rooms
by
Sean William Kortschot
A thesis submitted in conformity with the requirements for the degree of Masters of Applied Science
Mechanical and Industrial Engineering University of Toronto
© Copyright by Sean William Kortschot 2016
ii
A Simulator Study Evaluating the Efficacy of Group-View Displays in Nuclear Control Rooms
Sean William Kortschot
Masters of Applied Science
Mechanical and Industrial Engineering University of Toronto
2016
Abstract
Group View Displays (GVDs), have become prominent features in modern nuclear control room
(NCR) designs. Despite their widespread implementation, an operating experience review by
Myers & Jamieson (2014) revealed that the efficacy of these GVDs has yet to be thoroughly
evaluated. This thesis addresses this gap by presenting the analysis, design, and experimental
results of a full-scale nuclear simulator study designed to evaluate the purported benefits of
GVDs. The results of this study indicate that the current state of the art in NCRs can be improved
upon. We found evidence suggesting that the dominant GVD solution in NCRs was
outperformed in terms of communication by the experimental GVD alternative. Furthermore, we
demonstrated marked situation awareness improvements fostered by the ecological display
framework when compared against the advanced displays currently in use in NCRs. The
implications, limitations, and recommendations for future work from this study are discussed.
iii
Acknowledgments
I would like to thank CANDU Energy Inc. for their financial support and assistance throughout
this project . Specifically, I would like to thank Robert Leger who provided guidance and
assistance from start to finish. This work would not be possible without the help of Rick Bodner,
Brandon Elliott, Richard Brown, and the rest of the team that helped us set up the simulator and
interfaces at CANDU. I would like to also thank Raymond Dufrense, Radik Ixanov, and
Christian Runkowski, whose expertise was unparalleled.
I want to extend my gratitude to my family, who supported me through this entire process.
Special thanks to Cole Wheeler, whose friendship made the long hours at the simulator
enjoyable. Thanks to Billy Myers for laying down the groundwork of this project, Antony
Hilliard who listened to endless questions about everything from CWA to statistical analysis, and
to the rest of CEL. Special thanks to Maya Whitehead, for your friendship, insights, and
conversation.
I would also like to thank Birsen Donmez and Olivier St-Cyr for your assistance and guidance as
my committee members. I would particularly like to thank my supervisor, Greg Jamieson, for
your endless support and guidance. Your assistance and mentorship throughout my degree
showed the structure and patience required for good research.
iv
Table of Contents
ACKNOWLEDGMENTS.........................................................................................................................III
TABLEOFCONTENTS...........................................................................................................................IV
LISTOFTABLES..................................................................................................................................VII
LISTOFFIGURES................................................................................................................................VIII
LISTOFAPPENDICES...........................................................................................................................XI
INTRODUCTION.............................................................................................................................11
1.1 PLANTSAFETYANDEARLYDISPLAYS.......................................................................................................1
1.2 DISPLAYEVOLUTION............................................................................................................................1
1.2.1 DigitizationofControlRooms..................................................................................................2
1.2.2 InterfaceAdvancement...........................................................................................................3
1.3 GROUPVIEWDISPLAYS........................................................................................................................5
1.3.1 LargeScreenDisplays..............................................................................................................5
1.3.2 RedundantDisplays.................................................................................................................6
1.4 SELECTEDLITERATUREONGVDS...........................................................................................................8
1.5 RESEARCHQUESTIONS.........................................................................................................................8
1.6 DOCUMENTSTRUCTURE......................................................................................................................9
PROJECTSCOPE............................................................................................................................102
2.1 THECANDU6®NUCLEARPOWERPLANT.............................................................................................11
COGNITIVEWORKANALYSIS........................................................................................................123
3.1 STAGESOFANALYSIS.........................................................................................................................13
3.2 WORKDOMAINANALYSIS..................................................................................................................14
3.2.1 AnalyticalTool–TheAbstractionHierarchy.........................................................................14
3.2.2 ScopeofAnalysis...................................................................................................................15
3.2.3 ProductsofAnalysis..............................................................................................................17
3.2.4 SummaryofWorkDomainAnalysis......................................................................................22
3.3 CONTROLTASKANALYSIS...................................................................................................................23
3.3.1 AnalyticalTool–TheDecisionLadder...................................................................................23
v
3.3.2 ScopeofAnalysis...................................................................................................................24
3.3.3 ProductsofAnalysis..............................................................................................................25
3.3.4 SummaryofConTA................................................................................................................28
3.4 STRATEGIESANALYSIS........................................................................................................................28
3.4.1 AnalyticalTool–InformationFlowMap...............................................................................28
3.4.2 ScopeofAnalysis...................................................................................................................29
3.4.3 ProductsofAnalysis..............................................................................................................29
3.4.4 SummaryofStrategiesAnalysis............................................................................................33
3.5 SUMMARYOFCOGNITIVEWORKANALYSIS...........................................................................................33
DISPLAYDEVELOPMENT..............................................................................................................344
4.1 DISPLAYCOMPARISON.......................................................................................................................34
4.1.1 InformationalContent...........................................................................................................35
4.1.2 DisplayNavigation................................................................................................................35
4.1.3 DisplayFidelity......................................................................................................................35
4.1.4 DisplayAppearance...............................................................................................................35
4.2 DEVELOPMENTOFNOVELDISPLAYS.....................................................................................................36
4.2.1 Existingformusage...............................................................................................................37
4.2.2 DevelopmentofNovelForms................................................................................................38
4.2.3 DisplayFinalization................................................................................................................42
EXPERIMENTALMETHOD.............................................................................................................455
5.1 CONTROLROOMMOCK-UP...............................................................................................................45
5.2 DESIGN...........................................................................................................................................45
5.2.1 ExperimentalTasks................................................................................................................47
5.2.2 ExperimentalScenarios.........................................................................................................49
5.3 PARTICIPANTS..................................................................................................................................52
5.3.1 PlantRepresentation.............................................................................................................53
5.3.2 DisplaySimplification............................................................................................................55
5.3.3 ParticipantTrainingPackage................................................................................................56
5.4 MEASURES......................................................................................................................................56
5.4.1 SituationAwareness..............................................................................................................56
5.4.2 Communication.....................................................................................................................59
5.4.3 DiagnosticPerformance........................................................................................................60
vi
5.5 PROCEDURE.....................................................................................................................................61
5.5.1 Training.................................................................................................................................61
5.5.2 ExperimentalTrials................................................................................................................63
5.5.3 Post-TrialInterview&Debriefing..........................................................................................64
EXPERIMENTALRESULTS..............................................................................................................646
6.1 SITUATIONAWARENESS.....................................................................................................................64
6.1.1 InferredModel.......................................................................................................................69
6.1.2 DifferencesofLeastSquaresMeans......................................................................................69
6.2 COMMUNICATION.............................................................................................................................71
6.3 DIAGNOSTICPERFORMANCE...............................................................................................................72
DISCUSSION.................................................................................................................................747
7.1 SITUATIONAWARENESS.....................................................................................................................75
7.2 COMMUNICATION.............................................................................................................................77
7.3 DIAGNOSTICPERFORMANCE...............................................................................................................78
SUMMARY&CONCLUSIONS........................................................................................................798
8.1 SUMMARY.......................................................................................................................................79
8.2 CONCLUSIONS..................................................................................................................................81
8.2.1 Contributions.........................................................................................................................81
8.2.2 Limitations.............................................................................................................................82
8.2.3 FutureResearch.....................................................................................................................83
REFERENCES......................................................................................................................................85
APPENDICES......................................................................................................................................92
vii
List of Tables
Table 1. Relative advantages and disadvantages of GVD alternatives ........................................... 7
Table 2. Summary of Research Questions .................................................................................... 10
Table 3. Five abstraction levels of the AH (adapted from Rasmussen, 1985) .............................. 14
Table 4. Experimental Design ....................................................................................................... 46
Table 5. Example of one team's experimental schedule. .............................................................. 49
Table 6. List of selected faults for experimental scenarios ........................................................... 52
Table 7. Summary of measures ..................................................................................................... 61
Table 8. Number of PO measure queries for each Scenario X Setup ........................................... 65
Table 9. Type III Test for Fixed Effects ....................................................................................... 69
Table 10. Type III Test for Fixed Effects ..................................................................................... 71
Table 11. Type III Test for Fixed Effects ..................................................................................... 73
Table 12. Summary of main findings. .......................................................................................... 75
viii
List of Figures
Figure 1. Large screen displays used in the Qinshan control room. ............................................... 6
Figure 2. Redundant display configuration (Kiran Infosystems, 2014). ......................................... 7
Figure 3. Simplified diagram of the CANDU 6® main systems. .................................................. 12
Figure 4. Overall AH for the CANDU 6 nuclear power plant. Green boxes indicate the systems
included in this report. .................................................................................................................. 17
Figure 5. AH for HTS. .................................................................................................................. 18
Figure 6. AH for the P&IC. .......................................................................................................... 19
Figure 8. AH for the MSS. ............................................................................................................ 21
Figure 9. AH for the turbine system. ............................................................................................ 22
Figure 10. Typical decision ladder structure (Jenkins et al., 2010). ............................................. 24
Figure 11. DL for active search for fault without an alarm. ......................................................... 26
Figure 12. DL representing an active search for a deviation with the guidance of an alarm. ...... 28
Figure 13. IFM depicting the strategies used for searching for a fault without an alarm. ............ 30
Figure 14. IFM depicting the different strategies for searching for a fault with an alarm ............42
Figure 15. IFM depicting the different strategies within the task of online diagnosis of a fault. . 32
Figure 16. Unusable graphical forms ............................................................................................ 36
Figure 17. Mass-Energy-Saturation Ecological form ................................................................... 37
Figure 18. Rearrangement of existing charts. ............................................................................... 38
Figure 19. Mass-balance plots ...................................................................................................... 39
ix
Figure 20. Delta plot showing current inflow balancing with current outflow and the level at the
setpoint. ......................................................................................................................................... 40
Figure 21. Four quadrants of the delta plot ................................................................................... 41
Figure 22. Delta plot with trend information ................................................................................ 42
Figure 23. Ecological Critical Safety Parameter screen. .............................................................. 43
Figure 24. Ecological Plant Operations screen. ............................................................................ 43
Figure 25. Ecological Steam Generator screen. ............................................................................ 44
Figure 26. Ecological Primary Heat Transfer screen. ................................................................... 44
Figure 27. Schematic of the CANDU mock-up ............................................................................ 45
Figure 28. GVD configurations ................................................................................................... 46
Figure 29. Display types .............................................................................................................. 47
Figure 30. Operators performing primary monitoring task in the LSD configuration. ................ 48
Figure 31. Operators performing their secondary distractor task in the redundant display
configuration. ................................................................................................................................ 49
Figure 32. Scenario timelines. ...................................................................................................... 52
Figure 33. Final diagram presented to participants. ...................................................................... 55
Figure 34. Typical process overview query. ................................................................................. 57
Figure 35. Example of how recently was defined. ........................................................................ 58
Figure 36. Visible vs. Inferred means. .......................................................................................... 66
Figure 37. PO measure means for each scenario across the four conditions ................................ 67
Figure 38. Each group's PO measure scores across the four setups. ............................................ 67
x
Figure 39. Individual subjects' PO measure scores over the different setups. .............................. 68
Figure 40. LS Mean score for each experimental setup. ............................................................... 69
Figure 41. Multiple comparisons diffogram showing no significant contrasts. ........................... 70
Figure 42. Communication scores across the four experimental setups. ...................................... 72
Figure 43. Diagnostic performance mean estimates by experimental setups. .............................. 73
Figure 45. Difference in performance between the two display configurations. .......................... 74
Figure 46. Differences between advanced and ecological graphics. ............................................ 77
xi
List of Appendices
Appendix A. Information Requirements ....................................................................................... 92
Appendix B. Experimental Schedule ............................................................................................ 95
Appendix C. Process Overview Queries ....................................................................................... 96
Appendix D. Selected Statistical Outputs ................................................................................... 100
Appendix E. Selected SAS Code ................................................................................................ 101
1
Introduction 1Nuclear power generation represents a viable, stable method for producing carbon-free energy
and currently accounts for over 11% of global power supply (World Nuclear Association, 2016).
Unfortunately, accidents in the past such as Three-Mile Island, Chernobyl, and Fukushima, as
well as the initial war-related motives behind its development have tainted the global perception
of nuclear energy (Nuclear Energy Agency, 2010). In order to move past this public perception
and ensure the safe operation of this carbon-free resource, designers and engineers must
minimize the risk of operating the plants without compromising their functional purpose of
large-scale power production.
1.1 Plant Safety and Early Displays
From the beginning of nuclear power generation in the early 1950s, engineers were conscious of
the potential risks involved with fission power (World Nuclear Association, 2014). This
motivated the design and development of robust physical systems capable of containing the
radiation inherent to nuclear power generation. To ensure the proper operation of these physical
systems, engineers also needed to develop equally robust control and monitoring systems (IAEA,
2009).
The monitoring systems used to run early generation nuclear power plants were quite crude by
today’s standards, being composed of hardwired dials and gauges (IAEA, 2009). The human-
factors issues with these systems were first noticed after the Three-Mile Island incident in 1979,
wherein human error in monitoring allowed for coolant to boil away thereby causing the reactor
core to remain exposed for approximately fourteen hours (Joyce & Lapinsky, 1983).
Investigators concluded that these monitoring errors were largely the result of ineffective
displays, stating that the information on these displays was presented too disparately for the
operators to assemble meaningful mental models of the plant state.
1.2 Display Evolution
The findings from Three-Mile Island led to design recommendations for future safety parameter
displays. Investigators determined that to safely monitor plant operations a display must facilitate
(Joyce & Lapinsky, 1983):
2
1. Developing accurate and complete mental models of plant parameters,
2. Integrating information from dispersed areas of the control room,
3. Remembering that gathered information for later comparisons, and
4. Integrating all of this information to update the mental model of the plant.
Designers began to realize that presenting the values of individual parameters in separate
locations was not conducive to operators’ understanding of the behaviour of the plant (Joyce &
Lapinsky, 1983). In other words, simply ensuring that all of the requisite information is available
on the display is not sufficient. This can result in unnecessary taxation of the operators’ mental
resources, thereby inhibiting adequate situation awareness.
To ensure the presentation of sufficient content and suitable configuration of that content, the
Electric Power Research Institute put forth a three-step procedure for the development of
displays in control rooms (Beltracchi, 1988):
1. Task and functional analysis to ensure that the information required to perform each task
and subtask was available,
2. Synthesis, which is the process of determining appropriate display formats for the
information from the analysis phase, and
3. Evaluation, which is the rigorous assessment of the effectiveness of the display formats
developed in the synthesis phase
Of the three-phase program, the most innovative was the synthesis stage. Prior to the
implementation of Cathode-Ray Tube (CRT) displays, the constraints imposed by the hardwired
dials and gauges limited display synthesis. Advancing CRT and digital display technology
presented new opportunities in the control room to not only display information, but to make that
information more accessible to operators (Murch, 1984). They opened up the possibility of using
colour, active bar charts, and trends to relate meaningful information to operators (Beltracchi,
1988). Furthermore, they allowed for active-mimic displays to be used to show processes
occurring in real-time within the plant.
1.2.1 Digitization of Control Rooms
As display technology continued to develop, nuclear control rooms (NCRs) moved further away
from analog displays and control systems in favour of digital solutions (IAEA, 2009). As
3
discussed in the previous section, the first stage of this transition was the implementation of CRT
displays within the analog panels. The next phase of this transition saw the continued digitization
of many processes in the control room.
Digitized processes include emergency operating procedures, advanced alarm systems, graphic
display systems, and intelligent operator support (Roth & O’Hara, 2002). This shift was
motivated by the belief that reallocating these processes to the computer would lead to improved
control system performance in terms of accuracy, computation, increased capacity for data
handling, and storage (Lin, Yenn, & Yang, 2010). These improvements were meant to take some
of the load off of the operators allowing them to focus on important plant behaviours and thus
created more usable systems.
The transfer of many control room processes from analog to digital necessitated a parallel
transition in display technology. Operators needed to be able to monitor and control these newly
digitized processes, which led to the implementation of software interfaces, touch screen
controls, overview displays, and individual operator consoles (O'Hara, 2004).
1.2.2 Interface Advancement
With the newly ubiquitous software interfaces, designers found new areas of latitude for the
development and growth of display technology. They already knew that simply presenting the
necessary information to operators was insufficient (Joyce & Lapinsky, 1983) and therefore
focused their attention on improving the way that this information was being presented.
Traditional displays marked the first generation of software interfaces in NCRs. These displays
employed a mimic framework, wherein information presentation is based on the physical layout
of the plant (Lau, Jamieson, Skraaning, & Burns, 2008). This was the same framework that was
used on the hardware interfaces wherein dials and gauges would be mapped to the hardware
panel in accordance with the physical layout of the plant. Essentially, traditional displays
digitized the methods used on the hardwired panels without exploring the potential benefits
presented by the new technology.
The next stage in the evolution of display technology was initiated by the realization that
traditional displays were not taking full advantage of the available technology. To move past
this, the traditional mimic displays were supplemented with novel configural graphics such as
4
trend charts and other plots. The combination of these two graphical methods represented the
central characteristic of advanced displays (Kim & Kim, 2014). Advanced displays employ the
original mimic-style interfaces, but add configural graphics and new visualization methods to
illustrate how parameters change over time and in relation to one another (Bennett & Malek,
2000; Lau et al., 2008).
Mimic displays are based on the belief that they will help to develop and maintain operators’
mental models of plant operations (Jamieson & Miller, 2007). These have been found to be
useful for novice operators, who are still developing their mental models (Butcher, 2006; Gary &
Wood, 2011). However, as operators gain more experience and further their understanding of
their respective systems, their need for a mental-model-facilitating display is reduced since their
mental models have already become fully developed (Jenkins, Stanton, Salmon, Walker, &
Rafferty, 2010; Rasmussen, 1993; Zhang, 2008). This therefore compromises the utility of the
mimic aspect of advanced displays, especially in the nuclear domain where plants are run by
exclusively expert operators with fully formed mental models (Lee & Seong, 2009; Vicente,
Roth, & Mumaw, 2001).
This shortcoming of mimic-displays for expert operators gave rise to Ecological Interface
Design (EID), whose foundation is built on grouping functionally related information in cohesive
graphical forms to support the presentation of psychologically relevant information (Vicente &
Rasmussen, 1992). By doing this, ecological interfaces reduce the need for mimic-displays
unless some aspect of that mimicry is tied to the functional purpose of the system (e.g., process
sequences; Bennett & Malek, 2000). Ecological interfaces have demonstrated marked
improvements for both situation awareness and operator performance in simulator studies in the
nuclear domain (e.g., Burns et al., 2008), the hydrocarbon industry (e.g., Tharanathan et al.,
2012), and several others. Furthermore, ecological interfaces have demonstrated performance
improvements during unanticipated events (Lau et al., 2008). In spite of these performance
benefits, EID has yet to be fully integrated into NCRs (Vicente, 2002). In order to shift display
frameworks away from the status quo, research needs to continue evaluating the demonstrated
benefits of EID within full-scale simulator studies.
5
1.3 Group View Displays
Although the need for research evaluating the merits of different display frameworks has been,
and continues to be important, studying the configuration of the displays on which the
frameworks are presented is equally pressing. One of the main areas of this development focuses
on the different methods for presenting information to multiple operators within the control room
(Myers & Jamieson, 2014).
Group View Displays (GVDs) represent a recent trend in control room display technology that
was only made possible through the aforementioned digital evolution. GVDs are a class of
display configurations that enable the same information to be viewed by multiple operators in
different areas in the control room through a common point of reference (O’Hara, Brown, Lewis,
& Persensky, 2002). The motivation behind GVDs revolves around their believed improvements
towards operator situation awareness, communication between operators, and overall
performance (Roth et al., 1998). Although they come in multiple forms, the two most prominent,
are Large-Screen Displays and redundant displays.
1.3.1 Large Screen Displays
Large-Screen Displays (LSDs) are large (~100 X 100” or greater), wall mounted displays that are
typically positioned at the front of control rooms (Figure 1; Myers & Jamieson, 2014). They
achieve the GVD objective by locating the common point of reference at the same position in the
control room. This method therefore allows for the different operators viewing the LSD to be
looking not only at the same information, but also at the same geographic location within the
control room.
6
Figure 1. Large screen displays used in the Qinshan control room.
A key advantage of LSDs is that they are visible from almost anywhere in a control room (Roth
et al., 1998). Therefore, if an operator needs to check something at the panel, they don’t lose
sight of the GVD. However, the increased size of LSDs also introduces necessary tradeoffs.
LSDs require operators to be positioned further away, which increases the overall footprint of the
control room and therefore also the cost (Myers & Jamieson, 2015).
Although LSDs represent only one form of GVD, they have become the dominant technology
implemented in NCRs to achieve the GVD purpose (Myers & Jamieson, 2014). While the
posited benefits of LSDs are intuitive, there are viable alternatives such as redundant displays
that may be able to yield the same, or better results in terms of situation awareness,
communication, and operator performance.
1.3.2 Redundant Displays
Redundant displays are the primary GVD competition to LSDs in NCRs (Myers & Jamieson,
2014). Redundant displays are individual multi-display arrays that are located in front of each
operator’s work station. The arrays typically have a predetermined subset of screens dedicated to
the GVD and the content of these screens is identical for each operator. Therefore, the operators
still have common points of reference with which to look at matching information, but those
common points of reference are replicated in different spatial locations in the NCR. For example,
if an array consisted of four displays configured in a 2 X 2 arrangement, the top two displays
may fulfill the GVD function of the array, meaning that they would be the same for each
7
operator. This means that for both operators, their top two panels will always be identical,
thereby enabling the individual operators to always have a common point of reference with
which they can discuss information about the plant. Essentially, redundant displays relocate the
LSDs from the front wall of the NCR to the individual operator work stations. Figure 2 illustrates
this concept.
Figure 2. Redundant display configuration (Kiran Infosystems, 2014).
Redundant displays allow for smaller control rooms relative to LSDs since the operators do not
need to be located as far away from their screens. However, redundant displays are typically not
visible from all locations in the control room. This means that an operator may lose sight of the
GVD if he or she needs to check something at one of the panels. A summary of some of the
benefits and pitfalls of both GVD options is presented in Table 1.
Table 1. Relative advantages and disadvantages of GVD alternatives
GVD Type Advantages Disadvantages Large Screen Display Visible from any location in
the control room Increases the footprint of the control room
Redundant Display Allows for a smaller control room since
Not visible from any point in the control room
8
1.4 Selected Literature on GVDs
In two reports, Roth et al. (1997, 1998) assessed the benefits of LSDs under different display
frameworks. They found that the SA, workload, and performance benefits resulting from the
addition of functional information outweighed the costs of sacrificing extra screen space.
Although these results shed light on the possible benefits of functional information when
presented on wall-mounted displays, there is still a great deal to be learned with respect to
display frameworks and GVD alternatives. For example, these studies did not assess the
behaviour differences evoked from GVD alternatives, as they only studied LSDs. Furthermore,
they did not examine the interaction between display framework and GVD type to determine if
certain pairings between GVD and framework are superior to others. Finally, the statistical
evidence provided to support their claims regarding LSDs and the presentation of functional
information is scarce. For example, no F or t values are presented, nor are the details of their
functional displays. In spite of these limitations, their evidence converges with the body of
literature supporting functional displays (e.g., Burns et al., 2008) and provides a foundation for
new questions to be asked.
1.5 Research Questions
The literature described above describes the various options that are available to monitor a
nuclear power plant. This highlights two key areas of concern with respect to the current state of
the art in nuclear control rooms. The first is that the favouring of LSDs over redundant displays
appears to have no empirical backing (Myers & Jamieson, 2014). The second is that in spite of
the widespread support for ecological displays, advanced displays still represent the dominant
display framework used in nuclear control rooms (Lau et al., 2008).
This project aimed to address these two gaps in the literature with the underlying motivation of
assessing whether or not there is any empirical evidence supporting the current state of the art in
nuclear control rooms. To achieve this goal, we had two research questions:
1. Is the widespread favouring of LSDs over redundant displays rooted in improvements in
SA, communication, and overall performance?
9
2. Is the widespread implementation of advanced displays over other display frameworks
rooted in improvements in SA, communication, and overall performance?
We therefore set out to conduct a full-scale simulator study that would answer both of these
questions. The experiment included both LSDs and redundant displays, and both advanced and
ecological display frameworks. This 2 X 2 study included the necessary conditions for
addressing the gaps in the literature and allowed us to assess the empirical validity of the current
state of the art in nuclear control rooms.
1.6 Document Structure
This document outlines the analysis and design leading up to a summative experiment. It then
details the experimental design, training, method, and results of that experiment. Because a
comprehensive literature review (Myers & Jamieson, 2014) had already given rise to the research
questions, a detailed literature review is not included. This document therefore consists of five
main sections: analysis, design, experimentation, results, and discussion.
Chapter 2 provides details on the project’s scope. It summarizes some of the methods and
constraints that were involved in answering the two experimental questions mentioned above. It
also provides a brief overview of the domain of our analysis and experiment, the CANDU 6®
nuclear power plant.
Chapter 3 describes the Cognitive Work Analysis (CWA) that was conducted on the system. s
Although the individual phases of CWA are conducted for distinct purposes, each contributes to
attaining a more comprehensive understanding of the system. Our CWA analyzed the entirety of
a system and therefore proved to be a useful method for establishing the requirements necessary
for achieving our experimental goals within that system.
Chapter 4 details the design process and outcomes for the novel ecological interfaces. As
described above, we needed to create novel ecological interfaces against which to compare the
existing advanced displays. This stage was undertaken in service of the second experimental
question rather than to advance ecological interface design. We therefore relied as heavily as
possible on existing ecological forms as well as the current graphical suite in the CANDU
developer software.
10
Chapter 5 describes the experiment that was aimed at answering the two main research
questions. This includes descriptions of the experimental design, participant training, display
implementation, scenario design and implementation, and conducting the experiment.
Chapter 6 details the results of the study. This section encompasses the statistical methodology
and findings from the measures that were taken during experimentation.
The final portion of this document discusses the findings, limitations, and implications of this
study. Theories are presented that attempt to explain these findings with respect to the relevant
limitations.
Project Scope 2This research aimed to answer the two research questions through a full-scope simulator
experiment. We ran operators through four different display setups, each representing a unique
combination of display configuration and display type1. The four setups are summarized in Table
2 below.
Table 2. Summary of Research Questions
Question 1
Display Configuration
LSD Redundant
Question 2 Display Type
Advanced LSD-ADV RED-ADV ADV Ecological LSD-ECO RED-ECO ECO
LSD RED
The first research question asked what display configuration yields higher levels of operator
situation awareness, communication, and performance. To evaluate this we compared these three
dimensions across LSD and the redundant display configurations. Unfortunately, due to time and
resource limitations, we did not have a baseline condition wherein no GVD was present.
Including a baseline without any shared information screens would have contributed towards
1 The terms presented here to represent the different conditions will be used throughout the document. Display
Configuration refers to the LSD vs. redundant dimension. Display Type refers to the advanced vs. ecological dimension. Display Setup refers to the unique coupling of display configurations and display types (e.g., LSD-ADV, LSD-ECO, RED-ADV, & RED-ECO).
11
evaluating the GVD concept as a whole rather than evaluating the benefits of different
alternatives. Although this was primarily due to time constrictions, this was also motivated by
the fact that it has become an expectation that NCRs will satisfy the GVD requirement (O’Hara
et al., 2002). GVDs have thus become the standard not only in nuclear, but in many other process
control domains as well (Myers & Jamieson, 2014). Therefore, although this baseline condition
would have been helpful for either affirming or refuting the GVD concept, it would not represent
a state in the control room that exists in today’s NCR climate.
The second research question, asking what display type best elicits the postulated benefits of
GVDs, needed to be answered through a comparison of two distinct display frameworks. The
baseline condition for this question was the advanced displays that currently represent the state
of the art in the NCR (Lau et al., 2008). The comparison chosen to compare these against was
ecological displays. These were chosen because they fundamentally differ in the way that they
present information by focusing on depicting functional relationships between parameters
wherever possible. Therefore, ecological displays represent a framework that is sufficiently
different from advanced displays by simply altering the form rather than the content of the
information presented. This is particularly important for a valid comparison study because
maintaining the informational content across conditions allows for any performance differences
to be solely attributed to the different frameworks rather than different access to information
(Christoffersen, Hunter, & Vicente, 1996; Maddox, 1996).
2.1 The CANDU 6® Nuclear Power Plant
We conducted our analysis on a CANDU 6® advanced nuclear reactor. The reactor uses heavy
water (D2O) coolant, which allows for natural uranium to be used for fuel as opposed to
enhanced uranium. This enables online refueling, which reduces the frequency with which the
system has to be taken offline. A simplified block diagram of the system and its constituent
subsystems is presented in Figure 3.
12
Figure 3. Simplified diagram of the CANDU 6® main systems.
Cognitive Work Analysis 3Cognitive Work Analysis (CWA) is a framework used to systematically analyze complex
sociotechnical systems like the CANDU 6® nuclear power plant (Rasmussen, Pejtersen, &
Goodstein, 1994; Vicente, 1999). CWA analyzes the different classes of constraints that shape
13
work within a particular domain (Elix & Naikar, 2008). These classes of constraints represent the
key factors involved in governing the operations of a particular system.
Our CWA was conducted for three reasons. The first purpose was to understand and analyze the
system in order to develop tools and products that can improve the monitoring and operations of
those systems (Elix & Naikar, 2008). The second reason was to provide a structured approach for
the development of our own understanding of the system. Since both myself and my research
partner were new to the nuclear domain, a framework for studying the system that followed a
logical and proven structure was necessary (Jamieson & Miller, 2007). The third reason, which is
tied to the second, was to determine a suitable scope for the rest of the project. By fostering a
comprehensive understanding of the system, we were able to effectively use CWA to determine
what would be required in the design and experimentation phases in order to answer the research
questions.
3.1 Stages of Analysis
Although CWA has a canonical structure that consists of five phases, it also allows for
customization according to the specific demands of the analysis being conducted (Elix & Naikar,
2008; Vicente, 1999). This allows researchers to tailor the framework to fit the specific needs of
the project.
This study employed the first three phases of CWA: Work Domain Analysis (WDA), Control
Task Analysis (ConTA), and Strategies Analysis (StrA). These phases progress by initially
analyzing the physical and functional constraints of the system in WDA, then the control tasks
that operators use when operating according to these constraints in ConTA, and finally by
decomposing the different strategies that can be employed while performing these tasks in StrA
(Vicente, 1999). The final two phases, Social Organization and Cooperation Analysis, and
Workers Competencies Analysis were not formally conducted.
There were two clear task classes within our experimental scope: 1) Plant automation in response
to changing scenario characteristics, and 2) Operator observation and diagnosis. Since no control
actions were being performed, it was not necessary to analyze the functional allocation of these
tasks. This rendered Social Organization and Cooperation Analysis unnecessary for this
particular project. Workers Competencies Analysis was not omitted on the basis of it being
14
unnecessary, but rather it was integrated in all other phases of the CWA. We knew that the
eventual experimental participants would be students who were unfamiliar with the nuclear
domain. We therefore used the first three phases of CWA to determine exactly what areas of the
system to focus on in design and experimentation, based on the anticipated competencies of our
eventual participants. This method was also used during the experimental and display design
phases of the study, thereby encompassing an informal version of Workers Competencies
Analysis.
3.2 Work Domain Analysis
The first stage of CWA is Work Domain Analysis (WDA). WDA identifies purposive,
functional, and physical constraints of a particular system (Vicente, 1999). These constraints are
identified through the use of abstraction hierarchies.
3.2.1 Analytical Tool – The Abstraction Hierarchy
Abstraction hierarchies (AHs) trace the high level functional purposes of a system down to its
physical components through five levels of abstraction (Rasmussen, 1985). The elements in the
AH are connected via means-ends relationships, with elements at lower levels being the means to
achieving the ends that they are connected to at higher levels. By doing this, the AH illustrates
the role that each component serves within the higher-level system purposes. Descriptions of
each level as well as examples within the CANDU® plant are presented in Table 3.
Table 3. Five abstraction levels of the AH (adapted from Rasmussen, 1985)
Level Definition Example Abstraction Functional purpose
Objectives of the system Generate electricity How an element
is achieved
ê é
What an element achieves
Abstract function
Underlying laws and principles governing the system
H2O enthalpy
Generalized function
Standard functions and processes Heat transportation
Physical function
Functional descriptions of individual components
Heat transport system
Physical form The physical characteristics of individual components
Figure 8 configuration
15
The physical form level of the AH describes the physical characteristics of the individual
components in terms of appearance and configuration within the plant. The main concern with
respect to nuclear operation at the physical form level is the configuration of the systems within
the plant, especially for the novice operators who are still developing their mental model of the
operations (Butcher, 2006; Gary & Wood, 2011). Therefore, rather than including typical
physical form information in the AHs, we used a physical form schematic of the plant (see
Figure 3) to represent the bottom layer of each of the systems that were analyzed.
We used a somewhat unique approach to the abstract function level of our analysis by adopting
the Source, Store, Sink nomenclature. This is adopted from Reising and Sanderson (2002) and
describes the mass and energy balances of a system in terms of where the energy or mass came
from (the source), where mass or energy is contained in a system (the store), and how that mass
or energy is removed from a system (the sink). This nomenclature is well-suited for closed
systems like the CANDU 6® power plant since virtually all mass and energy in and out of the
system can be accounted for.
3.2.2 Scope of Analysis
The purpose of the CWA and its constituent stages was to answer the two research questions.
Therefore, the scope of the WDA needed to include the systems that would be necessary for
achieving this goal. This afforded the freedom of removing certain systems from the analysis and
thus also from the final representation of the plant.
A system was deemed eligible for removal if its removal did not impact the understanding of the
rest of the system. This meant that all systems in the primary and secondary loops of the system
had to be included in the plant representation (see Figure 3). This is because understanding each
of these systems, particularly at the abstract function level, requires an understanding of the input
and output systems that are connected to it.
We therefore focused our analytic efforts on the systems that nuclear operators tend to pay the
most attention to. These are outlined in Davey, (2000) as:
1. The nuclear reactor
2. The heat transport system (HTS)
3. The pressure and inventory control (P&IC)
16
4. The four steam generators (SG)
5. The main steam system (MSS)
6. The turbine
7. The feedwater system
The main systems that we eliminated from analysis were the moderator system, the refueling
system, and most importantly, the power grid. Again, the accurate understanding of how these
systems operated was unnecessary for understanding how the rest of the systems work together
to achieve the overarching goals of the plant.
Once we had identified the seven key systems in the plant, we determined the level of specificity
at which these systems would be represented. The most influential constraint when considering
this was the skills and competencies of our eventual experimental operators, which illustrates the
integration of Workers Competencies Analysis in the WDA. Due to limitations with respect to
access to either current or former nuclear operators, students from Lambton College’s Chemical
Production and Power Engineering courses represented the eventual participants. These students
had experience with other process control systems, but had no experience in the nuclear domain.
Because of this, and because we would have a limited window in which to train the operators,
the system did not need to be represented at its most detailed level.
We therefore reduced the plant to a level of specificity whereby all of the core functionalities of
each of the seven constituent systems was captured. For example, if a valve was integral to the
operation of a system, it was included in the analysis. However, if there were multiple
redundancies in the design of these valves, some were removed from our representation. The
diagram presented in Figure 3 illustrates the level at which the plant was represented in the
WDA.
It is important to note that as we approached the actual experiment, the PFn diagram was
significantly simplified to meet the specific constraints of our participants and limited training
window. Therefore, the WDA and resulting AHs presented in this section are more detailed than
what the participants ended up being trained on.
Although this analysis can be considered excessive, it contributed to the successful completion of
the project in two key ways. Firstly, initial over-analysis affords the freedom to selectively
simplify in subsequent stages. Secondly, the initial over-analysis bolstered our own
17
understanding of the dynamics of the system, which allowed us to assess what aspects of the
plant were necessary in the training regimen.
The following sections present the AHs for the individual systems that I analyzed. The
functionality of each system is briefly outlined in sufficient detail to contextualize later
descriptions of the experiment. Rather than presenting the lowest level of the AH, the physical
form level, diagrams are shown to illustrate the general structure of the systems.
3.2.3 Products of Analysis
The following sections consist of the final products from the WDA that I conducted for several
of the main subsystems of the CANDU 6® plant. Figure 4 presents the overall abstraction
hierarchy for the CANDU 6® plant. The systems highlighted in green represent the systems that I
analyzed while those with grey coloring show the systems that my partners analyzed. Because a
description of the work conducted on these systems is not necessary for understanding the
remainder of the paper, I have not included their AHs in this section.
Figure 4. Overall AH for the CANDU 6 nuclear power plant. Green boxes indicate the
systems included in this report.
18
3.2.3.1 Heat Transport System
The Heat Transport System (HTS) is responsible for circulating heavy-water coolant (D2O)
through the reactor and taking it to the steam generators where it transfers its heat to H2O. Once
it has transferred its heat to the H2O it is circulated back through the reactor where it regains its
heat. This process is continuously repeated while the plant is running. Figure 5 presents the AH
for the HTS.
Figure 5. AH for HTS.
3.2.3.2 Pressure & Inventory Control
The Pressure and Inventory Control (P&IC) system maintains the heavy water system pressure
at a setpoint throughout a variety of operating modes. The P&IC is principally controlled through
the pressurizer, which is a large tank connected to the primary circuit of the HTS with a large
pipe whose direction of flow is determined by pressure differential between the HTS and the
pressurizer. If the pressure in the HTS drops below setpoint, heaters in the pressurizer activate,
19
increasing the heat and therefore pressure in the pressurizer. This forces inventory into the HTS
thereby increasing the pressure in the HTS. If the HTS is above its setpoint, vents above the
pressurizer open, releasing pressure to a secondary tank. This causes inventory to flow from the
HTS to the pressurizer thereby reducing the pressure in the HTS. Figure 6 presents the AH for
the P&IC system.
Figure 6. AH for the P&IC.
3.2.3.3 Steam Generator
There are four steam generators (SGs) in the CANDU 6® plant. These serve as the link between
the primary and the secondary sides of the system by acting as a heat transfer. Hot D2O flows
into the primary side of the SGs where it goes through an inverted U-tube bundle. Liquid H2O
surrounds the tube-bundle, receiving the heat energy and vapourizing. The H2O steam then
leaves the SGs and flows towards the turbine. Figure 7 presents the AH for the SG.
20
Figu
re 7
. AH
for
the
SG.
21
3.2.3.4 Main Steam System
The Main Steam System (MSS) is responsible for transferring dry steam leaving the SGs to the
turbine. It has a series of valve-sets called the atmospheric discharge valves (ASDVs) and the
main steam safety valves (MSSVs), which vent steam to the atmosphere in conditions where the
pressure in the steam lines or the turbine is higher than the setpoint. A third type of valve, the
condensate steam discharge valve (CSDV), reroutes steam to the condenser, thereby bypassing
the turbine. This valve is used when the turbine is producing excess energy or in shutdown
scenarios. The electricity being generated by the turbine will decrease if any of these valves are
opened. Figure 8 presents the AH for the MSS.
Figure 8. AH for the MSS.
3.2.3.5 Turbine System
The turbine receives hot, dry steam from the MSS and converts it to rotational energy. The
turbine system in the CANDU 6® plant has one high-pressure (HP) turbine and two low-pressure
(LP) turbines. The HP turbine is the first in the sequence and takes a large portion of the energy
out of the steam. Steam leaves the HP turbine and is directed to the first LP turbine. Before it
22
reaches the LP turbine it goes through a reheater, which is powered by extraction steam taken
directly out of the steam generators. The steam passes through the first LP turbine and the
process is repeated before it reaches the second LP turbine. Upon leaving the second LP turbine
the steam has lost the vast majority of its energy. It is then fed into the feedwater system where it
is condensed and recycled back into the steam generators. Figure 9 presents the AH for the
turbine system.
Figure 9. AH for the turbine system.
3.2.4 Summary of Work Domain Analysis
Work Domain Analysis decomposed the CANDU 6® plant, illustrating how each subsystem and
their constituent components contribute to the plant’s functional purpose of consistent and
reliable electricity generation. Not only did it provide the basis for developing a comprehensive
set of information requirements (see Appendix A), but it also acted as a filter, revealing
important insights with respect to what systems were necessary for inclusion, and what systems
could be removed. This filtering limited the potential for unnecessary design and training.
23
3.3 Control Task Analysis
The aim of Control Task Analysis (ConTA) is to identify the recurring tasks that operators need
to perform within a work domain (Vicente, 1999). These tasks are examined independently of
who they are performed by and are analyzed according to the sequence of data-processing
activities and their resulting states of knowledge. Therefore, ConTA identifies and describes the
cognitive activities that occur while an operator performs a task.
3.3.1 Analytical Tool – The Decision Ladder
Rasmussen (1974) developed ConTA to trace the cognitive activities that transform an initial call
to action, to a comparison between the current system state against the targeted system state, to
the system manipulation that will achieve that target. The tool that he developed to do this was
the Decision Ladder (DL). DLs are structured flow diagrams that alternate between cognitive
activities and their resulting states of knowledge. The typical structure of a DL is presented in
Figure 10.
24
Figure 10. Typical decision ladder structure (Jenkins et al., 2010).
3.3.2 Scope of Analysis
ConTA identifies the recurrent control tasks that are performed within an operating environment
(Kilgore, St-Cyr, & Jamieson, 2008). Unfortunately, because novice operators would be used
during experimentation, control actions were omitted from the experimental design. This was
largely based on subject matter expert recommendation. Including control actions would have
complicated the scenarios too much to elicit accurate performance data for our novice operators
25
while also increasing the complexity of the training regimen. Therefore, a new application of
ConTA, within this study’s operating environment, needed to be found.
There were two classes of tasks within the scope of the present study. The first class includes
search behaviour while the second includes diagnostic processes. Since control actions fell
beyond the present study’s scope, the DLs essentially stop after the comparison between the
current state and the target state thereby removing the bulk of the second leg of the DL. This
stretch of the second leg of the DL, which covers definition of task, task, planning of procedure,
procedure, and execution is most helpful in domains with loosely defined tasks (Jenkins et al.,
2010). In the nuclear domain, where every task is regulated and every procedure is carefully
defined, this portion of the DL provides less value. Therefore, omission of this leg was deemed
suitable given the characteristics of the present study.
Within the high level class of search tasks, there were two categories: 1) Active search without
an alarm, and 2) active search with an alarm. Examining these two tasks independently revealed
the differences and similarities between them. This stage was informative for the experimental
design in that it revealed the underlying behaviours involved in plant operations which could
therefore be examined.
Diagnostic behaviour, the second type of action our participants would need to perform, was
omitted from this phase of the CWA. The decision ladder did not appear to be a suitable tool for
modeling this task, and therefore we focused more heavily on applying strategies analysis to
characterize diagnosis.
3.3.3 Products of Analysis
3.3.3.1 Active Search for Deviation (without alarm)
Searching for a system fault without the instigation and guidance of an alarm requires operators
to scan their displays to determine if any parameter or group of parameters had significantly
deviated from setpoint. Although there are typically advanced alarm systems integrated into the
CANDU instrumentation and control systems, there are still instances where the alarms either
fail to indicate the specific system where there is a deviation, or fail to indicate the breadth of the
problem. One of the reasons that this can occur is because alarms are presented in a list format,
which requires the operator to perceive individual alarms piece them together to determine the
26
problem. Therefore, regardless of the presence of an alarm system, a critical task within nuclear
monitoring involves searching for symptoms within the full array of parameters. Figure 11
illustrates this process in a decision ladder.
Figure 11. DL for active search for fault without an alarm.
As the above figure illustrates, many stages of the second leg of the decision ladder are bypassed
since there is no new procedure development during operation at a power plant. Instead,
operators would detect something and immediately jump to the appropriate procedure to respond
to that issue.
27
3.3.3.2 Active Search for Deviation (with alarm)
We analyzed this task for the purpose of contrasting the task of searching for a fault without the
aid of alarms. The analysis revealed that the only major differences between the two tasks are at
the activation and alert stages. In this task, the operators are called to action via the aid of the
alarm system rather than simply noticing a significant deviation in a parameter manually. After
this, the two tasks converge since the alarm system provides minimal guidance with respect to
how best to respond to a situation. The decision ladder for active alarm search is presented in
Figure 12.
28
Figure 12. DL representing an active search for a deviation with the guidance of an alarm.
3.3.4 Summary of ConTA
Understanding control tasks represents a key step towards understanding the nature of operations
within a complex sociotechnical system. Our ConTA helped us to identify the two main classes
of control tasks within our experimental scope: Search and diagnosis. The ConTA highlighted
the areas of operation that require the most support from an interface. This therefore informed
the display and the experimental design process by showing not only what areas to focus on
designing for, but by also revealing what areas are the most important to evaluate in an
experiment.
3.4 Strategies Analysis
The third and final stage of our CWA was Strategies Analysis (StrA). StrA illustrates the
different strategies that can be used to accomplish a control task (Vicente, 1999). Strategies, as
defined by Rasmussen (1981), are a series of actions that transform an initial state of knowledge
into a final state. Because there are a multitude of strategies for any given task, and each strategy
can be influenced by many factors, it is difficult to identify highly specific strategies. Instead,
StrA aims to identify generic strategy classes for the purpose of facilitating the design of displays
capable of supporting these strategies (Kilgore et al., 2008).
3.4.1 Analytical Tool – Information Flow Map
Information Flow Maps (IFMs) are graphical representations of the different strategies that an
operator can take to achieve a certain task (Vicente, 1999). IFMs have been presented in many
different ways in the past. Some studies have presented each distinct strategy as an individual
IFM (e.g., Kilgore et al., 2008). Others have shown all strategies on a single IFM to illustrate
where the individual strategies diverge from one another (e.g., Cornelissen, Salmon, Jenkins, &
Lenne, 2013). While there are merits to each method, unified IFMs are more capable of
illustrating the points where strategies can diverge from one another.
We adopted the “question notation” from the Jenkins et al. (2010) work on decision ladders. This
notation describes how a situation is assessed according to the presence or absence of a set of
variables. For example, a question may be “Is an alarm currently active?” From this point the
29
different strategies would diverge according to the presence or absence of an alarm. This
notation therefore illustrates not only the different strategies, but also the factors that influence
the selection of those strategies.
3.4.2 Scope of Analysis
We performed StrA on both of the search strategies that were analyzed in ConTA. As stated
earlier, we also used StrA for online diagnosis of operational faults. Due to the ambiguous nature
of the tasks being described, the strategies are highly interwoven and operators can often switch
between strategies during a task. For example, in a search task, the strategies can diverge
according to the level of specificity at which an operator views the system. However, during
holistic perception of a system, an operator may focus in on one area, thereby switching to a
more localized strategy. These relationships are depicted in unified IFMs.
3.4.3 Products of Analysis
3.4.3.1 Search for Fault (without alarm)
The strategies identified in the IFM for searching for a fault with the assistance of an alarm are
largely informed by the strategies identified in Mumaw, Roth, Vicente, & Burns (2000). The
strategies identified in this paper were derived from field observations in nuclear power plants.
They therefore represent strategies that expert operators would use and are categorized in three
broad classes of strategy, each with several sub-strategies:
1. Strategies used to maximize information extraction
2. Strategies used to create information
3. Strategies used to offload cognitive demands
Of the three classes identified in Mumaw et al. (2000), the most pertinent to this study are those
that maximize information extraction. These strategies were built into a unified IFM presented in
Figure 13.
30
Figure 13. IFM depicting the strategies used within the task of searching for a fault without
an alarm. Note that the legend listed here applies to all IFMs.
3.4.3.2 Search for Fault (with alarm)
The main difference caused by the presence of an alarm is that it provides preliminary guidance
towards the source. However, it does not necessarily identify the underlying cause of the alarm,
nor does it detail the full scale of the problem. Therefore, the strategies illustrated in the IFM
presented in Figure 14 details how an operator would use the aid of an alarm to answer these
questions.
31
Figu
re 1
4. IF
M d
epic
ting
the
diff
eren
t str
ateg
ies w
ithin
the
task
of s
earc
hing
for
a fa
ult w
ith a
n al
arm
.
32
3.4.3.3 Online Diagnosis of Fault
This strategy depicts the process of forming and testing hypotheses about the potential cause of a
fault. It is assumed that the input to this strategy is the successful assessment of the full scope of
the manifestation of the fault. As Figure 15 illustrates, there are two main strategies used in this
task and they differ in accordance with the level of specificity at which the operator observes the
system.
Figure 15. IFM depicting the different strategies within the task of online diagnosis of a
fault.
33
A critical aspect of this IFM is representation of the interconnectedness of the strategies within
the search box. In this task, operators would likely be switching rapidly between holistic and
localized processing. This is depicted by essentially resetting the search process after the
operator asks themselves the question of whether or not they have sufficient evidence to
formulate a preliminary hypothesis of the potential cause for the event.
Once they have that hypothesis, they need to weigh costs and benefits of the urgency with which
they respond. If they perceive a fault to be both time sensitive and safety critical upon initial
hypothesis formation, then they would be very likely to resort to an emergency operating
procedure rather than waiting to confirm their hypothesis. This actually represents a secondary
area where there are multiple strategies in spite of the fact that only one is shown. This is
because this safety-first strategy is the only strategy that exists in the nuclear domain (Meshkati,
1998).
3.4.4 Summary of Strategies Analysis
Although StrA is often underemphasized in CWA, it served a critical role in our project. The
graphical representations of the various tasks provided an illustration of the structure of nuclear
operations during loosely defined tasks. We were able to use these graphical representations to
build interfaces capable of supporting the strategies that they depict.
Our StrA also helped with the experimental design. StrA highlighted many of the behaviours that
are most important to nuclear operation, such as checking on referent systems, confirming initial
hypotheses, and many more. These findings were built into our experiment to ensure that the
tests would be evaluating key areas of operations.
3.5 Summary of Cognitive Work Analysis
Conducting the CWA on the CANDU 6® nuclear power plant represented a critical step towards
our goal of evaluating the efficacy of group view displays. CWA fostered our own understanding
of the system and operations, which directly led to the development of our experiment and
training materials for our participants. Furthermore, our CWA helped to develop the content and
architecture of the novel ecological interfaces against which we would be evaluating the existing
advanced design interfaces.
34
Display Development 4The existing interfaces in the CANDU 6® plants employ an advanced display methodology,
which combines mimic-elements with graphical trends and relatively simple configural graphics
(Lau, Jamieson, et al., 2008; Myers & Jamieson, 2014). To answer the experimental question that
asked what display type yields higher levels of the postulated benefits of GVDs, comparison
displays needed to be developed.
Ecological displays were selected as the comparison condition because they differ on a
philosophical level from advanced displays. They focus on depicting the functional relationships
between parameters (Vicente & Rasmussen, 1992). Physical relationships can still be depicted in
instances where a mimic diagram reflects a functional purpose of a system, as is the case in many
chemical processes where a specific sequence of processes is necessary to achieve the functional
purpose of a system (e.g., Paulsen display from Jamieson & Vicente, 2001).
As NCR display technology continues to develop, the displays focus more and more on depicting
information in a way that is psychologically relevant and accessible to operators. Traditional
displays focused entirely on mimicking the physical layout and hardware interfaces of the plant.
Advanced displays improved upon this by adding some functional information to make the
behaviour of parameters in the plant more accessible to operators. Ecological displays further
this abstraction by focusing on depicting the functional relationships between parameters in a
way that allows for the behaviour governing a system’s dynamics to be understood (Vicente &
Rasmussen, 1992).
4.1 Display Comparison
The comparison that we were making regarded the way that information is presented on an
interface. Issues such as informational content, information salience, and control methodology
were beyond the scope of this study and therefore needed to be controlled for. To do this, several
assurances were taken into account regarding the similarities and differences between the two
display type conditions.
35
4.1.1 Informational Content
Our principal concern in developing the comparison displays was maintaining the informational
content across conditions. Maddox (1996) challenged the purported benefits of ecological
displays by attributing the differences in the Christoffersen et al. (1996) results to the operators
simply having access to more information. In order to head off such criticisms, we needed to
ensure that information content was identical between the existing advanced displays and the
novel ecological displays.
4.1.2 Display Navigation
A second assurance had to be taken into account with respect to the way that operators navigate
through the displays. Again, we did not want potential differences between the conditions to be
attributed to anything other than the different display frameworks. We therefore implemented a
universal menu bar, which had tabs for each of the pages of the interface and made sure that the
controls to navigate between these tabs were identical between the two display conditions.
4.1.3 Display Fidelity
A critical aspect for comparing any novel design to an existing design is the degree of fidelity of
the two designs. Fidelity can come in many forms in display design, but the main areas of
concern were graphics, refresh rate, and accuracy. If, for example, the graphics on the novel
displays took longer to refresh than on the existing displays, there could have been far reaching
usability issues.
4.1.4 Display Appearance
The final assurance that had to be made came with respect to the appearance of the displays,
namely, in the salience of the colour indicators. The existing advanced displays use a light grey
background with black plots and predominantly green, white, or orange trend lines. Again, if we
were to use a completely different colour scheme, there may have been unforeseen consequences
with respect to situation awareness, diagnostic performance, or communication. To prevent any
potential confounds, we maintained the colour scheme for the novel displays.
36
4.2 Development of Novel Displays
The display development stage of the project was only done in service of assessing the
differences in task performance, situation awareness, and communication between ecological and
advanced displays. In other words, we did not develop these displays to advance the field of
ecological interface design. Therefore, we used existing ecological forms and principles
wherever possible rather than creating our own.
Unfortunately, limitations in the system used to build the display rendered certain forms
unfeasible. The principal limitations in the software were:
1. Unable to anchor lines to parameter values,
2. Unable to overlay anything on top of charts,
3. Unable to use dynamic angled lines, and
4. Complicated and time consuming coding process
Many ecological forms rely heavily on anchoring connections between parameter values (see
Figure 16). This meant we were unable to use connected bar charts, polar stars, and several other
ecological forms. Furthermore, the software prevented the use of dynamic angled lines, meaning
that certain forms such as the one presented in Figure 17 were unusable as well.
Figure 16. a) Connected bar chart from Lau et al. (2008); b) Polar star from Jamieson &
Miller (2007).
37
Figure 17. Chart depicting relationship between mass, energy, and saturation. The angled
line pivots in response to changes in mass. The three states depicted here are shown in the
legend in the bottom corner.
4.2.1 Existing form usage
While it was evident that certain forms were simply impossible to implement given the
restrictions imposed by the developer software, there were also issues pertaining to forms that
were technically possible. These issues stemmed from the unique programming language used
for this software, resulting in time constraints with respect to creating any novel forms. We
therefore attempted to reconfigure the existing forms within the software’s graphical suite
according to ecological principles.
We relied heavily on multivariate display forms, which compare the values of functionally
related variables to assess for any meaningful differences (Burns & Hajdukiewicz, 2004). These
forms were useful because they can emerge from simply rearranging existing bar charts on the
advanced displays. The existing advanced displays often presented values according to their
physical layout even for variables that had to be equal to one another. This meant that their
values were often presented in isolated locations on the screen. Therefore, in transforming these
38
into ecological displays, we simply rearranged the values to place them beside one another to
allow for easy comparison (see Figure 18).
Figure 18. Rearrangement of existing charts.
4.2.2 Development of Novel Forms
As described above, the purpose of this stage of the project was not to advance ecological
interface design. We therefore focused on utilizing existing tools wherever possible, both from
the collection of forms in CANDU’s development software and from the current ecological suite.
However, the aforementioned software limitations and perceived limitations with existing
ecological graphics necessitated the development of a novel graphical form.
As stated in Section 4.1, we were only interested in comparing the impact of displaying identical
information in different ways. Because the configuration contrasts were the only differences of
interest, we needed to ensure that the ecological displays depicted as many useful functional
relationships as possible. This means that wherever possible, the information on the displays
should reflect the underlying functional properties of the system.
Current advanced displays tend to focus on mass and energy balances in the system. They
therefore predominantly depict the abstract function level of the abstraction hierarchy. In order to
develop comparison displays, the ecological forms needed to also focus on these balances. The
ecological form that illustrates the abstract function level is presented in Figure 19.
39
Figure 19. a) Mass balance plot with more outflow than inflow. b) Mass balance plot with
more inflow than outflow.
There are three main problems with this plot. The first problem is that we were unable to anchor
the comparison line between inflow and outflow values. This isn’t necessarily a problem with the
form, but it was a problem for our project. This inability to anchor lines is particularly
problematic for this form since the values that need comparing are located relatively far apart
thereby rendering the connecting line essential for determining slight imbalances.
The second problem is that the point where the line connecting current inflow and current
outflow intersects with the line indicating the tank’s level is meaningless. This fails to take full
advantage of the relationship between these highly related values. Furthermore, if an operator is
accustomed to the relationships depicted in an interface being meaningful, this can serve as a
distraction.
The third problem is the lack of time-based information to indicate how the balance of these
three parameters lead to different tank states. For example, in Figure 19b the inflow is greater
than the outflow, which means that the volume in the tank should be increasing. If there was a
leak, the volume may stay the same because the extra inflow may be exiting the tank somewhere
before the sensor that measures the outflow. Without some trend information it would be very
difficult for an operator to assess this. A more problematic issue could arise if the inflow and the
outflow were balanced since the operator would likely take this to mean that the tank is stable.
40
Similarly to the previous example if there was a leak in the tank, the volume would gradually
decrease over time in spite of the balanced inflow and outflow.
To graphically address the first problem we used delta flow instead of presenting inflow and
outflow separately. Figure 20 shows the Delta Plot, which presents the current level along the Y-
Axis, and delta flow along the X-Axis. The vertical line at the 0 point along the X-Axis
represents the point where the current inflow and current outflow are equal. Therefore, if the
indicator is along this line, level in the tank should not be changing. The horizontal line in the
plot represents the current setpoint of the system, which can be set anywhere along the vertical
axis as determined by the system’s requirements.
Figure 20. Delta plot showing current inflow balancing with current outflow and the level
at the setpoint.
By intersecting these two lines, four quadrants are formed. These four quadrants represent the
four principal states of any closed system with respect to mass or energy balance. Quadrants Q2
and Q4 indicate that the level in the tank is moving towards the setpoint while Quadrants Q1 and
Q3 indicate that the level in the tank is moving away from the setpoint (see Figure 21).
Therefore, by looking at the location of the single point, an operator is able to determine if the
system is in an acceptable state or not.
41
Figure 21. Four quadrants of the delta plot
Although this form addresses the problem in the mass balance display presented in Figure 19 of
relating current level to inflow and outflow, it does not reveal how the individual parameters
making up this aggregated information relate to one another over time. This trend information
was therefore added as a secondary component of the graph, which allows the operator to see the
inflow, outflow, delta, and volume separately. Furthermore, it allows the operator to see how
these parameters have changed over time in relation to one another.
The outflow is presented on the bottom of the trend, sharing a common zero-point with the
inflow, but protruding in opposite directions (see Figure 22). The orange bar in the middle
illustrates the difference between these two values, thereby showing the current delta flow in or
out of the tank. The value of this orange bar is reflected about the angled line below the delta
plot, thereby providing the X-coordinate for the point on the delta plot. There is also a tail on the
point in the delta chart to show activity in the past ten seconds. The tail presented in Figure 22b
shows a recent increase in volume with changes in the delta flow occurring around eight seconds
ago.
42
Figure 22. a) Delta trend plot showing balanced state over the past 10 minutes. b) Delta
trend plot showing an unbalanced state with 76 kg/s greater inflow than outflow along with
the resulting changes in level.
4.2.3 Display Finalization
The delta chart proved to be very useful for representing multiple aspects of the CANDU 6®
plant at the abstract function level. In areas where the delta chart was unnecessary, the trend
portion of the chart was used, which included the mirrored bar charts showing either the energy
or mass input and output as well as the delta between those values.
Four screens were developed for the experiment, which covered the majority of the plant
parameters that would be involved in the experimental scenarios. Figure 23, Figure 24, Figure
25, and Figure 26 show the four ecological screens that were used during experimentation.
43
Figure 23. Ecological Critical Safety Parameter screen.
Figure 24. Ecological Plant Operations screen.
44
Figure 25. Ecological Steam Generator screen.
Figure 26. Ecological Primary Heat Transfer screen.
45
Experimental Method 5
5.1 Control Room Mock-UP
The experiment was conducted in a CANDU® Main Control Room (MCR) mock-up. The mock-
up is a scale representation of an actual MCR in a CANDU® plant. Some of the main features of
the MCR mock-up are (Candu Energy Inc., 2014):
• 19 touch screen panel displays
• Two 100” large screen display units
• Individual reconfigurable operator workstations
• Full-scope simulator control logic
The simulator control logic allowed for accurate representations of the plant operations during
experimentation while the high-fidelity nature of the rest of the MCR mock-up ensured that the
operating environment would be as close to real-world operating environments as possible.
Figure 27 shows a schematic of the MCR mock-up.
Figure 27. Schematic of the CANDU mock-up. Note that although this is not to scale, the
relative positions of relevant elements in the mock-up are accurate.
5.2 Design
The experiment’s aim was to evaluate the two research questions, which were what display
configuration and what display type yielded the highest levels of situation awareness,
46
communication, and diagnostic performance. Furthermore, we were interested in evaluating
whether certain interactions between display configuration and display type facilitated higher
levels of the three primary metrics. Table 4 illustrates this experimental design. The two
principal comparisons are shown in Figure 28 and Figure 29 respectively.
Table 4. Experimental Design
Question 1
Display Configuration
LSD Redundant
Question 2 Display Type
Advanced LSD-ADV RED-ADV ADV Ecological LSD-ECO RED-ECO ECO
LSD RED
Figure 28. a) LSD configuration; b) Redundant configuration.
47
Figure 29. a) Advanced display; b) Ecological display. Both images present identical
informational content.
5.2.1 Experimental Tasks
Experimentation consisted of two-person operating teams with each operator having a set of
predefined tasks. Both operators’ primary task throughout the entirety of the trial was
monitoring. This encompassed both observation of the plant parameters to determine if anything
was wrong as well as the diagnosis of a fault in cases when they did notice systematic deviations.
Since responsive control actions were outside of the scope of this experiment, once an operator
noticed a deviation, their principal role shifted to determining the underlying cause of the
deviation. Figure 30 shows a participant performing his primary monitoring task in the LSD
configuration.
48
Figure 30. Operators performing primary monitoring task in the LSD configuration.
In addition to monitoring the plant, the operators performed distractor tasks. These were included
in an attempt to portray operator activities as accurately as possible. In real-world operation,
operators are responsible for performing a number of activities beyond simply monitoring the
systems. Examples of these tasks include logging current values and testing systems (Davey,
2000). Each participant in an experimental team had to perform a different distractor task. One
operator was responsible for testing shutdown system one, which involved them approaching the
panels, reading and recording the levels of a variety of indicators, followed by testing the correct
functioning of the control logic. The second operator was responsible for checking the liquid
zone levels in the reactor. This task allowed them to remain at their workstations, but required
them to allocate one of their screens to the liquid zone levels display, thereby taking one of their
available monitoring screens offline for the duration of their distractor task. Figure 31 shows the
operators performing their individual distractor tasks.
49
Figure 31. Operators performing their secondary distractor task in the redundant display
configuration.
5.2.2 Experimental Scenarios
Each experimental team participated in four trials. Each trial involved a unique pairing between
scenario and display setup. Scenario refers to the type of fault that was introduced into the
system and display setup refers to the pairing between display configuration and display type.
There were two display configurations (LSD, RED) and two display types (ADV, ECO), and
thus four possible display setups. Therefore, the experimental teams had a different scenario and
a different display setup in each of their four trials such that there was no repetition of either. An
example of one team’s schedule is presented in Table 5.
Table 5. Example of one team's experimental schedule.
Display Setup Trial Scenario Configuration Type
1 3 LSD ECO 3 2 LSD ADV 2 4 RED ADV 4 1 RED ECO
50
The randomization of the pairings between scenario and display setup was controlled such that
each setup was paired with each scenario an equal number of times. Furthermore, the ordering of
these pairings was balanced to counteract the expected learning effects. The full schedule for all
eight teams can be found in Appendix B.
The scenarios were similar to one another in structure, but differed in the type of fault that was
introduced into the system. In order to accurately assess the performance differences caused by
the different experimental conditions, four considerations were taken into account when
designing and selecting the experimental scenarios.
The first consideration related to the novice operators. The scenarios needed to be complex
enough to challenge the operators but not so complex that they would be overwhelmed.
Candidate scenarios were therefore selected from normal operations rather than emergency, and
revolved around a set of parameters that the participants would have encountered in their college
program. This meant focusing on pressures and flows instead of the electrical grid or nuclear
processes.
The second criterion for scenario selection was that the scenarios needed to call into question the
postulated benefits of GVDs: 1) That they increase operator situation awareness, 2) That they
increase operator communication and collaboration, and 3) That they improve overall
performance (Roth et al., 1997, 1998). In an effort to meet these requirements, scenarios needed
to require a holistic understanding of the behavior of different systems. This was assessed by
examining the degrees of separation between the actual fault and the manifestation of that fault.
For example, a leak in the heat transport system will reveal itself in the D2O storage tank, which
is several tanks removed from the actual source of the problem (see Figure 3). In order to
diagnose this problem, an operator would need a sufficient understanding of the overall behavior
of the plant, which requires situation awareness of multiple related systems. The second and third
claims, pertaining to communication, collaboration, and performance, can be slightly more
difficult to ensure in an experimental scenario since these are often dependent on the individuals
in a team and the dynamic of that team as a whole. However, it has been shown that performance
in problem-solving situations, especially those in complex, high-risk environments, is positively
correlated with communication behaviour (Johansson & Persson, 2009). Therefore, having a
51
diagnosis stage of the trials should call into question the communication and performance claims
about GVDs.
The third scenario inclusion criterion was that no control actions will be required. This criterion
is largely based on subject matter expert recommendation and experimental constraints.
Including control actions would likely have complicated the scenarios too much to elicit accurate
performance data for our novice operators. Requiring control actions would also increase the
complexity of the training regimen. Furthermore, since the experiment’s aim was to evaluate the
efficacy of GVD display types and configurations, including control actions that would be made
at the panel would detract from the goals of the experiment. Therefore, control actions were
omitted from the scenarios.
Finally, the scenarios needed to be relatively similar to one another in difficulty. Differences in
scenario difficulty would likely result in significant performance differences independent of the
experimental manipulations. This is very difficult to control for in simulator studies, but by
focusing the scenarios on similar types of faults we were able to provide some element of
control. Notwithstanding these efforts, the scenario was specified as a random factor in statistical
analysis.
We selected our scenarios both through consultation with operational subject matter experts and
by reviewing the parameters used to initiate a setback or stepback on the reference CANDU 6®
design. Furthermore, we consulted a list of Enhanced CANDU 6® design basis events that would
lead to an Anticipated Operational Occurrence (AOO) plant state. An AOO is defined as a
deviation from normal operation that is expected to occur once or several times during the
operating lifetime of the NPP but that is unlikely to cause any significant damage to items
important to safety, nor lead to accident conditions. Past research has demonstrated the benefits
of functional information in unexpected operating scenarios (e.g., Burns et al., 2008). We
believed that these AOOs meet this criteria for the novice operators since any changes in plant
state would be unexpected.
The structure of each of the scenarios involved a period of steady-state, a fault introduction, a
freeze, and a period after the freeze where the fault continued to manifest itself. Freezes are
pauses in the simulation where the participants complete a measure to assess situation awareness.
52
This measure is discussed in detail in Section 5.4.1. The list of faults that were used in our
scenarios is presented in Table 6 and their timelines follow in Figure 32.
Table 6. List of selected faults for experimental scenarios
Fault Primary Symptom* D2O/ H2O Minor loss of coolant accident Loss of inventory in the D2O storage
tank D2O
Partially closing feedwater control valve
Steam Generator level decrease H2O
Inadvertent opening of MSSVs Decrease in Turbine power H2O Spurious closure of P&IC bleed path
Increase in flow into P&IC D2O
*Only the main indicator of the fault is listed here. The faults will impact many areas of the plant beyond what is listed
Figure 32. Scenario timelines.
As Table 6 indicates, two of the faults occur on the heavy water side of the plant, while two of
them occur on the light water side. We believe that this distributed sampling of faults ensured
good representativeness of different plant operations while also standardizing the difficulty of
each scenario to allow for fair comparison between scenarios.
5.3 Participants
Due to access limitations, licensed nuclear operators were not available for the experiment. We
therefore recruited students from Lambton College’s Process-Operations Program to participate
in the study. A total of 16 process operations students were recruited. Participants were paid
53
$250 for their participation with all expenses covered on top of this. The reason for the large
payment was due to the limited pool that we were able to draw from, and because we required
them to spend roughly six hours in transit, a night in a hotel, and a full day at the simulator
facility.
The participants were organized into eight teams of two, with one team being used for the pilot
and the other seven in the actual study. Some participants knew each other prior to
experimentation and formed a team beforehand. The others were put into teams by the
experimenters using their dates of availability as the sole selection criterion. Although this
presents a potential confound for our results, this was the only feasible way to recruit the
participants since they were active students who had class schedules to balance.
The students had simulator experience and a level of familiarity with other process systems such
as petrochemical or refinery plants. None of the students had any experience with nuclear
operations and were therefore considered novice operators during experimentation. This was
problematic from an ecological validity standpoint because nuclear plants are exclusively
operated by expert operators in the real-world (Juhasz & Soos, 2007).
The challenge therefore became determining the most effective way to capitalize on their
previous experience and training. Due to the time constraints on training, the only feasible way
to meet this challenge was to reduce the complexity of how the plant was represented during
experimentation. This decision was made on the basis that mimicking the expertise of real-world
operators in a simplified plant is more valid than putting novice operators in a system that they
are unqualified to monitor. Although this point can be contested, the simplification of the plant
was performed systematically to ensure that the processes governing plant operations were
accurately represented.
5.3.1 Plant Representation
The first step in reducing the complexity at which the plant was represented was to determine the
areas of operations that were integral to plant functionality. Integral, as used in this context,
refers to the areas that make up the two main circuits in the plant. Because of the closed nature of
these circuits, the understanding of any one system hinges on an understanding of both the inputs
and the outputs for that system. For example, although the reactor is obviously a critical system
54
within the plant, understanding the processes within that reactor is not actually necessary to
understand the behaviour of the rest of the plant at the depth required for our trials. In essence,
one only needs to understand the output of the reactor. Similarly, one does not actually need to
understand how the electricity being produced in the system is fed into the grid, but rather that
the power grid is the output system for electricity generated in the plant.
Any system where understanding both the input and the output were required for a complete
understanding of the behaviour of the plant was considered to be integral. For example, the heat
transport system receives heat energy from the reactor and transfers it to the steam generator
where it converts liquid water to steam. Since understanding this critical energy balance is
necessary for understanding the overall behaviour of that plant, the participants needed to be
trained on this system.
By removing the systems deemed superfluous to understanding plant dynamics, the participants
were able to focus their efforts during training and experimentation on the systems that played
key roles in the scenarios. This allowed them to gain a deeper understanding of the relevant
systems, which essentially increased their relative level of expertise. Figure 33 shows the final
diagram that was given to participants.
55
Figure 33. Final diagram presented to participants which illustrates the level of complexity
of plant representation.
5.3.2 Display Simplification
The interfaces also needed to be simplified to reflect the tailored representation of the plant.
Therefore, any display components relating to parameters beyond the scope of our experimental
scenarios were removed. Again, this allowed the participants to focus their attention on the areas
of the interfaces that were relevant during experimentation in a similar fashion to what an expert
operator would do. Since the participants did not have experience discerning what parameters
were relevant given a current operational state, this stage was necessary for maximizing the
relative expertise levels of these operators.
REACTOR
Deaerator
Deaerator Storage Tank
Condenser
HotwellHotwell
LP Feedheater
Condensate Storage Tank
Condenser
Emergency Stop Valve
Governor Valve
Release Valves
SG Isolation Valves
MSSVs
CSDV
ASDVs
PZR
D2O Storage
Tank
Deg.Cond
Outlet Headers
SGFP
SGFP
SGFP
Turbine GeneratorLegend
D2O PathH2O PathNormally closed valveNormally open valve
Pump
Cooler
Fail Open Valve
Fail Close Valve
HP Feedheater
SG1
SG4SG2
SG3
SG1
SG4
SG2
SG3
56
5.3.3 Participant Training Package
Once the appropriate level of system complexity was determined, I developed a training manual
focused on the areas of the plant that the experimental scenarios would focus on. The training
package was based on an existing introductory training package developed for CANDU
(Bereznai & Harvel, 2011). The pre-existing package was 258 pages long and consisted of
training on everything from the fuel handling systems to the control systems. Due to the limited
time window in which the participants had access to the materials, I selected the content that was
relevant to the simplified representation of the plant in the context of the experimental scenarios.
This allowed for the 258 page training manual to be tailored to 27 pages of essential
experimental training material.
The training package covered only the systems that would be relevant to the experiment,
focusing equally on those systems. The material took the eventual scenarios into account to
ensure that any participant would have the requisite knowledge for adequate performance in that
scenario. However, no extra information or focus was placed on the specific areas that caused the
faults in the different scenarios. This ensured that the participants would not have clues as to
what the cause of a fault in any of the scenarios would be. Details on the administration of the
training package will be covered in Section 5.5.1.
5.4 Measures
5.4.1 Situation Awareness
Situation Awareness (SA) was evaluated using the Process Overview (PO) measure (Lau et al.,
2011). The process overview measure was designed specifically for measuring SA in process
control plants. It uses context-dependent, top-down queries to assess operators’ overall
comprehension of the behaviour in the plant. The PO measure was selected over other measures
of situation awareness, such as the Situation Awareness Global Assessment Technique (SAGAT;
Endsley, 1995) because it was designed specifically for process control. SAGAT is based on
Endsley’s 3-level model of SA, which views SA as an information processing activity that is
assumed to generalize across different domains. SAGAT therefore uses the same approach to SA
measurement irrespective of the domain. In process control, SA is viewed as a creative problem
57
solving activity that is heavily influenced by domain context (Lau et al., 2011). Since SA is
characterized differently in process-control, the PO measure is tuned to this situational context.
To use the PO measure, we developed queries that ask questions about the behavior of specific
aspects of the plant in the given scenario. An example of the structure of a PO query is presented
in Figure 34. Each trial had two sets of 12 PO queries administered to each participant
individually. Participants were not allowed to communicate to one another nor did they have
access to the displays while completing these tests. The first set was presented to participants at
the freeze in the trial and the second was presented at the end of each trial (see Figure 32). The
queries asked about the recent behaviour of various parameters relevant to that particular
scenario. Although there can be some ambiguity with respect to how recently is defined (Lau et
al., 2011), we defined it to our participants for the first set (i.e., the set presented at the freeze) as
since the beginning of the trial. For the second set of PO queries, recently was defined as the
time since the freeze point. Therefore, if a particular value is currently lower than it was at the
beginning of that portion of the trial (i.e., either from the start until the freeze, or from the freeze
until the end), irrespective of the behaviour in the middle, the correct answer to that query is
decreased. An example of this is presented in Figure 35.
Figure 34. Typical process overview query.
58
Figure 35. Example of how recently was defined.
Each set of queries consisted of 12 questions. Eight of these were visible, meaning that the value
that the query was inquiring about was displayed somewhere on the display. The other four
questions were inferred. This means that their value was not visible anywhere in either the
advanced or the ecological displays. Correctly answering these was based on the participants’
understanding of the recent behaviour of related systems in the plant.
The queries should inquire about parameters in the plant that are integral to the present state (Lau
et al., 2011). For example, if a scenario involves a leak in the heat transport system, the queries
should focus on the heavy water side of the plant rather than ask questions about what is
happening in the steam system. To select the components and the queries, we consulted with a
SME. The SME was a former nuclear operator who was familiar with multiple CANDU® plants.
Although the PO measure is typically scored during experimentation by an SME, we were able
score the queries after the data collection. Most studies employing the PO measure involve
control actions, which can impact the plant’s state and therefore change the values of the
respective parameters and thus the answers to the respective PO queries. Since our study did not
have control actions, the behaviour of parameters in each scenario was identical. Therefore, all
scenarios could be scored using a reference key that represented the objective answers to each
query. This also circumvented any potential issues with interrater reliability (Lau, Jamieson, &
Skraaning, 2014). Further description of the scoring and coding of PO measure queries follows
in the results section. The list of queries is presented in Appendix C.
59
5.4.2 Communication
There were two sources of communication data. The data presented in this report was derived
from subjective ratings of communication during each trial. These ratings were based off of
experimental logs that were kept by the experimenters during testing. The logs included notes on
operator communication, performance, and overall monitoring behaviour. Ratings were therefore
meant to indicate a general communication metric. They were loosely defined and intended to
simply provide an initial impression of how well the operators communicated with one another
throughout a trial. All recording was done discretely from the experimenter recording station (see
Figure 27) to prevent any interference or to influence the communication behaviour of the
participants. These scores were done on a 7-point scale. This method of recording
communication served as a preliminary assessment of communication patterns within the various
setups. Although all efforts were made to accurately reflect the communication in each trial, this
data is likely subject to biases.
The second measure of communication employed Behaviorally Anchored Rating Scales
(BARS). Montgomery, Gaddy, and Toquam (1991) used BARS to rate team interaction in
nuclear power plant control rooms along the following five dimensions: communication, task
coordination, team spirit, maintaining task focus in transition, and adaptability. Using these five
dimensions in conjunction with an extensive rater-training system, they found interrater
reliability of 0.89, which is considered very high by interrater reliability standards (Gwet, 2014).
We recruited three students from the Mechanical and Industrial Engineering programs at The
University of Toronto to act as raters. Montgomery et al. (1991) state that non-expert raters can
still achieve high interrater reliability readings, provided they have sufficient training. We used
example scripts to train them on the five dimensions. These included both positive and negative
examples of behaviour along each of these dimensions to provide benchmarks. The raters were
trained in the system at a level sufficient to detect communication patterns. For example, the
raters needed to know when an operator noticed a problem. In order to do this, the raters needed
to know what the problem was and examples of things that the operators may be saying when
they notice the problem. The raters were paid $20 per hour to perform the rating task.
The raters used recordings from each trial to score the BARS. Raters were blind to the
experimental setups, but were aware of the scenarios in order to determine whether or not the
60
communications between operators were focused on the actual fault from the respective scenario.
Each team completed four trials, which amounted for a total of 75 minutes of recordings per
team and thus 525 total minutes for all 7 teams combined. This therefore took several weeks for
the raters to work through. Unfortunately, time restraints with respect to receiving the rating data
prevented them from being included in this report. This information will be included in future
publications.
5.4.3 Diagnostic Performance
To measure operator diagnostic performance, we adapted the method used by Lang, Roth, Bladh,
& Hine (2002), who used two principal metrics for evaluating performance: Detection and
diagnosis. In their study detection performance was determined by whether or not the crews
detected the target event (e.g., loss of coolant accident). This was done online and recorded
through observation by raters. Their second metric was diagnosis, was also done through
observation by raters who noted whether one or more of the team members explicitly stated the
cause of the event during the trial. We adapted this to make it post hoc, which better met the
needs of our novice participants. We used a semi-structured interview after each trial. The
interview consisted of the following questions:
1. Did you notice a problem? 2. Was the problem on the primary or the secondary side of the plant?
3. What were the main symptoms of the problem? 4. What systems did those symptoms manifest in?
5. What would you say was the exact cause of the problem?
Formalizing the measure in the form of a semi-structured interview following each trial
prevented us from relying entirely on the participants to explicitly state the cause of the fault.
Although this method is less formal than some of the other operator performance measures, it
was best suited for our needs since our participants did not engage in any control actions.
We also recorded temporal performance metrics using the same team of raters from the BARS
ratings. Although time restrictions prevented this data from being used in this report, it is
worthwhile detailing how temporal performance will be evaluated. These were metrics
representing the sequence of actions that an operator would go through when they first notice a
fault: 1) Detection, 2) problem solving, and 3) diagnosis. The raters were given criteria to
61
determine what constitutes each of the stages. Detection was defined as a participant verbalizing
that they noticed a problem. Problem solving was defined as the point in the trial when
participants begin verbally attempting to determine the cause of a problem. Accurate diagnosis
was defined as the point in the trial when the team verbalizes the actual cause of the problem
Again, since these definitions are slightly ambiguous, interrater reliability will be evaluated.
Using this data will eventually allow us to answer more questions relating to process monitoring
and control. While the detection time data may provide valuable insight towards the benefits of
the different experimental setups, finding significant relationships between experimental setup
and problem solving time and diagnosis can be extremely valuable for understanding the types of
behaviour that the different displays and configurations facilitate.
Table 7. Summary of measures
Construct Measure Description Included?
Situation Awareness
Process overview measure
A probe-based query system developed specifically for process control situation awareness
Yes
Operator Performance
Lang et al., (2002) method
Semi structured interviews to determine participants’ ability to accurately detect and diagnose a fault
Yes
Temporal metrics Latency of fault detection, troubleshooting, and accurate diagnosis No
Team Communication
Subjective ratings
Preliminary rating of communication based on experimental logs Yes
Behaviorally anchored rating scales
External raters judge operator interaction along a predetermined set of dimensions No
5.5 Procedure
5.5.1 Training
Participants received the training manual described in Section 5.3.3 four days before their
scheduled date at the simulator. They were instructed to read it over and make note of any
questions that they had. Finally, they were also told that they would be receiving a test on the
content of the training package to help motivate them to study the package closely.
62
Upon arrival at the simulator facility, participants were greeted and asked to sign non-disclosure
agreements and letters of informed consent. Once this was completed, the experimenters would
perform a walk-through of the schematic of the system to make sure that everyone had a basic
level of understanding of how the system worked. During this time, participants were
encouraged to ask any questions that may not have been addressed in the training package.
Once the systems walkthrough was completed, participants were given a brief test that evaluated
their understanding of certain aspects of the system. This test was included for two main reasons.
The first, was that since the participants were informed of this test before signing up for the
study, they would likely be motivated to read the training manual carefully than if they didn’t
think they were going to be evaluated. The second reason was to determine if there were any
areas of plant operations that the participants still did not understand. The tests were reviewed
with the participants upon completion and any points of confusion were discussed.
After the test debriefing, participants were trained on how to read both the ecological and
advanced display elements. They were introduced to the elements as standalone graphical forms
rather than being introduced to the full displays right away. This allowed them to focus on the
individual forms rather than potentially becoming overwhelmed by the breadth of the displays.
The delta plots, a prominent feature on the ecological interfaces, required more instruction than
did the elements used in the advanced displays. However, it was important for the participants to
spend equal amounts of time with both displays to prevent any preconceived biases from
developing. We therefore spent the same amount of time on both display types.
Once participants had been trained on the individual graphical forms, they were introduced to the
full displays. The experimenters guided the participants through both the ecological and
advanced display types in both the LSD and redundant configurations. Again, the ordering of this
training was randomized and counterbalanced to prevent any bias. After the participants had
been sufficiently introduced to the interfaces they were given 10-20 minutes to freely explore
them.
Following the interface exploration, participants began test trials. The trials were ordered in
terms of the severity of their symptom manifestation. Prior to the first trial, participants were
informed that changes in system state during actual experimentation would be likely to be much
more subtle than the ones in the test trials. The first test trial induced a reactor trip in less than a
63
minute. This was to illustrate the types of parameter behaviours that the participants should
expect in the actual trials. The system was then reset at the second test trial began. The test trials
were designed to be similar to the actual experimental trials, but without having overlap in the
potential causes for the symptoms. Because of this, we were limited in the number and scope of
test trials.
During the second test trial the participants were introduced to the freeze, which are simulator
pauses, during which time the PO measure was administered. Participants were instructed to
leave their workstations during the freeze and to approach a conference table to fill out the PO
measure (see Figure 27). During their completion of the PO measure the operators were not
given access to any displays. They were informed ahead of time that the PO measure relied on
their ability to maintain an accurate and up-to-date mental model of the operations of the plant.
We then reviewed their responses to the queries and discussed the answers to ensure that all
parties had the same concept of how to correctly interpret the queries.
Training was finished when the experimenters determined that the participants had attained a
sufficient level of proficiency with the system. This varied from team to team as some
participants differed significantly in the speed with which they became comfortable with the
system. There was no formal assessment to ensure an adequate level of proficiency prior to the
commencement of the experimental trials, but due to time restrictions, this was the only feasible
method of experimentation. Both subject and group were specified in all statistical analyses as
random variables in an effort to account for this limitation.
5.5.2 Experimental Trials
Prior to each trial, both participants in a team were instructed to activate their voice recorders
and to leave the recorders running until the interview following completion of the trial was
finished. Participants were instructed to go to their respective workstations while the
experimenters ensured that the necessary forms were in place. These forms included those
required to complete both of the distractor tasks, as well as the process overview measures.
The trials began with one of the experimenters giving the participants a countdown, after which
the trial was active. Once the trials were active the experimenters took their position at the shift
supervisors desk (see Figure 27) and began recording log notes.
64
Trials ran uninterrupted until the freeze. The simulator was programmed to automatically pause
the operations at the freeze. When the freeze occurred the participants left their workstations and
walked to the conference table at the back of the room (see Figure 27). Each participant filled out
the PO measure for that portion of the trial with no set time limit. During this time they were not
allowed to communicate with one another nor did they have access to the displays. When they
were finished they returned to their workstations, at which point one of the experimenters gave
them a second countdown marking the commencement of the second leg of the trial.
Again, that experimenter returned to the shift supervisor desk and resumed taking log notes for
the remainder of the trial. The trial ran uninterrupted until it was complete. When complete, the
simulator was programmed to restore its baseline conditions and the participants filled out the
second PO measure at the conference table.
5.5.3 Post-Trial Interview & Debriefing
After each participant was finished with their second PO measure (i.e., the set of queries at the
end of the trials) they were given individual post-trial semi-structured interviews. The purpose of
these interviews, as described in Section 5.4.2, was to assess diagnostic performance (Lang et al.,
2002). The interview consisted of five questions, which gradually got more specific as to the
nature of the fault from the preceding trial.
Once the interview was complete the participants were instructed to stop and save their voice
recordings. They were then debriefed on the actual cause of the problem. Since the problems
were unique, this did not present learning effect confounds.
This process was repeated for each of the four trials. Participants were never told the true nature
of the testing because of concerns surrounding the close-knit community at Lambton College and
the potential interference that could occur if a team told another team the motivations behind the
study.
Experimental Results 6
6.1 Situation Awareness
The data used to measure situation awareness (SA) came from process overview (PO) queries
that were given to the operators at both the freeze and endpoint of each trial. As stated above,
65
each set of queries consisted of 12 questions pertaining to characteristics of the plant that were
relevant to that particular scenario. Of the 12 questions, four were inferred, meaning that the
parameters they were inquiring about were not actually visible on any of the display screens.
These therefore required the participants’ deeper understanding of the behaviour of the plant.
The other eight queries were visible, meaning that the information for the parameter inquired
about was available when they were at their operating stations.
A total of 1344 process overview responses were recorded, 896 of which were visible and 448
inferred. A summary of the data collected is presented in Table 8. Due to one of the experimental
teams serving as the crew used in the pilot study, their data was not usable in this analysis. This
resulted in only seven teams being used for actual experimentation. Therefore, some Setup by
Scenarios were only tested by one team, resulting in 48 total PO queries for those conditions
rather than the 96 for all others.
Table 8. Number of PO measure queries for each Scenario X Setup
Setup LSD-ADV LSD-ECO RED-ADV RED-ECO Total
Scen
ario
HTS Leak 48 96 96 96 336 FW Valve Closure 96 48 96 96 336 MSSV Open 96 96 48 96 336 P&IC Path Closure 96 96 96 48 336
Total 336 336 336 336 1344
A mixed linear regression was performed on the visibility of the queries. As expected, the means
on the inferred queries (m = 0.44, SE = 0.06) were significantly lower than the visible queries (m
= 0.75, SE = 0.05; F(1323) = 111.52, p < .0001). This significant performance difference held
across all setups and scenarios (see Figure 36). The results of these t-tests are presented in
Appendix D.
66
Figure 36. Visible vs. Inferred means.
As the above figure illustrates, the participants’ responses on the inferred queries were
considerably lower than they were for the visible queries. The initial mixed generalized linear
model using the PROC GLIMMIX procedure in SAS University Edition utilized both the
inferred and visible queries. However, it failed to converge. Although there are many reasons for
this, the most likely is that there was either complete or semi-complete separation in the data
preventing the maximum likelihood estimates from being calculated (Allison, 2008). This failure
is likely the result of guess behaviour on the inferred queries counteracting any predictable form
in the data. Because the recommended number of queries per PO measure was eight, as per
recommendation from the developer of the PO measure (Lau, n.d.), subsequent analyses were
able to omit inferred queries from the models while maintaining an adequate sample size.
Therefore the pool of responses used to evaluate SA performance consisted of 224 query
responses from each experimental setup and scenario, all of which were visible.
A mixed generalized linear model was used to investigate the two experimental questions using
the PROC GLIMMIX procedure in SAS University Edition. This method examined the
probability of any given query being answered correctly in the various experimental setups,
irrespective of the scenario, group, or subject. Ideally, the scenarios, groups, and subjects would
67
be identical in terms of performance. However, as Figure 37, Figure 38, and Figure 39 illustrate,
there was a large amount of uncontrolled variance resulting from these variables. These were
thus specified as random variables, and were controlled for in the model.
Figure 37. PO measure means for each scenario across the four conditions. SCN refers to
the scenario (see Table 6).
Figure 38. Each group's PO measure scores across the four setups. GID refers to the group
identifier.
68
Figure 39. Individual subjects' PO measure scores over the different setups. SID refers to
the unique identifier for each of the 14 subjects.
After the random effects described by the three above figures were controlled for, the final model
used in the PROC GLIMMIX procedure examined a binomial distribution through a logit link
function. The SAS code used in this model is presented in Appendix E. A Type III test for fixed
effects revealed a significant effect of display type on PO measure scores, showing that the
likelihood of any given question being answered correctly was significantly higher in the
ecological displays (m = 0.80, SE = 0.08) compared to the advanced displays (m = 0.72, SE =
0.06; F(873) = 5.76, p < .02). Neither display configuration nor the interaction between display
configuration (F(873) = 0.07, p > .05) and display type (F(873) = 0.92, p > .05) were found to be
significant. Table 9 and Figure 40 illustrates these findings.
69
Table 9. Type III Test for Fixed Effects
Effect Num DF Den DF F Value Pr > F Configuration 1 873 0.07 0.79 Display Type 1 873 5.76 0.017 Interaction 1 873 0.92 0.34
Figure 40. LS Mean score for each experimental setup. LS Means have been transformed
to be on an interpretable scale (i.e., 0 < LSM < 1). Red bars indicate the 95% confidence
interval.
6.1.1 Inferred Model
A separate generalized linear mixed model was performed using only the inferred data to see if
there were any setups that yielded higher scores on inferred queries. As expected, there were no
significant effects found for either display configuration (F = 0.36, p > .05), display type (F =
0.41, p > .05), or the interactions between configuration and type (F = 1.89, p > .05).
6.1.2 Differences of Least Squares Means
Testing the differences of least squares means is a process that can be requested within the
PROC GLIMMIX procedure in SAS. This test conducts pairwise comparisons to evaluate
whether specific setups (i.e., Configuration X Display type) significantly differ from one another.
70
This can reveal whether certain display types perform better when coupled with certain display
configurations. Two differences were initially found to be significant: LSD-ECO vs. RED-ADV
(t(873) = 1.98, p < .05) and RED-ADV vs. RED-ECO (t(873) = -2.46, p < .02). However, after a
simulated adjustment for multiple comparisons, these differences were found to be not
significant. A full table showing the multiple comparisons is presented in Appendix D. The
differences are presented in a diffogram in Figure 41. Diffograms plot the mean of each setup
against the mean of every other setup to illustrate if the difference is significant (High, 2014).
When two means are very similar, their point of intersection will fall along the dashed angled
line projecting from the lower left corner to the upper right corner on the diffogram. When two
means are significantly different, their point of intersection in the diffogram will be further away
from this line. As Figure 41illustrates, no differences were found to be significant.
Figure 41. Multiple comparisons diffogram showing no significant contrasts.
71
6.2 Communication
A mixed linear regression was performed using PROC MIXED in SAS that specified the
scenario and group as random effects. As stated earlier, due to time constraints, the only data
used in this analysis are from subjective ratings given by the experimenters. Although all efforts
were made to provide unbiased ratings, the results presented in this section are considered
preliminary until the data from the BARS has been coded and analyzed.
A square root transformation was performed on the communication scores to normality
assumption. This compressed the scale such that the maximum score, which was originally 7,
became 2.65 with the minimum remaining at 0.
The mixed linear regression revealed a significant main effect of display configuration on
communication, indicating that the redundant displays (m = 2.10, SE = 0.17) elicited higher
communication scores compared to LSDs (m = 1.92, SE = 0.16; F(15) = 5.8, p < .05). There was
no significant effect found for display type (F(15) = 0.04, p > .05) or for the interaction between
configuration and display type (F(15) = 0.01, p > .05; see Table 10). The least squares means
estimates for the four experimental setups are presented in Figure 42. As the figure illustrates,
both of the redundant conditions yielded higher communication scores than both of the large
screen conditions. The SAS code used for this test is presented in Appendix E.
Table 10. Type III Test for Fixed Effects
Effect Num DF Den DF F Value Pr > F Configuration 1 15 5.80 0.029 Display Type 1 15 0.04 0.849 Interaction 1 15 0.01 0.943
72
Figure 42. Communication scores across the four experimental setups. Red bars indicate
the 95% confidence interval.
6.3 Diagnostic Performance
The data used to evaluate diagnostic performance was based on the method described in 5.4.2. A
mixed linear regression was performed using PROC MIXED in SAS, which once again
identified the scenario, group, and subject nested within group as random variables. The SAS
code used for this mixed linear regression is presented in Appendix E.
The mixed linear regression revealed no significant main effects. This indicates that there were
no significant differences between diagnostic performance mean estimates for GVD
configurations (F(36) = 2.61, p > .05), display types (F(36) = 0.31, p > .05), or for the interaction
between configuration and display type (F(36) = 0.73, p > .05; see Table 11 and Figure 43).
Furthermore, there were no significant pairwise contrasts.
73
Table 11. Type III Test for Fixed Effects
Effect Num DF Den DF F Value Pr > F Configuration 1 36 2.61 0.11 Display Type 1 36 0.39 0.54 Interaction 1 36 0.90 0.35
Figure 43. Diagnostic performance mean estimates by experimental setups. Red bars
indicate the 95% confidence interval.
To further investigate diagnostic performance, a mixed linear regression using the was
performed which examined the relationship between each trial’s performance score and how well
they did on that trial’s process overview queries. For each trial, each participant’s process
overview average was calculated and paired with their performance score on that trial. The
model revealed that process overview scores significantly predicted performance scores (F(45) =
6.46, p < .02).
It should be noted, that although the difference between configurations was not significant (F(36)
= 2.61, p = .11, the LSD display configuration achieved much higher performance estimates (m =
0.63, SE = 0.11) in the final model compared to the redundant display configuration (0.46, SE =
0.11). This difference was particularly large when comparing performance in the LSD-ECO
74
setup (m = 0.65, SE = 0.13) to performance in the RED-ECO setup (m = 0.38, SE = 0.13),
showing a 27% increase when the ecological displays were presented on the LSD compared to
redundant configurations. Although these results initially appeared systematic, an adjustment for
multiple comparisons rendered them insignificant (t = 1.81, p = 0.28). Although insignificant,
these sizable differences should warrant further evaluation in future studies. Figure 44 presents
the results.
Figure 44. Difference in performance between the two display configurations. Red bars
indicate the 95% confidence interval.
Discussion 7This study aimed to evaluate the efficacy of two GVD alternatives, and two display types within
a nuclear control room mock up. We evaluated how well different GVD and display framework
alternatives fostered SA, communication, and diagnostic performance. Although there is still
much to be learned about the different GVD and display alternatives in nuclear control rooms,
our study sheds light on the benefits of some of the foremost candidates.
Going into this study we had two principal research questions:
75
1. Is the widespread favouring of LSDs over redundant displays rooted in improvements in
SA, communication, and overall performance?
2. Is the widespread implementation of advanced displays over other display frameworks
rooted in improvements in SA, communication, and overall performance?
These questions focus on the three claimed benefits of GVD implementation for both GVDs and
display frameworks. Our findings are summarized in Table 12 and will be discussed according to
these three claims.
Table 12. Summary of main findings.
Dimension Main Findings
Display Configuration Display Type Situation Awareness
No significant difference for display configuration
Higher levels elicited by the ecological displays
Communication Significantly higher communication scores yielded by the redundant display configuration
No significant difference for display type
Performance No significant difference for display configuration
No significant difference for display type on communication
7.1 Situation Awareness
The process overview (PO) measure evaluated participants’ situation awareness (SA) under the
different experimental conditions. Participants scored significantly higher on the PO measure
under ecological conditions compared to advanced conditions, indicating that this display type
fostered a better understanding of the recent behaviour of the plant. There were no differences in
SA between the LSD and redundant configurations.
These results support previous research that has demonstrated the benefits of functional displays
on operator SA (Burns et al., 2008; Lau et al., 2008; Roth et al., 1998; Tharanathan et al., 2012).
As expected, the advanced displays performed well, scoring a 73% probability that any given PO
query would be answered correctly. This score is actually the exact score that Burns et al. (2008)
found under similar conditions using the PO measure. Contrary to the Burns et al. (2008) finding,
however, our participants demonstrated better PO scores in similar conditions using the
ecological displays than did their expert operators who scored 61% in similar operating
76
conditions. Of course, these results cannot be directly compared since their study employed
expert operators, different displays, and a different methodology, but it does serve as a helpful
reference point. Therefore, while both displays performed at a sufficient level, the significant
improvement elicited by the ecological displays indicates that improvements can be made to the
current advanced displays.
The higher levels of SA elicited by the ecological interfaces can be explained in a variety of
ways. Since there was no significant effect of display configuration, these differences in SA are
solely the result of display type. Furthermore, by employing the array of controls ensuring that
the only thing separating the advanced from the ecological displays was their driving framework
(see section 4.1), we can safely attribute these SA differences to differences between the display
frameworks.
The most likely explanation, given the control over the different conditions, revolves around the
integration of functionally related values into consolidated graphical forms in the ecological
displays. Instead of having to recall individual parameters when filling out the PO measure,
participants were able to simply recall the behaviour of a unified form. For example, the delta
plot (see section 4.2.2), which plots delta flow (i.e., inflow – outflow = delta flow) against
current level, was a prominent feature in the ecological displays for showing both energy and
mass balances. This plot used a single point with a trendline to represent the majority of the
necessary information pertaining to a system of interest. Participants could theoretically use this
plot to answer a variety of PO queries by simply remembering the behaviour of this single point
on that chart. The advanced displays presented the same information as individual values for
input, output, and level. This would therefore require them to recall three elements in the display
rather than one (see Figure 45). This contrast captures some of the essential differences between
the frameworks and is the likely factor driving the contrast in SA across the two display types.
77
Figure 45. Differences between advanced and ecological graphics.
7.2 Communication
Participants demonstrated significantly improved communication in the redundant display
configurations compared to the LSD configurations. Although there is an assumed benefit of
GVDs on communication (Roth et al., 1998), there were little expectations regarding the
difference between GVD alternatives since these had yet to be evaluated. Communication scores
were significantly higher in both the ecological and advanced display conditions for redundant
configurations. Since the screen content and navigation functionality of the two configurations
was identical, this finding suggests that the difference found between GVD types was systematic
and likely the direct result of differences in the location of the shared informational content in the
control room.
There are many possible explanations for this effect. I believe that the most likely reason stems
from differences in awareness of the behaviour of the other operator in a team. When looking at
the LSDs, operators were facing the same location. This allowed them to maintain sight of each
other which may have resulted in an implicit assumption of team-level situation awareness.
Because of this assumption, operators may have felt less need to communicate verbally with one
another. In the redundant conditions, the operating partners were out of each other’s peripheral
vision. This may have forced them to verbalize what they perceive to be meaningful behaviour in
the system because they were not assuming that their operating partner is seeing the same
information. These hypotheses will be discussed in further detail in the Future Research section
(see section 8.2.3).
78
There were significant limitations in the communication data. The communication metric was
based on ratings given by the experimenters during and after experimentation. Although we
attempted to be as unbiased as possible with our ratings, they were likely influenced by implicit
preconceptions. The same team of raters that recorded the time-based performance metrics was
used to rate each trial using behaviourally BARS. This method will yield much more reliable and
valid data describing the communication behaviour of the participants in the different
experimental trials. However, due to time restrictions, this data has not been included in the
present report.
7.3 Diagnostic Performance
Contrary to expectations, our results revealed no significant differences in diagnostic
performance between either display types or the display configurations. This unexpected result
was compounded by the fact that there was a significant positive correlation linking increased
SA to improved diagnostic performance. This finding suggests that although there was no direct
evidence linking improved performance to display type or configuration, there may be an
indirect link. In other words, since the ecological displays yielded better SA, and better SA
correlated with better performance, ecological displays may have indirectly improved diagnostic
performance.
The insignificant findings are likely due to unexpectedly low performance scores in the
redundant-ecological setup. This experimental setup yielded a mean estimate of just under 40%,
while the others were at 53%, 60%, and 63%, respectively. Since these performance metrics are
based on the averages of just 56 data points, they are more significantly impacted by variation in
the data. This surprising result should be examined in future studies.
There were two main limitations in the diagnostic performance data. The first is that the sole
indicator of performance presented in this thesis stems from an adaptation of a Lang et al. (2002)
method, which collapses all possible performance metrics into a single data point. We also
recorded time-based metrics for each group, including the time that they detected the fault, the
time after that detection that they began trouble-shooting the fault. Due to time limitations with
respect to receiving this data, it has not been included in the present report. However, we believe
that it will likely reveal interesting results linking the different display types to impacts on
performance.
79
The second limitation in performance data is the result of control actions being omitted from the
experiment. Unfortunately this was an unavoidable limitation in our study stemming from the
use of novice participants. Although this was inherent to our study, future work should look at
the same experimental questions with the addition of control actions.
Summary & Conclusions 8
8.1 Summary
This research was motivated by a finding from an operating experience review conducted by
Myers & Jamieson (2014) that there had yet to be a formal evaluation of the postulated benefits
of GVDs in nuclear control rooms. These claims are that GVDs will improve (Roth et al., 1998):
1. Operator situation awareness, 2. Communication within operating crews, and
3. Overall performance
This study therefore focused on developing and conducting an experiment capable of evaluating
different GVD alternatives along these three dimensions. Furthermore, we evaluated the
structure of the information presented on those GVDs to determine if ecological displays
outperformed the advanced displays along the same three dimensions.
We began the process of answering these questions by conducting a three-phased CWA. The
three phases we completed were work domain analysis, control task analysis, and strategies
analysis. Because the competencies of our experimental participants were known prior to
analysis, worker competencies analysis was interwoven into the three overt phases. Furthermore,
we were able to use these limited worker competencies to restrict operator actions to monitoring
and diagnosis. This removed the need for the social organization and cooperation analysis in our
CWA. This CWA provided us with the foundation on which to develop novel displays and
design a comprehensive experiment evaluating how these displays interact with different GVD
alternatives.
After completing the CWA, we developed a set of novel ecological displays that mirrored the
existing advanced displays in terms of informational content, navigation, and fidelity. They
differed significantly, however, in terms of how that content was presented to the operators,
focusing on depicting functional rather than physical relationships. While we attempted to use
80
existing ecological forms and elements that already existed in the display software’s graphical
suite, limitations in that software as well as perceived limitations in some ecological forms
necessitated the development of a novel ecological form.
This form depicts the abstract function level of the abstraction hierarchies from the work domain
analysis stage of our CWA. The abstract function level of the hierarchies describes the
underlying principles that drive system behaviour. Our analysis described these principles in
terms of their mass and energy balances focusing on the source, store, and sink of that mass or
energy. The form that we created illustrates these balances by presenting input, output, and how
they relate to the current level in a unified display. Furthermore, it shows how these individual
parameters change over time, both individually and in relation to one another.
Once the ecological displays had been created, we ran an experiment evaluating them against
advanced displays across two GVD configurations, LSD and redundant. This resulted in a 2X2
experimental design. Each operational team participated in each of the four possible display
setups (i.e., Configuration X Display Type) under a unique experimental scenario such that no
setup or scenario was repeated within an operating team.
The experimental data was categorized and recorded along the three posited benefits of GVDs.
Situation awareness data was recorded through the use of the process overview measure at two
locations in each trial. The communication scores that are presented here are based on ratings
given by the experimenters and are based on logs recorded during the experimentation. Finally,
the performance metrics were based on post-trial interviews wherein participants had to identify
the location and cause of a fault from the preceding scenario.
Results revealed marked SA improvement yielded by the ecological interfaces compared to the
advanced interfaces. There was a significant difference in communication between the display
configurations, with redundant displays outperforming the LSDs. Finally, although no significant
effects were found for diagnostic performance for either display configuration nor display type,
there was a significant correlation between diagnostic performance and SA. Therefore, we
believe that there may be an indirect benefit of ecological displays over advanced displays on
performance.
81
While the results describing the differences in display type are illuminating, the insignificant
difference between the LSD and redundant displays is also interesting. To our knowledge, no
previous study has examined the differences in SA, communication, and performance across
different GVD configurations. Our findings suggest that LSDs do not foster greater levels of SA,
communication, or performance compared to redundant displays. In fact, due to the increased
footprint required for LSD implementation, our results suggest that redundant displays may be
preferred in future control room designs.
8.2 Conclusions
8.2.1 Contributions
Although considerable improvements have been made to nuclear control rooms since the Three-
Mile Island accident, many of these improvements are driven by assumed benefits of new
technology rather than by empirical evidence. This thesis addresses this issue for the next stage
of control room evolution by systematically evaluating the configuration and framework by
which overview information is presented to operators. Our results suggest that the recent trend
towards the implementation of LSDs may not be the best option compared to redundant displays.
Not only does this result have direct implications for GVD technology, but it also illustrates the
need for empirical evaluation of safety-critical control room improvements.
This study did not find any evidence to suggest that the increased footprint required by LSDs is
warranted by SA, communication, or performance improvements. Further, we found evidence
suggesting that team communication is better in redundant conditions compared to LSD. In real-
world operating scenarios and in particular, unanticipated operating occurrences, this increased
communication may represent a critical aspect of performance and team situation awareness
(Juhasz & Soos, 2007). This suggests that although we did not find performance or SA benefits
from the redundant displays, there could be an indirect link that is formed by the communication
benefits of redundant displays. Furthermore, since both communication and performance are
multi-faceted constructs, it is possible that we simply did not measure the aspects of them that
would reveal correlation.
Our study also provides further support to the body of literature that has demonstrated SA
improvements elicited by ecological displays (e.g., Burns et al., 2008). We have shown these
82
benefits in a complex, full-scope environment, which further supports the ecological validity of
this framework.
In addition to substantiating the literature supporting EID and CWA, this study also advanced
EID by creating a novel, widely applicable graphical form. Although this design work was a
byproduct of the larger project, it resulted in a tool that is capable of displaying mass or energy
balances across domains. The form allows for easy problem detection, future state anticipation,
and online diagnosis. Operators are able to assess whether a system is balanced, unbalanced,
leaking, or blocked by looking at the behaviour of a single point on the delta plot. Since this
form was completely new to the participants, it is reasonable to assume that with more time to
adapt to the chart, further SA and performance benefits would accrue. The fact that the
ecological displays, which relied heavily on a form that the participants would have had very
little experience with, outperformed the advanced displays, who used forms the participants
would have familiarity with, on the first and only day of each team’s testing is a testament to the
potential utility of this tool in future applications.
8.2.2 Limitations
Several of the limitations of this study have been discussed in detail in the preceding sections,
but this section will summarize the major areas of concern. The principal limitation was the fact
that we used novice participants. Fortunately, we were aware of this prior to the analysis and
design phases of the study and were therefore able to accommodate their competencies in the
early stages of the study. However, studies in a domain that is exclusively operated by experts
should better reflect that user base. While we made all efforts to simplify the system and bring
our participants up to a reasonable level of expertise on that system, the results are still based on
a novice user base. As stated above, it should be noted that the participants scored similar levels
on the PO measure compared to previous simulator studies that have used expert operators (e.g.,
Burns et al., 2008). It should also be noted that if we had used expert operators, they would likely
have had experience with the existing advanced displays as well as the LSDs. This would have
likely biased the results. Therefore, using novice participants on a simplified system was not
without some benefits.
Another limitation related to our participants revolved around the amount of time we had with
them. Since we were drawing from a very select pool of active students, they were only able to
83
miss one day of classes. This meant that their training was only able to consist of a training
manual and one morning of in-person training on the systems. Furthermore, participants had a
very limited amount of time with which to familiarize themselves with the control interfaces and
navigation.
A third limitation related to our participants, was the limited sample size. This was significant
from a statistical analysis perspective, especially in terms of performance and communication
where there was less data per participant. Unfortunately, due to the nature of simulator research
in nuclear domains, a limited sample size is a common hindrance. Therefore, although our study
would have benefited from more participants, our sample size was similar to other studies in the
nuclear domain (e.g., Burns et al., 2008). This means that our methods and results are
comparable to other studies within the nuclear domain.
A final limitation in this report comes from the incomplete data presented in both the
communication and diagnostic performance sections. Although the main focus of our analysis
was on SA, this affected the validity of our conclusions regarding communication and
performance. This limitation will be amended once the BARS and performance timing data have
been coded and analyzed.
8.2.3 Future Research
Our study provided evidence suggesting that the current status quo in nuclear control rooms can
be improved upon. Future studies should continue this thread of research by not only examining
the benefits of different GVD alternatives, but also compare performance, situation awareness,
and communication between conditions with GVDs against conditions without GVDs.
Furthermore, studies should also longitudinally examine the same experimental question using
expert operators with the addition of control actions.
Future research should examine more thoroughly the pairings between GVD and display type. It
is possible that different display types perform better in different display configurations, as
demonstrated by the nearly significant contrast between the redundant-advanced and redundant-
ecological setups. This could demonstrate that the redundant displays are most effective when
combined with ecological displays.
84
In order to accurately capture the reason for communication differences between GVD
alternatives, future studies should examine the effect of different redundant display
configurations. We hypothesized that the improved communication scores in redundant
configurations was the result of not being able to see one’s operating partner. If this is correct,
then there should be a decrease in communication as the redundant displays are moved closer
together. Future studies should also assess the impact of these metrics on performance and team-
level situation awareness. It is possible that there are negative effects of assuming team-wide
situation awareness and that it is actually better to force the operators to verbally communicate
with one another. It should be noted that communications within nuclear are highly regulated,
and therefore with expert operators there may be less variation resulting from GVD
configurations. Therefore, future studies should also look at communication differences elicited
by GVD alternatives in fields with less regulated communication behaviour.
Again, despite the fact that design was not one of the principal concerns of this study, the tool
that was created proved to be very useful. Future research should examine the utility of this tool
over time. The tool appears to be a useful method of displaying information at the abstract
function level and we believe it can be used in a wide array of domains, particularly, in domains
where control actions are necessary.
85
References
Allison, P. D. (2008). Convergence Failures in Logistic Regression. SAS Global Forum, (5), 1–
11.
Beltracchi, L. (1988). An Expert Display System and Nuclear Power Plant Control Rooms. IEEE
Transactions on Nuclear Science, 35(2), 991–1000.
Bennett, K. B., & Malek, D. A. (2000). Evaluation of alternative waveforms for animated mimic
displays. Human Factors, 42(3), 432–450. doi:10.1518/001872000779698114
Bereznai, G. T., & Harvel, G. (2011). Workshop on NUCLEAR POWER PLANT SIMULATORS:
INTRODUCTION TO CANDU Systems and Applications.
Burns, C. M., & Hajdukiewicz, J. R. (2004). Ecological Interface Design. Boca Raton, Florida:
CRC Press.
Burns, C. M., Skraaning, G., Jamieson, G. a, Lau, N., Kwok, J., Welch, R., & Andresen, G.
(2008). Evaluation of ecological interface design for nuclear process control: situation
awareness effects. Human Factors, 50(4), 663–679. doi:10.1518/001872008X312305.
Butcher, K. R. (2006). Learning from text with diagrams: Promoting mental model development
and inference generation. Journal of Educational Psychology, 98(1), 182–197.
doi:10.1037/0022-0663.98.1.182
Candu Energy Inc. (2014). Main Control Room Mock-up Candu Energy.
Christoffersen, K., Hunter, C. N., & Vicente, K. J. (1996). A Longitudinal Study of the Effects of
Ecological Interface Design on Skill Acquisition. Human Factors, 38(3), 523–541.
Cornelissen, M., Salmon, P. M., Jenkins, D. P., & Lenne, M. G. (2013). How can they do it? A
structured approach to the strategies analysis phase of Cognitive Work Analysis.
Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 55(1), 340–
344. doi:10.1177/1071181311551070
Davey, E. (2000). Process Monitoring during Normal Operations at Canadian Nuclear Power
86
Plants. Proceedings of the Human Factors and Ergonomics Society Annual Meeting,
44(22), 811–814. doi:10.1177/154193120004402282
Elix, B., & Naikar, N. (2008). Designing safe and effective future systems: A new approach for
modelling decisions in future systems with Cognitive Work Analysis. Proceedings of the
8th International Symposium of the Australian Aviation Psychology Association.
Endsley, M. R. (1995). Toward a Theory of Situation Awareness in Dynamic Systems. Human
Factors, 37(1), 32–64. doi:10.1518/001872095779049543
Gary, M. S., & Wood, R. E. (2011). Mental Models, Decision Rules, and Performance
Heterogeneity. Strategic Management Journal, 32, 569–594. doi:10.1002/smj
Gwet, K. L. (2014). Handbook of Inter-Rater Reliability (4th ed.). Gaithersburg, MD: Advanced
Analytics, LLC.
High, R. (2014). Plotting Differences among LSMEANS in Generalized Linear Models, 1902–
2014. Retrieved from
https://sasglobalforum.activeevents.com/connect/fileDownload/session/7F75189251321B22
63401619B4AB46FB/1902_High_FinalPaper.pdf\nhttp://support.sas.com/resources/papers/
proceedings14/1902-2014.pdf
IAEA. (2009). Instrumentation and Control ( I & C ) Systems in Nuclear Power Plants : A Time
of Transition.
Infosystems, K. (2014). RGB Spectrum – Kiran Infosystems. Retrieved March 4, 2016, from
http://www.kiraninfo.com/rgb-spectrum/
Jamieson, G. A., & Vicente, K. J. (2001). Ecological interface design for petrochemical
applications: Supporting operator adaptation, continuous learning, and distributed,
collaborative work. Computers and Chemical Engineering, 25(7-8), 1055–1074.
doi:10.1016/S0098-1354(01)00678-0
Jamieson, G., & Miller, C. (2007). Integrating task-and work domain-based work analyses in
ecological interface design: a process control case study. Systems, Man and …, 37(6), 887–
905. Retrieved from http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=4344953
87
Jenkins, D. P., Stanton, N. A., Salmon, P. M., Walker, G. H., & Rafferty, L. (2010). Using the
Decision-Ladder to Add a Formative Element to Naturalistic Decision-Making Research.
International Journal of Human-Computer Interaction, 26(2-3), 132–146.
doi:10.1080/10447310903498700
Johansson, B. J. E., & Persson, P. A. (2009). Reduced uncertainty through human
communication in complex environments. Cognition, Technology and Work, 11(3), 205–
214. doi:10.1007/s10111-007-0108-6
Joyce, J. P., & Lapinsky, G. W. (1983). A History and Overview of the Safety Parameter Display
System Concept. IEEE Transactions on Nuclear Science, NS-30(1), 744–749.
Juhasz, M., & Soos, J. K. (2007). Impact of non-technical skills on NPP Teams’
performance: Task load effects on communication. Human Factors and Power Plants and
HPRCT 13th Annual Meeting, 2007 IEEE 8th, 225–232. doi:10.1109/hfpp.2007.4413210
Kilgore, R. M., St-Cyr, O., & Jamieson, G. A. (2008). From Work Domains to Worker
Competencies: A Five-Phase CWA. Applications of Cognitive Work Analysis, 15–48.
doi:10.1201/9781420063059.ch2
Kim, D. Y., & Kim, J. (2014). Journal of Nuclear Science and Technology How does a change in
the control room design affect diagnostic strategies in nuclear power plants ? Journal of
Nuclear Science and Technology, (January 2015), 37–41.
doi:10.1080/00223131.2014.923792
Lang, A. W., Roth, E. M., Bladh, K., & Hine, R. (2002). Using a Benchmark-Referenced
Approach for Validating a Power Plant Control Room: Results of the Baseline Study.
Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 46(23), 1878–
1882. doi:10.1177/154193120204602302
Lau, N. (n.d.). Re: A few More Questions.
Lau, N., Jamieson, G. A., & Skraaning Jr., G. (2014). Inter-rater reliability of query/probe-based
techniques for measuring situation awareness. Ergonomics. Taylor & Francis.
doi:10.1080/00140139.2014.910612
88
Lau, N., Jamieson, G. A., Skraaning Jr., G., & Burns, C. M. (2008). Ecological Interface Design
in the Nuclear Domain: An Empirical Evaluation of Ecological Displays for the Secondary
Subsystems of a Boiling Water Reactor Plant Simulator. IEEE Transactions on Nuclear
Science, 55(6), 3597–3610. doi:10.1109/TNS.2008.2005725
Lau, N., Skraaning Jr., G., Eitrheim, M. H. R., Karlsson, T., Nihlwing, C., & Jamieson, G. A.
(2011). Situation Awareness in Monitoring Nuclear Power Plants – The Process Overview
Concept and Measure. Toronto, Ontario.
Lau, N., Veland, Ø., Kwok, J., Jamieson, G. A., Burns, C. M., Braseth, A. O., & Welch, R.
(2008). Ecological Interface Design in the Nuclear Domain : An Application to the
Secondary Subsystems of a Boiling Water Reactor Plant Simulator, 55(6), 3579–3596.
Lee, H.-C., & Seong, P.-H. (2009). A computational model for evaluating the effects of
attention, memory, and mental models on situation assessment of nuclear power plant
operators. Reliability Engineering & System Safety, 94(11), 1796–1805.
doi:http://dx.doi.org/10.1016/j.ress.2009.05.012
Lin, C. J., Yenn, T.-C., & Yang, C.-W. (2010). Evaluation of Operators’ Performance for
Automation Design in the Fully Digital Control Room of Nuclear Power Plants. Human
Factors and Ergonomics in Manufacturing, 20(1), 10–23. doi:10.1002/hfm
Maddox, M. E. (1996). Critique of “A longitudinal study of the effects of ecological interface
design on skill acquisition” by Christoffersen, Hunter, and Vicente. Human Factors, 38(3),
542–546. Retrieved from
http://go.galegroup.com.myaccess.library.utoronto.ca/ps/i.do?id=GALE%7CA19027801&si
d=googleScholar&v=2.1&it=r&linkaccess=fulltext&issn=00187208&p=AONE&sw=w
Meshkati, N. (1998). Lessons of Chernobyl and Beyond: Creation of the Safety Culture in
Nuclear Power Plants. Proceedings of the Human Factors and Ergonomics Society Annual
Meeting, 42(10), 745–749. doi:10.1177/154193129804201019
Montgomery, J., Gaddy, C., & Toquam, J. (1991). Team interaction skills evaluation criteria for
nuclear power plant control room operators. In Proceedings of the Human Factors Society
35th Annual Meeting (Vol. 15, pp. 918–922).
89
Mumaw, R. J., Roth, E. M., Vicente, K. J., & Burns, C. M. (2000). There Is More to Monitoring
a Nuclear Power Plant than Meets the Eye. Human Factors: The Journal of the Human
Factors and Ergonomics Society, 42, 36–55. doi:10.1518/001872000779656651
Murch, G. M. (1984). Physiological priniciples for the effective use of color. IEEE Computer
Graphics and Applications, 4(11), 49–54.
Myers, W. P., & Jamieson, G. A. (2014). Operating Experience Review of Large-Screen
Displays in Nuclear Power Plant Control.
Myers, W. P., & Jamieson, G. A. (2015). RETHINKING GROUP-VIEW DISPLAY
ALTERNATIVES, 2570–2577.
Nuclear Energy Agency. (2010). Public attitudes to nuclear power.
O’Hara, J. (2004). Plant Modernization Programs. Nuclear Plant Journal, 22(2), 47–48.
O’Hara, J. M., Brown, W. S., Lewis, P. M., & Persensky, J. J. (2002). Human-System Interface
Design Review Guidelines Human-System Interface Design Review Guidelines.
Washington, DC.
Rasmussen, J. (1974). The human data processor as a system component: Bits and pieces of a
model (Report No. Risø-M-1722). Roskilde, Denmark.
Rasmussen, J. (1985). The role of hierarchical knowledge representation in decisionmaking and
system management. IEEE Transactions on Systems, Man, and Cybernetics, SMC-15(2),
234–243. doi:10.1109/TSMC.1985.6313353
Rasmussen, J. (1993). Diagnostic reasoning in action. IEEE Transactions on Systems, Man and
Cybernetics, 23(4), 981–992. doi:10.1109/21.247883
Rasmussen, J., Pejtersen, A. M., & Goodstein, L. P. (1994). Cognitive systems engineering. New
York: Wiley.
Roth, E. M., Brockhoff, C. S., Rusnica, L. A., Kenney, S., Kerch, S. P., Thomas, V. M., &
Sugibayachi, N. (1997). Rapid prototyping and simulator evaluation of a wall panel
90
overview display. In IEEE 6th Annual Human Factors Meeting (Vol. 18, pp. 14–19).
Orlando. doi:10.1017/CBO9781107415324.004
Roth, E. M., Lin, L., Thomas, V. M., Kerch, S., Kenney, S. J., & Sugibayashi, N. (1998).
Supporting situation awareness of individuals and teams using group view displays.
Proceedings of the Human Factors and Ergonomics Society 42nd Annual Meeting, 1, pp.
244–248.
Roth, E., & O’Hara, J. (2002). Integrating digital and conventional human-system interfaces:
Lessons learned from a control room modernization program. Retrieved from
http://scholar.google.com/scholar?hl=en&btnG=Search&q=intitle:Integrating+Digital+and+
Conventional+Human-
System+Interfaces:+Lessons+Learned+from+a+Control+Room+Modernization+Program#0
Tharanathan, a., Bullemer, P., Laberge, J., Reising, D. V., & Mclain, R. (2012). Impact of
Functional and Schematic Overview Displays on Console Operators’ Situation Awareness.
Journal of Cognitive Engineering and Decision Making, 6(2), 141–164.
doi:10.1177/1555343412440694
Vicente, K. (1999). Cognitive Work Anlaysis: Toward Safe, Productive, and Healthy Computer-
Based WOrk. Mahwah, N.J.: Lawrence Erlbaum Associate.
Vicente, K. (2002). Ecological interface design: progress and challenges. Human Factors, 44(1),
62–78. doi:10.1518/0018720024494829
Vicente, K., & Rasmussen, J. (1992). Ecological interface design: Theoretical foundations. IEEE
Transactions on Systems, Man, and Cybernetics, 22(4), 589–606. doi:10.1109/21.156574
Vicente, K., Roth, E. M., & Mumaw, R. J. (2001). How do operators monitor a complex,
dynamic work domain? The impact of control room technology. International Journal of
Human-Computer Studies, 54(6), 831–856. doi:http://dx.doi.org/10.1006/ijhc.2001.0463
World Nuclear Association. (2014). History of Nuclear Energy. Retrieved January 8, 2016, from
http://www.world-nuclear.org/info/Current-and-Future-Generation/Outline-History-of-
Nuclear-Energy/
91
World Nuclear Association. (2016). Nuclear Power Today | Nuclear Energy. Retrieved January
8, 2016, from http://www.world-nuclear.org/info/Current-and-Future-Generation/Nuclear-
Power-in-the-World-Today/
Zhang, Y. (2008). The influence of mental models on undergraduate students’ searching
behavior on the Web. Information Processing and Management, 44(3), 1330–1345.
doi:10.1016/j.ipm.2007.09.002
92
Appendices
Appendix A. Information Requirements
HEATTRANSPORTSYSTEMLevel Box InformationRequirements
Reactivity PoweroutputTemperature???SteamGenerators Levelcontrol Pressurewithin Steamwithin H2OIN/OUT D2OIN/OUT Steamout BlowdownRate
D2OCoolant TotalmassReactorInletheaders Pressureatheaders Volumethrough ratethroughReactoroutletheaders Pressureatheaders Volumethrough Ratethrough
FlowthroughPressurewithinHeatgeneration EnthalpyinsystemFuelCooling Efficiencyofexchangesteamgeneration QuantityofsteamgeneratedCoolantCirculation Massdistribution TotalvolumeCavitationprevention Pressuredistribuiton
Fuelin PoweroutFlowpumpenergy Massthrough
Massout HeatoutTotalmassinsystem
TotalSystemEnthalpy Totalsystempressure(~10Mpa)CoolantCirculation VolumeincirculationMassfromP&IC InputRateMasstoP&IC OutputRate
AveragetemperaturePhysicalFunction
GeneratlizedFunction
AbstractFunction
Reactivity
Averagetemperature
CoolantflowSteamGeneratorSink
HTPumps
Reactor
Pressure&InventoryControlLevel Box InformationRequirements
DegasserCondenser Level BleedRate FeedRate Steambleedrate Pressure TemperatureOpen/Close Flowthrough PressureatTemperature Tempbefore/after Flowthrough
steamD2Obleedvalves Open/Close FlowthroughShutdownCoolingSystem
PressureInside Pressureoutside Levelinside Steamlevelinside Inputrate outputrateLiquidD2Obleedvalves Open/Close FlowThrough
FeedrateOpen/Close Flowthrough
D2OStoragetank Level Input OutputPurificationcircuit FissionproductlevelsinD2O
LiquidD2OtemperaturereductionCondenseD2OSteam SteamD2Olevel
SteamD2OcoolingFlowthrough
D2OtemperatureincreaseMaintainPZRpressureatsetpoint PZRPressureLCwithincondenserPressureabsorptionfromD2OswellCoolantinsertionExcesscoolantremovalAcceptexcessinventoryPreventdrainingPreventliquidoverflowRemoveradioactivefissionproductsformD2O
Pressureloss HeatlossHeatloss
EnergyinLiquidD2OEnergyinsteamD2O
HeatinCoolantinflow
MassrecycleMassfromHTS Volumein
DegasserCooler
FeedValves
Refluxflow
Cooling
Heaters
GeneralizedFunction
AbstractFunction
Physicalfunction
TempbeforeandafterTotalsystempressure
VolumeinstoragetankRadioactivityofD2O
LiquidD2OsteamlevelD2Otemperature
Tempbeforeandafter
Temperaturebefore/after
VolumeofD2Oincondenser
Feedpumps
Pressurizer
refluxvlaveheaters
VolumeofcoolantinsystemVolumeofcoolantinsystem
Volumeinstoragetank
D2OstorageD2OtoHTS
TemperaturebroughtinfromD2O
RadioactivityofD2OcontentsD2Opurity
PressureofD2OTemperatureinD2O
TotalvolumeinstorageTotalvolumecyclinginthesystem
VolumeofinventorycirculatingintheHTS
VolumeinsystemVolumeinsystemVolumeinstoragetank
TemperatureinD2O
Spray
PressureofD2O
93
Table2.3.BOILERANDSTEAMGENERATORLevel Box InformationRequirements
Percentfull TemperaturePressurewithin
Levelwithin Presurewithin
Feedwaterinlet Pressureat Flowthrough
Temperature
Capacity
Steamscrubbers Integrity Moistersaturation
Cycloneseparators On/off Rotationrate
Downcomerannulus Level
Level PercentFull
Heavywaterinlet Pressureat Flowrate
U-Tube Pressureat Flowrate
Heavywateroutlet Pressureat FlowRate
HeatExchange Heatexchangerate
Lightwatervaporization TemperatureofD2O TemperatureofH2O PressureinBSG
Lightwatercirculation Overalflowrate Pressurewithintheboiler TemperatureofH2OTemperatureatreheaterdrainsinlet
Impurityremoval Removalrate
Moistureremoval Moistercontentbefore/after
Waterrecycling Flowthroughrecyclingcomponents
Steamrelease Outflowrate Presssureatsteamoutletnozzle
Heavywatercirculation Overalflowrate
Inputheavywaterenergy
Inputlightwaterenergy
Heavywaterenergy(withinboiler)
Lightwaterenergy(withinboiler)
Steamenergy(withinboiler)
Outputsteamenergy
Outputheavywaterenergy
Inputlightwatermass Inputrate
Lightwatermass(withinboiler)
Steammass(withinboiler)
Waterextractedfromsteammass
Outputsteammass
Outputlightwatermass
PressurePrinciples Pressure-temperature-boilingpointrelationship
Inputheavywatermass
Heavywatermass(totalinsystem)
Outputheavywatermass
DeliverrequiredquantityofsteamtoMainsteamsystemTargetsteamquantity Currentsteamquantity
RemoveheatfromHTS Targetheatremoval Currentheatremoval
PreventoverpressurizationofSG Pressurelimit Currentpressure
Abstract
Function
EnergyBalance
Lightwatermassbalance
Heavywatermassbalance
Functional
Purpose
Physical
Function
Generalized
Function
Steamoutlet
Blowdown
Preheater
Tubebundle
Risersection
94
Level Box InformationRequirements NotesGenerator Generatoroutput 680MWeLPTurbine Flowrate,output kg/h,MWHPTurbine Flowrate,output kg/h,MWEmergencyStopValve Open/closeGovernorvalve %openSafetyValve Open/close Combinedreliefcapacityof3/4is115%ofsteamflowfromeachSGASDVs Open/closeCSDVs Open/closeCondenserandhotwell CondenserlevelMoistureseparatorHPThrottlevalves steamflowthrough 957kg/stotheturbinethrottlevalvesSteamreheater livesteamflowtothesteamreheater 90kg/slivesteamtothesteamreheaterConvertlatentheatofsteamtomechanicalenergyEntropy(T,Pofsteamenteringtheturbines) steamenteringat250˚Cand4000kPa(density-.;05m^3/kg),leavingat35˚Cand5kPa(Density-25.2m^3/kg)Steamquantitycontrol flowthroughthegovernorvalvesandASDVs PositionofgovernorvalvessetbycombinationofspeedersettingandfrequencyerrorRapidlycutflowtoturbine Emergencystopvalvesandinterceptvalvesclose-openstatusCondensatecreation Pincondenser ~5kPa(absolutepressure),35˚C,10%moistureCondensatecollection T,Pofsteam Condensedsteamat5kPaiscollectedinthehotwellMoistureremoval %moisture/moistureremovalrate Steamleavesturbineat900kPa,170˚Cand10%moisture
Designedtoachievefullliftat4%abovesetpressuresSteamheatenergy energytransferfromtheteamgeneratorstothehpturbineSystempressure systempressureacrosscomponentsElectricaloutput enthalpy(energytransferfromtheLPturbinestothegenerator)
Atm.PressuredischargeenergyreleasetotheatmospherethroughASDVsSteamfromSG SteammassenteringfromSGDryvapormass DryvapormassWetVaporMass WetVaporMassPooledcondensate CondenstaemassFeedwatersystem MassinfeedwatersystemAtmosphericdischargesteamflowthroughtheASDVsifany
Maintainpressure:temp.ratio.Steaminjectionheat Flowin/outofreheaterMitigatepotentialfordamagetothesystemASDVs Potentiallyfaultysystems&componentsConvertheatintomechanicalenergyforthegeneratorGeneratoroutput,turbinespeedanderrorProvideCondensatetotheLPfeedwatersystemOutputrate
EnergyBalance
MassbalanceAbstractFunction
Functionalpurpose
total=1047kg/s
PhysicalFunction
Generalizedfunction
Table2.4.STEAMSUPPLYSYSTEM
Level Box InformationRequirements NotesGlandseal Integrity PotentiallygeneralIRforsystemintegrity?Diaphragm Wear,integrityCasing(upper)Casing(lower)ShaftBlades RotationalcapacityBearingsFixedbladesLowpressureoutlets PressureExtractionsteam(pump) PumpcapacityCondenser Coolingarea(sqm)Frictionprevention TempofbearingsLeakageprevention Pressuredrop(deviationfromnormalvalues)Steam-drivenshaftrotation RPM(revolutionperminute)Heattransfertohighvelocitykineticenergy P,TofheatTransfersteamtolowpressureturbines Outletpressure OutputrateExhaustlowpressuresteamtocondenser PressureRemovemoisturecontentfromturbines Moisture%Vibration mm/s
Shaftrotation TorqueLeakage Energybalance(conservationofenergy)/mass,energyHeatenergy InflowSteamenergy SteamtransferratetoturbinesRecycleenergyRotationalenergy MomentumSteaminflow SteamflowratefrombalanceheaderRecycledsteam Flowrate(kg/s)Condensate(wetsteam)Transferratetothecondenser
Content Moisture MoistureremovalrateConvertsteampressuretorotationalenergyCurrentMWoutputvssetpointPreventsystemdamage SystempressureMaximizeefficiencyofrotation CurrentRPMvsgoal
PhysicalFunction
GeneralizedFunction
Massbalance
Energybalance
AbstractFunction
Table2.5.TURBINECONTROL
Functionalpurpose
95
Appendix B. Experimental Schedule
TRIALNO.Scenario Setup Scenario Setup Scenario Setup
1 1 1 2 3 3 12 2 2 3 4 4 23 3 3 4 1 1 34 4 4 1 2 2 4
TRIALNO.Scenario Setup Scenario Setup Scenario Setup Scenario Setup Scenario Setup
1 4 3 4 4 1 2 2 4 3 22 1 4 1 1 2 3 3 1 4 33 2 1 2 2 3 4 4 2 1 44 3 2 3 3 4 1 1 3 2 1
SCENARIOS Setups1 HTSLeak 1 QinshanLSD2 FWvalveclosure 2 QinshanRedundant3 SpuriousMSSVOpening 3 EIDLSD4 PZRBleedpathClosure 4 EIDRedundant
Team4 Team5 Team6 NOV.5 NOV.6
OCT.26 PILOT OCT.28 Team2 Team3
96
Appendix C. Process Overview Queries
HTSLeak
Time 11m
Decreased Increased Inferred?
X
X
X
X *
X
X
X
X *
X *
X *
X
X
Time 20:00
Decreased Increased
X
X
X
X
X *
X
X
X
X *
X *
X *
X
Stayedthe
Same
Recently,themass(kg)leavingtheprimaryheattransportcircuithas
Recently,thedifferencebetweenreactorpowerandturbinepower(%FP)has
Recently,themassinsidetheD2Ostoragetank(kg)has
Recently,themassinsidetheperssurizer(kg)has
Recently,thedegasser-condenserlevel(m)has
Recently,thepressurizerpressure(MPa)has
Recently,theHTSpressure(MPa)has
Recently,theheatouputfromthepressurizer'sheaters(kj/s)has
Recently,theturbineoutput(MW)has
Recently,thesteamgeneratorpressure(MPa)has
Recently,thepressurizerlevel(m)has
Recently,theD2OStorageTanklevel(m)has
Stayedthe
Same
Recently,theD2OStorageTanklevel(m)has
Recently,thepressurizerlevel(m)has
Recently,themassinsidetheperssurizer(kg)has
Recently,theheatouputfromthepressurizer'sheaters(kj/s)has
Recently,thedifferencebetweenreactorpowerandturbinepower(%FP)has
Recently,themass(kg)leavingtheprimaryheattransportcircuithas
Recently,thesteamgeneratorpressure(MPa)has
Recently,theturbineoutput(MW)has
Recently,themassinsidetheD2Ostoragetank(kg)has
Recently,theHTSpressure(MPa)has
Recently,thepressurizerpressure(MPa)has
Recently,thedegasser-condenserlevel(m)has
97
FEEDWATERVALVECLOSURETime 8:45
Decreased Increased
X
X
X
X
X
X
X
X
X
X
X
Recently,thevalveopeningtosteamgenerator1(%)has X
Time 17:00Decreased Increased
X
X
X
X
Recently,thevalveopeningtosteamgenerator1(%)has X
X
X
X
X
X
X
X
Recently,thelevelinsteamgenerator4(m)has
StayedtheSame
Recently,theTurbineOutput(MW)has
Recently,thedifferencebetweenthesteamgeneratorlevels(m)has
Recently,thelevelinsteamgenerator1(m)has
Recently,theaveragesteampressure(KPa)has
Recently,thefeedflowintosteamgenerator4(kg/s)has
Recently,theaveragesteamflowleavingthesteamgenerators(kg/s)has
Recently,thesteamgeneratorfeedpumppressurehas
Recently,thePressurizer(m)levelhas
Recently,thelevelinsteamgenerator2(m)has
Recently,theaveragemassinthesteamgeneratorshas
Recently,thePressurizerlevel(m)has
Recently,thelevelinsteamgenerator2(m)has
Recently,theaveragemassinthesteamgeneratorshas
Recently,thelevelinsteamgenerator4(m)has
StayedtheSame
Recently,theTurbineOutput(MW)has
Recently,thediffernecebetweenthesteamgeneratorlevels(m)has
Recently,thelevelinsteamgenerator1(m)has
Recently,thesteampressure(KPa)has
Recently,thefeedflowintosteamgenerator4(kg/s)has
Recently,thesteamflowleavingthesteamgenerators(kg/s)has
Recently,theaveragesteamgeneratorfeedpumppressurehas
98
MSSVOpeningTime11m Decreased Increased
X
X
X
X
X
X
X
X
X
X
X
X
Time20m Decreased Increased
X
X
X
X
X
X
X
X
X
X
X
X
StayedtheSame
Recently,theTurbineOutput(MW)has
Recently,theaveragelevelinthesteamgenerators(m)has
Recently,theaverageflowoffeedwaterintothesteamgenerators(kg/s)has
Recently,themassinthecondensatestoragetank(kg)has
Recently,thepressureofthesteam(MPa)leavingtheturbineshas
Recently,thelevelinthecondensatestoragetank(m)has
Recently,thePressurizerlevel(m)has
Recently,theHTSpressure(MPa)has
Recently,thesteambeingdischargedtotheatmosphere(Kg/s)has
Recently,thesteamflowleavingsteamgenerator1(kg/s)has
StayedtheSame
Recently,theTurbineOutput(MW)has
Recently,theaveragelevelinthesteamgenerators(m)has
Recently,theaveragepressureinthesteamgenerators(MPa)has
Recently,thepressureofthesteam(MPa)leavingtheturbineshas
Recently,theaveragepressureinthesteamgenerators(MPa)has
Recently,thedifferencebetweenreactorpowerandturbinepower(%)has
Recently,thedifferencebetweenreactorpowerandturbinepower(%)has
Recently,theaverageflowoffeedwaterintothesteamgenerators(kg/s)has
Recently,thesteamflowleavingsteamgenerator1(kg/s)has
Recently,thesteambeingdischargedtotheatmosphere(Kg/s)has
Recently,theHTSpressure(MPa)has
Recently,thePressurizerlevel(m)has
Recently,thelevelinthecondensatestoragetank(m)has
Recently,themassinthecondensatestoragetank(kg)has
99
PZRBleedClosureTime 10:30
Decreased Increased
Recently,theD2OStorageTanklevel(m)has X
Recently,thedegasser-condenserlevel(m)has X
Recently,theflowleavingthepressurizer(kg/s)has X
Recently,theheatoutputfromthepressurizer'sheaters(Kj/s)has X
Recently,theHTSpressure(MPa)has X
Recently,themassinsidetheD2OStorageTank(kg)has X
Recently,themassinsidethepressurizer(kg)has X
Recently,thepressurizerlevel(m)has X
Recently,thepressurizerpressure(MPa)has X
Recently,thesteamgeneratorlevel(m)has X
Recently,thesteamgeneratorpressure(KPa)has X
Recently,theturbineoutput(MW)has X
Time 18:00Decreased Increased
Recently,theturbineoutput(MW)has X
Recently,thesteamgeneratorpressure(KPa)has X
Recently,thesteamgeneratorlevel(m)has X
Recently,thepressurizerpressure(MPa)has X
Recently,thepressurizerlevel(m)has X
Recently,themassinsidethepressurizer(kg)has X
Recently,themassinsidetheD2OStorageTank(kg)has X
Recently,theHTSpressure(MPa)has X
Recently,theheatoutputfromthepressurizer'sheaters(Kj/s)has X
Recently,theflowleavingthepressurizer(kg/s)has X
Recently,thedegasser-condenserlevel(m)has X
Recently,theD2OStorageTanklevel(m)has X
StayedtheSame
StayedtheSame
100
Appendix D. Selected Statistical Outputs
a) GLIMMIX output comparing visible to inferred queries.
0 refers to visible queries, while 1 refers to inferred.
b) LS mean comparisons for PO scores.
101
Appendix E. Selected SAS Code
a) Mixed Generalized Linear model for SA proc GLIMMIX data=subresults plots=anomplot plots=controlplot plots=diffplot ic=Q; class LR AE LRAE SET CON GID SID trial quesno; model score = LR AE LR*AE/ dist=binomial link=logit oddsratio solution; random con; random GID; random SID(GID); random trial; lsmeans AE LR AE*LR/cl ilink adjust=sim plot=diffplot plot=meanplot(plotby=LR join ilink) ; ods output lsmeans=lsmeans; output out=glimresid (drop=TID QQ QUESNO INF FLAG CQQ) pred=pred2 / allstats; run;
b) Mixed Linear Model for Communication
proc mixed data=commdat plots=none cl; class set trial lr AE con GID; model comsqrt = LR|AE; random con/ s; random GID /s; random trial/ s; lsmeans LR|AE/pdiff cl; ods output lsmeans=lsmeansout2; run;
c) Mixed Linear Model for Performance
proc mixed data=perfdata2 plots=none; class TID GID SID LR AE LRAE CON; model IDSCORE=LR|AE; Random con; Random GID; Random SID(GID); lsmeans LR|AE/pdiff cl; ods output lsmeans=pertable2; run;