wiki.cis.unisa.edu.au€¦ · Web viewThis dissertation presents a study of the appropriate...
-
Upload
vuongxuyen -
Category
Documents
-
view
212 -
download
0
Transcript of wiki.cis.unisa.edu.au€¦ · Web viewThis dissertation presents a study of the appropriate...
Visual Cues for the Instructed Arrangement of Physical Objects Using Spatial Augmented Reality (SAR)JESSICA A. TSIMERIS
A thesis submitted for the degree of Bachelor of Computer Science (Honours)
School of Computer and Information Science | University of South Australia
Supervisor: Professor Bruce H. Thomas | [email protected]
November 2010
Disclaimer
I declare that:
this thesis presents work carried out by myself and does not incorporate without
acknowledgment any material previously submitted for a degree or diploma in any
university, and
to the best of my knowledge this thesis does not contain any materials previously published
or written by another person except where due reference is made in the text; and all
substantive contributions by others to the work presented, including jointly authored
publications, is clearly acknowledged.
Adelaide, October 2010.
Jessica Anne Tsimeris
i
Acknowledgements
My honours year has been incredibly challenging but the fulfilment that I have felt this year has
been amazing. I know that I could not have completed this alone.
I would like to thank Bruce Hunter Thomas and the members of the Wearable Computer Lab at the
University of South Australia for helping me with new concepts, giving me timely advice, and
providing a welcomed distraction at times.
I would also like to thank my friends and family, for their unrelenting support that has continued
for the duration of my degree.
ii
Abstract
This dissertation presents a study of the appropriate visual cues to instruct a user on how to arrange
physical objects using Spatial Augmented Reality. Spatial Augmented Reality (SAR) systems
enhance a user’s physical environment via the integration of virtual objects into the physical
environment using projectors. SAR differs from traditional Augmented Reality (AR) systems
which typically enhance a user’s environment by utilising a head mounted device or by using a
mobile display. The SAR visual cues developed overcome a number of the current limitations
found in the area of object manipulation using AR and SAR systems; a major limitation is that
object manipulation using AR and SAR has been largely focused on virtual objects. The research
presented in this dissertation addresses this limitation as it provides a method for directly arranging
physical objects. In order to instruct a user on how to arrange physical objects using SAR, two
instructions are defined to arrange a physical object to a position; rotation and translation. A set of
SAR visual cues have been developed which utilise these two instructions or variations of them. A
top down arrangement application has been developed in which the user arranges representations of
physical objects from a top down view and the resulting arrangement can then be performed on the
physical objects using SAR visual cues. This process can be undertaken locally or remotely, in real
time by one user or as a collaboration task and the process can also be stored to provide
instructions on demand at a later date.
A user study was undertaken to determine the effectiveness of these SAR visual cues when
compared to a manual arrangement method and when compared to one another. The results of this
user study determined that SAR visual cues have been developed that are faster and more accurate
than a manual object arrangement method. A revised visual cue has been developed by taking into
account the results of the user study, and a pilot study regarding this new visual cue has indicated
speed and accuracy improvements. Furthermore, the SAR visual cues developed have been shown
to facilitate the arrangement of physical objects of a variety of sizes, including large objects. Using
SAR in this way could be useful for the arrangement of large objects, such as furniture or theatre
sets, and when multiple objects are being arranged. The SAR visual cues in isolation are not a
unique innovation, but in conjunction with the context in which the cues are being used – to
instruct users on how to arrange a physical object using SAR, a unique approach is presented.
iii
Table of Contents
1 Introduction...............................................................................................................................1
1.1 Motivation.......................................................................................................................2
1.2 Research Question...........................................................................................................4
1.3 Scope...............................................................................................................................4
1.4 Contribution.....................................................................................................................5
1.5 Research Structure...........................................................................................................5
1.6 Dissertation Structure......................................................................................................7
2 Background...............................................................................................................................9
2.1 Classification.................................................................................................................10
2.2 Benefits..........................................................................................................................11
2.3 Object Manipulation......................................................................................................13
2.4 User Interaction.............................................................................................................15
2.5 Tracking.........................................................................................................................17
3 SAR Object Arrangement Techniques.................................................................................19
3.1 Interaction......................................................................................................................19
3.2 Object Arrangement Techniques...................................................................................20
3.2.1 Translation Instruction...........................................................................................21
3.2.2 Rotation Instruction...............................................................................................23
3.2.3 SAR Visual Cues...................................................................................................25
3.2.3.1 Translation First Cue.........................................................................................27
3.2.3.2 Rotation First Cue.............................................................................................30
3.2.3.3 Both Instructions Cue........................................................................................30
3.2.3.4 Circle Cue.........................................................................................................32
3.2.3.5 Wireframe Cue..................................................................................................35
3.2.3.6 Square Cue........................................................................................................38
3.2.4 Discussion..............................................................................................................42
iv
3.2.4.1 Projection upon Tracking Markers...................................................................42
3.2.4.2 Distortion..........................................................................................................43
4 User Study...............................................................................................................................46
4.1 Configuration.................................................................................................................46
4.2 Tasks Undertaken for the User Study............................................................................47
4.3 Structure........................................................................................................................50
4.4 Data Gathering...............................................................................................................50
4.4.1 Quantitative Data Gathering..................................................................................50
4.4.2 Qualitative Data Gathering....................................................................................51
4.4.3 Manual Timer and SAR Timer Data Gathering....................................................52
5 Results......................................................................................................................................54
5.1 User Information...........................................................................................................54
5.2 Quantitative Data...........................................................................................................54
5.2.1 Task Completion Times........................................................................................55
5.2.2 Accuracy................................................................................................................56
5.3 Qualitative Data.............................................................................................................59
5.3.1 Intuitiveness Survey..............................................................................................59
5.3.2 User Opinion Survey.............................................................................................64
5.4 Discussion......................................................................................................................69
5.4.1 Notes on the Manual Timer and SAR Timer Results............................................70
6 Revised Visual Cue.................................................................................................................72
6.1 Analysis.........................................................................................................................72
6.2 Methodology and Implementation................................................................................73
6.3 Discussion......................................................................................................................77
7 Pilot Study...............................................................................................................................79
7.1 Structure........................................................................................................................79
7.2 Preliminary Results.......................................................................................................80
7.2.1 User Information...................................................................................................80v
7.2.2 Quantitative Data...................................................................................................80
7.2.2.1 Task Completion Times....................................................................................81
7.2.2.2 Accuracy...........................................................................................................82
7.2.3 Qualitative Data.....................................................................................................84
7.2.3.1 Intuitiveness Survey..........................................................................................84
7.2.3.2 User Opinion Survey.........................................................................................86
7.2.4 Discussion..............................................................................................................86
8 Large Object Arrangement....................................................................................................87
9 Implementation.......................................................................................................................95
9.1 Overall Structure...........................................................................................................95
9.2 Existing Components.....................................................................................................96
9.2.1 SAR System...........................................................................................................96
9.2.2 Tracking.................................................................................................................97
9.2.2.1 Infrared Tracking..............................................................................................98
9.3 Developed Components.................................................................................................98
9.3.1 SAR Module..........................................................................................................98
9.3.2 Top Down Arrangement Application..................................................................100
10 Conclusion.............................................................................................................................102
10.1 Future Work.................................................................................................................103
11 References..............................................................................................................................105
Appendix A User Study Survey.............................................................................................111
Appendix B Pilot Study Survey.............................................................................................117
Appendix C Data Tables for Included Results.....................................................................120
vi
Table of Figures
Figure 1 Research Structure................................................................................................................6
Figure 2 A Vase (Left), and the Vase Being Augmented by Projectors (Right) (Raskar, R et al.
2001)...............................................................................................................................10
Figure 3 The Reality-Virtuality Continuum (Drascic & Milgram 1996)..........................................11
Figure 4 Virtual Object Manipulation Using Physical Objects (Fjeld 1999)....................................14
Figure 5 The metaDESK System (Ullmer & Ishii 1997)..................................................................16
Figure 6 Object Arrangement Process...............................................................................................21
Figure 7 Translation Instruction: Starting Position...........................................................................22
Figure 8 Translation Instruction: Position After Part of the Translation has been Completed.........22
Figure 9 Translation Instruction: Final Position................................................................................23
Figure 10 Rotation Instruction: Starting Position..............................................................................24
Figure 11 Rotation Instruction: Position After Part of the Rotation has been Completed................24
Figure 12 Rotation Instruction: Final Position..................................................................................25
Figure 13 Translation First Cue: Translation (the Initial State)........................................................27
Figure 14 Translation Cue: Translation Implementation (the Initial State)......................................28
Figure 15 Translation First Cue: Rotation.........................................................................................29
Figure 16 Translation First Cue: Rotation Implementation..............................................................29
Figure 17 Both Instructions Cue: Scenario of Translation Performed First......................................31
Figure 18 Both Instructions Cue: Scenario of Rotation Performed First..........................................31
Figure 19 Both Instructions Cue: Implementation............................................................................32
Figure 20 Circle Cue: Translation (the Initial State).........................................................................33
Figure 21 Circle Cue: Translation Implementation (the Initial State)...............................................33
Figure 22 Circle Cue: Rotation..........................................................................................................34
Figure 23 Circle Cue: Rotation Implementation...............................................................................35
Figure 24 Wireframe Cue: Translation Implementation (the Initial State).......................................36
Figure 25 Wireframe Cue: Translation (the Initial State).................................................................37
Figure 26 Wireframe Cue: Rotation..................................................................................................37
Figure 27 Wireframe Cue: Rotation Implementation........................................................................38
Figure 28 Square Cue: Translation (the Initial State)........................................................................39
Figure 29 Square Cue: Translation Implementation (the Initial State).............................................40
Figure 30 Square Cue: Rotation........................................................................................................41
Figure 31 Square Cue: Rotation Implementation..............................................................................41
Figure 32 SAR Visual Cue Distortion on the Side of the Physical Object.......................................43
Figure 33 SAR Visual Cue Distortion from the Projector Position..................................................44vii
Figure 34 User Study Setup...............................................................................................................47
Figure 35 Projected Grid Utilised in Task 1......................................................................................48
Figure 36 The Grid used for Task 1 (left) and a Participant Undertaking Task 1 (right).................48
Figure 37 Average Completion Time for all User Study Tasks as Timed with the Manual Timer
and the SAR Timer.........................................................................................................55
Figure 38 Average x and y Axis Variation for all User Study Tasks................................................56
Figure 39 Average Rotation Variation of all User Study Tasks........................................................58
Figure 40 Translation Instruction Intuitiveness Diagram..................................................................59
Figure 41 Results of the Intuitiveness Survey Regarding the Translation Instruction......................60
Figure 42 Rotation Instruction Intuitiveness Diagram......................................................................60
Figure 43 Results of the Intuitiveness Survey Regarding the Rotation Instruction..........................61
Figure 44 Circle Cue Translation Instruction Intuitiveness Diagram...............................................62
Figure 45 Results of the Intuitiveness Survey Regarding the Circle Cue Translation......................62
Figure 46 Circle Cue Rotation Instruction Intuitiveness Diagram....................................................63
Figure 47 Results of the Intuitiveness Survey Regarding the Circle Cue Rotation..........................64
Figure 48 User Preference Between a Manual Object Arrangement and an Object Arrangement
Using SAR......................................................................................................................65
Figure 49 User Preference Between Translation First, Rotation First and Both Instructions Cue. . .65
Figure 50 User Preference between Translation First and Circle Cue..............................................66
Figure 51 User Preference Between Wireframe Cue and Wireframe Cue with No Destination......67
Figure 52 User Preference Between Square Cue (Task 8) and Square Cue with No Destination
(Task 9)...........................................................................................................................67
Figure 53 User Preference Between Square Cues (Task 8 and Task 9) and Wireframe Cues (Task 6
and Task 7)......................................................................................................................68
Figure 54 User Preference of Visual Cues that Utilise the Translation and Rotation Instructions...69
Figure 55 Revised Visual Cue: Translation.......................................................................................74
Figure 56 Revised Visual Cue: Translation Implementation............................................................74
Figure 57 Revised Visual Cue: Translation.......................................................................................75
Figure 58 Revised Visual Cue: Rotation...........................................................................................76
Figure 59 Revised Visual Cue: Rotation Implementation.................................................................76
Figure 60 Revised Visual Cue: Rotation...........................................................................................77
Figure 61 Average Completion Time of all User Study Tasks and the Revised Visual Cue (Manual
Timer and SAR Timer)...................................................................................................81
Figure 62 Average x Axis and y Axis Variation of all User Study Tasks and the Revised Visual
Cue..................................................................................................................................82
Figure 63 Average Rotation Variation of all User Study Tasks and the Revised Visual Cue..........83
viii
Figure 64 Revised Visual Cue Translation Instruction Intuitiveness Diagram.................................85
Figure 65 Revised Visual Cue Rotation Instruction Intuitiveness Diagram.....................................85
Figure 66 SAR Visual Cue Distortion...............................................................................................88
Figure 67 Large Object Arrangement: Initial Position......................................................................90
Figure 68 Large Object Arrangement: After a Translation...............................................................91
Figure 69 Large Object Arrangement: Translation Complete...........................................................92
Figure 70 Large Object Arrangement: Rotation................................................................................93
Figure 71 Large Object Arrangement: Successfully Arranged.........................................................94
Figure 72 Overall Implemented System Structure............................................................................95
Figure 73 Top Down Arrangement Application Screenshot...........................................................100
ix
1 Introduction
Azuma et al. (2001) describe Augmented Reality by defining an AR system as being a system that
enhances the user’s real world with virtual objects that appear to exist in the real world. According
to Azuma et al. (2001), AR systems:
blend real and virtual objects in a real environment,
are real time interactive, and
involve three dimensional registration [thus, the real and virtual objects are aligned, which
can be achieved through tracking] (Azuma et al. 2001, p.1).
Raskar et al. (2005) differentiate between Virtual Reality (VR) and AR by stating that in AR, the
user’s real world environment is not suppressed as it traditionally is in VR. Spatial Augmented
Reality (SAR) (Bimber & Raskar 2005) systems enhance a user’s physical environment via the
integration of virtual objects into the physical environment using projectors (Raskar, R et al. 2001).
SAR differs from traditional Augmented Reality (AR) (Azuma, RT 1997; Azuma, R et al. 2001)
systems which typically enhance a user’s environment by utilising a head mounted device or by
using a mobile display. SAR has been used in the past for object illumination (Raskar, R et al.
2001) and AR can be used for annotating objects (Feiner, Macintyre & Seligmann 1993). The
research described in this dissertation utilises a combination of both to produce SAR visual cues to
instruct a user on how to arrange tracked physical objects.
To further assist the arrangement process, a top down arrangement application has been developed
from which SAR visual cues for physical object manipulation can be generated. The user arranges
representations of physical objects, and the resulting arrangement is performed on the physical
objects using SAR visual cues.
As a means of illustrating the concept of this research, the following example is provided. A
teacher would like to arrange several furniture objects within a classroom; desks, whiteboards and
chairs. The SAR visual cues described within this dissertation can aid with this task. The teacher
devises an appropriate layout using representations of these objects within a top down arrangement 1
application running on a computer. Upon completion, the teacher, with the help of their students, is
instructed to arrange each of the objects to their correct positions according to the arrangement
specified in the top down arrangement application. This is completed using SAR visual cues, which
instruct the user to arrange an object to its correct position using instructions for translation and
rotation. At any time, the teacher may alter the arrangement in the top down arrangement
application and the changes will be incorporated into the current SAR object arrangement process.
The teacher may choose to store this configuration and access it when a specific class is being
undertaken. It is also possible for another staff member to utilise the top down arrangement
application from a remote location, and have the teacher undertake the arrangement task according
to the SAR visual cues generated.
This dissertation presents both an exploration of appropriate visual SAR cues that guide users in
the arrangement tasks of physical objects, and a detailed analysis of the effectiveness of these SAR
visual cues. Two instructions are needed to arrange a physical object to a position; rotation and
translation. The SAR visual cues developed utilise these instructions or variations of these
instructions.
The SAR visual cues in isolation are not a unique innovation, but in conjunction with the context in
which the cues are being used – to instruct users on how to arrange a physical object using SAR, a
unique approach is presented. The SAR visual cues described in this dissertation allow for the
direct manipulation of physical objects in a manner which is faster and more accurate when
compared to a manual method of physical object arrangement.
1.1 MotivationThe motivation in selecting this topic was sparked by limitations in the area of object arrangement
using SAR. The SAR visual cues developed overcome a number of the current limitations found in
the area of physical object manipulation using AR and SAR systems. A major limitation is that
object manipulation using AR and SAR has been largely focused on virtual objects. As investigated
in Section 2, AR has been used to enable the manipulation of virtual objects in physical
environments (Webster et al. 1996; MacIntyre et al. 2004; Wang & Gong 2007) and the
manipulation of virtual objects in virtual environments (Kato, H, Billinghurst, M, Poupyrev, K,
Imamoto, K & Tachibana, K 2000; Irawati 2006b). SAR systems have been developed that allow
for virtual objects to be selected, positioned, rotated and fixed in accordance to the user’s
manipulation of a corresponding physical object (Fjeld 1999). Thus, previous work is largely
focused on the manipulation and arrangement of virtual objects rather than physical objects.
2
There are many limitations with virtual object arrangement when compared with physical object
arrangement. There may be an additional task of applying the virtual arrangement to corresponding
physical objects. Achieving a virtual arrangement and then applying this arrangement to physical
objects is likely to be more time consuming and error prone than directly arranging physical
objects. For example, a user would like to arrange furniture in a room of their home and do so by
utilising an AR implementation that allows for the arrangement of virtual representations of their
furniture in a virtual room (Irawati 2006b). A limitation of this is that once the user devises the
arrangement with the virtual representations of the furniture, the user will have to arrange the
corresponding physical furniture items. This process is completely avoided if a direct manipulation
of the physical objects is undertaken. This process is likely to be error prone, as the user is offered
little guidance to arrange the physical objects in their correct locations: furniture will have to be
arranged using some estimation of their positions. The user may obtain a printed layout of the
locations of the virtual objects that they have arranged. This printed layout is then difficult to
change if the user changes their mind regarding where they would like the furniture. Alternatively,
the AR system may provide some measurements to instruct the user on how to arrange the furniture
pieces that corresponds with a virtual object, but this is also time consuming and not ideal.
The research presented in this dissertation addresses this limitation as it provides a method for
directly arranging physical objects with SAR assistance. When considering the same example and
utilising the system described in this dissertation, the user is able to interact with top down
representations of the physical objects, which can be arranged to achieve a suitable layout. The user
is then instructed on how to arrange each physical object with the use of SAR visual cues. The
physical objects can then be placed in their correct positions more accurately as this research
removes the need for estimation, and thus is less error prone. Using the system described in this
dissertation is also less time consuming as there is no estimation and calculation to be undertaken
by the user; the user must simply follow instructions from the SAR visual cues. The top down
arrangement application can be updated if the user changes their mind regarding where they would
like the furniture. Altering the top down arrangement application is then immediately reflected by
the SAR visual cues, which are updated dynamically. The lack of consideration for these
limitations was an important factor in deciding to undertake the research described in this
dissertation.
SAR visual cues allow physical objects to be arranged by giving a user several instructions to
arrange a physical object to a position specified. This is useful when:
1. The alignment of a physical object to a predetermined position is required.
2. There is a lack of physical cues to help position a physical object.
3
In these cases, having instructions for the physical object arrangement systematically displayed to
the user, and having each instruction verified upon successful completion may assist the user.
Utilising SAR in the way presented in this dissertation could be useful for the arrangement of large
objects, such as furniture or theatre sets, or in situations where multiple objects are being arranged.
There could also be applications for this research within the areas of games and edutainment. It has
also been identified that a SAR implementation such as this could be useful for the remote
arrangement of real objects.
1.2 Research QuestionThe question that this research will answer is:
What are the appropriate Spatial Augmented Reality visual cues to instruct a user on how
to arrange real objects?
1.3 Scope
This research is focused on the arrangement of physical objects upon a ground plane. Thus,
translation is bound to the x axis and y axis. Similarly, rotation around the z axis is addressed.
Future work can be undertaken to allow for more arrangement options. For example, a vertical
translation along the z axis would allow physical objects to be stacked and the ability to perform
rotations around the x axis and y axis would allow physical objects to be flipped.
The implementation produced allows for multiple physical objects to be arranged if so desired, but
the order in which the physical objects are arranged is not taken into consideration. For example,
when arranging multiple physical objects, the user may be presented with a situation where the
object to be arranged must be translated to a position that intersects other physical objects, making
the instruction difficult to perform. More work can be undertaken which addresses this. For
example, a technique may be implemented which generates a path for each object to be arranged,
and discerns the order in which the objects should be arranged.
As AR systems involve 3D registration (Azuma, R et al. 2001), this research employs the use of a
tracking framework to provide this. As the focus of this research was on the effectiveness of SAR
visual cues for physical object arrangement, an existing tracking framework was utilised. This
research makes use of the ARToolKitPlus tracking framework (Wagner & Schmalstieg 2007).
4
1.4 Contribution
This dissertation makes several contributions to the area of object arrangement using AR.
Contributions described in this dissertation include:
a survey of existing technologies,
SAR instructions for rotation and translation that can be used in conjunction to arrange a
physical object,
a set of validated and implemented SAR visual cues that utilise the developed instructions
or versions of these instructions,
the integration of the SAR visual cues to form a system that can instruct a user to arrange
physical objects,
a two-dimensional application that has been incorporated into the working system which
aids the physical object arrangement process,
an evaluation of the effectiveness of the SAR visual cues developed,
a revised SAR visual cue that considers the results of the user study, and
an adaptation of a SAR visual cue which can arrange large physical objects.
1.5 Research Structure
The research undertaken resembles, but does not strictly adhere to, the traditional waterfall
software development process. Figure 1 depicts a diagram of the structure of the research
undertaken.
5
Figure 1 Research Structure
The first component of the research undertaken was an analysis phase. As part of the analysis
phase, a literature review was undertaken which provided an overview of previous work that is
related to the research at hand. This literature review is presented in Section 2. The information
gathered from the literature review uncovered gaps in the research in the area and helped form the
research questions being addressed by this dissertation. The research question formed is presented
in Section 1.2. The literature review, devised research questions, a proposed methodology and
other analytical components were summated to produce a proposal of the research to be
undertaken.
After the analysis was complete, development of the system began and further research was
undertaken. This development and further research consisted of the tasks of designing the system
and its components, implementation of these designs and an evaluation of the system produced.
This cycle of tasks was repeated over four iterations. Upon completion of these iterations, a user
study was undertaken and then the iteration cycle was then performed another two times to produce
an improved SAR visual cue that incorporated the data gathered from the user study. Once the
improved SAR visual cue was developed, the iteration cycle was completed a further two times to
produce an adapted SAR visual cue to arrange large objects. Once the system development and
research had been undertaken, the results were written up and compiled to form this dissertation.
6
1.6 Dissertation Structure
This chapter provides an introduction to the research undertaken. This is achieved by a statement of
the motivation for the research, a definition of the research question that has been addressed, an
outline of the scope of the research and a description of the structure of the research undertaken.
Chapter 2 presents a study of the relevant previous work undertaken in the area. This serves to
define the context and provide background information on the research undertaken which supports
the validity of the topic chosen. This is achieved by presenting a survey of research that has been
previously undertaken within the areas of AR and SAR. This survey classifies the research and
states the benefits of the utilisation of SAR in this manner. The study also provides examples of
previous AR and SAR research undertaken in relation to object manipulation, user interaction and
tracking methods.
Chapter 3 presents the SAR object arrangement techniques that have been developed. To facilitate
the arrangement of physical objects using SAR, techniques for user interaction with the developed
system have been devised and are presented. Also presented are a description of the translation and
rotation instructions developed and a discussion of how they can be used in conjunction with other
augmentations to develop a set of SAR visual cues. Furthermore, The SAR visual cues developed
are presented, with descriptions and advantages of each. This is followed by a discussion of the
SAR visual cues which serves to compare the visual cues and address implementation issues.
Chapter 4 describes a user study that was undertaken to determine the effectiveness of the SAR
visual cues developed. This description includes a definition of the tasks undertaken, the structure
in which they were undertaken, and the data that was gathered.
Chapter 5 presents the results of the user study described. These results include quantitative data in
the form of manual timer results, SAR timer results and accuracy results, and qualitative data in the
form of intuitiveness survey results and user opinion survey results. This is followed by a
discussion which serves to clarify the results and concludes with a recommendation on the most
appropriate SAR visual cues.
Chapter 6 describes a revised SAR visual cue that was developed whilst taking into account the
results of the user study. Presented is the methodology undertaken to develop this new SAR visual
cue and a discussion of the new SAR visual cue.
Chapter 7 presents details of the pilot study undertaken to gather preliminary results on the
effectiveness of the Revised Visual Cue developed. The pilot study resembles the user study
undertaken. Also presented are the preliminary results gathered. The preliminary results are
7
compared with the user study data gathered, to give an indication of the effectiveness of the
Revised Visual Cue.
Chapter 8 provides a discussion of a scenario in which a modified SAR visual cue is utilised for the
instructed arrangement of a large object, as this scenario varies from the scenarios previously
described in this dissertation.
Chapter 9 describes the implementation of the overall system, providing a context of the developed
and existing components utilised to implement the functioning system. Details of the existing
components utilised are provided, which include a SAR system and a tracking framework. Details
are also provided of new components developed, which include a module for the existing SAR
system and a top down arrangement application that aids the object arrangement process.
Finally, Chapter 10 provides a conclusion which serves as a summary and discussion of the
research presented in this dissertation. Suggestions for future work that can be undertaken
regarding this research are also presented.
8
2 Background
This section of the dissertation serves to define the context and provide background information of
the research described in this dissertation. This is achieved by presenting a survey of research that
has been previously undertaken within the area of the research topic. The survey encompasses both
AR and SAR applications as the roots of AR and SAR are related (Bimber & Raskar 2005).
A background of the areas of AR and SAR are presented. This includes classification and benefits
of utilising SAR in this manner. Also presented are studies of the specific areas of AR and SAR
research; object arrangement, user interaction and tracking methods. These specific areas are
presented in Sections 2.3, Section 2.4 and Section 2.5 respectively.
The research described within this dissertation is an application of SAR. In order to define the
context of the research, related areas are addressed. Augmented Reality systems enhance the real
world with virtual objects (Azuma, R et al. 2001). To the user, these virtual objects appear to co-
exist with physical objects. AR typically utilises head mounted displays to enable the user to view
the virtual enhancements (Voida et al. 2005). Thus, the creation of the first head mounted display
(Sutherland 1968) has been considered as the birth of Augmented Reality (Bimber & Raskar 2005).
Raskar et al. (1998) introduced the paradigm of Spatial Augmented Reality (SAR) over a decade
ago. Since this time, many different implementations of SAR have been realised. SAR is
recognised as a branch of AR which defies AR conventions by enhancing the user’s real world with
virtual objects that are integrated into the user’s real world environment, rather than requiring the
use of a head mounted display to view enhancements to the real world. This integration can be
undertaken using a projector, as in the research described in this dissertation, or by other methods
such as the integration of a flat panel display directly into the environment (Raskar, R, Welch, G &
Chen, W 1999).
The Shader Lamps system is an early SAR implementation (Raskar, R, Welch, G & Chen, W
1999). Shader Lamps utilised projectors to illuminate physical models of buildings. These
buildings, situated on a tabletop, were scaled models, white in colour, upon which textures could be 9
projected. This implementation allowed for a flexible fast-prototyping method for architects and
designers. Figure 2 depicts the Shader Lamps system. The physical object is augmented with
textures via the use of projectors. AR can also be used for annotating environments (Feiner,
Macintyre & Seligmann 1993). The research presented in this dissertation utilises a combination of
both.
Figure 2 A Vase (Left), and the Vase Being Augmented by Projectors (Right) (Raskar, R et al. 2001)
The SAR research described in this dissertation utilises both projectors and cameras. Projectors
have been used in conjunction with cameras in previous related work (Bimber et al. 2003; Cotting
et al. 2004). Bimber et al. (2003) utilise optical see through displays to allow a user to view
physical and virtual objects with a consistent illumination made possible by techniques employing
the use of projectors and cameras. Cotting et al. (2004) also utilise projectors and cameras in their
method of embedding binary patterns into colour images displayed by a projector; the patterns are
visible to cameras synchronised with the projectors.
Implementations of augmented desks have been developed, which have utilised both projectors and
cameras to enhance surfaces and physical objects with virtual information (Wellner 1991, 1993;
Ishii & Ullmer 1997). The metaDESK utilised cameras and projectors to allow a user to manipulate
a map projected upon a desk (Ishii & Ullmer 1997). The DigitalDesk aimed to add electronic
features to ordinary paper. A camera enables a user to point at a portion of the desk and have the
system read the document located on the desk. A projector is utilised to display feedback to the
user (Wellner 1991). These are fixed setups where the surface being projected upon is in a fixed
position. However, research has been undertaken to allow a projection upon the surface of a
tracked physical object to be aligned whilst the object is moved (Ehnes, Hirota & Hirose 2004).
2.1 ClassificationThere has been a distinction between visual displays to produce a taxonomy for the area of AR
(Milgram & Kishino 1994). This taxonomy classifies AR implementations based on several
10
concepts (Milgram et al. 1994). With regards to these classifications, the research described in this
dissertation would be considered to be partially immersive.
Figure 3 The Reality-Virtuality Continuum (Drascic & Milgram 1996)
The research described in this dissertation would be considered to be located towards the left of the
Reality-Virtuality Continuum (Drascic & Milgram 1996) depicted in Figure 3. This is apparent as
this research relies heavily on reality, whilst applying some augmentations. AR has also been
categorised into three different augmentation approaches: user augmentation, object augmentation
and environment augmentation (Mackay 1998). User augmentation requires the use of head
mounted devices, object augmentation requires the embedding of devices within objects and
environmental augmentation requires the projecting of images within the environment. This
research falls within environment augmentation approach. AR research can be classified in terms of
the display technique employed. These techniques are see-through head-mounted displays,
projection-based displays and handheld displays. The research described is a projection-based
display.
2.2 BenefitsThe advantages of utilising SAR in preference to other techniques that fall within the areas of AR
and VR are well documented (Raskar, R, Welch, G & Chen, W 1999; Raskar, R & Low 2001). As
SAR is a hybrid of the physical and virtual environments, using SAR allows for advantages to be
drawn from each (Raskar, R et al. 2001).
The research described in this dissertation has the goal of instructing a user to arrange physical
objects. There are techniques in AR, VR and SAR that can allow this goal to be attained. Utilising
a physical model as in SAR allows for a model which is higher resolution, easier for the user to
view, more responsive and therefore better than a virtual model such as those used in VR.
However, VR allows for the user to be shown anything (Raskar, R et al. 2001); including scenarios
which may not make sense in the physical world. This may be beneficial to some applications;
however it is not a property that is useful to the research described in this dissertation.11
A user’s physical environment can be enhanced with virtual objects by the utilisation of AR video
and optical see through techniques. However, these techniques require the user to wear a head
mounted display, which is cumbersome. The ability to enhance the user’s environment with a
projector rather than the user having to physically utilise a head mounted display is the main
advantage of SAR over traditional AR (Raskar, R, Welch & Fuchs 1998). Utilising projectors also
negates the need to compensate for the user’s head movements (Voida et al. 2005) and levels of
user fatigue are lower (Bryson et al. 1996). Multiple users are able to view an enhancement to the
physical world using SAR, however in AR and VR, each user must utilise an additional head
mounted device.
Using VR or AR with video see through techniques requires the user to view the environment with
a limited resolution and at a limited frame rate. However, when utilising SAR, the visual fidelity of
the environment remains unchanged, and only the enhancements integrated into the environment
have any such limitations (Raskar, R, Welch, G & Chen, W 1999).
In addition, when compared to techniques that require the use of head mounted displays, the use of
SAR allows for large images to be generated which may span a user’s field of view. This provides
increased integration of virtual objects into the user’s real world. SAR can also allow for increased
immersion into the environment and improved user interaction (Raskar, R, Welch & Fuchs 1998).
SAR allows for virtual objects to be rendered closer to their intended real world location, which
allows for improved accommodation for the human eye (Raskar, R, Welch, G & Chen, W 1999).
Mobile Augmented Reality is another branch of AR, which utilises mobile devices. Mobile AR
restricts the resolution of the user’s world to the resolution of the device being used (Feiner et al.
1997).
Feiner et al. (1997) state that:
“We feel that augmented reality systems will become commonplace only when they
truly add to reality, rather than subtract from it” [as the limited resolution of mobile
devices does] (Feiner et al. 1997, p.5).
The disadvantage of being restricted to a low resolution device is avoided when using projectors in
SAR. As well as allowing for a higher resolution, SAR also negates the need for a user to hold an
interaction device. As mentioned previously regarding AR, multiple users are able to view an
enhancement to the physical world using SAR, however in Mobile AR, multiple mobile devices
may be needed.
12
2.3 Object ManipulationPrevious research in the area of object manipulation in AR environments has identified a lack of
research in the area of physical object manipulation (Kitamura, Ogata & Kishino 2002). However,
physical objects have been used to achieve virtual object manipulation (Gorbet, Orth & Ishii 1998;
Fjeld 1999; Kitamura, Itoh & Kishino 2001; Rekimoto, Ullmer & Oba 2001; Marner & Thomas
2010).
Previous AR work that does involve both physical and virtual objects focuses on applications that
vary from the research described in this dissertation. One such implementation involves applying
limited physics laws upon virtual objects so that consolidated manipulation techniques can be taken
upon both physical and virtual objects (Kitamura & Kishino 1997). Another implementation
involves occlusion and collision calculation with both physical and virtual objects (Breen et al.
1996).
Users have been equipped with a see-through HMD, allowing them to have their view of the real
world enhanced with virtual objects which can be manipulated (Webster et al. 1996). The objects
were visualised on a virtual sphere surrounding the user. A similar implementation was developed
whereby users also equipped with HMDs were able to arrange virtual objects within these spaces,
but were not restricted to a sphere (Wang & Gong 2007). The Designer’s Augmented Reality
Toolkit (DART) was devised to allow the arrangement of virtual objects upon the user’s view of
their physical environment (MacIntyre et al. 2004). A distributed AR system was developed
whereby multiple users were able to arrange virtual objects upon a video stream (Ahlers et al.
1995).
A SAR system was developed entitled BUILD-IT (Fjeld 1999). This tabletop system allowed users
to interact with real objects to control a virtual world, as depicted in Figure 4. Two views of this
world were projected; one on the table and one on the wall. The users were able to manipulate and
control the views using several special real objects. A virtual object was selected when a user
placed a real object at the position of a virtual object. The virtual object could then be positioned,
rotated and fixed by the user manipulating the real object. Subsequent SAR systems have been
developed that implement refinements to such a system, including a restructuring such that the user
no longer has to turn their head to view a wall projection (Gausemeier, Fruend & Matysczok 2002).
13
Figure 4 Virtual Object Manipulation Using Physical Objects (Fjeld 1999)
The Multiuser Augmented Reality Environment (MARE) tabletop AR system was developed
(Grasset & Gascuel 2002). It allows multiple users with HMDs to manipulate virtual objects using
real objects in a shared space. As HMDs were used, this implementation allowed for research into a
private space for each user to keep their own private virtual objects, and measures for access
controls. AR systems such as The Studierstube Project (Schmalstieg, D, Fuhrmann, A, Hesina, G,
Szalavari, Z, Encarnacao, L.M, Gervautz, M & Purgathofer, W 2002) allow interaction with
multiple users. Using SAR with projectors does not allow for each user to see a different display.
However, multiuser interaction and viewing can be achieved using SAR with projectors, without
each user requiring a HMD or a mobile device.
Implementations have been developed to allow a user equipped with a HMD to manipulate virtual
furniture in a virtual room (Kato, H, Billinghurst, M, Poupyrev, K, Imamoto, K & Tachibana, K
2000; Irawati 2006b, a). Irawati et al. (2006) utilised a multimodal user interface which utilised
speech and gestures with a physical paddle to arrange the virtual furniture objects. Similar
implementations exist which do not utilise multimodal input (Kato, H, Billinghurst, M, Poupyrev,
K, Imamoto, K & Tachibana, K 2000). The research described in this dissertation addresses a
similar problem; the arrangement of objects. The research proposed in this dissertation performs a
direct manipulation which will integrate the real and virtual worlds further by using real objects at
their typical scale, rather than scaled down virtual representations.
14
2.4 User InteractionShortcomings have been identified in user interface interaction in applications that utilise the
physical environment (Rekimoto & Nagao 1995). Rekimoto et al. (1995) stated that the focus is on
the user’s interaction with the computer device, rather than the human’s interaction with the
physical environment.
Research has been undertaken to address into the integration of information with the user’s
environment within AR user interface implementations (Höllerer 2001). One of the outcomes of
this research was the establishment of several methods in which an AR user interface can be
managed. An AR user interface can be managed by:
filtering information to show only the relevant information,
formatting the information suitably, and
correctly managing virtual objects [ensuring that virtual objects are laid out appropriately
in the real environment] (Höllerer 2001).
These methods were taken into account for the development of the SAR visual cues presented in
this dissertation. However, the user interfaces developed by Höllerer et al. (2001) resemble the user
interfaces of two dimensional applications and there may be more research that can be undertaken
to make the user interfaces more suited to three dimensional applications. Many early AR
implementations focused on registration with the environment and correct display of information,
and less on how users interact with the system. Thus, these systems also had user interfaces that
were based on the two dimensional desktop metaphor (Azuma, R et al. 2001).
Research into interaction techniques with projector based AR implementations has been undertaken
(Raskar, R & Low 2001). Physical object illumination has been achieved upon moving objects and
this has allowed for the development of a three dimensional painting system which allows a user to
paint textures with a tracked paintbrush upon a tracked moving object that is held by the user. The
paintbrush and object are projected upon with two projectors in real time and are tracked
independently by sensors that are attached to them. The user is able to change the virtual colour and
shape of the brush. The resulting coloured model is able to be stored to work on at a later date.
Physical objects have been used to facilitate user interaction with virtual information (Ishii &
Ullmer 1997; Szalavári & Gervautz 1997). The use of physical objects in this way has been
referred to as Tangible User Interfaces (Ishii & Ullmer 1997). The aforementioned metaDESK
allows interaction with a map by moving several real objects which correspond with landmarks on
the map, and a Luminous Table has been developed which integrates 2D drawings and real objects,
15
and can be used to complete urban design tasks (Ishii et al. 2002). Figure 5 depicts an overview of
the components that form the metaDESK.
Figure 5 The metaDESK System (Ullmer & Ishii 1997)
The Personal Interaction Panel was introduced and developed as an interaction device that is an
extension of a physical notebook, and it allows for a user to perform AR manipulation tasks
(Szalavári & Gervautz 1997; Schmalstieg, D, Fuhrmann & Hesina 2000). Building upon the notion
of Tangible User Interfaces, occlusion techniques have also been utilised in AR user interfaces.
Conventional interaction methods such as the pushing of a button can be achieved with the use of
occlusion (Lee 2004). Thus, by the user physically pressing the button, a marker was covered,
indicating that the button was pushed.
A system has been devised whereby a user is instructed on how to play billiards with the use of
projected visual cues (Suganuma et al. 2008). A camera enables the detection of the location of the
billiard balls and a projector then projects instructions to the user on how the white ball should be
hit and the direction the ball will travel. Further instructions are provided to allow the user to
assume a correct posture. One of these instructions consists of a projected rectangle in which the
user should place one of their hands. Another of these instructions consists of a projected circular
shape, which prompts the user to move their head to a position where the circular shape is
occluded.
16
2.5 TrackingThe research described in this dissertation requires physical objects to be tracked in order to allow
for their arrangement. The research does not aim to develop a unique tracking method; rather an
existing tracking framework is utilised as a component that provides infrastructure.
The ARToolKit tracking framework (Kato, H & Billinghurst 1999) has been used to allow users to
create and manipulate 3D models by using a keyboard and mouse in a tracked workspace (Do &
Lee 2010). The placement of markers determined the action that would be performed such as
creating, selecting and editing a 3D model. The aforementioned study into interaction with tangible
user interfaces made use of ARToolKit for implementing an occlusion technique for the pressing of
a virtual button (Lee 2004). The accuracy of ARToolKit has been measured over distances of one
to three metres (Malbezin, Piekarski & Thomas 2002). This was undertaken by comparing the
physically measured distance from a camera to a marker with the distance extracted from
ARToolKit. Results show an error with the accuracy of the marker's position that increases as the
distance between the camera and the marker increases. The error also varies in the x axis and the y
axis. However, a comparison has been undertaken that determined that, compared with several
other tracking frameworks, ARToolKit provided the best performance for detecting a marker from
a great distance (Zhang, Fronz & Navab 2002). The outcome of their research indicated that the
best existing tracking framework to use depended on the given situation, but ARToolKit performed
well in most situations.
Other tracking methods have been used; the aforementioned BUILD-IT system used bricks covered
in paper that could reflect infra red light which could then be detected by an infra-red camera (Fjeld
1999). It has been claimed that there is an apparent recent move towards utilising model based
tracking in AR (Zhou, Duh & Billinghurst 2008). Model-based tracking systems use
distinguishable features gathered from a model of the object to implement tracking. A method of
model-based tracking has been devised which allows camera pose to be calculated from several
different features of models such as lines, cylinders and spheres. This research focused on
achieving tracking with a low error rate, which was aided by having an increased knowledge of
objects present in the scene.
Research has been undertaken into using natural features of objects for tracking, instead of
introducing artificial markers to the scene (Park, You & Neumann 1999). This method of tracking
requires the camera pose calculation to be undertaken from known features, but then the system is
able to dynamically add features such as textures and corners detected in the scene to aid in
updating the pose calculation.
17
Tracking has also been achieved using encoded LEDs instead of paper based markers (Naimark &
Foxlin 2005). The use of LED tracking is beneficial in several situations, such as for applications
on a small scale which require high precision tracking, and for tracking in the dark.
In recent years, there has been a trend of research focused on vision-based tracking techniques
rather than sensor-based tracking techniques (Zhou, Duh & Billinghurst 2008). However, vision
based techniques are more computationally demanding, and often have errors caused by factors
such as occlusion (You, Neumann & Azuma 1999). In one of the only sensor-based, non-camera
based tracking techniques to be developed in recent years, ultrasonic sensors are used for tracking
indoors with the Bat system (Newman 2001). When utilising the Bat system, users are fitted with
small wireless devices called Bats. Receivers are located in known, fixed positions on ceilings and
the Bats emit signals which are detected by these receivers. The distance from the Bats to the
receiver can be calculated from the time taken for the signals to be received.
18
3 SAR Object Arrangement Techniques
This section presents the SAR visual cues developed. The majority of the SAR visual cues
implement a Translation Instruction and a Rotation Instruction and follow a developed process for
arranging objects to a desired position, all of which are described.
Translation and rotation have been identified as instructions that are to be performed to arrange a
physical object to a desired position, and thus a Translation Instruction and a Rotation Instruction
have been developed. These two instructions act as building blocks; they are utilised for the
majority of the SAR visual cues, either on their own or incorporated with other augmentations. The
Translation Instruction is described in Section 3.2.1 and the Rotation Instruction is described in
Section 3.2.2. A process has been developed to instruct the arrangement of a physical object to a
desired position by a series of moves utilising the Translation and Rotation instructions; this
process is described in Section 3.2 and the majority of the SAR visual cues developed follow this
process.
A set of SAR visual cues have been developed, each providing various advantages and a different
user experience; they are described in Section 3.2.3. The intention is to develop a variety of SAR
visual cues with different strengths, and then assess their performance with the user study described
in Chapter 4. It is envisaged that these SAR visual cues can be applied to different physical objects
in a number of different application domains, such as manufacturing, warehousing or theatrical
sets. A discussion of the SAR visual cues developed is provided in Section 3.2.4. When
considering the theoretical ideas of the SAR visual cues versus their implementations, various
differences are evident which are also discussed. This section begins with a description, in Section
3.1, of user interaction techniques which enable users to arrange physical objects using SAR.
3.1 Interaction
A top down arrangement application has been developed, from which SAR visual cues for physical
object manipulation can be generated. The application provides a user interface in which the user
19
arranges representations of physical objects from a top down view. The resulting arrangement can
then be performed on the physical objects using SAR visual cues.
This process can be undertaken in real time by one user or as a collaboration task. This process can
also be stored to provide instructions on demand at a later date. When collaborating on a task, a
remote user can provide arrangement instructions of objects to a user physically present with the
physical objects. For example, the remote user can specify the desired arrangement of the physical
objects in the top down arrangement application, and then have this arrangement carried out by a
second party who follow the resulting SAR visual cues to achieve the arrangement that has been
specified. The instructions can also be stored for use at a later date. For example, a user arranging
theatre sets may want to store a set of specific arrangements for the objects that can be utilised
when the need arises.
3.2 Object Arrangement Techniques
The overall system has a notion that the objects to be arranged are constrained to the ground plane.
This notion does restrict the movement of physical objects to a single planar surface, such as a
floor or table-top, but does allow for simpler SAR visual cues to be provided to the user. This
design decision was developed to support the arrangement of large physical objects such as
furniture or theatre sets. The rotation instructions are performed within the tracked physical
object’s local coordinate system whereas the translation instructions are performed within the
world coordinate system.
The instructions are carried out systematically; the user begins with one instruction, a translation or
a rotation. For example, if the translation is undertaken first, then the Translation Instruction is
displayed to the user guide the user to move the physical object to a position that corresponds with
the desired final translation. Once the user has moved the physical object so that the translation is
correct, the Translation Instruction ceases to be displayed. The Rotation Instruction is then
displayed to guide the user to rotate the physical object to a position that corresponds with the
desired final rotation. Once the user has moved the physical object to a position that corresponds
with the desired final translation and rotation, the physical object has been successfully arranged.
Once both the rotation and translation instructions have been completed successfully, the SAR
visual cue will no longer be displayed, indicating that the physical object is successfully arranged.
A state machine of the process can be seen in Figure 6. Once the physical object has been arranged,
the next physical object can be arranged if there are multiple objects to consider.
20
Figure 6 Object Arrangement Process
Section 3.2.1 describes an instruction for the translation of a tracked physical object. Similarly,
Section 3.2.2 describes an instruction for the rotation of a tracked physical object. The utilisation of
these instructions or variations of these instructions, to arrange an object is referred to as a SAR
visual cue. A discussion of the SAR visual cues developed is provided in Section 3.2.3.
3.2.1 Translation Instruction
Translation is achieved by instructing the user to move a tracked physical object on a ground plane,
along the x and y axes, in a z axis up world. The translation of the tracked physical object is
undertaken within the world coordinate system. A full range of translations are possible along the x
and y axes, with a portion of the projection following the tracked physical object. Figure 7, Figure
8 and Figure 9 depict, from a top down view, the process of undertaking a translation using the
translation instruction. The tracked physical object is represented by a solid shape whilst the
desired final position of the object is represented by a dotted shape.
21
Figure 7 Translation Instruction: Starting Position.
The translation instruction shows a circular projection upon the tracked object to be arranged, and
the same circular shape projection at the desired final position of the object. The two circular
shapes are connected with a line which serves to show the user a path to the circle located at the
physical object’s desired final position. This line is useful for situations when the desired final
position is difficult to locate. For example, the desired final position may be located at a distance
from the object’s current position. It is also useful in situations when the shape of the physical
object can obscure the projection at the desired final position.
Figure 8 Translation Instruction: Position After Part of the Translation has been Completed.
As the user translates the tracked physical object, the projection displayed is updated in accordance
to the object’s new position.
22
Figure 9 Translation Instruction: Final Position.
The user moves the tracked object until the circular shapes are aligned; once the object has been
successfully translated, the instruction will cease to be displayed. This is depicted in Figure 9.
3.2.2 Rotation Instruction
Rotation is achieved by instructing the user rotate a tracked physical object on a ground plane,
around a centre axis. The rotation of the tracked physical object is in the object’s local coordinate
system. A 360 degree range of rotation can be achieved. Translating the object whilst the rotation
instruction is displayed does not cause a conflict, as the rotation projection follows the tracked
object.
Figure 10, Figure 11 and Figure 12 depict, from a top down view, the process of undertaking a
rotation using the rotation instruction. The tracked physical object is represented by a solid shape
whilst the desired final position of the object is represented by a dotted shape.
23
Figure 10 Rotation Instruction: Starting Position.
The rotation instruction developed follows a similar structure to that of the translation instruction.
The rotation instruction shows a two lines originating from the centre of the tracked physical
object. One of the lines is projected along the x axis of the object, and follows this axis in
accordance to the rotation that the user is performing. The other line is immovable and represents
the destination rotation which can be achieved when the movable line is aligned with the
immovable line.
In Figure 10 and Figure 11, the immovable line is represented by the upper line of the projection in
the diagrams. In Figure 12, the lines are indeterminable as they overlap. A circular projection is
projected at the end of each of the two lines. The appearance of the lines from the centre of the
object can be likened to the hands of a clock.
Figure 11 Rotation Instruction: Position After Part of the Rotation has been Completed.
24
As the user rotates the tracked physical object, the projection displayed is updated in accordance to
the object’s new position.
Figure 12 Rotation Instruction: Final Position
The user rotates the tracked object until the movable line is aligned with the immovable line, which
is also the case when the circular shapes are aligned; once the object has been successfully rotated,
the instruction will cease to be displayed. This is depicted in Figure 12.
3.2.3 SAR Visual Cues
A set of SAR visual cues have been developed which allow for different experiences when
arranging tracked physical objects. The majority of the SAR visual cues utilise the Translation
Instruction as described in Section 3.2.1 and the Rotation Instruction as described in Section 3.2.2,
however the Circle Cue implements variations of the translation and rotation instructions. Many of
the SAR visual cues developed have additional augmentations projected with the goal of aiding the
physical object arrangement process.
The SAR visual cues implement different advantages which are summarised in Table 1; for
example, the Circle Cue allowed for the most simplistic projection which may be advantageous in
some situations, but this simple projection also conveyed the least information to the user. The
Wireframe Cue allowed for a larger amount of information to be conveyed to the user; wireframe
projections were present which the user could align. The presence of these wireframe projections
allow for rotation information to be conveyed at all times through the orientation of the wireframe
projections, however the use of the wireframe projections may result in the SAR visual cue
seeming cluttered.
25
SAR Visual Cue and Description Advantages Disadvantages
Translation First Cue: Utilises
only the Translation and Rotation
Instructions
Simplistic No further projections are
displayed, so no extra
information is conveyed.
Rotation First Cue: Utilises only
the Translation Instruction and
Rotation Instruction
Simplistic. No further projections are
displayed, so no extra
information is conveyed.
Both Instructions Cue: Utilises
only the Translation Instruction and
Rotation Instruction, displayed at
the same time.
User can perform the instructions in
either order.
More cluttered than many of the
other SAR visual cues.
Circle Cue: Translation and
rotation are achieved with only the
use of a circle projection.
Very simplistic. Conveys less information than
the other SAR visual cues.
Wireframe Cue: The Translation
and Rotation Instructions are used,
along with two wireframe
projections that can be aligned.
The wireframe projections convey both
translation and rotation information at
all times, and this is shown in addition
to the Translation and Rotation
Instructions.
More cluttered than many of the
other SAR visual cues.
Square Cue: The Translation and
Rotation Instructions are used,
along with two square projections
that can be aligned.
The square projections convey both
translation information and some
rotation information at all times, and
this is shown in addition to the
Translation and Rotation Instructions.
Does not convey as much
rotation information as the
Wireframe Cue.
Table 1 SAR Visual Cue Overview of Advantages and Disadvantages
The intention is to develop a variety of SAR visual cues with different strengths, and then assess
their performance with the user study described in Chapter 4; a process which results in a
recommendation of the best SAR visual cue developed. In addition to the visual cues described, a
revised SAR visual cue has been developed, taking into consideration the results of the user study.
Information regarding the revised SAR visual cue is presented in Section 6.
26
3.2.3.1 Translation First Cue
The Translation Instruction and Rotation Instruction are utilised, with no further augmentations
present. The Translation Instruction is projected in yellow and the Rotation Instruction is projected
in green. The SAR visual cue follows the process described by Figure 6 with the Translation
Instruction occurring first.
When compared to many of the other SAR visual cues developed, such as the Wireframe Cue
described in Section 3.2.3.5, the Translation First Cue achieves a higher level of simplicity as no
further augmentations are provided to the SAR visual cue; only the Translation and Rotation
Instructions are utilised. However, the Translation First Cue is not as simplistic as the Circle Cue
described in Section 3.2.3.4.
Figure 13 Translation First Cue: Translation (the Initial State)
Figure 13 depicts the initial state of the physical object. The implementation of this initial state is
presented in Figure 14. The initial state of the Translation First Cue is the Translation Instruction
presented in Section 3.2.1.
27
Figure 14 Translation Cue: Translation Implementation (the Initial State)
The Translation Instruction is displayed to guide the user to move the physical object to a position
that corresponds with the translation coordinates of the desired final position. In the scenario
presented by Figure 14, the user must translate the physical object towards the camera, so that the
circles are aligned. This can be achieved by following the line projection. Once the user has moved
the physical object so that the translation is correct, the Translation Instruction ceases to be
displayed.
28
Figure 15 Translation First Cue: Rotation
Figure 15 depicts the state of the physical object once the initial translation has been completed and
rotation is now being performed. The implementation of this state is presented in Figure 16. This
state corresponds with the Rotation Instruction presented in Section 3.2.2.
Figure 16 Translation First Cue: Rotation Implementation
29
The Rotation Instruction is now displayed to guide the user to rotate the physical object to a
position that corresponds with the desired final rotation. In the scenario presented by Figure 16, the
user must rotate the object further clockwise, so that the two projected lines are aligned. Once the
user has moved the physical object to a position that corresponds with the translation and rotation
of the desired final position, the SAR visual cue will no longer be displayed, indicating that the
physical object has been successfully arranged.
3.2.3.2 Rotation First Cue
The Rotation First Cue is similar to the Translation First Cue, however the deviation is that the
Rotation Instruction is displayed first, followed by the Translation Instruction. This order of the
instructions is continued until the physical object is correctly aligned. The Rotation First Cue has
the advantages specified for the Translation First Cue in Section 3.2.3.1. The Rotation First Cue
and Translation First Cue were both implemented to provide some variety and discover if one order
of the instructions is more effective than the other.
3.2.3.3 Both Instructions Cue
The Both Instructions Cue is similar to the aforementioned SAR visual cues, however the variation
is that both the Translation Instruction and Rotation Instruction are displayed at the same time.
The Both Instructions Cue is more cluttered than the Translation First Cue described in Section
3.2.3.1 and Rotation First Cue described in Section 3.2.3.2 as it displays both of the instructions at
the same time. However, this allows for more flexibility in the method that the user undertakes
when arranging the object. Displaying both translation and rotation information at the same time
offers the advantage of the user being able to choose which instruction to perform first as in some
situations it may be better to perform the translation first and in other situations it may be more
efficient to perform the rotation first.
30
Figure 17 Both Instructions Cue: Scenario of Translation Performed First
For example, the user may choose to undertake the translation of the physical object first. In this
case, the user will utilise the Translation Instruction to move the physical object to a position that
corresponds with the desired final translation. Once the user has moved the physical object so that
the translation is correct, the user can then rotate the object according to the Rotation Instruction.
The Translation Instruction continues to be displayed although the translation has been successfully
completed. This allows the user to correct the translation of the object if the translation becomes
incorrect whilst the rotation is being performed. This scenario is depicted in Figure 17.
Figure 18 Both Instructions Cue: Scenario of Rotation Performed First
Alternatively, the user may choose to undertake the rotation of the physical object first. In this case,
the user will utilise the Rotation Instruction to rotate the physical object to a position that
corresponds with the desired final rotation. Once the user has rotated the physical object so that the
31
rotation is correct, the user can then translate the object according to the Translation Instruction.
The Rotation Instruction continues to be displayed although the rotation has been successfully
completed. This allows the user to correct the rotation of the object if the rotation becomes
incorrect whilst the translation is being performed. This scenario is depicted in Figure 18.
Figure 19 Both Instructions Cue: Implementation
Figure 19 presents the implementation of the Both Instructions Cue. In this scenario the user has
rotated the physical object such that the rotation is almost correct. This is evident by the two lines
of the Rotation Instruction being almost aligned. In order to successfully arrange the physical
object, the user must now translate the object in a south westerly direction and ensure that the
rotation is aligned.
Once the user has moved the physical object to a position that corresponds with the desired final
translation and rotation, the SAR visual cue will no longer be displayed, indicating that the physical
object has been successfully arranged.
3.2.3.4 Circle Cue
The Translation Instruction and Rotation Instruction are not utilised for this SAR visual cue. The
SAR visual cue follows the process described by Figure 6 with translation occurring first. This
SAR visual cue allows for simplicity and was designed with the aim that the circle projection will
32
be clearer when projected upon a variety of different physical objects, as compared with SAR
visual cues that incorporate more complicated projections.
Figure 20 Circle Cue: Translation (the Initial State)
Figure 20 depicts the initial state of the physical object. The implementation of this initial state is
presented in Figure 21.
Figure 21 Circle Cue: Translation Implementation (the Initial State)
33
The only projection utilised is of a circle. This circle has a radius larger than the circle projection
used for the Translation Instruction and Rotation Instruction. The circle is displayed upon the
tracked physical object and at the desired final position to guide the user to move the physical
object to a position that corresponds with the translation coordinates of the object’s desired final
position. This is achieved in a similar fashion to the Translation Instruction; the user moves the
object so that the two circles are aligned. In the scenario presented by Figure 21, the user must
translate the object in a south westerly direction, so that the two projected circles are aligned. Once
the user has moved the physical object so that the translation is correct, the two circles cease to be
displayed.
Figure 22 Circle Cue: Rotation
Figure 22 depicts the state of the physical object once the initial translation has been completed and
rotation is now being performed. The implementation of this state is presented in Figure 23.
34
Figure 23 Circle Cue: Rotation Implementation
A variation of the Rotation Instruction is displayed which consists of one circle of a differing
colour; green. This green circle is displayed upon the physical object to guide the user to rotate the
physical object to a position that corresponds with the desired final rotation. This is achieved by the
user rotating the object until the green circle is no longer displayed. Once the user has moved the
physical object to a position that corresponds with the desired final translation and rotation, the
SAR visual cue will no longer be displayed, indicating that the physical object has been
successfully arranged.
The user is not given any more visual cues to aid the rotation other than the circle, and the
indication that the rotation is correct. This results in a simplistic and consistent projection being
displayed when utilising this SAR visual cue, but the lack of information is a disadvantage to this
increased simplicity.
3.2.3.5 Wireframe Cue
The Wireframe Cue is similar to the Translation First Cue, however the deviation is that two
wireframe projections are present, which are projected in grey. One wireframe projection is aligned
with the physical object and follows the physical object as it is moved. The other wireframe
projection is present at the desired final position. The existence of the wireframe projections may
assist the user in arranging the object. The wireframe projections convey both rotation and
translation information to the user, regardless of which state the arrangement process is in. The user 35
is aided by being able to align the wireframe projection following the physical object with the static
wireframe projection located at the desired final position, as when the two wireframe projections
are aligned, the physical object is successfully arranged.
Figure 24 Wireframe Cue: Translation Implementation (the Initial State)
Figure 24 depicts the initial state of the physical object. The implementation of this initial state is
presented in Figure 25.
36
Figure 25 Wireframe Cue: Translation (the Initial State)
As with the Translation First Cue described in Section 3.2.3.1, the Translation Instruction is
displayed and, in addition, the wireframe projections are displayed. In the scenario presented by
Figure 25, the user must translate the object in a south westerly direction. This can be achieved by
using the Translation Instruction, and can also be achieved by aligning the two wireframe
projections.
Figure 26 Wireframe Cue: Rotation
37
Figure 26 depicts the state of the physical object once the initial translation has been completed and
rotation is now being performed. The implementation of this state is presented in Figure 27.
Figure 27 Wireframe Cue: Rotation Implementation
As with the Translation First Cue described in Section 3.2.3.1, the Rotation Instruction is displayed
and, in addition, the wireframe projections remain present. In the scenario presented by Figure 27,
the user must rotate the object further clockwise. This can be achieved by using the Rotation
Instruction, and can also be achieved by aligning the two wireframe projections.
3.2.3.6 Square Cue
The Square Cue is similar to the Translation First Cue, however the deviation is that two square
projections are present, which are projected in cyan. Thus, the Square Cue is the same as the
Wireframe Cue, but with square projections instead of wireframe projections. One square
projection is aligned with the centre of the top of the physical object and follows the physical
object as it is moved. The other square projection is present at the desired final position.
The existence of the square projections may assist the user in arranging the object. The square
projections convey translation information and some rotation information to the user, regardless of
which state the arrangement process is in. The user is aided by being able to align the square
projection following the physical object with the static square projection located at the desired final
position, as when the user translates the physical object so that the two square projections are
38
aligned, the physical object is successfully translated and the rotation is either correct or it may
have a variation of either 90, 180 or 270 degrees, which can then be corrected by the user when the
Rotation Instruction is displayed. This SAR visual cue allows for a degree of simplicity as the only
projections present at one time are the two square projections and one of the Translation Instruction
or the Rotation Instruction. It does not convey as much information as the Wireframe Cue
described in Section 3.2.3.5 as a more exact rotation can be gathered from the wireframe
projections, however the Square Cue is comparatively less cluttered. In comparison to the Circle
Cue described in Section 3.2.3.4, the Square Cue offers more information but is more cluttered and
less simplistic.
Figure 28 Square Cue: Translation (the Initial State)
Figure 28 depicts the initial state of the physical object. The implementation of this initial state is
presented in Figure 29.
39
Figure 29 Square Cue: Translation Implementation (the Initial State)
As with the Translation First Cue described in Section 3.2.3.1, the Translation Instruction is
displayed and, in addition, the square projections are displayed. In the scenario presented by Figure
29, the user must translate the object in a south westerly direction, so that the circles are aligned.
This can be achieved by using the Translation Instruction, and can also be achieved by aligning the
two square projections.
40
Figure 30 Square Cue: Rotation
Figure 30 depicts the state of the physical object once the initial translation has been completed and
rotation is now being performed. The implementation of this state is presented in Figure 31.
Figure 31 Square Cue: Rotation Implementation
As with the Translation First Cue described in Section 3.2.3.1, the Rotation Instruction is displayed
and, in addition, the square projections remain present. In the scenario presented by Figure 31, the
41
user must rotate the object clockwise. This can be achieved by using the Rotation Instruction, and
may be aided by using the two square projections.
3.2.4 Discussion
Multiple SAR visual cues have been developed to provide variety and different advantages. Many
of the SAR visual cues undertake the object arrangement process outlined in Figure 6 with either
the translation or rotation occurring first. However, the Both Instructions Cue described in Section
3.2.3.3 displays both translation and rotation information at the same time. As stated, this allows
for more flexibility in the method that the user undertakes when arranging the object and may be
advantageous.
For example, a situation in which the Both Instructions Cue may be advantageous is if the user
must navigate a couch through a doorway in order to reach the desired final position. In this
situation, performing the translation first would be the most efficient method, as any rotation
applied to the couch may need to be undone as the couch may need to be rotated in a different way
to fit through the doorway. In this situation, the user can decide that the translation should be
undertaken first, and can choose to perform the translation followed by the rotation.
If the same example were undertaken with the Rotation First Cue described in Section 3.2.3.2, the
user would first rotate the couch, and then translate it through the doorway. In order to translate the
couch through the doorway, the couch must be rotated to a different position, undoing the rotation
already performed. Once the translation is correct, the Rotation Instruction is displayed once again.
Thus, the object will still be arranged to the desired destination position, but the rotation
undertaken first was not needed and thus the process was not efficient. While the SAR visual cues
developed may all allow the physical object to be correctly arranged, some have advantages in
different situations.
The design of the SAR visual cues described has lead to issues regarding the location of the
projections upon the physical object, in terms of the marker fixed to the physical object and the
distortion caused in several situations when utilising the SAR visual cues described. Projection
upon tracking markers is addressed in Section 3.2.4.1 and distortion is addressed in Section 3.2.4.2
3.2.4.1 Projection upon Tracking Markers
The SAR visual cues developed have been implemented in several bright colours in the interest of
being easier to see. As the tracking framework utilised required the use of markers attached to the
physical object, the projection overlapped the markers frequently. This made portions of the SAR
visual cues difficult to see, when projected upon the black of the markers. As this research focuses
on SAR visual cues for the arrangement of physical objects, there was no endeavour to consider
42
limitations caused by the tracking framework. The SAR visual cues were not designed according to
the limitations of the tracking framework utilised. In future work, a tracking framework could be
utilised that does not require markers to be located in positions where projections are made.
3.2.4.2 Distortion
When considering both the theoretical ideas of these SAR visual cues versus their implementations,
differences are evident. These fundamental differences are clear when viewing Figure 7 which
depicts a top down diagram of the Translation Instruction and Figure 14 which depicts the
implementation translation of the Translation First Cue (which is also the Translation Instruction
with no further augmentations). In the implementations these visual cues, occlusion and distortions
are present.
The design of the visual cues has been undertaken with these fundamental SAR issues in mind. A
measure taken to limit the problems of occlusion and distortion is that the Translation Instruction
utilises a line between the two circles. In a situation where the circle projected at the desired final
translation cannot be seen, the user is able to follow the line and arrive at the desired final
translation.
Figure 32 SAR Visual Cue Distortion on the Side of the Physical Object
For example, when considering the translation scenario presented in Figure 32, the projection of
the circle at the desired final translation is distorted. If it is assumed that the user’s view of the
43
physical object is from the camera position, it is evident that the object must be moved south in
order to complete the Translation Instruction. However, if the user were viewing this SAR visual
cue at the same angle but from the opposite side of the physical object, the user would not see the
circle projection at the desired final translation. This distortion scenario has been addressed by the
utilisation of the line projection between the physical objects current translation and the desired
final translation. The user is to move the physical object in the direction that the line is indicating.
As the physical object nears the desired final translation, the projection of the circle at the desired
final translation will be seen upon the top of the physical object, enabling the user to align the two
circles.
Another measure taken to limit the problems of occlusion and distortion is that the Rotation
Instruction utilises two lines from the centre point of the physical object. It can be imagined that
there is a vertical line through this point, which represents the axis around which the physical
object is rotated. Two lines are drawn from this centre point, and two circles that must be aligned;
one indicating the current rotation and another indicating the desired final rotation. In a situation
where one or more of the projected circles cannot be seen, the user is able to align the two lines and
still achieve the desired final rotation.
Figure 33 SAR Visual Cue Distortion from the Projector Position
When considering the rotation scenario presented in Figure 33, the projection of the circle
indicating the desired final rotation is distorted, in a manner that differs from the previous example.
44
The location of the projector causes the projection to be disjointed. This has been addressed by the
utilisation of line projections from the centre of the physical object to the circles that must be
aligned in order to achieve the desired final rotation. As the user can see a portion of these lines as
they stem from the centre point of the object, the user is to rotate the physical object so that the
lines overlap. As the physical object nears the desired final rotation, the projection of the circle at
the desired final rotation will be seen upon the top of the physical object, allowing the user to align
the two circles.
45
4 User Study
A user study was performed to determine:
the effectiveness, in terms of task time and accuracy, of different SAR visual cues versus a
manual method to instruct the arrangement of a physical object,
the intuitiveness of several different SAR visual cues, and
user opinion of the tasks undertaken.
This data was gathered to determine the effectiveness of the SAR visual cues developed, in
comparison to one another and in comparison to a manual object arrangement. A system was
utilised in accordance to the configuration specified in Section 4.1. The hypothesis of the user
study is that there is an improvement in speed and accuracy when using the SAR visual cues in
comparison to the manual arrangement method. The participants were invited to utilise this system
to undertake physical object arrangement tasks. These tasks are outlined in Section 4.2 and form
part of the structure outlined in Section 4.3. Section 4.4 describes the data that was gathered from
the user study, as well as the comparisons made with this data. Also discussed is the gathering of
data from both the manual timer and SAR timer, as the use of two timers may cause confusion. The
results of this user study are presented in Section 5.
4.1 ConfigurationThe system consisted of the components that are described in Chapter 8, however the top down
arrangement application was not utilised in the user study. One projector and one camera were
used. Figure 34 depicts the configuration of the components utilised for the user study.
46
Figure 34 User Study Setup
Projector calibration was undertaken such that a one millimetre distance along a physical model
corresponded with a one unit distance along the corresponding virtual model. Thus, a unit in the
projector coordinate space corresponded with a millimetre in the physical world.
The participants were asked to arrange the same tracked physical object; a foam box with the
dimensions of 220 millimetres (length), 100 millimetres (width) and 120 millimetres (height). The
physical object was tracked using the ARToolKitPlus framework (Wagner & Schmalstieg 2007).
4.2 Tasks Undertaken for the User StudyOne of the tasks addressed physical object arrangement using a manual arrangement technique
which did not utilise SAR visual cues, and the remainder of the tasks addressed physical object
arrangement using SAR visual cues developed.
47
Figure 35 Projected Grid Utilised in Task 1
The desired final positions of the physical objects were randomly generated. For Task 1 (No SAR),
the desired final positions were one of four randomly generated positions located on a grid, with
each position being one hundred projector units apart. For the tasks that utilised SAR visual cues,
this was a randomly generated position within an area of six hundred by six hundred projector
units.
Figure 36 The Grid used for Task 1 (left) and a Participant Undertaking Task 1 (right)
The tasks undertaken by each participant consisted of the following structure and in the following
order:
1. No SAR: A three by three grid was projected upon a ground plane, upon which the
participant was asked to place the centre point of the tracked physical object at one of four
positions that corresponded to points at the corners of the interior square of the grid. These
positions were located one hundred units apart, which corresponds to approximately one
48
hundred millimetres according to the calibration method utilised. The projected grid did
not show any numbering; rather the numbers and corresponding locations were explained
to the participant verbally and with a demonstration of the physical object placed at each of
the four possible positions. The position of where the participant was to arrange the
physical object was randomly generated, as was the angle that the participant was to rotate
the object to. For example, a participant may be told to place the physical object at position
2, at 45 degrees. A protractor was available for use if desired. The grid was utilised as the
specified points were known coordinates. This allowed for a comparison between the
achieved position of the tracked physical object and the known coordinates. Figure 35
depicts a diagram of the grid used. Figure 36 presents the implementation of the grid used,
along with a participant arranging a physical object using the grid.
2. Translation First: The participant arranged the physical object to a randomly generated
position using the Translation First Cue specified in Section 3.2.3.1.
3. Rotation First: The participant arranged the physical object to a randomly generated
position using the Rotation First Cue specified in Section 3.2.3.2.
4. Both Instructions Cue: The participant arranged the physical object to a randomly
generated position using the Both Instructions Cue specified in Section 3.2.3.3.
5. Circle Cue: The participant arranged the physical object to a randomly generated position
using the Circle Cue specified in Section 3.2.3.4.
6. Wireframe Cue: The participant arranged the physical object to a randomly generated
position using the Wireframe Cue specified in Section 3.2.3.5.
7. Wireframe Cue with No Destination: The participant arranged the physical object to a
randomly generated position using the Wireframe Cue specified in Section 3.2.3.5, with the
variation that the wireframe upon the desired final position was not shown.
8. Square Cue: The participant arranged the physical object to a randomly generated position
using the Square Cue specified in Section 3.2.3.6.
9. Square Cue with No Destination: The participant arranged the physical object to a
randomly generated position using the Square Cue specified in Section 3.2.3.6, with the
variation that the square upon the desired final position was not shown.
On occasion, user study tasks were repeated if a problem occurred during the undertaking of the
task. This occurred on several occasions when participants inadvertently moved or removed the
tracking markers.
49
4.3 StructureThe study was taken with one participant at a time. Each participant undertook the following
procedure:
1. The participant read the information sheet containing information of the study.
2. Any questions that the participant had regarding the study were addressed.
3. The participant read and signed the consent form.
4. The participant completed a survey designed to gather information about the participant,
such as age and their AR experience level.
5. The participant completed a survey to gather information on the intuitiveness of the SAR
visual cues.
6. The participant completed nine tasks adhering to the structure and order described. The
tasks were not timed during the first run through. The participant was given an explanation
of each task and encouraged to ask questions.
7. The participant completed the nine tasks again, still adhering to the structure and order
described. The tasks were timed during the second run through. The participant activated a
manual timer before undertaking each task, and stopped the timer once the task was
completed. The timer was activated by the participant pushing a key on a keyboard located
on the same surface as the physical object to be arranged. The physical object was
collected by the participant from a consistent location after the timer was activated.
8. The participant completed a survey to gather their opinions of the tasks they had
undertaken.
The participant was able to indicate that they would like a break at any time during the user study.
The study took place within the School of Computer and Information Science at the University of
South Australia.
4.4 Data Gathering
A range of both quantitative and qualitative data was gathered from the user study. Provided are
details of the data gathered, as well as the comparisons that will be made with the data gathered.
4.4.1 Quantitative Data Gathering
With regards to quantitative data, the user study allowed for the gathering of data pertaining to:
the time that the task took to complete, using a manual timer (for all tasks),
the time that the task took to complete, using a SAR timer (for Task 2 to Task 9),
50
the randomly generated desired final position of the physical object, for all tasks, which
consisted of rotation and translation values, and
the achieved position of the physical object once the task was complete, for all tasks, which
consisted of rotation and translation values.
The quantitative data gathered allows for comparisons between:
the desired final x position and the achieved final x position of the physical object (x axis
accuracy),
the desired final y position and the achieved final y position of the physical object (y axis
accuracy),
the desired final rotation position and the achieved final rotation position of the physical
object (rotation accuracy),
the time taken to complete each task using a manual timer (task completion time), and
the time taken to complete each SAR visual cue task (Task 2 to Task 9) using a SAR timer
(SAR visual cue task completion time).
4.4.2 Qualitative Data Gathering
With regards to qualitative data, the intuitiveness survey undertaken as part of the user study
allowed for the gathering of data pertaining to:
the participant’s attempt to determine the outcome of a diagram depicting a physical object
with a SAR visual cue instructing a translation, according to the translation instruction
described in Section 3.2.1,
the participant’s attempt to determine the outcome of a diagram depicting a physical object
with a SAR visual cue instructing a 45 degree rotation, according to the rotation instruction
described in Section 3.2.2,
the participant’s attempt to determine the outcome of a diagram depicting a physical object
with a SAR visual cue instructing a translation, according to the translation instruction
described in the Circle Cue description in Section 3.2.3, and
the participant’s attempt to determine the outcome of a diagram depicting a physical object
with a SAR visual cue instructing a 45 degree rotation, according to the rotation instruction
described in the Circle Cue description in Section 3.2.3.
Also with regards to qualitative data, the user opinion survey undertaken as part of the user study
allowed for the gathering of data pertaining to:
the participant’s preference between Task 1 (No SAR) and Task 2 (Translation First Cue),
51
the participant’s preference between Task 2 (Translation First Cue), Task 3 (Rotation First
Cue) and Task 4 (Both Instructions Cue),
the participant’s preference between Task 2 (Translation First Cue) and Task 5 (Circle
Cue),
the participant’s preference between Task 6 (Wireframe Cue) and Task 7 (Wireframe Cue
with No Destination),
the participant’s preference between Task 8 (Square Cue) and Task 9 (Square Cue with No
Destination),
the participant’s preference between the Wireframe tasks (Task 6 and Task 7) and the
Square tasks (Task 8 and Task 9), and
the participant’s preference between the tasks that utilised the translation instruction as
described in Section 3.2.1 and the rotation instruction as described in Section 3.2.2 (Task 2
to Task 4, and Task 6 to Task 9).
Appendix A presents the surveys given to the participants to gather this qualitative data.
4.4.3 Manual Timer and SAR Timer Data Gathering
Data was recorded for all tasks using a manual timer. Data was also recorded with a SAR timer for
all tasks that utilised SAR visual cues. Thus, Task 1 (No SAR) was not timed with the SAR timer
and all other tasks were timed with the SAR timer. The manual timer was required for the
comparison of the task completion times of the tasks that utilise SAR visual cues with Task 1 (No
SAR) which did not utilise SAR visual cues. The SAR timer was utilised as it allows for more
accurate comparisons between the SAR visual cue tasks as less human error is involved.
The manual timer was implemented within the SAR module developed, which is further discussed
in Section 9.3.1. As stated, the manual timer was activated by the user before each task was
undertaken, and stopped once the task was completed. The manual timer was controlled by the user
pushing keys on a keyboard located on the same surface as the physical object to be arranged. To
activate the manual timer, the user pressed the “1” key. To stop the manual timer, the user pressed
the “2” key. Thus, for all tasks during the second run through, the following process was
undertaken:
1. The user presses “1” to activate the timer.
2. The user retrieves the physical object from a consistent location.
3. The user undertakes the arrangement task, leaving the object in an arranged position.
4. The user presses “2” to stop the timer.
52
The SAR timer was activated within the SAR system when the user brought the physical object
towards the centre of the table, within the range of the camera, allowing the marker to be
recognised by the SAR system. The SAR timer was stopped within the SAR system when the
object had been successfully arranged. If the physical object was moved again, after being
successfully arranged, the timer would resume timing. Once the user had completed the
arrangement task, the SAR timer value was recorded and reset to zero in anticipation for the next
task.
A user may operate the manual timer at varying speeds compared to other users. A user may also
operate the timer at varying speeds for each task that they performed. Utilising a SAR timer
provided decreased opportunity for varying timer values as a result of human error and other
human factors.
As the two timers gathered different data in different manners, it is best to view them standalone,
and not draw comparisons between manual timer results and SAR timer results. It is intended that
the SAR timer results give a more accurate indication of the task time difference between the
different SAR visual cues. It is intended that the manual timer results give an indication of the how
the manual arrangement task time varies compared to the task completion times of the SAR visual
cues, when values are gathered in the same manner.
53
5 Results
This section presents results gathered from the user study described in Section 4. Information
collected about the participants, including the average age of the participants and their previous
experience with AR and SAR systems is presented in Section 5.1. Quantitative data such as the
manual and SAR timer results and accuracy results are presented in Section 5.2, whereas
qualitative data such as the intuitiveness survey and user opinion survey results are presented in
Section 5.3. A discussion of the results is then provided in Section 5.4, including summation of the
major trends in the results gathered. This discussion also includes clarification of the manual timer
and SAR timer results. Appendix C presents the data gathered to form these results.
5.1 User InformationThe study involved twenty participants, consisting of nineteen males and one female. The average
age of the participants was 24.85, and the standard deviation of this data is 4.475. The participants
were asked to rate their experience level with any type of AR systems. Ten participants (or fifty
percent) of the user study stated they had no experience with AR systems, whilst seven participants
(or thirty-five percent) claimed they had some experience and three participants (or fifteen percent)
claimed they were experienced in using AR systems. Participants were also queried on whether
they had interacted with a SAR system previously. Fourteen participants (or seventy percent) of the
user study stated they had never interacted with a SAR system, whilst four participants (or twenty
percent) claimed they had interacted with a SAR system and two participants (or ten percent) were
unsure.
5.2 Quantitative Data
One of the aims of the user study undertaken was to determine:
the effectiveness, in terms of task time and accuracy, of different SAR visual cues versus a
manual method to instruct the arrangement of a physical object.
54
Task time was determined using a manual timer for all tasks and a SAR timer for all tasks that
utilise SAR visual cues. Accuracy was determined by gathering x axis, y axis and rotation values
for the final position and achieved final positions of the physical object after each arrangement
task.
5.2.1 Task Completion Times
For all tasks performed, values gathered with a manual timer will be presented. In addition, for the
tasks that utilise SAR visual cues, values gathered with a SAR timer will be presented. As Task 1
(No SAR) did not utilise a SAR visual cue, no SAR timer values were gathered for that task. Figure
37 compares task completion times gathered from the manual timer and the SAR timer.
1 2 3 4 5 6 7 8 905
1015202530354045
Average Completion Time of User Study Tasks as Timed with the Manual Timer and the SAR
Timer
Manual Timer
SAR Timer
Task Number
Tim
e (s
econ
ds)
Figure 37 Average Completion Time for all User Study Tasks as Timed with the Manual Timer and
the SAR Timer
When considering the manual timer results, Task 9 (Square Cue with No Destination) took the
shortest to complete, with an average completion time of 17.776 seconds and a standard deviation
of 8.679 seconds. Task 5 (Circle Cue) took the longest to complete, with an average completion
time of 41.154 seconds and a standard deviation of 25.229 seconds. The average completion time
of Task 1 (No SAR) was 32.348 seconds with a standard deviation of 13.773 seconds, making it
the second longest task to complete.
A pair wise T-test of the manual timer value for Task 1 (No SAR) with the best performing SAR
task (Task 9: Square Cue with No Destination) was performed. As a result, a significant effect on
55
completion time (p < 0.05) can be observed, indicating that Task 9 was significantly faster than
Task 1.
The notable difference with the values from the SAR timer when compared to the values of the
manual timer is that Task 8 (Square Cue) has been completed faster on average than Task 3
(Rotation First), Task 6 (Wireframe Cue) and Task 7 (Wireframe Cue with No Destination). Task 9
(Square Cue with No Destination) remains the fastest completed task. A discussion regarding the
manual timer results and the SAR timer results is present in Section 5.4.1.
5.2.2 Accuracy
For all tasks performed, values gathered regarding the accuracy along each axis; the x axis and the
y axis, and the rotation accuracy will be presented. The difference in the desired final translation
position and the achieved final translation position in terms of each axis were calculated and are
presented in Figure 38.
1 2 3 4 5 6 7 8 90
5
10
15
20
25
30
35
Average x and y Axis Variation for all User Study Tasks
X Axis Variation
Y Axis Variation
Task Number
Aver
age
Varia
tion
(uni
ts)
Figure 38 Average x and y Axis Variation for all User Study Tasks
Task 1 (No SAR) had the least accurate x axis translation: the achieved position diverged from the
desired final position by an average of 16.955 units with standard deviation of 22.708 units. This
was followed by Task 2 (Translation First) which had an average of 5.929 units of variation with a
standard deviation of 4.251. The most accurate task with regards to x axis translation was Task 4
(Both Instructions Cue) which had an average of 3.628 units of variation with a standard deviation
of 2.195.
56
The average x axis variation of all SAR visual cue tasks is 4.724 units with a standard deviation of
1.142 units, whereas the average x axis translation variation for Task 1 (No SAR) is 16.955 units
with a standard deviation of 22.708 units, a higher variation.
It can be stated with 95% confidence that the translation variation along the x axis when arranging
a physical object using the developed SAR visual cues will occur between 4.223 units and 5.224
units. It can also be stated with 95% confidence that the translation variation along the x axis for
Task 1 (No SAR) will occur within a higher range; between 7.003 units and 26.907 units.
As with the variation with the x axis translation, Task 1 (No SAR) also had the least accurate y axis
translation: the achieved position diverged from the desired final position by an average of 32.981
units with a standard deviation of 11.899 units. This was followed by Task 9 (Square Cue with No
Destination) which had an average of 5.290 units of variation with a standard deviation of 3.991
units. The most accurate task with regards to y axis translation was Task 3 (Rotation First) which
had an average of 4.260 units of variation with a standard deviation of 2.977 units.
The average translation variation along the y axis when arranging a physical object using the SAR
visual cues is 4.776 units with a standard deviation of 1.167 units, whereas the average y axis
translation variation for Task 1 (No SAR) is 32.981 units with a standard deviation of 11.899 units,
a higher variation.
It can be stated with 95% confidence that the translation variation along the y axis when arranging
a physical object using the developed SAR visual cues will occur between 4.265 units and 5.288
units. It can also be stated with 95% confidence that the translation variation along the y axis for
Task 1 (No SAR) will occur within a higher range; between 27.767 units and 38.196 units.
Rotation accuracy is now presented. The differences between the desired final rotation position and
the achieved final rotation position were calculated. Figure 39 shows the average variation of the
rotation for each task.
57
1 2 3 4 5 6 7 8 90
1
2
3
4
5
6
7
8
Average Rotation Variation of all User Study Tasks
Task Number
Aver
age
Rota
tion
Varia
tion
(deg
rees
)
Figure 39 Average Rotation Variation of all User Study Tasks
Task 1 (No SAR) also had the least accurate rotation: the achieved final position diverged from the
desired final position by an average of 7.467 degrees with a standard deviation of 9.208 degrees.
This was followed by Task 2 (Translation First) which had an average of 6.903 degrees of variation
with a standard deviation of 3.299 degrees. The most accurate task with regards to rotation was
Task 8 (Square Cue) which had an average of 3.319 degrees of variation with a standard deviation
of 2.368 degrees.
The average rotation variation when arranging a physical object using the developed SAR visual
cues is 5.280 degrees with a standard deviation of 0.899 degrees, whereas the average rotation
variation for Task 1 (No SAR) is 7.467 degrees with a standard deviation of 9.208 degrees, a higher
variation.
It can be stated with 95% confidence that the rotation variation when arranging a physical object
using the developed SAR visual cues will occur between 4.886 degrees and 5.675 degrees. It can
also be stated with 95% confidence that the rotation variation for Task 1 (No SAR) will occur
between 3.432 degrees and 11.502 degrees. The confidence values indicate that the variation for
Task 1 (No SAR) is likely to occur within a higher range; there may be many occurrences within
the range of 5.675 degrees and 11.502 degrees which is higher than the range indicated regarding
the SAR visual cues.
A pair wise T-test of the rotation variation for Task 1 (No SAR) with the SAR task with the most
accurate rotation (Task 8: Square Cue) was performed. As a result, a significant effect on rotation
58
accuracy (p < 0.05) can be observed, indicating that Task 8 was significantly more accurate than
Task 1.
5.3 Qualitative Data
The remaining aims of the user study undertaken were to determine:
the intuitiveness of several different SAR visual cues in SAR, and
user opinion of the tasks undertaken.
The intuitiveness of several different SAR visual cues was determined by a written activity
preceding the arrangement tasks, and the user’s opinion of the tasks undertaken was gathered upon
the participant’s completion of the user study.
5.3.1 Intuitiveness Survey
Results of the intuitiveness of the Translation Instruction described in Section 3.2.1 are presented
first.
Figure 40 Translation Instruction Intuitiveness Diagram
Participants were presented with written instructions and the intuitiveness diagram depicted in
Figure 40 as part of the survey presented in Appendix A. Sixteen participants (or eighty percent)
correctly identified the translation instruction being performed upon the physical object. The
remaining four participants (or twenty percent) were incorrect. Three of the participants who
answered incorrectly stated that the SAR visual cue was instructing both a rotation and a
59
translation. The other participant who answered incorrectly stated that the SAR visual cue was
instructing a translation, but to an incorrect location.
Results of Intuitiveness Study Regarding the Translaton Instruction
CorrectIncorrect
Figure 41 Results of the Intuitiveness Survey Regarding the Translation Instruction
Figure 41 shows a graphical depiction of the number of participants who were correct and the
number of participants who were incorrect when undertaking this intuitiveness task regarding the
Translation Instruction.
Results of the intuitiveness of the Rotation Instruction described in Section 3.2.2 are now
presented.
Figure 42 Rotation Instruction Intuitiveness Diagram
60
Participants were presented with written instructions and the intuitiveness diagram depicted in
Figure 42 as part of the survey presented in Appendix A. Fourteen participants (or seventy percent)
correctly identified the rotation instruction being performed upon the physical object. The
remaining six participants (or thirty percent) were incorrect. Three of the participants who
answered incorrectly thought the SAR visual cue was instructing both a rotation and a translation –
one of these participants had the correct rotation but had translated the object also. Two of the
participants who answered incorrectly stated that the SAR visual cue was instructing a rotation of
forty-five degrees, but in the incorrect direction. The other participant who answered incorrectly
stated that the SAR visual cue was instructing a translation.
Results of Intuitiveness Study Regarding the Rotation Instruction
CorrectIncorrect, but indicated a translationIncorrect
Figure 43 Results of the Intuitiveness Survey Regarding the Rotation Instruction
Figure 43 shows a graphical depiction of the number of participants who were correct and the
number of participants who were incorrect when undertaking this intuitiveness task regarding the
Rotation Instruction.
In addition to gathering data on the intuitiveness of the translation instruction described in Section
3.2.1 and the rotation instruction described in Section 3.2.2, data regarding the intuitiveness of the
modified translation and rotation instructions utilised for the Circle Cue was also gathered. Results
of the intuitiveness of the Circle Cue translation instruction described in Section 3.2.3 are now
presented.
61
Figure 44 Circle Cue Translation Instruction Intuitiveness Diagram
Participants were presented with written instructions and the intuitiveness diagram depicted in
Figure 44 as part of the survey presented in Appendix A. Eighteen participants (or ninety percent)
correctly identified the translation instruction being performed upon the physical object. The
remaining two participants (or ten percent) were incorrect. The remaining two participants who
answered incorrectly thought the SAR visual cue was instructing a rotation. Both participants
indicated the same rotation whereby the physical object would be rotated so that one of the ends of
it would point in the direction of the circle not projected upon the object.
Results of Intuitiveness Study Regarding the Circle Cue Translation Instruction
CorrectIncorrect
Figure 45 Results of the Intuitiveness Survey Regarding the Circle Cue Translation
62
Figure 45 shows a graphical depiction of the number of participants who were correct and the
number of participants who were incorrect when undertaking this intuitiveness task regarding the
Circle Cue translation instruction.
Results of the intuitiveness of the Circle Cue rotation instruction described in Section 3.2.3 are now
presented.
Figure 46 Circle Cue Rotation Instruction Intuitiveness Diagram
Participants were presented with written instructions and the intuitiveness diagram depicted in
Figure 46 as part of the survey presented in Appendix A. One participant (or five percent) correctly
identified the rotation instruction being performed upon the physical object. The remaining
nineteen participants (or ninety-five percent) were incorrect. Of the nineteen participants who
answered incorrectly, one thought that the SAR visual cue was instructing a rotation, but of 90
degrees, and the remaining participants thought that the SAR visual cue was instructing no
movement, and thus the physical object should remain in an unchanged position.
63
Results of Intuitiveness Study Regarding the Circle Cue Rotation Instruction
CorrectIncorrectIncorrect, but indicated a ro-tation
Figure 47 Results of the Intuitiveness Survey Regarding the Circle Cue Rotation
Figure 47 shows a graphical depiction of the number of participants who were correct and the
number of participants who were incorrect when undertaking this intuitiveness task regarding the
Circle Cue rotation instruction.
5.3.2 User Opinion Survey
Upon completion of the tasks, participants were asked to complete a series of questions gauging
their opinions of the different SAR visual cues in the tasks undertaken; results of this survey are
now presented.
When asked which of the tasks was better, out of Task 1 (No SAR) and Task 2 (Translation First),
nineteen participants (or ninety-five percent) claimed to prefer Task 2. Figure 48 shows a graphical
depiction of the preferences of the participants.
64
User Preference Between a Manual Ob-ject Arrangement and an Object Ar-
rangement Using SARPreference for Task 1 (No SAR)Preference for Task 2 (Trans-lation First)
Figure 48 User Preference Between a Manual Object Arrangement and an Object Arrangement Using
SAR
When asked which of the tasks was better, out of Task 2 (Translation First), Task 3 (Rotation First)
and Task 4 (Both Instructions Cue), four participants (or twenty percent) claimed to prefer Task 2,
one participant (or five percent) claimed to prefer Task 3, and fourteen participants (or seventy
percent) claimed to prefer Task 4. The remaining participant had no preference, claiming that Task
2 and Task 3 allowed the user to be stepped through the object arrangement process if there is
unfamiliarity with the arrangement process, but once familiarity with the arrangement process is
gained, the tasks can be completed faster when rotation and translation instructions are both visible
as in Task 4. Figure 49 shows a graphical depiction of the preferences of the participants.
User Preference Between Translation First, Rotation First and Both Instructions
Cue Preference for Task 2 (Trans-lation First)Preference for Task 3 (Ro-tation First)Preference for Task 4 (Both Instructions Cue)No Preference
Figure 49 User Preference Between Translation First, Rotation First and Both Instructions Cue
65
When asked which of the tasks was better, out of Task 2 (Translation First) and Task 5 (Circle
Cue), nineteen participants (or ninety-five percent) claimed to prefer Task 2. The remaining
participant had no preference, and claimed that Task 2 and Task 5 were as good as each other.
Figure 50 shows a graphical depiction of the preferences of the participants.
User Preference Between Translation First and Circle Cue
Preference for Task 2 (Trans-lation First)Preference for Task 5 (Circle Cue)No Preference
Figure 50 User Preference between Translation First and Circle Cue
When asked which of the tasks was better, out of Task 6 (Wireframe Cue) and Task 7 (Wireframe
Cue with No Destination), twelve participants (or sixty percent) claimed to prefer Task 6 and five
participants (or twenty-five percent) claimed to prefer Task 7. The remaining three participants had
no preference. One of the participants who claimed that they had no preference stated that the
wireframe was difficult to see in both tasks and this was a factor in the participant having no
preference. Another of the participants who claimed that they had no preference stated that they did
not utilise the wireframe during the arrangement process. Rather, this participant claimed that they
used only the translation instruction as described in Section 3.2.1 and rotation instruction as
described in Section 3.2.2 that were also projected upon the physical object. Figure 51 shows a
graphical depiction of the preferences of the participants.
66
User Preference Between Wireframe Cue and Wireframe Cue with No Destination
Preference for Task 6 (Wire-frame Cue)Preference for Task 7 (Wire-frame Cue with No Destina-tion)No Preference
Figure 51 User Preference Between Wireframe Cue and Wireframe Cue with No Destination
When asked which of the tasks was better, out of Task 8 (Square Cue) and Task 9 (Square Cue
with No Destination), eighteen participants claimed to prefer Task 8, and one participant claimed to
prefer Task 9. The remaining participant had no preference. Figure 52 shows a graphical depiction
of the preferences of the participants.
User Preference Between Square Cue (Task 8) and Square Cue with No Des-
tination (Task 9)Preference for Task 8 (Square Cue)Preference for Task 9 (Square Cue with No Destination)No Preference
Figure 52 User Preference Between Square Cue (Task 8) and Square Cue with No Destination (Task 9)
The participants were also asked which of the groups of tasks they preferred out of the tasks that
utilised a wireframe projection (Task 6 and Task 7) and the tasks that utilised a square projection
(Task 8 and Task 9). Fifteen of the participants claimed to prefer the tasks that utilised a square
projection and five of the participants claimed to prefer the tasks that utilised the wireframe
projection. Figure 53 shows a graphical depiction of the preferences of the participants.
67
User Preference Between Square Cues (Task 8 and Task 9) and Wireframe Cues
(Task 6 and Task 7)Preference for the Wireframe Cues (Task 6 and Task 7)Preference for the Square Cues (Task 8 and Task 9)
Figure 53 User Preference Between Square Cues (Task 8 and Task 9) and Wireframe Cues (Task 6
and Task 7)
Finally, the users were asked for their preference of the tasks that utilised the translation instruction
as described in Section 3.2.1 and the rotation instruction as described in Section 3.2.2. Thus, Task 1
(No SAR) and Task 5 (Circle Cue) were omitted from this question. This question determined that:
one participant (or five percent) claimed to prefer Task 2 (Translation First),
no participants (or zero percent) claimed to prefer Task 3 (Rotation First),
six participants (or thirty percent) claimed to prefer Task 4 (Both Instructions Cue),
two participants (or ten percent) claimed to prefer Task 6 (Wireframe Cue),
no participants (or zero percent) claimed to prefer Task 7 (Wireframe Cue with No
Destination),
seven participants (or thirty-five percent) claimed to prefer Task 8 (Square Cue), and
no participants (or zero percent) claimed to prefer Task 9 (Square Cue with No
Destination).
The remaining four participants indicated that they had no clear preference of the SAR visual cues
encountered in the tasks undertaken. One of the remaining participants claimed that they preferred
both Task 4 (Both Instructions Cue) and Task 6 (Wireframe Cue) as they both allowed for natural
movement. Another of the remaining participants had a similar claim. They claimed that they
would have preferred a combination of Task 4 (Both Instructions Cue) and Task 7 (Wireframe Cue
68
with No Destination) as they felt it would provide a generic approach that is applicable in a range
of situations.
2 3 4 6 7 8 902468
1012
User Preference of Visual Cues that Util-ise the Translation and Rotation In-
structionsPreferences for the Visual Cue as Part of a Com-bination
Preference for the Visual Cue
Task Number
Num
ber o
f Pre
fere
nces
Figure 54 User Preference of Visual Cues that Utilise the Translation and Rotation Instructions
Another of the remaining participants claimed that they would have preferred a combination of
Task 4 (Both Instructions Cue) and Task 8 (Square Cue). Another of the remaining participants had
a similar claim. They claimed that they preferred Task 4 (Both Instructions Cue) as they felt it
would be applicable to a wider variety of situations, such as the arrangement of round physical
objects; however they also preferred Task 8 (Square Cue) in the tasks undertaken during the user
study. Figure 54 shows a graphical depiction of the preferences of the participants.
5.4 Discussion
When considering the manual timer results, Task 9 (Square Cue with No Destination) was the
quickest task completed. Since Task 9 is also the last task undertaken, it can be suggested that this
quick task time is attributed to a learning effect. However, the manual timer results of the
remainder of the tasks do not appear to conform to this trend and there is also no trend present
regarding the accuracy values that suggests a learning effect.
When further considering the manual timer results, all of the tasks that utilised SAR visual cues
were quicker than Task 1 (No SAR) except for the Task 5 (Circle Cue). Unlike the remainder of the
SAR visual cues, the Circle Cue implements variations of the Translation and Rotation Instructions
described in Section 3.2.1 and Section 3.2.2. Thus, it can be stated that all of the cues that utilised
the Translation and Rotation Instructions as described in Section 3.2.1 and Section 3.2.2 where
quicker than Task 1 (No SAR). In addition, a pair wise t-test determined that Task 9 (Square Cue
with No Destination) was significantly faster than Task 1. 69
In addition, Task 1 (No SAR) also resulted in the least accurate physical object arrangements. This
inaccuracy was present when considering all accuracy data gathered: the x axis accuracy, y axis
accuracy and the rotation accuracy.
When considering the intuitiveness of the Translation Instruction described in Section 3.2.1 and the
Rotation Instruction described in Section 3.2.2, the majority of the participants were able to
correctly identify both of the instructions when presented with a diagram of each. However, when
considering the intuitiveness of the translation and rotation instructions that were utilised for the
Circle Cue, the majority of the participants could identify the translation when presented with a
diagram, but only one participant was able to identify the rotation when presented with a diagram.
When considering the results of the user opinion survey undertaken, Task 2 (Translation First Cue)
was rated favourably when compared with tasks in which the arrangement method varied, such as
the manual arrangement in Task 1 (No SAR) and Task 5 (Circle Cue) as the Circle Cue
implemented a differing translation and rotation than the rest of the SAR visual cues. However,
Task 4 (Both Instructions Cue) was preferred over Task 2 (Translation First Cue). The SAR visual
cues which utilised square projections were also rated favourably, and out of these, Task 8 (Square
Cue) was preferred.
In summation, the SAR visual cues that implement the Translation Instruction and Rotation
Instruction provide a quicker arrangement than a manual arrangement method, and the SAR visual
cues provide a more accurate arrangement than a manual arrangement method. The Translation
Instruction and Rotation Instruction were more intuitive than variations of these that were utilised.
The Both Instructions Cue and Square Cue were considered favourably. From these results, a
recommendation for either the Both Instructions Cue or the Square Cue can be made. In addition to
receiving a favourable user opinion, they are consistent with the advantages found with the results;
they are both SAR visual cues that implement the Translation and Rotation Instructions and thus
are quicker and more accurate than a manual arrangement, as well as being more intuitive. The
hypothesis of the user study is validated as there is an improvement in speed and accuracy when
comparing the SAR visual cues with the manual timer method.
5.4.1 Notes on the Manual Timer and SAR Timer Results
Data was gathered from both a manual timer and a SAR timer. The gathering of this data was
described in Section 4.4.3. The manual timer results were higher than the corresponding SAR timer
results gathered. To explain this, the steps of undertaking a task in the user study are revisited:
1. The user presses “1” to activate the timer.
2. The user retrieves the physical object from a consistent location.
70
3. The user undertakes the arrangement task, leaving the object in an arranged position.
4. The user presses “2” to stop the timer.
With regards to this process, the manual timer is activated when the user presses “1”, but the SAR
timer is activated later, when the user brings the physical object to the centre of the table. Similarly,
with regards to this process, the manual timer is stopped when the user presses “2”, but the SAR
timer is stopped earlier, when the physical object is successfully arranged. As manual timer is
activated earlier and stopped later, it is evident that this is a reason why the manual timer values are
larger.
However, the manual timer values are not consistently larger. For example, when considering the
SAR timer values, Task 8 (Square Cue) is performed quicker than Task 3 (Rotation First Cue),
Task 6 (Wireframe Cue) and Task 7 (Wireframe Cue with No Destination). However, when
considering the manual timer values for these cues, Task 8 (Square Cue) is performed slower than
the other specified tasks. This is attributed to human factors. As stated in Section 4.4.3, a user may
operate the timer at varying speeds for each task that they performed.
To reiterate, as the two timers gathered different data in different manners, it is best to view them
standalone, and not draw comparisons between manual timer results and SAR timer results. It is
intended that the SAR timer results give a more accurate indication of the task time difference
between the different SAR visual cues. It is intended that the manual timer results give an
indication of the how the manual arrangement task time varies compared to the task completion
times of the SAR visual cues when values are gathered in the same manner.
71
6 Revised Visual Cue
A revised visual cue was developed with consideration to the results presented in Section 5.
Analysis of the user study results indicated several trends which were taken into account when
developing the revised visual cue. The analysis performed is detailed in Section 6.1. Details of the
methodology and implementation undertaken to develop the Revised Visual Cue are presented in
Section 6.2. A discussion of the Revised Visual Cue is presented in 6.3, including reasons for its
particular implementation. A pilot study was then undertaken to gather preliminary results
regarding the revised SAR visual cue. A description of this pilot study and the results gathered are
presented in Section 7.
6.1 Analysis
A revised SAR visual cue was developed by considering the:
manual timer results and the SAR timer results: Task 9 (Square Cue with No Destination)
was the quickest completed task when assessed with both the manual timer and the SAR
timer. Task 8 (Square Cue) was the second quickest completed task when assessed with the
SAR timer. All SAR visual cues that utilised the Translation Instruction and Rotation
Instruction were completed quicker than Task 1 (No SAR),
accuracy results: Task 4 (Both Instructions Cue) had the most accurate x axis translation.
Task 3 (Rotation First Cue) had the most accurate y axis translation. Task 8 (Square Cue)
was completed with the most accurate rotation,
intuitiveness results: The translation instruction described in Section 3.2.1 was more
intuitive than the Circle Cue translation as described in Section 3.2.3. Similarly, the
rotation instruction described in Section 3.2.2 was more intuitive than the Circle Cue
rotation as described in Section 3.2.3, and
72
user opinion survey results: SAR visual cues are preferred over a manual arrangement.
Participants preferred Task 4 (Both Instructions Cue) over SAR visual cues that displayed
either a translation instruction or rotation instruction at any one time. Participants preferred
the SAR visual cues which utilised the Translation Instruction as described in Section 3.2.1
and the Rotation Instruction as described in Section 3.2.2, over the Circle Cue which
utilised variations of those instructions as described in Section 3.2.3. There was a
preference for SAR visual cues that displayed a projection upon the physical object and the
same projection at the destination position. There was a seventy five percent preference for
the SAR visual cues that utilised the Square projection over SAR visual cues that utilised
the wireframe projections. Task 4 (Both Instructions Cue) and Task 8 (Square Cue) utilised
the most preferred visual and second most preferred SAR visual cue.
Thus, it was determined that the revised SAR visual cue should build upon elements of the Square
Cue, the Both Instructions Cue, the Translation Instruction as described in Section 3.2.1 and the
Rotation Instruction as described in Section 3.2.2.
6.2 Methodology and ImplementationThe Translation Instruction is utilised along with a variation of the Rotation Instruction and two
projections of a revised shape. One shape projection is aligned with the centre of the top of the
physical object and follows the physical object as it is moved. The other shape projection is present
at the desired final position. The SAR visual cue follows the process described by Figure 6 with the
Translation Instruction occurring first.
73
Figure 55 Revised Visual Cue: Translation
Figure 55 depicts the initial state of the physical object. The implementation of this initial state is
presented in Figure 56.
Figure 56 Revised Visual Cue: Translation Implementation
As with the Translation First Cue described in Section 3.2.3.1, the Translation Instruction is
displayed and, in addition, the revised shape projections are displayed. The Translation Instruction
74
will guide the user to move the physical object to a position that corresponds with the desired final
translation. In the scenario presented by Figure 56, the user must translate the object in a south
westerly direction, so that the circles are aligned. This can be achieved by following the line
projections. It can also be achieved by aligning the two shape projections. Once the user has moved
the physical object so that the translation is correct, the Translation Instruction ceases to be
displayed.
Figure 57 depicts, from a top down view, the process of undertaking a translation using the Revised
Visual Cue. The tracked physical object is represented by a solid shape whilst the desired final
position of the object is represented by a dotted shape.
Figure 57 Revised Visual Cue: Translation
Upon completion of the scenario depicted in Figure 57, the tracked physical object would be
successfully arranged as the revised shape is aligned and thus both the translation and rotation have
been completed successfully. However, if the user has arranged the tracked physical object at a
position with a correct translation but an incorrect rotation, a rotation instruction is displayed.
75
Figure 58 Revised Visual Cue: Rotation
Figure 58 depicts the state of the physical object once the initial translation has been completed and
rotation is now being performed. The implementation of this state is presented in Figure 59.
Figure 59 Revised Visual Cue: Rotation Implementation
A variation of the Rotation Instruction is now displayed to prompt the user to rotate the physical
object; an arrow is displayed in a similar distance from the centre of the object as the circles
76
displayed for the Rotation Instruction described in Section 3.2.2. The arrow indicates the direction
that the user must rotate the physical object in order to achieve a position that corresponds with the
desired final rotation. The current rotation follows the x axis of the tracked physical object’s local
coordinate system. If it is imagined that the destination rotation is at zero, then when the object is
rotated to any degree between 0 and 180, the arrow will point left towards zero. If the object is
rotated past 180, to 270 degrees, for example, the arrow will instead be inverted, in which case it
will point right towards zero.
In the scenario presented by Figure 59, the user must rotate the object clockwise, as indicated by
the arrow. Once the user has moved the physical object to a position that corresponds with the
desired final translation and rotation, the SAR visual cue will no longer be displayed, indicating
that the physical object has been successfully arranged.
Figure 60 Revised Visual Cue: Rotation
Figure 60 depicts the process of undertaking a rotation using the Revised Visual Cue. The tracked
physical object is represented by a solid shape whilst the desired final position of the object is
represented by a dotted shape.
6.3 DiscussionA square has four axes of symmetry, and as a result, the Square Cue communicated rotation
information poorly as when the user attempted to align the square shapes, the rotation of the
squares was ambiguous. As the revised shape utilised in the revised visual has only one axis of
symmetry, rotation information can be communicated in a less ambiguous manner.
For the shape to be successfully arranged the user is required to align the revised shape and there is
only one position in which this alignment can be achieved correctly, which can be verified visually.
Although the desired final rotation can be gathered from the revised shape, the arrow is displayed if
the user arranges the object in a way such that the translation is complete but the rotation is
77
incomplete. In this scenario, the user has moved the object so that the revised shapes are
overlapping but not aligned, the display of the arrow aids the user in rotating the object if the
desired final rotation has not been achieved.
78
7 Pilot Study
This section presents details of the pilot study undertaken to gather preliminary results on the
effectiveness of the Revised Visual Cue developed. Results gathered from the pilot study are then
presented. The pilot study was undertaken in a similar manner to the user study performed. The
pilot study consisted of six participants. All six participants had previously participated in the user
study. The pilot study endeavoured to determine preliminary results regarding:
the effectiveness, in terms of task time and accuracy, of the Revised Visual Cue,
the intuitiveness of the Revised Visual Cue, and
the user opinion of the task undertaken.
Data was gathered using the methods utilised for the user study which are described in Section 4.4.
The pilot study configuration resembles the configuration of the user study which is defined in
Section 4.1. The only task undertaken was a physical object arrangement task utilising the Revised
Visual Cue. The structure of the pilot study is outlined in Section 7.1 and the preliminary results
gathered are presented in Section 7.2.
7.1 StructureThe pilot study was taken with one participant at a time. Each participant undertook the following
procedure:
1. The participant completed a survey to gather information on the intuitiveness of the
Revised Visual Cue
2. The participant completed a task resembling the structure of the tasks undertaken during
the user study. The task was to arrange a tracked physical object utilising the Revised
Visual Cue. The task was not timed during the first run through.
3. The participant completed the task again, which followed the aforementioned structure,
however the task was timed. The participant activated a manual timer before undertaking
the task, and stopped the timer once the task was completed. The timer was activated by 79
the user pushing a key on a keyboard located on the same surface as the physical object to
be arranged. The physical object was collected by the user from a consistent location after
the timer was activated.
4. The participant completed a survey to gather their opinions of the tasks they had
undertaken.
Appendix B presents the surveys given to the participants to gather the qualitative data.
7.2 Preliminary Results
This section presents the results gathered from the pilot study. Information specific to the
participants is presented in Section 7.2.1. Quantitative data such as the manual and SAR timer
results are presented in Section 7.2.2, whereas qualitative data such as the intuitiveness survey and
user opinion survey results are presented in Section 7.2.3. A discussion of the results is then
presented in Section 7.2.4.
The pilot study data is shown on the graphs alongside the data gathered from the user study tasks,
to provide an indication of the effectiveness of the Revised Visual Cue. The Revised Visual Cue
task is represented on graphs as Task 10.
7.2.1 User Information
The pilot study involved six participants, consisting of six males and zero females. The average age
of the participants was 24 with a standard deviation of 3.521. All participants had previously
participated in the user study, thus they all had experience with the SAR system utilised. Appendix
C presents the data gathered to form these preliminary results.
7.2.2 Quantitative Data
One of the aims of the pilot study was to determine:
the effectiveness, in terms of task time and accuracy, of the Revised Visual Cue.
This was achieved in the same manner as the user study undertaken. Task time was determined
using a manual timer and a SAR timer. Accuracy was determined by gathering x axis, y axis and
rotation values for the desired final position and achieved final positions of the physical object after
the task.
80
7.2.2.1 Task Completion Times
For the task undertaken, values gathered with a manual timer and values gathered with a SAR timer
will be presented. Figure 61 compares the values gathered from the manual timer and the SAR
timer for the Revised Visual Cue with the values gathered from the manual timer and the SAR
timer for the tasks undertaken in the user study. The Revised Visual Cue is represented as Task 10.
1 2 3 4 5 6 7 8 9 1005
101520253035404550
Average Completion Time of all User Study Tasks and the Revised Visual Cue (Manual
Timer and SAR Timer)
Manual Timer
SAR Timer
Task Number
Tim
e (s
econ
ds)
Figure 61 Average Completion Time of all User Study Tasks and the Revised Visual Cue (Manual
Timer and SAR Timer)
Task 10 (Revised Visual Cue) took the shortest to complete, with an average completion time of
12.570 seconds with a standard deviation of 4.321 seconds. According to the manual timer, Task
10 (Revised Visual Cue) was completed 5.205 seconds faster than Task 9 (Square Cue with No
Destination), which was the task with the shortest completion time in the user study.
With regards to the values gathered with a manual timer, it can be stated with 95% confidence that
the task completion time when arranging a physical object using the Revised Visual Cue will occur
between 9.112 seconds and 16.027 seconds. From the results gathered from the user study, it can
also be stated with 95% confidence that the task completion time for Task 1 (No SAR) will occur
within a higher range; between 26.312 seconds and 38.384 seconds.
Task 10 (Revised Visual Cue) also took the shortest to complete when timed with the SAR timer.
An average completion time of 5.597 seconds with a standard deviation of 2.939 seconds was
recorded. According to the SAR timer, Task 10 (Revised Visual Cue) was completed 7.042
81
seconds faster than Task 9 (Square Cue with No Destination), which was the task with the shortest
completion time in the user study.
7.2.2.2 Accuracy
For the task undertaken, results regarding the translation accuracy along each axis; the x axis and
the y axis, and the rotation accuracy will be presented.
As with the user study, the difference in the desired final translation and the achieved final
translation in terms of each axis were calculated. Figure 62 compares the average variation of the
translation along the x axis and y axis for the Revised Visual Cue with the values gathered for the
tasks undertaken in the user study. The Revised Visual Cue is represented as Task 10.
1 2 3 4 5 6 7 8 9 100
5
10
15
20
25
30
35
Average x Axis and y Axis Variation of all User Study Tasks and the Revised Visual Cue
x Axis Variationy Axis Variation
Task Number
Aver
age
x Ax
is Va
riatio
n (u
nits
)
Figure 62 Average x Axis and y Axis Variation of all User Study Tasks and the Revised Visual Cue
Task 10 (Revised Visual Cue) had the most accurate x axis translation: the achieved final
translation diverged from the desired final translation by an average of 2.770 units with a standard
deviation of 2.538 units. The most accurate task undertaken in the user study regarding the x axis
variation was Task 4 (Both Instructions Cue), which had an average x axis variation of 3.628 units.
Thus, Task 10 (Revised Visual Cue) obtains an x axis translation that is, on average, 0.858 units
more accurate than Task 4 (Both Instructions Cue).
It can be stated with 95% confidence that the translation variation along the x axis when arranging
a physical object using the Revised Visual Cue will occur between 0.739 units and 4.801 units.
From the results gathered from the user study, it can also be stated with 95% confidence that the
82
translation variation along the x axis for Task 1 (No SAR) will occur within a higher range;
between 7.003 units and 26.907 units.
Task 10 (Revised Visual Cue) also had the most accurate y axis translation: the achieved final
translation diverged from the desired final translation by an average of 1.839 units with a standard
deviation of 1.651 units. The most accurate task undertaken in the user study regarding the y axis
variation was Task 3 (Rotation First Cue), which had an average y axis variation of 4.260 units.
Thus, Task 10 (Revised Visual Cue) obtains a y axis translation that is, on average, 2.420 units
more accurate than Task 3 (Rotation First Cue).
It can be stated with 95% confidence that the translation variation along the y axis when arranging
a physical object using the Revised Visual Cue will occur between 0.519 units and 3.160 units.
From the results gathered from the user study, it can also be stated with 95% confidence that the
translation variation along the y axis for Task 1 (No SAR) will occur within a higher range;
between 27.767 units and 38.196 units.
Results regarding the rotation accuracy are now presented. As with the user study, the difference in
the desired final rotation and the achieved final rotation were calculated. Figure 63 compares the
average variation of the rotation for the Revised Visual Cue with the values fathered for the tasks
undertaken in the user study. The standard deviation of this data is 9.076. The Revised Visual Cue
is represented as Task 10. The standard deviation of this data is 1.253.
1 2 3 4 5 6 7 8 9 100
1
2
3
4
5
6
7
8
Average Rotation Variation of all User Study Tasks and the Revised Visual Cue
Task Number
Aver
age
Rota
tion
Varia
tion
(deg
rees
)
Figure 63 Average Rotation Variation of all User Study Tasks and the Revised Visual Cue
83
Task 10 (Revised Visual Cue) had the fourth most accurate rotation: the achieved position diverged
from the desired final rotation by an average of 4.835 degrees with a standard deviation of 1.079
units. Thus, Task 10 (Revised Visual Cue) obtains a rotation that is, on average, more accurate than
six of the object arrangement tasks undertaken. The most accurate task undertaken in the user study
regarding the rotation variation was Task 8 (Square Cue), which had an average rotation variation
of 3.319 degrees. Thus, Task 10 (Revised Visual Cue) obtains a rotation that is, on average, 1.516
degrees less accurate than Task 8 (Square Cue).
It can be stated with 95% confidence that the rotation variation when arranging a physical object
using the Revised Visual Cue will occur between 3.971 units and 5.698 units. It can also be stated
with 95% confidence that the rotation variation for Task 1 (No SAR) will occur within 3.431 units
and 11.502 units. The confidence values indicate that the rotation variation for Task 1 (No SAR) is
likely to occur within a higher range; there may be many occurrences within the range of 5.698
degrees and 11.502 degrees which is higher than the range indicated regarding the Revised Visual
Cue.
7.2.3 Qualitative Data
The remaining aims of the pilot study are to determine:
the intuitiveness of the Revised Visual Cue, and
user opinion of the task undertaken.
This was achieved in the same manner as the user study undertaken.
7.2.3.1 Intuitiveness Survey
An intuitiveness survey was undertaken before the Revised Visual Cue task. The survey utilised is
included in Appendix B.
84
Figure 64 Revised Visual Cue Translation Instruction Intuitiveness Diagram
When presented with the diagram depicted in Figure 64, six participants (or one hundred percent)
correctly identified the translation instruction being performed upon the physical object utilising
the Revised Visual Cue.
Figure 65 Revised Visual Cue Rotation Instruction Intuitiveness Diagram
Similarly, when presented with the diagram depicted in Figure 65, six participants (or one hundred
percent) correctly identified the rotation instruction being performed upon the physical object
utilising the Revised Visual Cue.
85
7.2.3.2 User Opinion Survey
Upon completion of the task, participants were asked to complete a series of questions gauging
their opinions of the Revised Visual Cue in the task undertaken. Participants were asked what was
better about the Revised Visual Cue when compared with the SAR visual cues experienced in the
user study. Two participants commented that the rotation was easier to achieve, and another
participant claimed that the arrow used was easier to understand than the Rotation Instruction
utilised previously. Participants were also asked what was worse about the Revised Visual Cue
compared with the SAR visual cues experienced in the user study. Three participants claimed that
nothing was worse, one participant claimed to have experienced confusion over which projection
was the projection for the current position. The comment was also made that, as the revised shape
was more complex than the square used previously, it caused some confusion when both of the
revised shape projections were aligned. When asked whether the Revised Visual Cue was preferred
over the SAR visual cues encountered in the user study, six participants (or one hundred percent)
stated that it is.
7.2.4 Discussion
The preliminary results from the pilot study have been presented alongside the user study results.
This allows for comparison between the two data sets. The preliminary results show promising
indications that the Revised Visual Cue allows for a faster physical object arrangement when
compared to a manual arrangement and the other SAR visual cues developed. In addition, the
preliminary results also show promising indications that the Revised Visual Cue allows for a more
accurate physical object arrangement along the x and y axes when compared to a manual
arrangement and the other SAR visual cues developed. Preliminary results regarding the rotation
accuracy are also promising in comparison to a manual arrangement; however the rotation accuracy
results gathered for the Revised Visual Cue are not as accurate as those gathered for the Square
Cue from the user study. A participant stated on the user opinion survey that the revised shape used
for the Revised Visual Cue was complex and therefore caused confusion when the projection at the
current position was overlapped with the projection at the desired final position; this may have
caused the rotation to be performed better with the Square Cue as a simpler shape is utilised. In
addition, preliminary results regarding the intuitiveness and user opinion of the Revised Visual Cue
are also positive. Thus, it appears that the Revised Visual Cue developed appears to be heading
toward an improvement regarding SAR object arrangement.
86
8 Large Object Arrangement
This section provides discussion of a scenario in which a modified SAR visual cue is utilised for
the instructed arrangement of a large physical object. The research described in this dissertation
until this point has addressed the arrangement of objects that can be easily arranged by hand and
that can be effectively projected upon. When considering the arrangement of large physical objects
using SAR visual cues, the scenario differs from that addressed previously in this dissertation. For
example, the physical object utilised for the user study had a maximum dimension of
approximately two hundred and fifty millimetres and was arranged upon a tabletop. Projections
were made upon the top of the object as the user was able to view these projections easily. If a
projection is made upon the top of a large physical object, it may be that the user is not able to view
it as they cannot see the top of the physical object due to its height.
87
Figure 66 SAR Visual Cue Distortion
To present another example, in the scenario where a SAR visual cue projection was to exceed the
surface area of the top of a physical object, portions of the projection would instead be projected
upon the surrounding surface. This is depicted in Figure 66, where the physical object utilised in
the user study is being projected upon. This issue is present with the arrangement of smaller
physical objects, addressed in Section 3.2.4.2, but is magnified when arranging large objects as the
distance to the ground plane may be further, resulting in the portion of the projection being
projected at a further distance from the projection upon the physical object, which may result in the
SAR visual cue being considerably less clear. This problem may also be undesirable in the
arrangement of smaller physical objects, and thus the adapted SAR visual cue for large object
arrangement may also be desirable for use with smaller physical objects.
In order to address these differences in the task of arranging large objects with SAR visual cues,
the Revised Visual Cue described in Section 6 was adapted to develop a new arrangement
technique. The developed SAR visual cue utilises the shape comprised of a square and semi-circle
that was utilised for the Revised Visual Cue. Four of these shapes are projected around the current
position of the physical object and around the desired final position; the user must align the shapes
in order to successfully arrange the large physical object. With the projection at the current position
of the physical object, four arrows are projected; one adjacent to each of the shapes, which indicate
88
the direction that the large physical object must be rotated. A projected line presents a path from
the centre point of the current position to the centre point of the desired final position, to aid the
user if the end point is not visible. As the user moves the physical object, the projection updates,
following the physical object.
The shape utilised has only one axis of symmetry; if a square projection was utilised, ambiguity
would be present as the rotation could be incorrect by 90 degrees. The rotation of the desired final
position shown in Figure 66 is currently at 45 degrees, if a square were used the projection of
shapes shown would appear the same at 45 degrees and 135 degrees, for example; making the
projection at the desired final position ambiguous. The presence of the arrows would aid in
correcting this issue, however utilising the shapes with only one axis of symmetry removes the
ambiguity in this situation as when the projection is rotated by 90 degrees it will not look the same.
To illustrate this, when considering the rotation of the final position in Figure 66, the semi-circle
shapes are directed towards the south east, but if the projection were to be rotated clockwise by 90
degrees, the semi-circle shapes would instead be directed towards the west.
Utilising four shapes at the current position and at the desired final position allows for occlusion;
even if one or more of the shapes is occluded or distorted by the large physical object or others in
the vicinity, the SAR visual cue is still able to be understood. The four shapes also aid the SAR
visual cue in being viewed from multiple vantage points which may be especially useful for the
arrangement of large physical objects as multiple users may partake in the arrangement process.
89
Figure 67 Large Object Arrangement: Initial Position
In the scenario presented in Figure 67, the user must translate the large physical object in a south
easterly direction. The four arrows are present on the projection at the large physical object’s
current position. These arrows are indicating that the large physical object should be rotated in a
clockwise direction. This rotation is also indicated by the rotation that has been applied to the
projection of the shapes at the desired final position.
90
Figure 68 Large Object Arrangement: After a Translation
Figure 68 presents the large physical object after a translation has been performed. The user has
translated the object in the correct direction, and the projection has updated to show the current
position of the large physical object.
91
Figure 69 Large Object Arrangement: Translation Complete
Figure 69 presents the large physical object after the translation has been completed. The arrows
remain, indicating to the user that the large physical object must be rotated in a clockwise direction.
Occlusion occurs frequently when utilising SAR visual cues to arrange large physical objects; this
occlusion is clearly evident in Figure 69. This illustrates one of the reasons for utilising four
projected shapes at the current position and the desired final position; the SAR visual cue is still
able to be understood even if one or more of the shapes is occluded or distorted.
92
Figure 70 Large Object Arrangement: Rotation
Figure 70 presents the large physical object after a rotation has been completed. The projection has
been updated as the user rotates the large physical object. It is evident in Figure 70 that the object is
almost successfully arranged, and the user can successfully arrange the object by aligning the
shapes which is achieved by rotating the large physical object in the direction indicated by the
arrows.
93
Figure 71 Large Object Arrangement: Successfully Arranged
Figure 71 presents the successful arrangement of the large physical object. The arrows indicating
that a rotation is to be performed are no longer projected, but will return if the physical object is
moved from the correct final position.
94
9 Implementation
The developed system described in this dissertation utilises SAR to instruct a user on how to
arrange a physical object. To achieve this, existing components have been utilised and new
components have been developed. A description of how the existing and developed components are
structured to form the developed system is provided as an overview of the developed system in
Section 9.1.
The existing components utilised are a SAR system and a tracking framework. These existing
components are described in Section 9.2. Several methods for tracking were also explored. The
new components that have been developed during this research are a module for the existing SAR
system that implements object arrangement techniques and a top down arrangement application
that aids the object arrangement process. These developed components are described in Section 9.3.
9.1 Overall Structure
Figure 72 Overall Implemented System Structure
95
The research described in this dissertation utilised several components. An overall structure of the
system incorporating these existing and new components is depicted in Figure 72. The research
utilises an existing SAR system for which a module was created and other existing modules were
utilised. An accompanying top down arrangement application was also implemented to aid the user
in arranging physical objects. A tracking framework and camera were utilised in conjunction with
the SAR system. These components required the use of an existing tracking calibration module. A
projector was also used in conjunction with the SAR system, which required the use of an existing
projector calibration module. As the existing SAR system was implemented and runs in UNIX, the
implemented system was developed in a UNIX environment, on a computer utilising Ubuntu 9.10.
9.2 Existing Components
Existing components were utilised to enable the system to function. These included a SAR system
described in Section 9.2.1, a tracking framework described in Section 9.2.2 and associated
hardware.
9.2.1 SAR System
The existing SAR system is a framework upon which SAR applications can be developed. The
SAR system is developed in C/C++ and OpenGL. A library has also been developed, libwcl, which
allows for further resources, including mathematical classes, that can be utilised directly, and
which are utilised by the SAR system. Modules can be developed and integrated in the SAR
system, allowing for the use of previously developed SAR resources. The module developed to
implement the research described in this dissertation is discussed in Section 9.3.1. The SAR system
utilised and the libwcl library were developed within the Wearable Computer Lab at the University
of South Australia1.
The existing SAR system allows for the use of one or more projectors. The implementation
described in this dissertation utilised one projector. Also present is a projector coordinate space,
which can be calibrated using a projector calibration module present within the existing SAR
system. This projector coordinate space allows for the precise projection of images upon a desired
position. For example, the projector coordinate space could be defined such that the x axis and the
y axis are parallel to a flat surface such as a table, and the z axis is perpendicular, allowing for the
projection of two dimensional images upon the surface and the projection of three dimensional
images upon a physical object located on the surface.
1 Developed by Michael Marner ([email protected]), Markus Broecker
([email protected]) and Benjamin Close ([email protected]).96
At least one camera and one projector are to be connected to the computer on which the SAR
system is installed. The camera and projector are to be positioned so that their field of view
overlaps, allowing for the projector coordinate space and the ARToolKitPlus coordinate space to
overlap.
Projector calibration is performed to define the projector coordinate space. This allows for the x
axis and y axis to be parallel with the ground plane and the z axis to be perpendicular to it. This
also allows for the scale to be defined according to the physical objects to be arranged. In the
system implemented the projector calibration was undertaken such that a one millimetre distance
along a physical model corresponded with a one unit distance along the corresponding virtual
model. Thus, a unit in the projector coordinate space corresponded with a millimetre in the physical
world. Tracking calibration is then performed to provide a transformation between the
ARToolKitPlus coordinate space and the previously defined projector coordinate space.
The existing SAR system provides predefined resources for the input of models and the rendering
and display of these models using the aforementioned projector coordinate space. Also provided is
a structure for implementing and utilising existing tracking frameworks. Support for input devices
was also present, of which the keyboard input was utilised for user study functionality. Support for
hardware such as cameras and projectors is also provided. The existing SAR system is configurable
using XML files. The modules to use can be defined within an XML file, settings for the loaded
models can be specified and the resources to be loaded can be determined. SAR can then be
executed with the appropriate XML configuration file.
9.2.2 Tracking
The physical objects to be arranged must be tracked to allow for the SAR visual cues to function as
described. The developed system utilises the ARToolKitPlus tracking framework (Wagner &
Schmalstieg 2007) which allows for the obtainment of a marker position and orientation in a real
time environment.
The tracking calibration process generates a transformation matrix that allows the mapping of the
ARToolKitPlus coordinate space to the SAR projector coordinate space. This allows a conversion
between a set of coordinates in the ARToolKitPlus coordinate space to the SAR projector
coordinate space. An example of the use of this conversion is in the scenario of projecting upon a
tracked physical object. By fixing fiduciary markers at a known position upon the physical object,
the tracking framework returns the coordinates of this position. Through applying the
transformation matrix, the corresponding SAR projector coordinate of this position can be
calculated, allowing for projection upon on this position using the SAR system.
97
By utilising tracked physical objects according to the described process, the SAR visual cues are
able to be updated as the physical object is moved. For example, if the user is prompted to translate
an object according to the Translation Instruction, the user may move the physical object and have
the circle projection follow the object and the line projection show the updated path to the final
position. As the tracked physical object is moved, the ARToolKitPlus coordinates of the tracked
physical object (obtained from the marker) are periodically converted to coordinates in the SAR
projector coordinate space, allowing the projection to follow the tracked physical object.
Utilising tracked physical objects according to the described process also enables the system to
verify whether a tracked physical object has been successfully rotated and translated. For example,
if the user is prompted to translate a tracked physical object according to the Translation
Instruction, the user may move the physical object so that the circle projections overlap. The
Translation Instruction will cease to be displayed when the ARToolKitPlus coordinates of the
tracked physical object (obtained from the marker) are equivalent to the coordinates of the physical
object’s desired final translation in the projector coordinate space.
9.2.2.1 Infrared Tracking
The possibility of utilising Hewlett Packard’s UV/IR Invisible Ink HP Q7485A was explored. It
was foreseen that by utilising this ink to produce ARToolKitPlus markers in conjunction with a
camera that could view infrared, physical object tracking could be undertaken. This possibility was
explored and deemed unfeasible at the current time as the ink and devices utilised did not allow for
this functionality, however it is a possible direction for future work.
9.3 Developed Components
A SAR module was developed for the existing SAR system described in Section 9.2.1, which
implements SAR visual cues to instruct a user on how to arrange a physical object. A top down
arrangement application was developed to aid the object arrangement process.
9.3.1 SAR Module
A SAR module has been developed in C/C++ and OpenGL to implement the SAR visual cues
described in Section 3.2.3. The utilisation of the SAR system allowed for the module developed to
utilise predefined rendering and display resources, the easy integration of the ARToolKitPlus
tracking framework, the easy integration of hardware, use of the definable projector coordinate
space, the use of input devices and a method for configuration using XML.
98
The module reads in a desired final translation and rotation position for each object, this can be
obtained from the top down arrangement application or from an external file. A SAR visual cue
then instructs a user on how to arrange each tracked physical object to their respective desired final
positions.
The module utilises the ARToolKitPlus tracking framework provided by the existing SAR system.
The positions of two ARToolKitPlus markers are tracked by the module, one representing the
centre of the current tracked physical object, and the other present to allow for rotation calculations
within the developed module.
The module implements an update method which is called by the SAR system periodically. Within
this method, a check is performed to determine whether the top down arrangement application has
been altered. If the top down arrangement application has been altered, the updated final position
information is gathered and the projection is updated.
The object arrangement process as defined in Section 3.2 is implemented. The update method also
performs a check to determine the state of the current tracked physical object to be arranged. The
current object to be arranged is assigned a state. The possible states of the current object are
translation, rotation, and successfully arranged. Once the object has been assigned the successfully
arranged state and it has been verified that the translation and rotation are correct, the next object
becomes the current object to be arranged. The order of the objects is determined by the order they
are entered in the XML configuration file.
When verifying that a tracked physical object has been correctly translated and rotated, some
leeway is given so that the user does not have to arrange the physical object precisely to the desired
final position. This was implemented as it was observed to be difficult and time consuming for the
user to arrange the physical object precisely to the desired final position, and the leeway given still
allows for the improved accuracy of SAR visual cues over a manual arrangement technique, as
determined from the accuracy information gathered from the user study (Section 5.2.2). When
translating a tracked physical object, the user is required to apply a translation to the object that is
within ten units of the desired final position of the object, along both the x axis and the y axis. In
the system implemented, this ten unit distance corresponds with approximately ten millimetres.
When rotating a tracked physical object, the user is required to apply a rotation to the object that is
within ten degrees of the desired final rotation.
The module also implements a draw method in which predefined rendering and display resources
provided by the SAR system are used along with OpenGL techniques to implement the SAR visual
cues. A SAR visual cue is displayed instructing the user to arrange the current object, in accordance
to the current object’s state.99
Keyboard input is utilised to undertake the user study. This functionality allows for the
specification of which SAR visual cue is utilised.
9.3.2 Top Down Arrangement Application
An application was developed to provide a user interface which allows a user to arrange top down
representations of the physical objects. The resulting arrangement can then be performed on the
physical objects using SAR cues. As described in Section 3.1, this application can be run in real
time for a collaboration task or the arrangement generated can be stored to provide instructions at a
later date.
Figure 73 Top Down Arrangement Application Screenshot
Figure 73 presents a screenshot of three representations of physical objects being arranged within
the application. The user is able to rotate and translate the representations with the arrow keys to
determine a desired arrangement. Multiple objects are able to be loaded and the current object to be
arranged is selected from the drop down menu. In the scenario depicted by Figure 73, the user has
added three representations of physical objects and has arranged them. The transformation matrices
for each object are output to a file periodically and information can either be communicated to the
developed module within the SAR system in real time, or can be saved in the file and retrieved
from the file at a later date. When the information is communicated in real time, this allows for the
representations to be manipulated even after the SAR visual cues have been carried out upon the 100
corresponding physical objects. This is possible as the arrangement of each physical object is
revisited if the physical object is not arranged in the correct position according to the
transformation matrices communicated to the module.
101
10 Conclusion
This dissertation presents several significant contributions which address limitations found in the
related research. A unique approach is presented to instruct a user on how to arrange physical
objects using visual cues implemented with SAR. This approach has been implemented, validated
with a user study and improved upon. Two instructions have been defined to arrange a physical
object into all possible positions along a ground plane. A set of SAR visual cues have been
developed which utilise these instructions or variations of them. A survey was undertaken of
related research, this showed that object manipulation using AR and SAR systems is largely
focused on the manipulation of virtual objects, which causes limitations. The research presented in
this dissertation addresses these limitations by providing a method for directly arranging physical
objects using SAR. Interaction with the system developed has been addressed which has resulted in
the development of a top down arrangement application to aid the arrangement of objects using
SAR. SAR object arrangement can be undertaken locally or remotely, in real time by one user or as
a collaboration task. The process can also be stored to provide instructions on demand at a later
date. A user study was undertaken to determine the effectiveness of these SAR visual cues when
compared to a manual arrangement method and when compared to one another.
A research question was initially proposed. This research question is:
What are the appropriate Spatial Augmented Reality visual cues to instruct a user on how
to arrange real objects?
This research question has been answered by the research described in this dissertation. From the
results of the user study, a recommendation for either the Both Instructions Cue or the Square Cue
was made. In addition to receiving a favourable user opinion, they implement features consistent
with the advantages found with the results; they are both SAR visual cues that implement the
Translation and Rotation Instructions and thus are quicker and more accurate than a manual
arrangement, as well as being more intuitive.
102
The research described in this dissertation has made a number of contributions. Contributions
described in this dissertation include:
a survey of existing technologies,
SAR instructions for rotation and translation that can be used in conjunction to arrange a
physical object,
a set of validated and implemented SAR visual cues that utilise the developed instructions
or versions of these instructions,
the integration of the SAR visual cues to form a system that can instruct a user to arrange
physical objects,
a two-dimensional application that has been incorporated into the working system which
aids the physical object arrangement process,
an evaluation of the effectiveness of the SAR visual cues developed,
a revised SAR visual cue that considers the results of the user study, and
an adaptation of a SAR visual cue which can arrange large physical objects.
SAR visual cues have been developed that are quicker and more accurate than a manual
arrangement method; validating the hypothesis presented by the user study. The Translation
Instruction and Rotation Instruction were more intuitive than variations of these that were utilised.
In addition, a revised visual cue has been developed by taking into account the results of the user
study, and a pilot study regarding this new visual cue has indicated improvements in all areas in
which data was gathered: in terms of speed, accuracy, intuitiveness and user opinion. Furthermore,
the SAR visual cues developed have been shown to facilitate the arrangement of physical objects of
a variety of sizes, including large objects. Using SAR in this way could be useful for the
arrangement of large objects, such as furniture or theatre sets, and when multiple objects are being
arranged.
10.1 Future Work
Although the techniques and concepts presented in this dissertation are implemented to form a
system that employs SAR visual cues to arrange physical objects, there is potential for the
improvement and continuation of this research. This section highlights several ideas.
103
This research is focused on the arrangement of physical object upon a ground plane. Future work
can be undertaken to allow for more arrangement options. For example, a vertical translation along
the z axis would allow physical objects to be stacked and the ability to perform rotations around the
x axis and y axis would allow physical objects to be flipped. It is envisaged that instructions such
as flip and stack could be implemented and incorporated into the object arrangement process
described in Section 3.2 along with the already implemented Translation and Rotation Instructions.
Additionally, the order in which multiple physical objects are arranged could be taken into account.
The implementation allows for multiple physical objects to be arranged, however the order in
which they are arranged is not currently considered. This could be useful in the following example:
when arranging multiple physical objects, the user may be presented with a situation where the
object to be arranged must be translated to a position that intersects other physical objects, making
the instruction difficult to perform.
As the focus of this research was on the effectiveness of SAR visual cues for physical object
arrangement, an existing tracking framework was utilised. There are limitations in the existing
tracking framework and thus further study into a tracking method that is better suited to the
application is warranted. In addition, the infra red tracking described in Section 9.2.2.1 could be
revisited with different components utilised.
104
11 References
Ahlers, KH, Kramer, A, Breen, DE, Chevalier, P-Y, Crampton, C, Rose, E, Tuceryan, M, Whitaker, RT & Greer, D 1995, Distributed Augmented Reality for Collaborative Design Applications, Maastricht.
Azuma, R, Baillot, Y, Behringer, R, Feiner, S, Julier, S & MacIntyre, B 2001, 'Recent Advances in Augmented Reality', IEEE Comput. Graph. Appl., vol. 21, no. 6, pp. 34-47.
Azuma, RT 1997, 'A Survey of Augmented Reality', Presence: Teleoperators and Virtual Environments, vol. 6, pp. 355–385.
Bimber, O, Grundhofer, A, Wetzstein, G & Knodel, S 2003, Consistent Illumination within Optical See-Through Augmented Environments, IEEE Computer Society, p. 198.
Bimber, O & Raskar, R 2005, Spatial Augmented Reality: Merging Real and Virtual Worlds, A K Peters, Wellesley, Massachusetts.
Breen, DE, Whitaker, RT, Rose, E & Tuceryan, M 1996, Interactive Occlusion and Automatic Object Placement for Augmented Reality.
Bryson, S, Zeltzer, D, Bolas, MT, Chapelle, BdL & Bennett, D 1996, The Future of Virtual Reality: Head Mounted Displays Versus Spatially Immersive Displays, ACM, pp. 485-486.
Cotting, D, Naef, M, Gross, M & Fuchs, H 2004, 'Embedding Imperceptible Patterns into Projected Images for Simultaneous Acquisition and Display', paper presented at the Mixed and Augmented Reality, 2004. ISMAR 2004. Third IEEE and ACM International Symposium on.
105
Do, TV & Lee, J-W 2010, '3DARModeler: a 3D Modeling System in Augmented Reality Environment', International Journal of Electrical, Computer, and Systems Engineering, vol. 4, no. 2, pp. 145-154.
Drascic, D & Milgram, P 1996, Perceptual Issues in Augmented Reality.
Ehnes, J, Hirota, K & Hirose, M 2004, Projected Augmentation - Augmented Reality using Rotatable Video Projectors, IEEE Computer Society, pp. 26-35.
Feiner, S, MacIntyre, B, Hollerer, T & Webster, A 1997, A Touring Machine: Prototyping 3D Mobile Augmented Reality Systems for Exploring the Urban Environment, IEEE Computer Society, p. 74.
Feiner, S, Macintyre, B & Seligmann, D 1993, 'Knowledge-based augmented reality', Communications of the ACM, vol. 36, no. 7, pp. 53-62.
Fjeld, M, Voorhorst, F, Bichsel, M, Lauche, M, Rauterberg, M & Krueger, H 1999, 'Exploring Brick-Based Navigation and Composition in an Augmented Reality', Handheld and Ubiquitous Computing, vol. 1707, pp. 102-116.
Gausemeier, J, Fruend, J & Matysczok, C 2002, AR-Planning Tool: Designing Flexible Manufacturing Systems With Augmented Reality, Eurographics Association, Barcelona, Spain, pp. 19-25.
Gorbet, MG, Orth, M & Ishii, H 1998, Triangles: Tangible Interface for Manipulation and Exploration of Digital Information Topography, ACM Press/Addison-Wesley Publishing Co., Los Angeles, California, United States, pp. 49-56.
Grasset, R & Gascuel, J-D 2002, MARE: Multiuser Augmented Reality Environment on Table Setup, ACM, San Antonio, Texas, pp. 213-213.
Höllerer, T 2001, 'User Interface Management Techniques for Collaborative Mobile Augmented Reality', Computers & Graphics, vol. 25, no. 5, pp. 799-810.
106
Irawati, S, Green, S, Billinghurst, M, Duenser, A & Ko, H 2006a, An Evaluation of an Augmented Reality Multimodal Interface Using Speech and Paddle Gestures, Hangzhou, China, pp. 272-283.
Irawati, S, Green, S, Billinghurst, M, Duenser, A & Ko, H 2006b, '"Move the Couch Where?": Developing an Augmented Reality Multimodal Interface', Symposium on Mixed and Augmented Reality, Proceedings of the 5th IEEE and ACM International Symposium on Mixed and Augmented Reality, pp. 183-186.
Ishii, H, Ben-Joseph, E, Underkoffler, J, Yeung, L, Chak, D, Kanji, Z & Piper, B 2002, Augmented Urban Planning Workbench: Overlaying Drawings, Physical Models and Digital Simulation, IEEE Computer Society, p. 203.
Ishii, H & Ullmer, B 1997, Tangible Bits: Towards Seamless Interfaces Between People, Bits and Atoms, ACM, Atlanta, Georgia, United States, pp. 234-241.
Kato, H & Billinghurst, M 1999, Marker Tracking and HMD Calibration for a Video-Based Augmented Reality Conferencing System, IEEE Computer Society, p. 85.
Kato, H, Billinghurst, M, Poupyrev, K, Imamoto, K & Tachibana, K 2000, 'Virtual Object Manipulation on a Table-Top AR Environment', Human Interface, pp. 275-278.
Kitamura, Y, Itoh, Y & Kishino, F 2001, Real-time 3D interaction with ActiveCube, ACM, Seattle, Washington, pp. 355-356.
Kitamura, Y & Kishino, F 1997, Consolidated Manipulation of Virtual and Real objects, ACM, Lausanne, Switzerland, pp. 133-138.
Kitamura, Y, Ogata, S & Kishino, F 2002, A Manipulation Environment of Virtual and Real Objects Using a Magnetic Metaphor, ACM, Hong Kong, China, pp. 201-207.
Lee, G, Billinghurst, M & Kim, GJ 2004, Occlusion Based Interaction Methods For Tangible Augmented Reality Environments, Singapore, pp. 419-426.
107
MacIntyre, B, Gandy, M, Dow, S & Bolter, JD 2004, DART: A Toolkit for Rapid Design Exploration of Augmented Reality Experiences, ACM, Santa Fe, NM, USA, pp. 197-206.
Mackay, WE 1998, Augmented Reality: Linking Real and Virtual Worlds: A New Paradigm for Interacting with Computers, L'Aquila, Italy.
Malbezin, P, Piekarski, W & Thomas, BH 2002, Measuring ARToolKit Accuracy in Long Distance Tracking Experiments, Darmstadt, Germany.
Marner, MR & Thomas, BH 2010, 'Augmented Foam Sculpting for Capturing 3D Models', paper presented at the 3D User Interfaces (3DUI), 2010 IEEE Symposium on.
Milgram, P & Kishino, F 1994, 'A Taxonomy of Mixed Reality Visual Displays', IEICE Transactions on Information and Systems, vol. E77-D, no. 12, pp. 1321-1329.
Milgram, P, Takemura, H, Utsumi, A & Kishino, F 1994, 'Augmented Reality: A Class of Displays on the Reality-Virtuality Continuum', Telemanipulator and Telepresence Technologies, vol. 2351, pp. 282-292.
Naimark, L & Foxlin, E 2005, 'Encoded LED System for Optical Trackers', paper presented at the Fourth IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR'05), Vienna, Austria.
Newman, J 2001, 'Augmented Reality in a Wide Area Sentient Environment', paper presented at the IEEE and ACM International Symposium on Augmented Reality.
Park, J, You, S & Neumann, U 1999, Natural Feature Tracking for Extendible Robust Augmented Realities, A. K. Peters, Ltd., Bellevue, Washington, United States, pp. 209-217.
Raskar, R & Low, K-L 2001, Interacting With Spatially Augmented Reality, ACM, Camps Bay, Cape Town, South Africa, pp. 101-108.
Raskar, R, Welch, G & Fuchs, H 1998, Spatially Augmented Reality, A. K. Peters, Ltd., Bellevue, Washington, United States, pp. 63-72.
108
Raskar, R, Welch, G, Low, K-l & Bandyopadhyay, D 2001, 'Shader Lamps: Animating Real Objects with Image-Based Illumination', paper presented at the 12th Eurographics Workshop on Rendering (EGWR) London, June.
Raskar, R, Welch, G & Chen, W 1999, Table-Top Spatially-Augmented Reality: Bringing Physical Models to Life with Projected Imagery, p. 64.
Rekimoto, J & Nagao, K 1995, The World through the Computer: Computer Augmented Interaction with Real World Environments, Pittsburgh, Pennsylvania, United States, pp. 29-36.
Rekimoto, J, Ullmer, B & Oba, H 2001, DataTiles: a modular platform for mixed physical and graphical interactions, ACM, Seattle, Washington, United States, pp. 269-276.
Schmalstieg, D, Fuhrmann, A & Hesina, G 2000, Bridging Multiple User Interface Dimensions with Augmented Reality.
Schmalstieg, D, Fuhrmann, A, Hesina, G, Szalavari, Z, Encarnacao, L.M, Gervautz, M & Purgathofer, W 2002, 'The Studierstube Augmented Reality Project', Presence, vol. 11, no. 1, pp. 33-54.
Suganuma, A, Ogata, Y, Shimada, A, Arita, D & Taniguchi, R-i 2008, Billiard Instruction System for Beginners with a Projector-Camera System, ACM, Yokohama, Japan, pp. 3-8.
Sutherland, IE 1968, A Head-Mounted Three Dimensional Display, ACM, San Francisco, California, pp. 757-764.
Szalavári, Z & Gervautz, M 1997, The Personal Interaction Panel - a Two-Handed Interface for Augmented Reality.
Ullmer, B & Ishii, H 1997, The metaDESK: models and prototypes for tangible user interfaces, ACM, Banff, Alberta, Canada, pp. 223-232.
109
Voida, S, Podlaseck, M, Kjeldsen, R & Pinhanez, C 2005, A Study on the Manipulation of 2D Objects in a Projector/Camera–Based Augmented Reality Environment, Portland, Oregon.
Wagner, D & Schmalstieg, D 2007, 'ARToolKitPlus for Pose Tracking on Mobile Devices', paper presented at the 12th Computer Vision Winter Workshop, St. Lambrecht, Austria.
Wang, X & Gong, Y 2007, Augmented Virtuality Space: Enriching Virtual Design Environments with Reality, Brisbane, Australia.
Webster, A, Feiner, S, MacIntyre, B, Massie, W & Krueger, T 1996, Augmented Reality in Architectural Construction, Inspection, and Renovation.
Wellner, P 1991, The DigitalDesk calculator: tangible manipulation on a desk top display, ACM, Hilton Head, South Carolina, United States, pp. 27-33.
Wellner, P 1993, 'Interacting with Paper on the DigitalDesk', Commun. ACM, vol. 36, no. 7, pp. 87-96.
You, S, Neumann, U & Azuma, R 1999, Hybrid Inertial and Vision Tracking for Augmented Reality Registration, IEEE Computer Society, p. 260.
Zhang, X, Fronz, S & Navab, N 2002, Visual Marker Detection and Decoding in AR Systems: A Comparative Study, IEEE Computer Society, p. 97.
Zhou, F, Duh, HB-L & Billinghurst, M 2008, Trends in Augmented Reality Tracking, Interaction and Display: A Review of Ten Years of ISMAR, IEEE Computer Society, pp. 193-202.
110
Appendix A User Study Survey
111
Visual Cues for Spatial Augmented Reality User Study Questionnaire
Age: ___Gender: M/F
What is your experience with Augmented Reality systems?None Some Experienced
Have you ever interacted with a Spatial Augmented Reality System before?
The following are visual cues that can be projected onto the top of a rectangular object. The view is a top down, “bird's eye” view.Each one of the four cues instructs the rectangle to be moved in one of the following ways:
rotate the object (by 45 degrees). translate the object (along an x and y axis which is flat on the page).
The only rotation allowed is 45 degrees.There is a movement performed in each scenario.Please draw what you believe the outcome of the cues might be, if they were to be applied to the rectangular object. You may draw over the drawings provided.
Cue #1:
Cue #2:
112
Cue #3 (the leftmost rectangle is the original object):
Cue #4
113
Scenarios: Scenario 1: No SAR Scenario 2: Line Cue: Translation First Scenario 3: Line Cue: Rotation First Scenario 4: Line Cue: Both Cues At The Same Time Scenario 5: Circle Cue Scenario 6: Line Cue: With wireframe, with final destination Scenario 7: Line Cue: With wireframe, without final destination Scenario 8: Line Cue: With square, with final destination Scenario 9: Line Cue: With square, without final destination
Which do you think was better: Scenario 1: No SAR, or Scenario 2: Line Cue: Translation first?
Why did you think this?
Which do you think was better: Scenario 2: Line Cue: Translation first, or Scenario 3: Line Cue: Rotation first, or Scenario 4: Line Cue: Both Cues At The Same Time?
Why did you think this?
Which do you think was better: Scenario 2: Line Cue: Translation first, or Scenario 5: Circle Cue?
Why did you think this?
114
Which do you think was better: Scenario 6: Line Cue: With wireframe, with final destination, or Scenario 7: Line Cue: With wireframe, without final destination?
Why did you think this?
Which do you think was better: Scenario 8: Line Cue: With square, with final destination, or Scenario 9: Line Cue: With square, without final destination?
Why did you think this?
Which do you think was better: The scenarios with the square cues (Scenario 8, Scenario 9) The scenarios with the wireframe cues (Scenario 6, Scenario 7)
Why did you think this?
115
Which of the line cues did you think was better: Scenario 2: Line Cue: Translation First, or Scenario 3: Line Cue: Rotation First, or Scenario 4: Line Cue: Both Cues At The Same Time, or Scenario 6: Line Cue: With wireframe, with final destination, or Scenario 7: Line Cue: With wireframe, without final destination, or Scenario 8: Line Cue: With square, with final destination, or Scenario 9: Line Cue: With square, without final destination?
Why did you think this?
Anything else you would like to add?
116
Appendix B Pilot Study Survey
117
The following is a visual cue that can be projected onto the top of a rectangular object. The
view is a top down, “bird’s eye” view.
The cue instructs the rectangle to be moved in one of the following ways:
Rotate the object (by 45 degrees)
Translate the object (along an x and y axis which are flat on the page).
The only rotation allowed is 45 degrees. There is a movement performed in each scenario.
Please draw what you believe the outcome of the cues might be, if they were to be applied
to the rectangular object. You may draw over the drawings provided.
Cue #1
Cue #2
118
Was this visual cue better than the ones experienced in the user study?
What was better with this one?
What was worse with this one?
Any other comments?
119
Appendix C Data Tables for Included Results
120
User Study Timer Data (milliseconds)
Manual Timer Participant 1 Participant 2 Participant 3 Participant 4Task 1 26652 16392 58099 23852Task 2 18922 13191 20921 23521Task 3 16924 10661 14660 68760Task 4 10393 29850 27716 23853Task 5 15857 36113 49171 41576Task 6 10527 40243 9861 32115Task 7 11326 14523 17723 31713Task 8 12391 8395 16391 24918Task 9 11060 18654 13326 19055Average of all Tasks 13425 21453.75 21221.125 33188.875
Manual Timer Participant 5 Participant 6 Participant 7 Participant 8Task 1 15591 44507 36113 23184Task 2 16790 46639 36779 22920Task 3 11726 30782 16258 32114Task 4 16123 77686 15991 30781Task 5 53302 67026 29450 25985Task 6 13458 24119 8395 17723Task 7 17056 16789 22121 27050Task 8 10927 20388 20521 19588Task 9 15458 14524 16123 13192Average of all Tasks 19355 37244.125 20704.75 23669.125
Manual Timer Participant 9 Participant 10 Participant 11 Participant 12Task 1 31182 50769 32381 16257Task 2 28517 33312 67959 10395Task 3 16656 17456 29450 26917Task 4 29182 98074 16124Task 5 37710 71025 54101Task 6 20520 25850 30381 13325Task 7 10394 23986 59965 19321Task 8 5997 15326 104336 16655Task 9 20522 15325 25452 23452Average of all Tasks 18826.85714 24137.85714 60830.25 22536.25
121
Manual Timer Participant 13 Participant 14 Participant 15 Participant 16Task 1 47838 33181 44106 22653Task 2 17589 79019 47038 21854Task 3 31980 29715 19320 10661Task 4 18789 71024 32514 16790Task 5 21852 118061 19855 26252Task 6 17856 67560 9594 27716Task 7 16258 13858 12394 15193Task 8 23586 77021 13859 18522Task 9 17456 48238 10659 8795Average of all Tasks 20670.75 63062 20654.125 18222.875
Manual Timer Participant 17 Participant 18 Participant 19 Participant 20 AverageTask 1 55568 14259 21586 32780 32347.5Task 2 21453 47438 20121 23985 30918.15Task 3 16923 22519 15858 11059 22519.95Task 4 12925 24252 11726 11860 30297.53Task 5 48904 8394 34247 23053 41154.42Task 6 26518 12126 9328 12394 21480.45Task 7 24784 4798 14257 11992 19275.05Task 8 15592 6263 13059 9195 22646.5Task 9 26385 10661 13991 13193 17776.05Average of all Tasks 24185.5 17056.375 16573.375 14591.375
122
SAR Timer Participant 1 Participant 2 Participant 3 Participant 4Task 2 15456 6527 21586 19589Task 3 11726 3730 9728 52368Task 4 6394 24519 19053 18655Task 5 11992 29849 42907 34112Task 6 4264 33445 13458 53567Task 7 8127 8794 11993 82218Task 8 8396 2930 11725 13989Task 9 7460 11993 9195 13192Average of all SAR Tasks 9226.875 15223.375 17455.625 35961.25
SAR Timer Participant 5 Participant 6 Participant 7 Participant 8Task 2 12258 35045 23718 16124Task 3 7461 24519 10262 25718Task 4 11726 71956 9860 24652Task 5 48903 61029 20920 20920Task 6 8260 18255 2663 11992Task 7 12126 11060 17457 21986Task 8 4930 14391 15857 14258Task 9 10793 9194 11725 8796Average of all SAR Tasks 14557.125 30681.125 14057.75 18055.75
SAR Timer Participant 9 Participant 10 Participant 11 Participant 12Task 2 21986 26917 57566 5331Task 3 12525 12257 22386 19454Task 4 23449 9728 89145 11593Task 5 24252 27717 63695 50102Task 6 16522 21186 22920 8530Task 7 5463 18121 50769 14792Task 8 2665 9994 77418 12526Task 9 15457 11327 20120 13589Average of all SAR Tasks 15289.875 17155.875 50502.375 16989.625
SAR Timer Participant 13 Participant 14 Participant 15 Participant 16Task 2 11859 69824 39579 17989Task 3 27583 24251 10526 7062Task 4 14392 66760 12124 13191Task 5 18122 107003 15456 23319Task 6 13992 53434 4797 24785Task 7 11859 8396 8928 11724Task 8 18921 64761 10794 12925
123
Task 9 13458 39713 7729 3463Average of all SAR Tasks 16273.25 54267.75 13741.625 14307.25
SAR Timer Participant 17 Participant 18 Participant 19 Participant 20 AverageTask 2 13591 42773 13991 19989 24584.9Task 3 10926 19722 10661 7329 16509.7Task 4 5996 21320 6262 6662 23371.85Task 5 42638 4797 28381 17989 34705.15Task 6 20254 9061 3598 7462 17622.25Task 7 18520 2133 8929 8262 17082.85Task 8 7595 3596 7728 5463 16043.1Task 9 20655 7329 8259 9329 12638.8Average of all SAR Tasks 17521.875 13841.375 10976.125 10310.625
124
User Study Accuracy Data (units)
Accuracy Participant 1 - Variation Participant 2 - Variation Participant 3 - VariationTask 1 x 1.039297 6.099629 13.084367Task 1 y 42.174904 44.340803 31.232368Task 1 Angle 14.130518 1.41048 3.495705Task 2 x 9.311109 4.637155 4.850228Task 2 y 3.332299 2.065624 1.994922Task 2 Angle 3.10447 8.024928 8.644944Task 3 x 4.334879 1.245961 7.552184Task 3 y 2.352554 0.86928 1.708735Task 3 Angle 1.311248 7.168728 7.480713Task 4 x 2.44157 3.501189 4.577567Task 4 y 5.850972 7.319244 10.311855Task 4 Angle 4.954148 2.886078 5.141129Task 5 x 2.64451 4.315219 4.204956Task 5 y 1.539177 0.647657 1.814049Task 5 Angle 6.377228 2.305633 3.691101Task 6 x 4.084566 1.593704 15.525131Task 6 y 5.527533 3.513853 10.870038Task 6 Angle 9.706848 8.089829 3.221551Task 7 x 5.523292 0.349215 0.565146Task 7 y 3.459471 1.137969 6.524606Task 7 Angle 8.326416 2.464569 6.985962Task 8 x 5.176577 5.492001 3.910662Task 8 y 2.807469 2.218419 2.753907Task 8 Angle 7.071961 5.27073 2.411071Task 9 x 5.016434 15.947254 0.78761Task 9 y 4.106385 16.885332 0.989237Task 9 Angle 6.443856 7.074723 8.928337
Accuracy Participant 4 - Variation Participant 5 - Variation Participant 6 - VariationTask 1 x 55.704788 12.779458 7.732371Task 1 y 48.640778 34.267438 12.991065Task 1 10.747589 4.417854 7.964401
125
AngleTask 2 x 1.791216 8.671446 4.540201Task 2 y 7.29361 0.661729 3.445666Task 2 Angle 8.386962 7.574997 7.924637Task 3 x 2.577838 5.25909 5.515763Task 3 y 0.868942 5.089246 3.847748Task 3 Angle 6.534317 5.190176 9.109069Task 4 x 1.401345 0.214374 4.96145Task 4 y 7.527944 4.250156 1.534501Task 4 Angle 4.360626 2.325547 4.54419Task 5 x 1.185389 1.390412 8.902074Task 5 y 0.262309 0.528527 5.192332Task 5 Angle 3.703746 8.057343 3.836151Task 6 x 19.97283 5.697926 1.112047Task 6 y 12.22551 5.176905 1.318666Task 6 Angle 3.221665 7.490673 8.192505Task 7 x 4.899351 0.866437 0.371613Task 7 y 12.796405 5.831572 8.08445Task 7 Angle 5.579986 8.33194 3.785754Task 8 x 7.664078 1.279692 9.282825Task 8 y 13.983102 3.367409 7.383993Task 8 Angle 1.873734 2.758267 2.538553Task 9 x 9.174715 0.767815 4.74629Task 9 y 8.187706 7.770797 3.326917Task 9 Angle 4.32663 4.660461 5.0009
Accuracy Participant 7 - Variation Participant 8 - Variation Participant 9 - VariationTask 1 x 6.993693 10.920417 98.685171Task 1 y 24.008659 19.016226 24.969531Task 1 Angle 4.509575 5.060087 2.397989Task 2 x 5.987828 19.011801 4.674298Task 2 y 3.929651 6.513794 7.673532Task 2 Angle 3.070907 16.490707 5.644791Task 3 x 1.346825 0.290998 9.080372Task 3 y 3.104461 1.908415 4.917646Task 3 7.949072 6.712524 8.499989
126
AngleTask 4 x 5.317802 2.871913 2.782842Task 4 y 2.960601 1.289673 5.512639Task 4 Angle 1.638122 5.017792 7.400635Task 5 x 4.842592 1.965225 1.069937Task 5 y 0.304081 4.254856 3.096462Task 5 Angle 4.189408 5.106949 6.84111Task 6 x 3.423531 1.507248 7.30256Task 6 y 0.533269 9.332173 5.236787Task 6 Angle 4.706054 7.595883 7.250984Task 7 x 1.591043 3.328602 0.704764Task 7 y 4.782579 3.122943 3.250757Task 7 Angle 4.019318 9.898727 5.972128Task 8 x 3.96838 2.605622 0.309635Task 8 y 4.48907 8.452465 5.935727Task 8 Angle 2.614822 2.446015 4.147103Task 9 x 1.936099 2.672111 1.420598Task 9 y 1.317336 1.30716 9.593448Task 9 Angle 7.436803 2.853844 5.345833
Accuracy Participant 10 - Variation Participant 11 - Variation Participant 12 - VariationTask 1 x 13.446533 7.569635 5.559771Task 1 y 20.19087 19.628868 22.961542Task 1 Angle 42.893341 3.735723 10.378162Task 2 x 3.048069 7.805073 9.282025Task 2 y 2.37344 8.359359 4.647328Task 2 Angle 4.724213 7.302918 4.737778Task 3 x 7.642148 7.28321 8.163282Task 3 y 1.468308 2.374198 4.814531Task 3 Angle 7.206512 8.916786 1.834259Task 4 x 1.166884 8.794916 5.223243Task 4 y 1.104884 6.549114 5.482608Task 4 Angle 8.071105 3.3833 5.477127Task 5 x 5.71743 6.566397 0.830942Task 5 y 11.816035 4.73257 9.109616Task 5 3.547265 1.815139 7.547706
127
AngleTask 6 x 9.061326 2.1272 2.571Task 6 y 4.953227 3.343865 8.618427Task 6 Angle 2.362732 7.303749 7.632049Task 7 x 6.951204 0.819095 3.643125Task 7 y 1.453997 5.919619 6.974047Task 7 Angle 7.377995 7.539597 2.552658Task 8 x 4.448137 5.329015 1.102177Task 8 y 2.06629 6.917297 0.182512Task 8 Angle 0.469849 5.334875 2.138718Task 9 x 6.050835 4.610652 8.218855Task 9 y 4.619998 8.748012 0.37383Task 9 Angle 4.635437 0.394714 7.936523
Accuracy Participant 13 - Variation Participant 14 - Variation Participant 15 - VariationTask 1 x 11.157458 10.919239 29.693233Task 1 y 27.891035 25.749081 42.769776Task 1 Angle 2.659973 0.92218 0.729692Task 2 x 0.028391 7.809859 4.064739Task 2 y 7.382892 2.951053 1.899705Task 2 Angle 5.315826 7.802017 12.42131Task 3 x 3.916131 3.233429 6.324694Task 3 y 3.178588 8.287995 0.073385Task 3 Angle 5.571777 0.700645 6.014999Task 4 x 2.653247 0.892778 2.825411Task 4 y 9.780528 7.015494 7.177656Task 4 Angle 4.648544 0.504275 4.779968Task 5 x 4.323891 10.543114 8.77073Task 5 y 7.428329 2.866711 7.51959Task 5 Angle 1.423861 2.263794 1.045472Task 6 x 2.83752 3.05858 1.708741Task 6 y 13.278838 2.362912 8.34893Task 6 Angle 3.216186 4.064606 1.836342Task 7 x 6.907535 4.355759 5.824256Task 7 y 0.310526 0.037548 5.623588Task 7 8.749832 7.445179 4.061065
128
AngleTask 8 x 0.502259 7.744605 8.183574Task 8 y 7.537782 2.599366 1.008305Task 8 Angle 2.310099 4.585144 1.851043Task 9 x 5.958924 4.587892 5.269618Task 9 y 4.001875 0.108859 3.957533Task 9 Angle 4.524475 10.274788 0.311661
Accuracy Participant 16 - Variation Participant 17 - Variation Participant 18 - VariationTask 1 x 1.475482 19.473617 8.029748Task 1 y 31.047636 51.602575 40.233751Task 1 Angle 10.577271 9.168022 1.172787Task 2 x 1.638791 8.933812 0.231718Task 2 y 6.98595 1.131072 7.478341Task 2 Angle 2.858475 8.854126 6.110062Task 3 x 7.057989 8.341644 3.694768Task 3 y 9.368799 7.953215 6.782873Task 3 Angle 9.31987 1.247924 1.760849Task 4 x 6.666618 2.2159 5.268515Task 4 y 5.195172 4.507716 3.654681Task 4 Angle 3.486313 8.522421 4.303718Task 5 x 0.44915 1.404236 8.534449Task 5 y 6.218363 8.901302 0.981715Task 5 Angle 4.020464 9.977737 4.844101Task 6 x 2.204614 0.68721 6.332388Task 6 y 0.558016 4.065189 1.610421Task 6 Angle 2.515915 9.327111 5.218079Task 7 x 4.793839 4.390802 30.98502Task 7 y 1.887394 6.188506 4.064856Task 7 Angle 7.475998 9.324257 2.024742Task 8 x 2.390844 0.454762 8.606223Task 8 y 7.858291 1.69462 5.24689Task 8 Angle 8.492493 0.054131 7.51738Task 9 x 6.401455 2.571915 2.173138Task 9 y 5.20792 6.811522 7.391329Task 9 5.81691 8.874908 2.178864
129
Angle
Accuracy Participant 19 - Variation Participant 20 - Variation AverageTask 1 x 16.035428 2.702092 16.95507135Task 1 y 51.064596 44.845894 32.9813698Task 1 Angle 8.782784 4.186607 7.467037Task 2 x 7.704435 4.5574 5.9289797Task 2 y 8.028665 5.788502 4.6968567Task 2 Angle 3.944412 5.117477 6.90279785Task 3 x 8.44112 1.279486 5.12909055Task 3 y 9.64036 6.585169 4.2597224Task 3 Angle 9.473953 7.154785 5.95790975Task 4 x 2.493928 6.288715 3.62801035Task 4 y 0.299595 5.789072 5.15570525Task 4 Angle 2.008194 9.326233 4.63897325Task 5 x 5.023185 3.053279 4.28685585Task 5 y 6.12265 5.912205 4.4624268Task 5 Angle 1.060852 4.738373 4.31967165Task 6 x 6.744991 1.796916 4.96750145Task 6 y 0.87351 2.796306 5.22721875Task 6 Angle 0.245133 2.133651 5.26657725Task 7 x 3.442507 2.461362 4.638698Task 7 y 5.900677 2.636251 4.49938805Task 7 Angle 8.31797 6.570614 6.34023535Task 8 x 0.902094 9.961924 4.4657543Task 8 y 3.366543 2.524759 4.6197108Task 8 Angle 0.59491 1.888672 3.3184785Task 9 x 8.799026 7.559643 5.23354395Task 9 y 3.826679 7.266434 5.28991545Task 9 Angle 5.293354 7.58754 5.49502805
130
User Study Survey Data
Scenario Preference Participant 1 Participant 2 Participant 3 Participant 4 Participant 5Scenario 1Scenario 2 1 1 1 1 1
Scenario 2Scenario 3Scenario 4 1 1 1 1 1
Scenario 2 1 1 1 1 1Scenario 5
Scenario 6 1 1 1 1Scenario 7 1
Scenario 8 1 1 1 1 1Scenario 9
Scenarios 8, 9 1 1 1Scenarios 6, 7 1 1
Scenario 2Scenario 3Scenario 4 1 1 1 1 1Scenario 6 1Scenario 7Scenario 8 1Scenario 9
Scenario Preference Participant 6 Participant 7 Participant 8 Participant 9 Participant 10Scenario 1 Scenario 2 1 1 1 1 1 Scenario 2 1 1Scenario 3 1 Scenario 4 1 1 Scenario 2 1 1 1 1 1
131
Scenario 5
Scenario 6 1 1 Scenario 7 1 1 Scenario 8 1 1 1 1Scenario 9 1
Scenarios 8, 9 1 1 1 1 1Scenarios 6, 7 Scenario 2 1Scenario 3 Scenario 4 1Scenario 6 Scenario 7Scenario 8 1 1 1 1Scenario 9
Scenario Preference Participant 11 Participant 12 Participant 13 Participant 14 Participant 15Scenario 1 1 Scenario 2 1 1 1 1 Scenario 2 1 1Scenario 3 Scenario 4 1 1 Scenario 2 1 1 1 1Scenario 5
Scenario 6 1 Scenario 7 1 1 Scenario 8 1 1 1 1Scenario 9
Scenarios 8, 9 1 1 1Scenarios 6, 7 1 1 Scenario 2
132
Scenario 3 Scenario 4 1 1 1Scenario 6 1 Scenario 7 1Scenario 8 1Scenario 9
Scenario Preference Participant 16 Participant 17 Participant 18 Participant 19 Participant 20Scenario 1 Scenario 2 1 1 1 1 1 Scenario 2Scenario 3 Scenario 4 1 1 1 1 1 Scenario 2 1 1 1 1 1Scenario 5
Scenario 6 1 1 1 1 1Scenario 7 Scenario 8 1 1 1 1 1Scenario 9
Scenarios 8, 9 1 1 1 1Scenarios 6, 7 1 Scenario 2Scenario 3 Scenario 4 1Scenario 6 1 Scenario 7Scenario 8 1 1 1Scenario 9
133
User Study Intuitiveness Data
Rotation Instruction No. of ParticipantsCorrect 14Incorrect, but indicated a translation 1Incorrect 5
Translation Instruction No. of ParticipantsCorrect 16Incorrect 4
Circle Translation No. of ParticipantsCorrect 18Incorrect 2
Circle Rotation No. of ParticipantsCorrect 1Incorrect 18Incorrect, but indicated a rotation 1
134
Pilot Study Data
Pilot Study Participant 1 Participant 2 Participant 3Manual Timer 17990 14657 15191SAR Timer 9994 6795 5062
Pilot Study Participant 4 Participant 5 Participant 6 AverageManual Timer 6928 12659 7996 12570.16667SAR Timer 1733 6796 3199 5596.5
Was the Revised Visual Cue Better? No. of ParticipantsYes 6No 0
135