Usability of Requirements Techniques - A Systematic Literature Review @ SAC 2016
A Systematic Review of Usability Studies in Augmented Reality between 2005 and 2014
-
Upload
dr-arindam-dey -
Category
Technology
-
view
261 -
download
1
Transcript of A Systematic Review of Usability Studies in Augmented Reality between 2005 and 2014
A Systematic Review of Usability Studies in Augmented Reality between 2005 and 2014
Arindam Dey, Mark Billinghurst, Robert W. Lindeman, J. Edward Swan II
• Earlier Surveys • Methodology • Results • Key Insights
A Systematic Review of Usability Studies in Augmented Reality between 2005 and 2014
Earlier Surveys of AR Usability Studies
Time Venues Papers#
Swan & Gabbard (2005) 1992 - 2004 ISMAR,ISWC,
VR, Presence 21
Dünser et al. (2007) 1992 - 2007 ACM, IEEE
digital libraries 165
Bai & Blackwell (2012) 2001 - 2010 IEEE ISMAR 71
Our Survey 2005 - 2014 Scopus 291
• Earlier Surveys • Methodology • Results • Key Insights
A Systematic Review of Usability Studies in Augmented Reality between 2005 and 2014
Methodology
Scopus Keyword Search Unique (1,147)
Reported User Study (604)
Quality Check (396)
Norming (5) Review / Data Collection
Impact (291)
Methodology
Scopus Keyword Search Unique (1,147)
Quality Check (396)
Impact (291)
Keywords from Denser et al. (2007):"augmented reality" AND "user evaluation(s)" "augmented reality" AND "user study/-ies" "augmented reality" AND "feedback" "augmented reality" AND "experiment(s)" "augmented reality" AND "pilot study" "augmented reality" AND participant AND study "augmented reality" AND participant AND experiment "augmented reality" AND subject AND study "augmented reality" AND subject AND experiment
Additional keywords for our survey: "augmented reality" AND "user experience(s)” "augmented reality" AND "usability” “augmented reality” AND “evaluation(s)” AND “participant” “augmented reality” AND “evaluation(s)” AND “user” “augmented reality” AND “empirical” AND “participant” “augmented reality” AND “empirical” AND “user” “augmented reality” AND “empirical”
Reported User Study(604)
Methodology
Scopus Keyword Search Unique (1,147)
Description of the user study included • Task • Design • Participant demographics
Quality Check(396)
Impact (291)
Reported User Study (604)
Methodology
Scopus Keyword Search Unique (1,147)
Average Citation Count (ACC) = Total Lifetime Citation / Lifetime in Years
ACC >= 1.5 included
Quality Check (396)
Impact(291)
Reported User Study (604)
Methodology
Norming(5) Review / Data Collection
• Five papers were randomly selected • All reviewers reviewed the papers • Discussed the process in a group • Agreed on reviewing consistency • Started reviewing papers individually
Methodology
• Distributed papers equally among four • Regular meetings (Skype) • Doubt resolution • Data collected through Google Form
291 papers with 369 studies were reviewed
Norming (5) Review / Data Collection
Methodology
• Application areas & keywords • Experimental design & data collection • Participant demographics • Experimental tasks & environments • Types of experiment (pilot, formal etc.) • Types of senses augmented and displays used
Focused Attributes
• Earlier Surveys • Methodology • Results • Key Insights
A Systematic Review of Usability Studies in Augmented Reality between 2005 and 2014
Results
Publication Year2014
20132012
20112010
20092008
20072006
2005
Num
ber o
f Pap
ers
60
50
40
30
20
10
0
InteractionCollaborationTourism and ExplorationNavigation and DrivingIndustrial ApplicationEntertainment and GamingEducationMedicalPerception
Application Areas
Page 1
• Most papers in Interaction (67) and Perception (51)
• Tourism (8), Collaboration (12), and Entertainment (14) are among the least
Results
• HMD (102) and Handheld (100) almost equal
• 2010: Handheld overtook
Num
ber o
f Pap
ers
30
25
20
15
10
5
0
Publication Year2014
20132012
20112010
20092008
20072006
2005
…SpatialDesktopHMDHandheld
Displays Used
Page 1
Results
• Primarily lab-based / controlled
Num
ber o
f Pap
ers
60
50
40
30
20
10
0
Publication Year2014
20132012
20112010
20092008
20072006
2005
Heuristic EvaluationPilot TestField StudyFormal User Study
Experiment Type
Page 1
Results
• Tourism and Collaboration most cited • Medical had most authors per paper on average • Most collected both quantitative and qualitative data
Results
• Majority indoor studies • Within-subjects studies are most common • Overall participant median 16 (12-28) • Young, male, and university-based participants
-> results generelizable?
Results
perception
depth
displayevaluation
uservisual
optical outdoor
cuesdevices
interfacemobile
see-through
study text
virtual
x-ray
contrast
distance drawing
feedback
handheld head-mounted
information
interaction
legibility
presence
styles
vision
blur calibration character color computing
design
empirical environment
focus hand-
held haptic
immersion
label
mr
presentation
rendering saliency
sensitivity
user-perspective view
accommodation activity acuity attention aug autostereoscopy awareness background bci
bone
consistency context depth-
of-field discrimination disparity
eeg embodied
exocentric expectations experience geo
human illusion image
layout
measures mented methodology metric mirror
navigation online
peripheral placement polarity predictive
projected proprio-perception quantitative
realism rubber saltation
sensation simulation situation stereoscopic
stiffness subtle systems three-dimensional
video
Perception
user
museum
humaninterface
virtual
gesture guide
interaction
mobile
study
activity audio aware behavioral collaborative computing context detection environment
expectations experience following
human-robot identification immersion information
investigation learning lunar
modeling multi-user ontologies
presence public recognition requirements robot rule-based science screen sensory
services social surface systems tabletop tour-guide tracking ubiquitous
usability visiting
Tourism
mobilenavigation
displays
studies
uservisual
indoor
virtual
acceptance
camera
computing devices
driver
interface
maps
audio
driving
field
handheld
interaction
localization
outdoor
phones processes rendering
spatial
activity-based aging ambience arrow artificial
authoring aware band binaural camera-based
controls cues design differences digital
disorientation distraction
dynamic editing electronic evaluation gap geospatial gps graphic guided
heads-up highlighting human inattention individual
input intelligent interac legged
locomotion magic management maneuver manipulation
movement
object occlusion output path
peephole perceptual presence roads
screen search selection sensory slam small super-
imposed system tags tangible techniques technology tion topometric transportation trust
turn ultar-wide urban user-centered view visibility
wayfinding wearable windshield-based
Navigation
rehabilitation
surgery
medical
simulation
trainingvirtual
feedback
laparoscopy
navigation
computer control
phobia
device endoscopy
forcegait guidance
laparoscopic
learning
motor needle
visual
animal biopsy box
ct
cues education
exposure
game haptic image
medicine mixed
neurosurgery none patient
phone physical
registration
small stroke
systems therapy
tracking trainers
treatmentuser
acrophobia augmented-reality closed-loop
computer-assisted
exercise fmri
image-
guided index insertion intravenous
load
mobile model
mri nasa
objective parkinson
placement pretend radio-
frequency
sinus skull soft
therapeutic
tissue tomography
variability
Medical
interactionmobile
user
displaydevice
interface
techniques
camera
computing
environments
gesture
lens
selection
tracking
virtual
magic
spatial study
tangible
visual
design
evaluation experience field
handheld
manipulation
multimodal
phone projection
attention based
communicative
head head-
mounted
menus
modeling mounted none
view wearable
acquisition active annotation application
conflict context depth
dynamic end
eye
finger fitts force gaze hand
human human-computer illusion information input intent
law
map
natural object online orientation oz
peephole point prototype pseudo-haptics recognition see-
through service survey system
target touch
usability wizard
Interaction
user
computing
guidance
interactioninterfacesmaintenance
mobile
studies
virtual
awareness
device
model
order
picking
task
assembly
building
display engineering
ergonomics
industrial
information instruction
multimedia none
real-time repair
service
tracked
aided algorithms analysis applications architecture attention audio-
haptic augmentedreality compass
computer-assisted contex data design
environments
everyday expectations fabrication facility geometries
geospatial gis gps handheld haptic hardware head-
mounted heart-rate helmet hmd hypermedia
interfaces--input large learning life logistics long-term
management methodology micro-projection
military mixed monocular mounted
non-visual optical participation
planning prototypes readability
scenarios screen see-through sensor signal sketching spatial strategies
styles surveying systems systems--artificial
techniques techniques--interaction text tree urban
video vision visualization wearable workpiece
Industrial
games
mobile
study
handheld
play
co-
located collaboration
experiences
interactive interface
phone
social sound
virtual
ar audio audio-augmented auditory avoidance based board capture cd
competitive computer computer-supported
cooperative cross-media cscp cscw depth devices displays drama empirical engagement
environments exploratory eyes-free field
garden imaginary
interference intergenerational lens linear
location magic mixed motion music narratives orientation
peephole perception performance physical presence
probabilistic qualitative spatial static
tabletop task trial user work
Gaming
learninginteractive
user
computing
education environments
interface
systems
teaching
collaborative
evaluation experience
information
mobile
multimedia
training
academic
human
tangible
virtual
acceptance achievement applications areas attitude authoring cal
classroom cognitive cooperative
design elementary
game guideline human-computer
improving
method motivation
none rehabilitation sensor simulations subject
task technology ubiquitous
visualization
Education
collaborativecommunication
interaction
video-mediated
computer
mobile
cscw depthhuman
telepresence
analysis ar around artificial automotive awareness camera
design device
games gesture handheld
interfaces interpretation markerless
mediated mixed mr out-of-body person phones presence
recognition referencing remote sensors social spatial streaming study support system
tangible telecollaboration touch tracking user
view virtual visual
vr
Collaboration
• Earlier Surveys • Methodology • Results • Key Insights
A Systematic Review of Usability Studies in Augmented Reality between 2005 and 2014
Key Insights
• Nine primary application areas emerged• Primarily young, male, and university-based participants used• Handheld displays had a rapid uptake and equalled HMDs• More user studies needed in collaboration and entertainment• Median of 12-28 participants used in studies (overall median 16)• Not enough execution of pilot studies• Heuristics for AR applications need to be developed• Subjective questionnaires are most widely used instrument• Medical interfaces less focused on usability• Work in progress
List of all papers reviewed are available on poster!
Arindam Dey [email protected]
Empathic Computing Lab University of South Australia
www.empathiccomputing.org