Designing Augmented Reality Experiences

80
Designing Augmented Reality Experiences Mark Billinghurst [email protected] The HIT Lab NZ, University of Canterbury June 5 th 2013

description

Talk on Designing Augmented Reality Experiences given by Mark Billinghurst at the AWE 2013 conference on June 5th 2013

Transcript of Designing Augmented Reality Experiences

Page 1: Designing Augmented Reality Experiences

Designing Augmented Reality Experiences

Mark Billinghurst

[email protected]

The HIT Lab NZ, University of Canterbury

June 5th 2013

Page 2: Designing Augmented Reality Experiences

How Would You Design This?

  Put nice AR Picture here – and video

Page 3: Designing Augmented Reality Experiences

Or This?

Page 4: Designing Augmented Reality Experiences

DARE 101 1.  Know the Technology 2.  Design for User Experience

  All aspects of user experience

3.  Follow good Interaction Design principles   Discover, Design, Evaluate

4.  Consider all the Design Elements   Physical, Virtual and Metaphorical

5.  Know Future Research Directions

Page 5: Designing Augmented Reality Experiences

Know the Technology

Page 6: Designing Augmented Reality Experiences

What is Augmented Reality?  Defining Characteristics (Azuma 97)

•  Combines Real and Virtual Images – Both can be seen at the same time

•  Interactive in real-time – The virtual content can be interacted with

•  Registered in 3D – Virtual objects appear fixed in space

Azuma, R., A Survey of Augmented Reality, Presence, Vol. 6, No. 4, August 1997, pp. 355-385.

Page 7: Designing Augmented Reality Experiences

AR From Science Fiction to Fact

1977 – Star Wars

2008 – CNN

Page 8: Designing Augmented Reality Experiences

AR Part of MR Continuum

Mixed Reality

Reality - Virtuality (RV) Continuum

Real Environment

Augmented Reality (AR)

Augmented Virtuality (AV)

Virtual Environment

"...anywhere between the extrema of the virtuality continuum."

P. Milgram and A. F. Kishino, Taxonomy of Mixed Reality Visual Displays IEICE Transactions on Information and Systems, E77-D(12), pp. 1321-1329, 1994.

Page 9: Designing Augmented Reality Experiences

Core Technologies  Combining Real and Virtual Images

•  Display technologies  Interactive in Real-Time

•  Input and interactive technologies  Registered in 3D

•  Viewpoint tracking technologies Display

Processing

Input Tracking

Page 10: Designing Augmented Reality Experiences

Display Technologies  Types (Bimber/Raskar 2003)

 Head attached •  Head mounted display/projector

 Body attached •  Handheld display/projector

 Spatial •  Spatially aligned projector/monitor

 HMD Optical vs. Video see-through   Optical: Direct view of real world -> safer, simpler   Video: Video overlay -> more image registration options

Page 11: Designing Augmented Reality Experiences

Display Taxonomy

Page 12: Designing Augmented Reality Experiences

AR Input Technologies  Tangible objects

•  Tracked items  Touch (HHD)

•  Glove, touch  Gesture

•  Glove, free-hand  Speech/Multimodal  Device motion

•  HHD + sensors

Page 13: Designing Augmented Reality Experiences

Tracking Technologies  Active

•  Mechanical, Magnetic, Ultrasonic •  GPS, Wifi, cell location

 Passive •  Inertial sensors (compass, accelerometer, gyro) •  Computer Vision

•  Marker based, Natural feature tracking, model based

 Hybrid Tracking •  Combined sensors (eg Vision + Inertial)

Page 14: Designing Augmented Reality Experiences

Design for User Experience

Page 15: Designing Augmented Reality Experiences

“The product is no longer the basis of value. The

experience is.”

Venkat Ramaswamy The Future of Competition.

Interaction Design

Page 16: Designing Augmented Reality Experiences

experiences

services

products

components

Valu

e

Gilmore + Pine: Experience Economy

Function

Emotion

Page 17: Designing Augmented Reality Experiences

experiences

applications

tools

components

Designing AR Experiences

Tracking, Display, Input

Authoring

Interaction

Usability

Page 18: Designing Augmented Reality Experiences

The Value of Good User Experience

Kenya: 20c

My house: 50c

Starbucks: $3.50

Page 19: Designing Augmented Reality Experiences

Good Experience Design   Reactrix

  Top down projection   Camera based input   Reactive Graphics   No instructions   No training

Page 20: Designing Augmented Reality Experiences

Would You Wear This?

Page 21: Designing Augmented Reality Experiences

User Experience is All About You   Designing good user

experience involves many aspects

  Consider all the needs of the user   Especially context of

use

Page 22: Designing Augmented Reality Experiences

  Web Based AR   Flash, HTML 5 based AR   Marketing, education

  Outdoor Mobile AR   GPS, compass tracking   Viewing Points of Interest in real world

  Handheld AR   Vision based tracking   Marketing, gaming

  Location Based Experiences   HMD, fixed screens   Museums, point of sale, advertising

Typical AR Experiences

Page 23: Designing Augmented Reality Experiences

What Makes a Good AR Experience?   Compelling

  Engaging, ‘Magic’ moment

  Intuitive, ease of use  Uses existing skills

  Anchored in physical world   Seamless combination of real and digital

Page 24: Designing Augmented Reality Experiences

Demo: colAR

  Turn colouring books pages into AR scenes  Markerless tracking, use your own colours..

  Try it yourself: http://www.colARapp.com/

Page 25: Designing Augmented Reality Experiences

Follow Good Interaction Design Principles

Page 26: Designing Augmented Reality Experiences

Interaction Design

  Answering three questions:   What do you do? - How do you affect the world?   What do you feel? – What do you sense of the world?   What do you know? – What do you learn?

 The Design of User Experience with Technology

“Designing interactive products to support people in their everyday and working lives”

Preece, J., (2002). Interaction Design

Page 27: Designing Augmented Reality Experiences

Interaction Design Process

Interaction Design

Page 28: Designing Augmented Reality Experiences

AR UI Design  Consider your user   Follow good HCI principles  Adapt HCI guidelines for AR  Design to device constraints  Using Design Patterns to Inform Design  Design for you interface metaphor  Design for evaluation

Page 29: Designing Augmented Reality Experiences

Consider Your User   Consider context of user

  Physical, social, emotional, cognitive, etc

  Mobile Phone AR User   Probably Mobile  One hand interaction   Short application use  Need to be able to multitask  Use in outdoor or indoor environment  Want to enhance interaction with real world

Page 30: Designing Augmented Reality Experiences

AR vs. Non AR Design

  Design Guidelines  Design for 3D graphics + Interaction  Consider elements of physical world   Support implicit interaction

Characteristics Non-AR Interfaces AR Interfaces

Object Graphics Mainly 2D Mainly 3D

Object Types Mainly virtual objects Both virtual and physical objects

Object behaviors Mainly passive objects Both passive and active objects

Communication Mainly simple Mainly complex

HCI methods Mainly explicit Both explicit and implicit

Page 31: Designing Augmented Reality Experiences

Maps vs. Junaio

  Google Maps   2D, mouse driven, text/image heavy, exocentric

  Junaio   3D, location driven, simple graphics, egocentric

Page 32: Designing Augmented Reality Experiences

Design to Device Constraints   Understand the platform and design for limitations

 Hardware, software platforms

  Eg Handheld AR game with visual tracking  Use large screen icons  Consider screen reflectivity   Support one-hand interaction  Consider the natural viewing angle  Do not tire users out physically  Do not encourage fast actions   Keep at least one tracking surface in view 32

Art of Defense Game

Page 33: Designing Augmented Reality Experiences

Design Patterns “Each pattern describes a problem which occurs over and over again in our environment, and then describes the core of the solution to that problem in such a way that you can use this solution a million times over, without ever doing it the same way twice.”

– Christopher Alexander et al.

Use Design Patterns to Address Reoccurring Problems

C.A. Alexander, A Pattern Language, Oxford Univ. Press, New York, 1977.

Page 34: Designing Augmented Reality Experiences

Handheld AR Patterns Title Meaning Embodied Skills Device Metaphors Using metaphor to suggest available player

actions Body A&S Naïve physics

Control Mapping Intuitive mapping between physical and digital objects

Body A&S Naïve physics

Seamful Design Making sense of and integrating the technological seams through game design

Body A&S

World Consistency Whether the laws and rules in physical world hold in digital world

Naïve physics Environmental A&S

Landmarks Reinforcing the connection between digital-physical space through landmarks

Environmental A&S

Personal Presence The way that a player is represented in the game decides how much they feel like living in the digital game world

Environmental A&S Naïve physics

Living Creatures Game characters that are responsive to physical, social events that mimic behaviours of living beings

Social A&S Body A&S

Body constraints Movement of one’s body position constrains another player’s action

Body A&S Social A&S

Hidden information The information that can be hidden and revealed can foster emergent social play

Social A&S Body A&S

Page 35: Designing Augmented Reality Experiences

Example: Seamless Design

  Design to reduce seams in the user experience   Eg: AR tracking failure, change in interaction mode

  Paparazzi Game  Change between AR tracking to accelerometer input

Yan Xu , et.al. , Pre-patterns for designing embodied interactions in handheld augmented reality games, Proceedings of the 2011 IEEE International Symposium on Mixed and Augmented Reality--Arts, Media, and Humanities, p.19-28, October 26-29, 2011

Page 36: Designing Augmented Reality Experiences

Example: Living Creatures

  Virtual creatures respond to real world events   eg. Player motion, wind, light, etc  Creates illusion creatures are alive in the real world

  Sony EyePet   Responds to player blowing on creature

36

Page 37: Designing Augmented Reality Experiences

Rapid Hardware Prototyping

  Speed development time by using quick hardware mockups   Handheld connected to PC, LCD screen, USB phone keypad,

Camera

  Can use PC tools for rapid application development   Flash, Visual Basic, etc

Page 38: Designing Augmented Reality Experiences

Build Your Own Google Glass

  Rapid Prototype Glass-Like HMD   Myvu HMD + headphone + iOS Device + basic glue skills

  $300 + less than 3 hours construction   http://www.instructables.com/id/DIY-Google-Glasses-AKA-the-Beady-i/

Page 39: Designing Augmented Reality Experiences

Why Evaluate AR Applications?   To test and compare interfaces, new technologies,

interaction techniques   To validate the efficiency and efficient the AR

interface and system   Test Usability (learnability, efficiency, satisfaction,...)   Get user feedback, Better understand your users   Refine interface design   Better understand your end users   ...

Page 40: Designing Augmented Reality Experiences

HIT Lab NZ Usability Survey   A Survey of Evaluation Techniques Used in

Augmented Reality Studies   Andreas Dünser, Raphaël Grasset, Mark Billinghurst

  reviewed publications from 1993 to 2007   Extracted 6071 papers which mentioned “Augmented

Reality”   Searched to find 165 AR papers with User Studies

Page 41: Designing Augmented Reality Experiences

Types of Experiments and topics   Sensation, Perception & Cognition

  How is virtual content perceived ?   What perceptual cues are most important ?   How to visualize augmented/overlay information on real environment?   Visual search/attention/salience issues of human performance

  Interaction   How can users interact with virtual content ?   Which interaction techniques are most efficient in certain context ?

  Collaboration & Social issues   How is collaboration in AR interface different ?   Which collaborative cues can be conveyed best ?   Privacy and security issues of AR interface

Page 42: Designing Augmented Reality Experiences

Gabbard Model for AR Design

1. user task analysis 2. expert guidelines-based evaluation 3. formative user-centered evaluation 4. summative comparative evaluations

Gabbard, J.L.; Swan, J.E.; , "Usability Engineering for Augmented Reality: Employing User-Based Studies to Inform Design,” Visualization and Computer Graphics, IEEE Transactions on, vol.14, no.3, pp.513-525, May-June 2008

Page 43: Designing Augmented Reality Experiences

Gabbard Model in Context

Page 44: Designing Augmented Reality Experiences

Consider All Design Elements

Page 45: Designing Augmented Reality Experiences

  Interface Components  Physical components  Display elements

- Visual/audio   Interaction metaphors

Physical Elements

Virtual Elements Interaction

Metaphor Input Output

AR Design Elements

Page 46: Designing Augmented Reality Experiences

AR Design Space

Reality Virtual Reality

Augmented Reality

Physical Design Virtual Design

Page 47: Designing Augmented Reality Experiences

Design of Objects   Objects

  Purposely built – affordances   “Found” – repurposed   Existing – already at use in marketplace

  Affordance   The quality of an object allowing an action-

relationship with an actor   An attribute of an object that allows people to

know how to use it -  e.g. a door handle affords pulling

Page 48: Designing Augmented Reality Experiences

Affordance Led Design   Make affordances perceivable

  Provide visual, haptic, tactile, auditory cues

  Affordance Led Usability   Give feedback   Provide constraints   Use natural mapping   Use good cognitive model

Page 49: Designing Augmented Reality Experiences

Example: AR Chemistry

  Tangible AR chemistry education (Fjeld) Fjeld, M., Juchli, P., and Voegtli, B. M. 2003. Chemistry education: A tangible interaction

approach. Proceedings of INTERACT 2003, September 1st -5th 2003, Zurich, Switzerland.

Page 50: Designing Augmented Reality Experiences

Input Devices

  Form informs function and use

Page 51: Designing Augmented Reality Experiences

AR Interaction Metaphors   Information Browsing

  View AR content

  3D AR Interfaces   3D UI interaction techniques

  Augmented Surfaces   Tangible UI techniques

  Tangible AR   Tangible UI input + AR output

Page 52: Designing Augmented Reality Experiences

1. Information Browsing   Information is registered to

real-world context  Hand held AR displays

  Interaction  Manipulation of a window

into information space   Applications

 Context-aware information displays

Rekimoto, et al. 1997

Page 53: Designing Augmented Reality Experiences

2. 3D AR Interfaces   Virtual objects displayed in 3D

physical space and manipulated   HMDs and 6DOF head-tracking   6DOF hand trackers for input

  Interaction   Viewpoint control   Traditional 3D user interface

interaction: manipulation, selection, etc.

Kiyokawa, et al. 2000

Page 54: Designing Augmented Reality Experiences

3. Augmented Surfaces   Basic principles

  Virtual objects are projected on a surface   Physical objects are used as controls for

virtual objects   Support for collaboration

  Rekimoto, et al. 1998   Front projection   Marker-based tracking   Multiple projection surfaces

Page 55: Designing Augmented Reality Experiences

Lessons from Tangible Interfaces

  Physical objects make us smart   Norman’s “Things that Make Us Smart”   encode affordances, constraints

  Objects aid collaboration   establish shared meaning

  Objects increase understanding   serve as cognitive artifacts

Page 56: Designing Augmented Reality Experiences

TUI Limitations   Difficult to change object properties

  Can’t tell state of digital data   Limited display capabilities

  projection screen = 2D   dependent on physical display surface

  Separation between object and display   Augmented Surfaces

Page 57: Designing Augmented Reality Experiences

4. Tangible AR Metaphor   AR overcomes limitation of TUIs

  enhance display possibilities   merge task/display space   provide public and private views

  TUI + AR = Tangible AR   Apply TUI methods to AR interface design

Page 58: Designing Augmented Reality Experiences

Tangible AR Demo

  Use of natural physical object manipulations to control virtual objects

  VOMAR Demo  Catalog book:

-  Turn over the page   Paddle operation:

-  Push, shake, incline, hit, scoop

Page 59: Designing Augmented Reality Experiences

Object Based Interaction: MagicCup   Intuitive Virtual Object Manipulation

on a Table-Top Workspace

  Time multiplexed  Multiple Markers

-  Robust Tracking   Tangible User Interface

-  Intuitive Manipulation   Stereo Display

-  Good Presence

Page 60: Designing Augmented Reality Experiences
Page 61: Designing Augmented Reality Experiences

Tangible AR Design Principles   Tangible AR Interfaces use TUI principles

  Physical controllers for moving virtual content   Support for spatial 3D interaction techniques   Time and space multiplexed interaction   Support for multi-handed interaction  Match object affordances to task requirements   Support parallel activity with multiple objects   Allow collaboration between multiple users

Page 62: Designing Augmented Reality Experiences

Example 1: AR Lens   Physical Components

  Lens handle -  Virtual lens attached to real object

  Display Elements   Lens view

-  Reveal layers in dataset

  Interaction Metaphor   Physically holding lens

Page 63: Designing Augmented Reality Experiences

Example 2: LevelHead

  Physical Components  Real blocks

  Display Elements   Virtual person and rooms

  Interaction Metaphor   Blocks are rooms

Page 64: Designing Augmented Reality Experiences

Know Future Research Directions

Page 65: Designing Augmented Reality Experiences

The Vision of AR

Page 66: Designing Augmented Reality Experiences

To Make the Vision Real..  Hardware/software requirements

 Contact lens displays  Free space hand/body tracking  Speech/gesture recognition  Etc..

 Most importantly  Usability/User Experience

Page 67: Designing Augmented Reality Experiences

Natural Interaction   Automatically detecting real environment

  Environmental awareness   Physically based interaction

  Gesture Input   Free-hand interaction

  Multimodal Input   Speech and gesture interaction   Implicit rather than Explicit interaction

Page 68: Designing Augmented Reality Experiences

AR MicroMachines   AR experience with environment awareness

and physically-based interaction   Based on MS Kinect RGB-D sensor

  Augmented environment supports   occlusion, shadows   physically-based interaction between real and

virtual objects

Page 69: Designing Augmented Reality Experiences

Physics Simulation

  Create virtual mesh over real world

  Update at 10 fps – can move real objects

  Use by physics engine for collision detection (virtual/real)

  Use by OpenScenegraph for occlusion and shadows

Page 70: Designing Augmented Reality Experiences

Rendering

Occlusion Shadows

Page 71: Designing Augmented Reality Experiences

Gesture Input Architecture 5. Gesture

•  Static Gestures • Dynamic Gestures •  Context based Gestures

4. Modeling

• Hand recognition/modeling •  Rigid-body modeling

3. Classification/Tracking

2. Segmentation

1. Hardware Interface

Page 72: Designing Augmented Reality Experiences

Results

Page 73: Designing Augmented Reality Experiences

Free Hand Multimodal Input   Use free hand to interact with AR content   Recognize simple gestures   No marker tracking

Point Move Pick/Drop

Page 74: Designing Augmented Reality Experiences

Multimodal Architecture

Page 75: Designing Augmented Reality Experiences

Multimodal Fusion

Page 76: Designing Augmented Reality Experiences

Hand Occlusion

Page 77: Designing Augmented Reality Experiences

Conclusion

Page 78: Designing Augmented Reality Experiences

Conclusion

  There is need for better designed AR experiences   Through

  use of Interaction Design principles   understanding of the technology   use of rapid prototyping tools   rigorous user evaluation

  There a number of important areas for future research   Natural interaction, Multimodal interfaces, Intelligent agents, …

Page 79: Designing Augmented Reality Experiences

More Information •  Mark Billinghurst

– [email protected]

•  Websites – www.hitlabnz.org

Page 80: Designing Augmented Reality Experiences

80

Resources