COSC 426: Augmented Reality

Mark Billinghurst

mark.billinghurst@hitlabnz.org

July 26th 2013

Lecture 3: AR Tracking

Key Points from Lecture 2

“The product is no longer the basis of value. The

experience is.”

Venkat Ramaswamy The Future of Competition.

experiences

services

products

components

Valu

e

Sony CSL © 2004

Gilmore + Pine: Experience Economy

Function

Emotion

Interaction Design is All About You

  Users should be involved throughout the Design Process

  Consider all the needs of the user

Interaction Design Process

experiences

applications

tools

components

Building Compelling AR Experiences

Tracking, Display

Authoring

Interaction

Usability

Optical see-through head-mounted display

Virtual images from monitors

Real World

Optical Combiners

Video see-through HMD Video cameras

Monitors

Graphics

Combiner

Video

Video Monitor AR

Video cameras Monitor

Graphics Combiner

Video

Stereo glasses

AR Tracking and Registration

  Registration   Positioning virtual object wrt real world

  Tracking  Continually locating the users viewpoint

-  Position (x,y,z) -  Orientation (r,p,y)

Tracking

Tracking Requirements

  Augmented Reality Information Display   World Stabilized   Body Stabilized   Head Stabilized

Increasing Tracking Requirements

Head Stabilized Body Stabilized World Stabilized

Tracking Technologies  Active

•  Mechanical, Magnetic, Ultrasonic •  GPS, Wifi, cell location

 Passive •  Inertial sensors (compass, accelerometer, gyro) •  Computer Vision

•  Marker based, Natural feature tracking

 Hybrid Tracking •  Combined sensors (eg Vision + Inertial)

AR Tracking Taxonomy

e.g. AR Toolkit

Low Accuracy at 15-60 Hz

e.g. IVRD

High Accuracy & High Speed

Hybrid Tracking

Limited Range

e.g. HiBall

Many Fiducials in space/time

but no GPS

Extended Range

Indoor Environment

e.g. WLVA

Not Hybridized GPS or

Camera or Compass

Low Accuracy & Not Robust

e.g. BARS

Hybrid Tracking GPS and

Camera and Compass

High Accuracy & Robust

Outdoor Environment

AR TRACKING

Tracking Types

Magnetic Tracker

Inertial Tracker

Ultrasonic Tracker

Optical Tracker

Marker-Based Tracking

Markerless Tracking

Specialized Tracking

Edge-Based Tracking

Template-Based Tracking

Interest Point Tracking

Mechanical Tracker

Mechanical Tracker   Idea: mechanical arms with joint sensors

  ++: high accuracy, haptic feedback   -- : cumbersome, expensive

Microscribe

Magnetic Tracker   Idea: difference between a magnetic transmitter

and a receiver

  ++: 6DOF, robust   -- : wired, sensible to metal, noisy, expensive

Flock of Birds (Ascension)

Magnetic Tracking Error

Ultrasonics Tracker   Idea: Time of Flight or Phase-Coherence Sound Waves

  ++: Small, Cheap   -- : 3DOF, Line of Sight, Low resolution, Affected

Environment Conditon (pressure, temperature)

Ultrasonic Logitech IS600

Inertial Tracker   Idea: measuring linear and angular orientation rates

(accelerometer/gyroscope)

  ++: no transmitter, cheap, small, high frequency, wireless   -- : drift, hysteris only 3DOF

IS300 (Intersense) Wii Remote

Mobile Sensors   Inertial compass

  Earth’s magnetic field  Measures absolute orientation

  Accelerometers  Measures acceleration about axis  Used for tilt, relative rotation  Can drift over time

Global Positioning System (GPS)

  Created by US in 1978   Currently 29 satellites

  Satellites send position + time   GPS Receiver positioning

  4 satellites need to be visible   Differential time of arrival   Triangulation

  Accuracy   5-30m+, blocked by weather, buildings etc

Problems with GPS   Takes time to get satellite fix

  Satellites moving around

  Earths atmosphere affects signal   Assumes consistent speed (the speed of light).   Delay depends where you are on Earth   Weather effects

  Signal reflection   Multi-path reflection off buildings

  Signal blocking   Trees, buildings, mountains

  Satellites send out bad data   Misreport their own position

Accurate to < 5cm close to base station (22m/100 km) Expensive - $20-40,000 USD

Assisted-GPS (A-GPS)   Use external location server to send GPS signal

  GPS receivers on cell towers, etc   Sends precise satellite position (Ephemeris)

  Speeds up GPS Tracking   Makes it faster to search for satellites   Provides navigation data (don’t decode on phone)

  Other benefits   Provides support for indoor positioning   Can use cheaper GPS hardware   Uses less battery power on device

Assisted GPS

Cell Tower Triangulation   Calculate phone position

from signal strength   < 50 m in cities   > 1 km in rural

WiFi Positioning   Estimate location by using WiFi access points

 Can use know locations of WiFi access points   Triangulate through signal strength

  Eg. PlaceEngine (www.placeengine.com)  Client software for PC and mobiles   SDK returns position

  Accuracy   5 – 100m (depends on WiFi density)

WiFi Hotspots in New York

Indoor WiFi Location Sensing   Indoor Location

  Asset, people tracking

  Aeroscout   http://aeroscout.com/  WiFi + RFID

  Ekahau   http://www.ekahau.com/  WiFi + LED tracking

Integrated Systems   Combine GPS, Cell tower, WiFi signals   Skyhook (www.skyhookwireless.com)

  Core Engine

  Database of known locations   700 million Wi-Fi access points and cellular towers.

Comparative Accuracies   Study testing iPhone 3GS cf. low cost GPS   A-GPS

  8 m error   WiFi

  74 m error

  Cell Tower Positioning   600 m error

Accuracy of iPhone Locations: A Comparison of Assisted GPS, WiFi, and Cellular Positioning

In GIScience on July 15, 2009 at 8:11 pm By Paul A Zandbergen Transactions in GIS, Volume 13 Issue s1, Pages 5 - 25

Optical Tracking

Optical Tracker   Idea: Image Processing and Computer Vision   Specialized

  Infrared, Retro-Reflective, Stereoscopic

  Monocular Based Vision Tracking

ART Hi-Ball

Outside-In vs. Inside-Out Tracking

Optical Tracking Technologies

  Scalable active trackers   InterSense IS-900, 3rd Tech HiBall

  Passive optical computer vision   Line of sight, may require landmarks  Can be brittle.  Computer vision is computationally-intensive

3rd Tech, Inc.

HiBall Tracking System (3rd Tech)   Inside-Out Tracker

  $50K USD

  Scalable over large area   Fast update (2000Hz)   Latency Less than 1 ms.

  Accurate   Position 0.4mm RMS  Orientation 0.02° RMS

Starting simple: Marker tracking   Has been done for more than 10 years   A square marker provides 4 corners

  Enough for pose estimation!

  Several open source solutions exist   Fairly simple to implement

  Standard computer vision methods

Marker Based Tracking: ARToolKit

http://artoolkit.sourceforge.net/

Tracking Range with Pattern Size

Rule of thumb – range = 10 x pattern width

Tracking Error with Range

Tracking Error with Angle

Tracking challenges in ARToolKit

False positives and inter-marker confusion (image by M. Fiala)

Image noise (e.g. poor lens, block coding /

compression, neon tube)

Unfocused camera, motion blur

Dark/unevenly lit scene, vignetting

Jittering (Photoshop illustration)

Occlusion (image by M. Fiala)

Limitations of ARToolKit   Partial occlusions cause tracking failure   Affected by lighting and shadows   Tracking range depends on marker size   Performance depends on number of markers

  cf artTag, ARToolKitPlus   Pose accuracy depends on distance to marker   Pose accuracy depends on angle to marker

Tracking, Tracking, Tracking

Other Marker Tracking Libraries   arTag

  http://www.artag.net/

  ARToolKitPlus [Discontinued]   http://studierstube.icg.tu-graz.ac.at/handheld_ar/

artoolkitplus.php   stbTracker

  http://studierstube.icg.tu-graz.ac.at/handheld_ar/stbtracker.php

  MXRToolKit   http://sourceforge.net/projects/mxrtoolkit/

Markerless Tracking

Markerless Tracking

Magnetic Tracker Inertial Tracker

Ultrasonic Tracker

Optical Tracker

Marker-Based Tracking

Markerless Tracking

Specialized Tracking

Edge-Based Tracking

Template-Based Tracking

Interest Point Tracking

  No more Markers! Markerless Tracking

Natural feature tracking   Tracking from features of the surrounding

environment   Corners, edges, blobs, ...

  Generally more difficult than marker tracking   Markers are designed for their purpose   The natural environment is not…

  Less well-established methods   Usually much slower than marker tracking

Natural Feature Tracking   Use Natural Cues of Real Elements

  Edges   Surface Texture   Interest Points

  Model or Model-Free   ++: no visual pollution

Contours

Features Points

Surfaces

Texture Tracking

Edge Based Tracking   RAPiD [Drummond et al. 02]

  Initialization, Control Points, Pose Prediction (Global Method)

Line Based Tracking   Visual Servoing [Comport et al. 2004]

Model Based Tracking   Track from 3D model   Eg OpenTL - www.opentl.org

  General purpose library for model based visual tracking

Marker vs. natural feature tracking   Marker tracking

  + Can require no image database to be stored   + Markers can be an eye-catcher   + Tracking is less demanding   - The environment must be instrumented with markers   - Markers usually work only when fully in view

  Natural feature tracking   - A database of keypoints must be stored/downloaded   + Natural feature targets might catch the attention less   + Natural feature targets are potentially everywhere   + Natural feature targets work also if partially in view

Hybrid Tracking

Sensor tracking   Used by many “AR browsers”   GPS, Compass, Accelerometer, (Gyroscope)   Not sufficient alone (drift, interference)

Outdoor Hybrid Tracking   Combines

  computer vision -  natural feature tracking

  inertial gyroscope sensors   Both correct for each other

  Inertial gyro - provides frame to frame prediction of camera orientation

  Computer vision - correct for gyro drift

Combining Sensors and Vision   Sensors

-  Produce noisy output (= jittering augmentations) -  Are not sufficiently accurate (= wrongly placed augmentations) -  Gives us first information on where we are in the world,

and what we are looking at   Vision

-  Is more accurate (= stable and correct augmentations) -  Requires choosing the correct keypoint database to track from -  Requires registering our local coordinate frame (online-

generated model) to the global one (world)

Outdoor AR Tracking System

You, Neumann, Azuma outdoor AR system (1999)

Robust Outdoor Tracking

  Hybrid Tracking  Computer Vision, GPS, inertial

  Going Out   Reitmayer & Drummond (Univ. Cambridge)

Handheld Display

Registration

Spatial Registration

The Registration Problem   Virtual and Real must stay properly aligned   If not:

  Breaks the illusion that the two coexist   Prevents acceptance of many serious applications

Sources of registration errors   Static errors

 Optical distortions  Mechanical misalignments   Tracker errors   Incorrect viewing parameters

  Dynamic errors   System delays (largest source of error)

-  1 ms delay = 1/3 mm registration error

Reducing static errors   Distortion compensation   Manual adjustments   View-based or direct measurements   Camera calibration (video)

View Based Calibration (Azuma 94)

Dynamic errors

  Total Delay = 50 + 2 + 33 + 17 = 102 ms   1 ms delay = 1/3 mm = 33mm error

Tracking Calculate Viewpoint Simulation

Render Scene

Draw to Display

x,y,z r,p,y

Application Loop

20 Hz = 50ms 500 Hz = 2ms 30 Hz = 33ms 60 Hz = 17ms

Reducing dynamic errors (1)

  Reduce system lag   Faster components/system modules

  Reduce apparent lag   Image deflection   Image warping

Reducing System Lag

Tracking Calculate Viewpoint Simulation

Render Scene

Draw to Display

x,y,z r,p,y

Application Loop

Faster Tracker Faster CPU Faster GPU Faster Display

Reducing Apparent Lag

Tracking Update

x,y,z r,p,y

Virtual Display

Physical Display

(640x480)

1280 x 960

Last known position

Virtual Display

Physical Display

(640x480)

1280 x 960

Latest position

Tracking Calculate Viewpoint Simulation

Render Scene

Draw to Display

x,y,z r,p,y

Application Loop

Reducing dynamic errors (2)   Match input streams (video)

 Delay video of real world to match system lag

  Predictive Tracking   Inertial sensors helpful

Azuma / Bishop 1994

Predictive Tracking

Time

Position

Past Future

Can predict up to 80 ms in future (Holloway)

Now

Predictive Tracking (Azuma 94)

Wrap-up   Tracking and Registration are key problems   Registration error

 Measures against static error  Measures against dynamic error

  AR typically requires multiple tracking technologies   Research Areas: Hybrid Markerless Techniques,

Deformable Surface, Mobile, Outdoors

Project List   Mobile

  Hybrid Tracking for Outdoor AR   City Scale AR Visualization   Outdoor AR Authoring Tool   Outdoor AR collaborative game   AR interaction for Google Glass

  Non-Mobile   AR Face Painting   AR Authoring Tool   Tangible AR puppeteer studio   Gesture based interaction with AR content

More Information •  Mark Billinghurst

– mark.billinghurst@hitlabnz.org •  Websites

– www.hitlabnz.org