7/29/2019 Week14 VR AR

1/26

Combining real imagery withcomputer generated imagery

Virtual reality;

Augmented reality;Teleorobotics

7/29/2019 Week14 VR AR

2/26

Combining real imagery with

computer generated imagery Robot-assisted surgery

Virtual real estate tours

Virtual medical tours

Urban planning

Map-assisted navigation Computer games

7/29/2019 Week14 VR AR

3/26

Virtual image of real data3D sensed data can bestudied for surgical paths tobe followed by a surgeon ora robot.

In the future, real-timesensing and registration canbe used for feedback in theprocess.

7/29/2019 Week14 VR AR

4/26

Human operating in a real

environment: brain surgery.

All objects are real; we cook food, chopwood, do brain surgery

7/29/2019 Week14 VR AR

5/26

Most computer games / videos

are entirely virtualIMMERSION, or engagement, canbe very high, however, with

Quality spatial resolutionStereo

Smooth motion

Little time delay between userinteractions and visual effects

Synchronized audio and forcefeedback are important

Courtesy of University ofWashington HIT Lab

7/29/2019 Week14 VR AR

6/26

Virtual immersive environments

7/29/2019 Week14 VR AR

7/26

Virtual environment schematic

Example: nurse gets training on giving injections usinga system with stereo imagery and haptic feedback

7/29/2019 Week14 VR AR

8/26

Virtual dextrous work

Medical personnel practice surgery or injection, etc. Artist can carvea virtual 3D object. Haptic system pushes back on tool appropriateto its penetration (intersection) of the model space. Users free hand

grabs a physical arm model under the table in injection training.

http://www.sensable.com/products-haptic-devices.htm

7/29/2019 Week14 VR AR

9/26

Augmented reality: views of

real objects + augmentation

7/29/2019 Week14 VR AR

10/26

AR in teleconferencing person works at real desk

remote collaboratorrepresented by picture or

video or talking head

objects of discussion; e.g.a patients brain image,might also be fused intovisual field

HOW IS THIS ACHIEVED?

From University of Washington HIT Lab

7/29/2019 Week14 VR AR

11/26

Imagine the virtual book Real book with empty identifiable pages

AR headset

Pay and download a story

System presents new stereo imageswhen the pages are turned

Is this better than a .pdf file?

Is this better than stereo .pdf?

7/29/2019 Week14 VR AR

12/26

Human operating with AR

Think of a heads up display on your auto windshield, or on theinstrument panel. What could be there to help you navigate?

(Vectors to nearby eating places? Blinking objects we mightcollide with? Congestion of nearby intersections? Web pages?)

7/29/2019 Week14 VR AR

13/26

Special devices needed to fuse/register real and

generated images

Human sees real environment optics design problem

Human sees graphicsgenerated from 3D/2D models computer graphics problem

Graphics system needs toknow how the human isviewing the 3D environmentdifficult pose sensing problem From University of Washington HIT Lab.

7/29/2019 Week14 VR AR

14/26

Devices that support AR

Need to fuse imagery;

Need to compute pose of userrelative to the real world

7/29/2019 Week14 VR AR

15/26

Fusing CAD models with real env.

Plumber marks the wall where the CAD blueprint shows thepipe to be.

7/29/2019 Week14 VR AR

16/26

Two types of HMD

7/29/2019 Week14 VR AR

17/26

Difficult augmentation problem

How does computer system know where toplace the graphic overlay?

Human very sensitive to misregistration Some applications OK such as circuit board

inspection.

Can use trackers on HMD to give approximate

head pose Tough calibration procedures for individuals

(see Charles Owens work)

7/29/2019 Week14 VR AR

18/26

Teleoperation

remotely guided police robot moves a suspected bomb

teleoperated robot cleans up nuclear reactor problem

surgeon in US performs surgery on a patient in France

Dr in Lansing does breast exam on woman in Escanaba (work

of Mutka, Xi, Mukergee, et al.)

7/29/2019 Week14 VR AR

19/26

Teleoperation on power lines

7/29/2019 Week14 VR AR

20/26

Face2face mobiletelecommunication

Concept HMD at left; actual images from our prototype HMD at right.

Problem is to communicate the face to a remote communicator.

7/29/2019 Week14 VR AR

21/26

Reddy/Stockman used geometrictransformation and mosaicking

Which 2 are real

video frames andwhich arecomposed of 2transformed andmosaicked views?

7/29/2019 Week14 VR AR

22/26

Miguel Figueroas system

Face image is fit as a blendof basis faces from trainingimages

c1F1+c2F2+ cnFn

Coefficients [c1, c2, , cn]sent to receiver embeddedin the voice encoding.

Receiver already has thebasis vectors F1, F2, , Fnand a mapping from sideview to frontal view and canreconstruct the currentframe.

7/29/2019 Week14 VR AR

23/26

Actual prototype in operation

Mirror size is exaggerated in these images by perspective;however they are larger than desired. Consider using theMotorola headsets that football coaches use with tinycamera on the microphone boom.

7/29/2019 Week14 VR AR

24/26

Captured side view projectedonto basis of training samples

7/29/2019 Week14 VR AR

25/26

Frontal views contructed bymapping from side views

This approach avoids geometrical reconstruction of distorted left andright face parts by using AAM methods -- training and mapping.

7/29/2019 Week14 VR AR

26/26

Summary of issues

All systems (VR,AR,TO) require sensingof human actions or robot actions

All systems need models of objects orthe environment

Difficult registration accuracy problem

for AR, especially for see-throughdisplays, where the fusion is done inthe humans visual system