Augmented-Reality

Augmented-RealityAugmented-Reality

Ping GaiPing Gai

HFE 760HFE 760

Augmented-RealityAugmented-Reality

Augmented- Reality DefinitionAugmented- Reality Definition Augmented Reality vs. Virtual RealityAugmented Reality vs. Virtual Reality Visual Display Systems for ARVisual Display Systems for AR Video Keying and Image RegistrationVideo Keying and Image Registration System Design IssuesSystem Design Issues Augmented Reality ApplicationAugmented Reality Application

Augmented Reality DefinitionAugmented Reality Definition

Augmented Reality is a growing area in Augmented Reality is a growing area in virtual reality area. virtual reality area.

An Augmented Reality system generates a An Augmented Reality system generates a composite view for the user. It’s a composite view for the user. It’s a combination of the real scene viewed by the combination of the real scene viewed by the user and a virtual scene generated by the user and a virtual scene generated by the computer that augments the scene generated computer that augments the scene generated by the computer that augmented the scene by the computer that augmented the scene with additional information.with additional information.

Augmented Reality DefinitionAugmented Reality Definition

Typically, the real-world visual scene in an AR Typically, the real-world visual scene in an AR display is captured by video or directly viewed. display is captured by video or directly viewed.

Most current AR displays are designed using see-Most current AR displays are designed using see-through HMDs which allow the observer to view through HMDs which allow the observer to view the real world directly with the naked eye. the real world directly with the naked eye.

If video is used to capture the real world, one may If video is used to capture the real world, one may use either an opaque HMD or screen-based system use either an opaque HMD or screen-based system to view the scene. to view the scene.

AR vs. VRAR vs. VR

Virtual Reality: a computer generated, interactive, Virtual Reality: a computer generated, interactive, three-dimensional environment in which a person three-dimensional environment in which a person is immersed.(Aukstakanis and Blatner, 1992)is immersed.(Aukstakanis and Blatner, 1992)– Virtual Environment is a computer generated three Virtual Environment is a computer generated three

dimensional scene which requires high performance dimensional scene which requires high performance computer graphics to provide an adequate level of computer graphics to provide an adequate level of realism.realism.

– The virtual world is interactive. A user requires real-The virtual world is interactive. A user requires real-time response from the system to be able to interact time response from the system to be able to interact with it in an effective manner. with it in an effective manner.

– The user is immersed in this virtual environment. The user is immersed in this virtual environment.

AR vs . VRAR vs . VR

VR: the user is completely immersed in an VR: the user is completely immersed in an artificial world and becomes divorced from artificial world and becomes divorced from the real environment. The generated world the real environment. The generated world consists entirely of computer graphics. consists entirely of computer graphics.

AR vs. ARAR vs. AR

VR strives for a totally immersive VR strives for a totally immersive environment. The visual, and in some environment. The visual, and in some systems aural and sense are under control of systems aural and sense are under control of the system. the system.

In contrast, an AR system is augmenting the In contrast, an AR system is augmenting the real world sense of presence in that world. real world sense of presence in that world. The virtual images are merged with the real The virtual images are merged with the real view to create the augmented display. view to create the augmented display.

AR vs. VRAR vs. VR

For some applications , it may be desirable For some applications , it may be desirable to use as much as possible real world in the to use as much as possible real world in the scene rather creating a new scene using scene rather creating a new scene using computer imagery. For example, in medical computer imagery. For example, in medical applications, the physician must view the applications, the physician must view the patient to perform surgery, in telerobotics patient to perform surgery, in telerobotics the operator must view the remote scene in the operator must view the remote scene in order to perform tasks. order to perform tasks.

AR vs. VRAR vs. VR

A main motivation for the use of AR relates A main motivation for the use of AR relates to the computational resources necessary to to the computational resources necessary to generate and update computer-generated generate and update computer-generated scene. In VR, The more complex the scene, scene. In VR, The more complex the scene, the more computational resource needed to the more computational resource needed to render the scene. render the scene.

AR can maintain the high-level of detail AR can maintain the high-level of detail and realistic shading that one finds in the and realistic shading that one finds in the real world. real world.

AR vs. VRAR vs. VR

NO simulator sickness. Vertigo, dizziness NO simulator sickness. Vertigo, dizziness introduced by sensory mismatch within introduced by sensory mismatch within display environment can be a problem when display environment can be a problem when one uses an HMD to view a virtual world. one uses an HMD to view a virtual world.

If the task is to show an annotation to the If the task is to show an annotation to the real world. real world.

Visual Display System for ARVisual Display System for AR

Hardware for display visual imagesHardware for display visual images A position and orientation sensing system A position and orientation sensing system Hardware for combining the computer Hardware for combining the computer

graphics and video images into one signalgraphics and video images into one signal The associated system softwareThe associated system software


There are two main ways in which the real world There are two main ways in which the real world and the computer generated imagery may be and the computer generated imagery may be combined to form an augmented scene. combined to form an augmented scene. – Direct viewing of the real world with overlaid Direct viewing of the real world with overlaid

computer generated imagery as an enhancement.In this computer generated imagery as an enhancement.In this case, the the real world and the CG images are case, the the real world and the CG images are combined optically. combined optically.

– Combining the camera-captured video of the real world Combining the camera-captured video of the real world with CG imagery viewed using either an opaque HMD, with CG imagery viewed using either an opaque HMD, or a screen-based display system.or a screen-based display system.


Two basic types of AR systemTwo basic types of AR system– Opaque HMD or screen-based AR.Opaque HMD or screen-based AR.

These systems can be used to view local or remote video views These systems can be used to view local or remote video views of real world scenes, combined with overlaid CG.The viewing of real world scenes, combined with overlaid CG.The viewing of a remote scene is an integral component of telepresence of a remote scene is an integral component of telepresence applications.applications.

– Transparent HMD AR. Transparent HMD AR. This system allows the observer to view the real world directly This system allows the observer to view the real world directly

using half-silvered mirrors with CG electronically composited using half-silvered mirrors with CG electronically composited into the image. An advantage id that the real-world can be into the image. An advantage id that the real-world can be directly viewed and manipulated. directly viewed and manipulated.

Video KeyingVideo Keying

Relevant when an opaque HMD with video Relevant when an opaque HMD with video input is used to create an AR scene. Video input is used to create an AR scene. Video and synthetic image are mixed using a and synthetic image are mixed using a video keyer to form an integrated scene. video keyer to form an integrated scene.

Video Keying is a process that is widely Video Keying is a process that is widely used in television, film production and CG. used in television, film production and CG. (weather report)(weather report)


When using video keying to design AR When using video keying to design AR scenes, one signal contains the foreground scenes, one signal contains the foreground image and the other one contains the image and the other one contains the background image. The keyer combines the background image. The keyer combines the two signal to produce a combined video two signal to produce a combined video which is then sent to the display device. which is then sent to the display device.


Keying can be done using composite or Keying can be done using composite or component video signals.component video signals.– A composite video signal contains information A composite video signal contains information

about color, luminance, and synchronization, about color, luminance, and synchronization, thus combining three piece of information into thus combining three piece of information into one signal. one signal.

– With component video, luminance With component video, luminance synchronization are combined, but chroma synchronization are combined, but chroma information is delivered separately. information is delivered separately.


Chroma keying involves specifying a desired Chroma keying involves specifying a desired foreground key color. Foreground areas foreground key color. Foreground areas containing the keying color are then electronically containing the keying color are then electronically replaced with the background image. This results replaced with the background image. This results in the background image being replaced with the in the background image being replaced with the fore ground image in areas where the background fore ground image in areas where the background image contains chroma color.image contains chroma color.

Blue is typically used for chroma keying Blue is typically used for chroma keying (Chromakey blue) rarely shows up in human skin (Chromakey blue) rarely shows up in human skin tones.tones.


If a video image of the real world is chosen If a video image of the real world is chosen as the foreground image, parts of the scene as the foreground image, parts of the scene that should show the computer-generated that should show the computer-generated world are rendered blue.world are rendered blue.

In contrast, if video of the real world is In contrast, if video of the real world is chosen as the background image, the chosen as the background image, the computer generated environment will be computer generated environment will be located in the foreground.located in the foreground.


A luminance keyer works in a similar manner to a A luminance keyer works in a similar manner to a chroma keyer, however, a luminance keyer chroma keyer, however, a luminance keyer combines the background image wherever the combines the background image wherever the luminance values are below a certain threshold.luminance values are below a certain threshold.

Luminance and chroma keyers both accomplish Luminance and chroma keyers both accomplish the same function but usa of a chroma keyer can the same function but usa of a chroma keyer can result in a sharper key and has greater flexibility, result in a sharper key and has greater flexibility, whereas a luminance keyer is typically lower whereas a luminance keyer is typically lower resolution and had less flexibility. resolution and had less flexibility.

Z-keyingZ-keying

Z-keyingZ-keying

Figure is a schema of the z-key method. The z-key Figure is a schema of the z-key method. The z-key method requires images with both depth information method requires images with both depth information (depth map) as inputs. The z-key switch compares (depth map) as inputs. The z-key switch compares depth information of two images for each pixel, and depth information of two images for each pixel, and connects output to the image which is the nearer one connects output to the image which is the nearer one to the camera. The result of this is that real and to the camera. The result of this is that real and virtual objects can occlude each other correctly. This virtual objects can occlude each other correctly. This kind of merging is impossible by the chroma-key kind of merging is impossible by the chroma-key method, even if it is accompanied with some other method, even if it is accompanied with some other positioning devices such as magnetic or acoustic positioning devices such as magnetic or acoustic sensor, since these devices provide only a gross sensor, since these devices provide only a gross measurement of position. measurement of position.

Image RegistrationImage Registration

It’s required that the computer generated images It’s required that the computer generated images accurately register with the surroundings in the accurately register with the surroundings in the real world. In certain applications, image real world. In certain applications, image registration is crucial.registration is crucial.

In terms of developing scenes for AR displays, the In terms of developing scenes for AR displays, the problem of image registration, or positioning of problem of image registration, or positioning of the synthetic objects within the scene in relation to the synthetic objects within the scene in relation to real objects, is both a difficult and important real objects, is both a difficult and important technical problem to solve.technical problem to solve.

Image RegistrationImage Registration

With applications that require close With applications that require close registration, accurate depth information has registration, accurate depth information has to be retrieved from the real world in order to be retrieved from the real world in order to carry out the calibration of the real and to carry out the calibration of the real and synthetic environments. Without an synthetic environments. Without an accurate knowledge of the geometry of the accurate knowledge of the geometry of the real world and computer-generated scene, real world and computer-generated scene, exact registration is not possible.exact registration is not possible.

System Design IssuesSystem Design Issues

Frame rate, update rate, system delays, and the Frame rate, update rate, system delays, and the range and sensitivity of the tracking sensors.range and sensitivity of the tracking sensors.

Frame rate is a hardware-controlled variable Frame rate is a hardware-controlled variable determining the number of images presented to the determining the number of images presented to the eye per second. AR displays which show stereo eye per second. AR displays which show stereo images alternatively to the left and right eye images alternatively to the left and right eye typically use a scan rate doubler to transmit 120 typically use a scan rate doubler to transmit 120 frames per second so that each eye has an effective frames per second so that each eye has an effective frame rate of 60 Hz.frame rate of 60 Hz.

Update rate of the display is the rate at which new images Update rate of the display is the rate at which new images are presented to the viewer. are presented to the viewer.

With a low update rate, if the user using an AR display With a low update rate, if the user using an AR display moves his head, the real and computer-generated images moves his head, the real and computer-generated images will no longer be registered until the next update. Small will no longer be registered until the next update. Small errors in registration are easily detectable by the visual errors in registration are easily detectable by the visual system. system.

What limits the update rate is the relationship between the What limits the update rate is the relationship between the complexity of the scene and the computational power of complexity of the scene and the computational power of the computer system used to generate the scene. This the computer system used to generate the scene. This relationship is esp. important for computationally intensive relationship is esp. important for computationally intensive applications such as medical imaging. applications such as medical imaging.



The lag in image generation and tracking is noticeable The lag in image generation and tracking is noticeable in all HMDs but is dramatically accentuated with see-in all HMDs but is dramatically accentuated with see-through HMDs. This is an crucial problem if exact through HMDs. This is an crucial problem if exact image registration is required. image registration is required.

There are two types of system delays which will affect There are two types of system delays which will affect performance in AR: computational and sensor delays.performance in AR: computational and sensor delays.– As the complexity of the CG image increases, the computational delay As the complexity of the CG image increases, the computational delay

is a major factor determining the update of a display. is a major factor determining the update of a display. – In addition, sensor delay, the time requires updating the display, is an In addition, sensor delay, the time requires updating the display, is an

important variable in determining performance in augmented reality. important variable in determining performance in augmented reality. – Many HMD-based systems have combined latencies over 100ms, which Many HMD-based systems have combined latencies over 100ms, which

become very noticeable. become very noticeable.


Sensor sensitivitySensor sensitivity– The head-tracking requirements for AR displays.The head-tracking requirements for AR displays.

– A tracker must be accurate to a small fraction of a A tracker must be accurate to a small fraction of a degree in orientation and a few millimeters in position. degree in orientation and a few millimeters in position.

– Errors in head orientation(pitch, roll, yaw) affect image Errors in head orientation(pitch, roll, yaw) affect image registration more so than error in position(x, y, z), registration more so than error in position(x, y, z), leading to the more stringent requirements for head-leading to the more stringent requirements for head-orientation tracking. orientation tracking.

– Positional tracking errors of no more than 1 to 2 mm Positional tracking errors of no more than 1 to 2 mm are maximum for AR system.are maximum for AR system.

In addition to visual factors, cognitive factors should be In addition to visual factors, cognitive factors should be considered in the design as well.considered in the design as well.

Users of systems form mental models of the system Users of systems form mental models of the system they interact with and the mental model they form they interact with and the mental model they form influence their performance. influence their performance.

With AR displays the designer must take into account With AR displays the designer must take into account two mental models of the environment, the mental two mental models of the environment, the mental model of the synthetic imagery and of the real image.model of the synthetic imagery and of the real image.

The challenge will be to integrate the two stimuli in The challenge will be to integrate the two stimuli in such a way that a single mental model will be formed of such a way that a single mental model will be formed of the augmented scene.the augmented scene.

System Design IssuesSystem Design Issues System Design IssuesSystem Design Issues


Integrated Mental Model

Mental Model of real envoronment

Mental Model of synthetic envoronment

Virtual world stimuli

Auditory, haptic, visual

Real world stimuli

Auditory, haptic, visual

Augmented-Reality ApplicationAugmented-Reality Application

MedicalMedical EntertainmentEntertainment Military TrainingMilitary Training Engineering DesignEngineering Design Robotics and TeleroboticsRobotics and Telerobotics Manufacture, Maintenance and RepairManufacture, Maintenance and Repair Consumer DesignConsumer Design


MedicalMedicalMost of the medical applications deal with image guided surgery. Most of the medical applications deal with image guided surgery.

Pre-operative imaging studies, such as CT or MRI scans, of the Pre-operative imaging studies, such as CT or MRI scans, of the patient provide the surgeon with the necessary view of the internal patient provide the surgeon with the necessary view of the internal anatomy. From these images the surgery is planned. Visualization anatomy. From these images the surgery is planned. Visualization of the path through the anatomy to the affected area where, for of the path through the anatomy to the affected area where, for example, a tumor must be removed is done by first creating a 3D example, a tumor must be removed is done by first creating a 3D model from the multiple views and slices in the preoperative model from the multiple views and slices in the preoperative study. AR can be applied so that the surgical team can see the CT study. AR can be applied so that the surgical team can see the CT or MRI data correctly registered on the patient in the operation or MRI data correctly registered on the patient in the operation theater while the procedure is progressing. Being able to correctly theater while the procedure is progressing. Being able to correctly register the images at the point will enhance the performance of register the images at the point will enhance the performance of the surgical team and eliminate the need for the painful and the surgical team and eliminate the need for the painful and cumbersome stereotactic frames currently used for registration.cumbersome stereotactic frames currently used for registration.


EntertainmentEntertainment– Weather reportWeather report– Virtual studioVirtual studio– Movie special effectMovie special effect– AdvertisementAdvertisement


Military TrainingMilitary TrainingThe military has been using display in cockpits that The military has been using display in cockpits that

present information to the pilot on the windshield present information to the pilot on the windshield of the cockpit or the visor of their flight helmet. of the cockpit or the visor of their flight helmet. This is a form fo AR display. This is a form fo AR display.


Engineering DesignEngineering DesignDistributed CoollaberationDistributed Coollaberation

Product visualizatoinProduct visualizatoin

The scenario for this application consists of an office manager who is working with an interior designer on the layout of a room. The office manager intends to order furniture for the room. On a computer monitor the pair see a picture of the room from the viewpoint of the camera. By interacting with various manufacturers over a network, they select furniture by querying databases using a graphical paradigm. The system provides descriptions and pictures of furniture that is available from the various manufactures who have made models available in their databases. Pieces or groups of furniture that meet certain requirements such as colour, manufacturer, or price may be requested. The users choose pieces from this "electronic catalogue" and 3D renderings of this furniture appear on the monitor along with the view of the room. The furniture is positioned using a 3D mouse. Furniture can be deleted, added, and rearranged until the users are satisfied with the result; they view these pieces on the monitor as they would appear in the actual room. As they move the camera they can see the furnished

room from different points of view.


Robotics and TeleroboticsRobotics and Telerobotics


Manufacturing, Manufacturing, Maintenance, and Maintenance, and RepairRepair

One application area that is currently being explored involves One application area that is currently being explored involves mechanical maintenance and repair. In this scenario a mechanic mechanical maintenance and repair. In this scenario a mechanic is assisted by an AR system while examining and repairing a is assisted by an AR system while examining and repairing a complex engine. The system may present a variety of complex engine. The system may present a variety of information to the mechanic. Annotations may identify the information to the mechanic. Annotations may identify the name of parts, describe their function, or present other name of parts, describe their function, or present other important information like maintenance or manufacturing important information like maintenance or manufacturing records. AR may lead the mechanic through a specific task by records. AR may lead the mechanic through a specific task by highlighting parts that must be sequentially removed and highlighting parts that must be sequentially removed and showing the path of extraction. The system may also provide showing the path of extraction. The system may also provide safety information. Parts that are hot or electrified can be safety information. Parts that are hot or electrified can be highlighted to constantly remind the mechanic of the danger of highlighted to constantly remind the mechanic of the danger of touching them. The mechanic may also be assisted by a remote touching them. The mechanic may also be assisted by a remote expert who can control what information is displayed on the expert who can control what information is displayed on the

mechanic's AR system.mechanic's AR system.


Consumer DesignConsumer DesignHouse DesignHouse Design

Fashion, beauty industryFashion, beauty industry

……..

ReferenceReference

http://www.cs.rit.edu/~jrv/research/ar/http://www.cs.rit.edu/~jrv/research/ar/ Virtual Environments and Advanced Virtual Environments and Advanced

Interface Design, edited by Woodrow Interface Design, edited by Woodrow Barfield, Thomas A.Furness IIIBarfield, Thomas A.Furness III

Augmented Reality Sites - North America Augmented Reality Sites - North America MIT MIT Image Guided Surgery home pageImage Guided Surgery home page Intelligent Room projectIntelligent Room project J P Mellor's home pageJ P Mellor's home page Media Lab Wearable Computer pageMedia Lab Wearable Computer page CMU CMU Z-Key projectZ-Key project Magic Eye projectMagic Eye project Columbia University Columbia University Virtual Worlds researchVirtual Worlds research Architectural AnatomyArchitectural Anatomy University of North Carolina - Chapel Hill University of North Carolina - Chapel Hill Ultrasound Visualization ResearchUltrasound Visualization Research Hybrid Tracking ResearchHybrid Tracking Research Latency in Augmented RealityLatency in Augmented Reality Ronald Azuma's Augmented Reality pageRonald Azuma's Augmented Reality page TelepresenceTelepresence Research Group Research Group Rich Holloway's Home PageRich Holloway's Home Page USC Computer Graphics andUSC Computer Graphics and Immersive Immersive Technologies Laboratory Technologies Laboratory University of Washington Human Interface Technology Lab (HITL)University of Washington Human Interface Technology Lab (HITL) Colorado School of MinesColorado School of Mines Hazardous waste management Bozidar Stojadinovi, Virtual Reality Lab, University of Michigan Hazardous waste management Bozidar Stojadinovi, Virtual Reality Lab, University of Michigan Augmented Reality work at the University of Toronto Augmented Reality work at the University of Toronto Argonne National Labs - Michael E. Jebb's Augmented Reality page Argonne National Labs - Michael E. Jebb's Augmented Reality page NIST description of the Boeing project NIST description of the Boeing project Colorado State Univ. - Michael L. Croswell's Augmented Reality page Colorado State Univ. - Michael L. Croswell's Augmented Reality page Ross Whitaker's Augmented Reality page Ross Whitaker's Augmented Reality page Mihran Tuceryan's Augmented Reality page Mihran Tuceryan's Augmented Reality page The WorldBoard Project The WorldBoard Project Vision-based Augmented Reality for Guiding Assembly Rajeev Sharma, Jose Molineros, University of Illinois at Urbana-Champaign Vision-based Augmented Reality for Guiding Assembly Rajeev Sharma, Jose Molineros, University of Illinois at Urbana-Champaign Alexander Chislenko's Intelligent Information Filters and Enhanced Reality Page Alexander Chislenko's Intelligent Information Filters and Enhanced Reality Page Microvision's Virtual Retinal Display Microvision's Virtual Retinal Display

Augmented-Reality

Documents

Transcript of Augmented-Reality