Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

100
Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment

Transcript of Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Page 1: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Tools for Edutainment

Sensory Interfaces, Augmented Reality and Edutainment 

Page 2: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

BackgroundWu Pin Fong

Virtual Reality Consulting company, Xtensory Inc, California, America

Total 25 years industrial experienceDone virtual reality a long time – 14 years

Emphasis mixed reality support for industry

Page 3: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Virtual Reality Background

Page 4: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Started

Worked at Digital Equipment (now part of Hewlett Packard) 1990

Start the virtual reality program at Digital

First major US company to support virtual reality

Page 5: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Undocumented VR History Working for a major american

computer company

Able to work with all the key VR developers

VR small community then

Page 6: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Relationships

Jaron Lanier, VPL ResearchScott Fisher, NASADr Tom Furness, HITL Labs, University

of WashingtonCarolina Cruz , CAVE inventor, Iowa

State UniversityRudy Darken, Naval Post Graduate

School

Page 7: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Others William Bricken

Autodesk HITL Labs

Foundation for WorldToolKit

Device manufacturers Ascension, Polhemius, Intersense, Immersion, etc

Page 8: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Jaron Lanier VPL Research Taught me Virtual Reality

Worked on VR project For SIGGRAPH 1990

Page 9: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

VPL Research in 1990 Very exciting

New technology

Mark Bolas (later to start FakeSpace)

Page 10: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Tom Furness Director, HITL Labs Came from Aerospace

Medical Research Laboratory (USAF), Wright-Patterson AFB, OH.

Helped get funding from Digital Equipemt for HITL Labs

Page 11: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Electronic Visualization Lab - CAVE The CAVE is a

projection based virtual reality system developed at the Electronic Visualization Lab created by Carolina Cruz-Neira

Page 12: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Naval Post Graduate School Rudy Darken, Director MOVES Institute,

Monterey Post Graduate School

Edutainment game - America's Army

Page 13: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.
Page 14: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.
Page 15: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.
Page 16: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Some ProjectsNASAAmerican army, navyWright patterson air force base, OHFAABank of AmericaMayo ClinicToshibaHITL labs, University of WashingtonHasbroLockheed SchlumbergerMonterey naval schoolHeitmanCompaq

Page 17: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

OverView Need make it easy to develop

mixed reality games

Provide Standard components

Page 18: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Why components Many mixed reality games involve people

who are not virtual reality experts

Most edutainment projects assume completely new software Control of actual real data Expert programmers

Real world need interface with legacy software

Page 19: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Edutainment games

Out of research into actual industrial use

Page 20: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Today Discuss one standard component -

device support Example of its use Special device support – Haptics A haptics example Component for building virtual cities

for use in mixed reality games

Page 21: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Device Support Component

Page 22: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

XvsLink Standard device component for device support

Paper - A Component-based Sensor Architecture for CAVEs (CAVE Automated Virtual Environments), Proceedings of Immersive Projection Technology Workshop, Iowa State University, 2000.

Page 23: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Industry tested since 1996

Not research Used in actual products

Toshiba University of Pennsylvania –virtual

human Jack Mayo Clinic

Page 24: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Standard Device Interface

Importance of components

Example of components – standardised device interfaces

Page 25: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

XvsLink Purpose Instrument existing legacy software

Linux, Windows, Sun Solaris, HP, SGI

Easy to use VR devices

Interfaces to non VR devices

Page 26: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Work Closely with Device Manufacturers So closely actually referenced in

their manuals Example Intersense IS600 manual

Page 27: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Focused - Does not provide user interaction techniques

scene graph (or transform tree)

graphics techniques

Page 28: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Provides handle the device control layer

provide communication between hosts

hook to higher levels of an edutainment system

Page 29: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

XvsLink Functional Diagram

Device supportSession capture &

playback

Network supportUsers

Developersadministrators

Virtual vehicle

Legacy software

User interaction analysis

Page 30: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

One industrial Difference Need to monitor user interaction

with the game

Capture user data for later playback and analysis

Page 31: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Layered Architecture

Application

device interface transport

XvsLink

User interaction tools

Legacy software

Page 32: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Component Device Interface Provide Easy Access

Common Interface to different devices

Access to VR peripherals from any application

Storage and replay of device data

Page 33: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

XvsLink

XvsLink is a C++ class library with both a C and C++ interface that provides a standard device interface for integrating devices into simulator applications.

Page 34: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Make devices invisible to user Writing a driver for a new device

usually a few days

Special commands for specific device drivers

Windows and linux kernel level drivers

Page 35: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Operation Open networked, serial, USB, shared

memory connections

Invisible to user Generic devices by functions

connection automatically restored if the device server is shut down and restarted

Page 36: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Design

generic devices polled or queued local or remote actual or virtual devices identified by logical name

Page 37: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Design

polled or queued- application can specify

polled reads data asynchronously typical example: tracker, valuator

queued circular typical example: keyboard, button

Page 38: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Local and Remote devices remote devices treated as local

devices

Local client and server

queries for device capabilities

Page 39: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Design

local or remote device application unaware of source

can use actual or virtual data Device data from hardware or

calculations

Page 40: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Device Mapping

interfaces specify a set of capabilities

device is mapped onto one or more interfaces

Client connects to the different interfaces

run-time bindings

Page 41: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Design

identified by logical name

Device can map to any name

new names can be created as needed

Page 42: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Extensible

Generic Type devices

Layered devices

Multiple-behavior devices

Page 43: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Design

generic devices locator valuator button

Page 44: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Device added dynamically

dynamic device support

examples Intersense tracker Fakespace wand, pinchGlove Phantom haptic

Page 45: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Support

Linux, windows, SUN, HP, SGI

hooks to other tools, packages

Hooks for legacy systems

Page 46: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Example of XvsLink’s Use

Page 47: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Virtual Collaborative Clinic

Page 48: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

NASA – Virtual Collaborative Clinic

1998

Dr. Muriel Ross, head of NASA's biometrics research at Ames

Out of research into deployment usually no news

Page 49: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Virtual Collaborative clinic

Page 50: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Collaboration Partners

Page 51: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

ObjectivesMedical support in remote locations

surgical try-outs in mixed reality after which the most adequate procedure can be stored to be used in the actual operation.

project computer images onto the patient in order to guide physicians during surgical interventions.

Page 52: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Special device support – Haptics

Page 53: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Haptic Rendering of Virtual Stimuli

Page 54: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Haptic

Page 55: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Haptic Information Haptic information is a form of sensory

stimulation that is acquired through collisions with objects

Page 56: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Using the Haptic Sensory Channel

Force, vibration, texture object shape, compliance, impact,

contact, sliding, slipping, torque and kinematic constraint.

encoding the feel and movement of real or virtual objects during manipulation or exploration

Page 57: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Haptic Exploratory Procedures

Page 58: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Human Sensing

Page 59: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Force Feedback

Page 60: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Force Feedback

Page 61: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Rutgers

Page 62: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Haptics

Page 63: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Example

Page 64: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Xtensory’s Tactile Controller

Page 65: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Shape Memory Alloy (SMA)

a system of long stroke vertical pin-rods, using a coil-form Shape Memory Alloy (SMA) as an actuator.

This material can be stretched or deformed from its original shape but would spontaneously return to its original shape when heated.

Page 66: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Tactor Sensor

Page 67: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Tactor Array

a 16 by 16 array of pin-rods

high-speed update rates

Page 68: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Tactor Array in Action

Page 69: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Operation

orchestrating the vertical motion of a dense array of pin-rods. The effect is similar to a children's "Pop up book", where a flat surface transforms into a 3D form.

the driving method is scalable so that tactor arrays of any size can be constructed

Page 70: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

A Haptics Example

Page 71: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Call Options Game

Done for major american bank to train its traders

Conceptual architecture and framework for mixing real and virtual data

Added haptically rendering virtual stimuli

Page 72: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.
Page 73: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.
Page 74: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

The Game - Call Options The options portfolio consist of options with

3 different times to expiration. The gamma value of an option is

increasing dramatically just before expiration if the option is about at-the-money.

Local peaks or valleys therefore occur for each strike, near to maturity . (Long positions give positive gamma and short positions give negative gamma).

Page 75: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Using Haptics Peaks and valleys shown visually

rendered haptically Mix real and virtual data

Page 76: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

In Operation

Page 77: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Another Haptics Example

Page 78: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Enclosed Hazardous Spaces

Page 79: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Narrow field of View

Page 80: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Constrained Space Need sense of touch to move haptic orienting cues play a critical

role

Page 81: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Inspection Game Mix real object – containment vessel

Virtual hazards

Tactile sensing

Training

Page 82: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Virtual City Support Component

Page 83: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Why Many mixed reality games will use

actual city buildings, streets, locations

Need make it easy to mix real and virtual for actual cities

Military – urban operations

Page 84: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Urban Operations In the last 10 years, however, the

U.S. has committed military forces to far-flung urban operations in Panama, Somalia, Liberia, Bosnia and now Kosovo.

The Marine Corps alone has been involved in urban operations in 237 of its last 250 deployments.

Page 85: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Urban Operations Chracterised by extremely short firing

ranges imposed by a city's long canyons of high-rise buildings and dense thickets of trackless housing and teeming alleys

Need for local infantry autonomy

No command and control capability

Page 86: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Intelligence & Reconnaissance

TelegraphOptics

Motion Imagery, Hyperspectral

Radio, ComputerImageAircraft, Camera

Horse,Drums,Smoke, Mirrors, etc…

WrittenEyes, Ears

FeetMouthEyes, Ears

TransmissionReportCollection

Page 87: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Requirements Quick and easy way to build virtual cities

for training games

Acuisition of motion imagery from camera phones, UAVs

Build virtual cities from analysis of motion imagery

Time Frame - Late 1990’s

Page 88: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.
Page 89: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Xtensory Contracts Enhanced Multiplatform, Integrated Virtual Urban

Warfare Simulation. Navy SBIR N99-063.

Soldier-Computer Interface for Enhanced Interaction and Communication. Army SBIR A95-035

Immersive Visualization of Complex Situations for Mission Rehearsal. Army SBIR A95-089.

Page 90: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Component Motion Imagery analysis (acquired

from another company)

Polygon processing into low polygon count models, autocad, 3DS file formats

Good way to build virtual cities

Page 91: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.

Conclusion

Shown how use of component tools help in making industrial strength edutainment games

Page 92: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.
Page 93: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.
Page 94: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.
Page 95: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.
Page 96: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.
Page 97: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.
Page 98: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.
Page 99: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.
Page 100: Tools for Edutainment Sensory Interfaces, Augmented Reality and Edutainment.