Tools for Edutainment

Sensory Interfaces, Augmented Reality and Edutainment 

BackgroundWu Pin Fong

Virtual Reality Consulting company, Xtensory Inc, California, America

Total 25 years industrial experienceDone virtual reality a long time – 14 years

Emphasis mixed reality support for industry

Virtual Reality Background

Started

Worked at Digital Equipment (now part of Hewlett Packard) 1990

Start the virtual reality program at Digital

First major US company to support virtual reality

Undocumented VR History Working for a major american

computer company

Able to work with all the key VR developers

VR small community then

Relationships

Jaron Lanier, VPL ResearchScott Fisher, NASADr Tom Furness, HITL Labs, University

of WashingtonCarolina Cruz , CAVE inventor, Iowa

State UniversityRudy Darken, Naval Post Graduate

School

Others William Bricken

Autodesk HITL Labs

Foundation for WorldToolKit

Device manufacturers Ascension, Polhemius, Intersense, Immersion, etc

Jaron Lanier VPL Research Taught me Virtual Reality

Worked on VR project For SIGGRAPH 1990

VPL Research in 1990 Very exciting

New technology

Mark Bolas (later to start FakeSpace)

Tom Furness Director, HITL Labs Came from Aerospace

Medical Research Laboratory (USAF), Wright-Patterson AFB, OH.

Helped get funding from Digital Equipemt for HITL Labs

Electronic Visualization Lab - CAVE The CAVE is a

projection based virtual reality system developed at the Electronic Visualization Lab created by Carolina Cruz-Neira

Naval Post Graduate School Rudy Darken, Director MOVES Institute,

Monterey Post Graduate School

Edutainment game - America's Army

Some ProjectsNASAAmerican army, navyWright patterson air force base, OHFAABank of AmericaMayo ClinicToshibaHITL labs, University of WashingtonHasbroLockheed SchlumbergerMonterey naval schoolHeitmanCompaq

OverView Need make it easy to develop

mixed reality games

Provide Standard components

Why components Many mixed reality games involve people

who are not virtual reality experts

Most edutainment projects assume completely new software Control of actual real data Expert programmers

Real world need interface with legacy software

Edutainment games

Out of research into actual industrial use

Today Discuss one standard component -

device support Example of its use Special device support – Haptics A haptics example Component for building virtual cities

for use in mixed reality games

Device Support Component

XvsLink Standard device component for device support

Paper - A Component-based Sensor Architecture for CAVEs (CAVE Automated Virtual Environments), Proceedings of Immersive Projection Technology Workshop, Iowa State University, 2000.

Industry tested since 1996

Not research Used in actual products

Toshiba University of Pennsylvania –virtual

human Jack Mayo Clinic

Standard Device Interface

Importance of components

Example of components – standardised device interfaces

XvsLink Purpose Instrument existing legacy software

Linux, Windows, Sun Solaris, HP, SGI

Easy to use VR devices

Interfaces to non VR devices

Work Closely with Device Manufacturers So closely actually referenced in

their manuals Example Intersense IS600 manual

Focused - Does not provide user interaction techniques

scene graph (or transform tree)

graphics techniques

Provides handle the device control layer

provide communication between hosts

hook to higher levels of an edutainment system

XvsLink Functional Diagram

Device supportSession capture &

playback

Network supportUsers

Developersadministrators

Virtual vehicle

Legacy software

User interaction analysis

One industrial Difference Need to monitor user interaction

with the game

Capture user data for later playback and analysis

Layered Architecture

Application

device interface transport

XvsLink

User interaction tools

Legacy software

Component Device Interface Provide Easy Access

Common Interface to different devices

Access to VR peripherals from any application

Storage and replay of device data

XvsLink

XvsLink is a C++ class library with both a C and C++ interface that provides a standard device interface for integrating devices into simulator applications.

Make devices invisible to user Writing a driver for a new device

usually a few days

Special commands for specific device drivers

Windows and linux kernel level drivers

Operation Open networked, serial, USB, shared

memory connections

Invisible to user Generic devices by functions

connection automatically restored if the device server is shut down and restarted

Design

generic devices polled or queued local or remote actual or virtual devices identified by logical name

Design

polled or queued- application can specify

polled reads data asynchronously typical example: tracker, valuator

queued circular typical example: keyboard, button

Local and Remote devices remote devices treated as local

devices

Local client and server

queries for device capabilities

Design

local or remote device application unaware of source

can use actual or virtual data Device data from hardware or

calculations

Device Mapping

interfaces specify a set of capabilities

device is mapped onto one or more interfaces

Client connects to the different interfaces

run-time bindings

Design

identified by logical name

Device can map to any name

new names can be created as needed

Extensible

Generic Type devices

Layered devices

Multiple-behavior devices

Design

generic devices locator valuator button

Device added dynamically

dynamic device support

examples Intersense tracker Fakespace wand, pinchGlove Phantom haptic

Support

Linux, windows, SUN, HP, SGI

hooks to other tools, packages

Hooks for legacy systems

Example of XvsLink’s Use

Virtual Collaborative Clinic

NASA – Virtual Collaborative Clinic

1998

Dr. Muriel Ross, head of NASA's biometrics research at Ames

Out of research into deployment usually no news

Virtual Collaborative clinic

Collaboration Partners

ObjectivesMedical support in remote locations

surgical try-outs in mixed reality after which the most adequate procedure can be stored to be used in the actual operation.

project computer images onto the patient in order to guide physicians during surgical interventions.

Special device support – Haptics

Haptic Rendering of Virtual Stimuli

Haptic

Haptic Information Haptic information is a form of sensory

stimulation that is acquired through collisions with objects

Using the Haptic Sensory Channel

Force, vibration, texture object shape, compliance, impact,

contact, sliding, slipping, torque and kinematic constraint.

encoding the feel and movement of real or virtual objects during manipulation or exploration

Haptic Exploratory Procedures

Human Sensing

Force Feedback

Force Feedback

Rutgers

Haptics

Example

Xtensory’s Tactile Controller

Shape Memory Alloy (SMA)

a system of long stroke vertical pin-rods, using a coil-form Shape Memory Alloy (SMA) as an actuator.

This material can be stretched or deformed from its original shape but would spontaneously return to its original shape when heated.

Tactor Sensor

Tactor Array

a 16 by 16 array of pin-rods

high-speed update rates

Tactor Array in Action

Operation

orchestrating the vertical motion of a dense array of pin-rods. The effect is similar to a children's "Pop up book", where a flat surface transforms into a 3D form.

the driving method is scalable so that tactor arrays of any size can be constructed

A Haptics Example

Call Options Game

Done for major american bank to train its traders

Conceptual architecture and framework for mixing real and virtual data

Added haptically rendering virtual stimuli

The Game - Call Options The options portfolio consist of options with

3 different times to expiration. The gamma value of an option is

increasing dramatically just before expiration if the option is about at-the-money.

Local peaks or valleys therefore occur for each strike, near to maturity . (Long positions give positive gamma and short positions give negative gamma).

Using Haptics Peaks and valleys shown visually

rendered haptically Mix real and virtual data

In Operation

Another Haptics Example

Enclosed Hazardous Spaces

Narrow field of View

Constrained Space Need sense of touch to move haptic orienting cues play a critical

role

Inspection Game Mix real object – containment vessel

Virtual hazards

Tactile sensing

Training

Virtual City Support Component

Why Many mixed reality games will use

actual city buildings, streets, locations

Need make it easy to mix real and virtual for actual cities

Military – urban operations

Urban Operations In the last 10 years, however, the

U.S. has committed military forces to far-flung urban operations in Panama, Somalia, Liberia, Bosnia and now Kosovo.

The Marine Corps alone has been involved in urban operations in 237 of its last 250 deployments.

Urban Operations Chracterised by extremely short firing

ranges imposed by a city's long canyons of high-rise buildings and dense thickets of trackless housing and teeming alleys

Need for local infantry autonomy

No command and control capability

Intelligence & Reconnaissance

TelegraphOptics

Motion Imagery, Hyperspectral

Radio, ComputerImageAircraft, Camera

Horse,Drums,Smoke, Mirrors, etc…

WrittenEyes, Ears

FeetMouthEyes, Ears

TransmissionReportCollection

Requirements Quick and easy way to build virtual cities

for training games

Acuisition of motion imagery from camera phones, UAVs

Build virtual cities from analysis of motion imagery

Time Frame - Late 1990’s

Xtensory Contracts Enhanced Multiplatform, Integrated Virtual Urban

Warfare Simulation. Navy SBIR N99-063.

Soldier-Computer Interface for Enhanced Interaction and Communication. Army SBIR A95-035

Immersive Visualization of Complex Situations for Mission Rehearsal. Army SBIR A95-089.

Component Motion Imagery analysis (acquired

from another company)

Polygon processing into low polygon count models, autocad, 3DS file formats

Good way to build virtual cities

Conclusion

Shown how use of component tools help in making industrial strength edutainment games