Electronic Visualization Laboratory University of Illinois at Chicago

Interaction between Real and Virtual Humans:

Playing Checkers

R. Torre, S. Balcisoy P. Fua, M. Ponder, and D. Thalmann  Computer Graphics Lab (LIG)

Swiss Federal Institute of Technology

Atul Nayak

Electronic Visualization Laboratory University of Illinois at Chicago

Virtual Humans

• Guided– Represent the user

• Autonomous– Act on their own

Electronic Visualization Laboratory University of Illinois at Chicago

Virtual Worlds

Virtual humans live in synthetic worlds

Electronic Visualization Laboratory University of Illinois at Chicago

So what’s new?

• Make the virtual human interact with the user

• 2-way communication• Real-time• Non-invasive• Vision-based

recognition system

Electronic Visualization Laboratory University of Illinois at Chicago

Playing checkers….

Electronic Visualization Laboratory University of Illinois at Chicago

Real time tracking of 3D objects

• To track a 3D convex object (a cube, a checkers board) we need– A camera– A target– The wire frame 3D model of the target– A powerful computer

Electronic Visualization Laboratory University of Illinois at Chicago

Model Based Tracking

• Create a mathematical model of the target• World coordinates are a function of the state

variables that control the object’s shape• An objective function is minimized to find the best

fit with image data

Electronic Visualization Laboratory University of Illinois at Chicago

3-D tracking algorithm

• Get a video image• Process the image• X(t+1) = X(t) + V(t).dT

– X = position ; V = speed ; dT = time between t and t+1

• Minimization of the objective function

Electronic Visualization Laboratory University of Illinois at Chicago

Detecting the checkers

• If the sum of the intensities of each pixel in the real image of a white square is less than a certain threshold value , it means that a piece is present on that square.

Electronic Visualization Laboratory University of Illinois at Chicago

Virtual Human Director (VHD)

• A system used to produce real-time multimedia products and TV shows

• Focuses on 2 issues– Fully integrated virtual humans with facial and

body animation, and speech– A straightforward user interface for designers

and directors

Electronic Visualization Laboratory University of Illinois at Chicago

The VHD interface

• Virtual human animation and interaction capabilities integrated into one environment

• Control multiple actors• Provide tools to create any scenery involving realistic virtual humans

Electronic Visualization Laboratory University of Illinois at Chicago

The Augmented Reality System

• Synchronize the rendering of synthetic scenes with the real camera output

• VHD animates the virtual player and updates the scene – input from tracker and checker game controller

Electronic Visualization Laboratory University of Illinois at Chicago

Image composition

• Chroma-keying : black is the key color• All real objects represented in black• Table’s virtual representation masks the virtual human• Blend the real filmed scene and the synthetic 3D scene to get

the mixed image• Real player sees the mixed image on a large monitor

Electronic Visualization Laboratory University of Illinois at Chicago

Comments

• Very realistic virtual humans will be widely used in film, games and interactive television

• Integrate VR with the Web – create 3D scenes with multimedia objects

• Replace checkers board with any other calibration object

• No extra devices – you can see yourself in the virtual world

Electronic Visualization Laboratory University of Illinois at Chicago

More cool demos….

Electronic Visualization Laboratory University of Illinois at Chicago

More cool demos….

Electronic Visualization Laboratory University of Illinois at Chicago

References

• VHD: A System for Directing Real-Time Virtual Actors, G. Sannier, S. Balcisoy, N. Magnenat-Thalmann, D. Thalmann, The Visual Computer, Springer, Vol.15, No 7/8, 1999, pp.320-329

• All images and movies in this presentation have been taken from the LIG’s website URL : http://ligwww.epfl.ch/

Electronic Visualization Laboratory University of Illinois at Chicago

Comments?