Mixed Reality
description
Transcript of Mixed Reality
Mixed Reality
University of Central FloridaMedia Convergence Laboratory @ IST
School of Electrical Engineering & Computer ScienceDigital Media Department
Media Convergence Laboratory Using mixed reality to solve real problems
The faces of MR
Physical Reality (PR) – real world Virtual Reality (VR) – purely synthetic Augmented Reality (AR) – virtual assets
registered in real world Augmented Virtuality (AV) – real (people,
props) layered in virtual space
Mixed Reality Properties
Blends the real and the synthetic into a single landscape & experience
Addresses multiple senses Requires proper registration of real and
virtual, relative to each other Typically requires complex behaviors of
virtual characters Enables buy-in through imagination
Limitations along the continuum
PR is constrained by physical space VR limits person-to-person expression and
context of PR AR often limits escape from real world AV allows rich layering In general, we want to move smoothly
along the MR continuum
Visual capture
OpticalOptical
VideVideoo
HMDHMDss
Light Light capturecapture
3d laser3d lasermocapmocap
Visual rendering
Dome ScreensDome Screens
Flat WorldFlat World
Demo Demo DomeDome
MR MR WindowsWindows
HMDHMDss
Audio capture / rendering
Surround Surround HydrophonesHydrophones
HolophonHolophonee
MixingMixing
DesigningDesigning
Delivery in constrained Delivery in constrained settingssettings
Pipeline: Audio for VR/MR
Planning & Prescripting
Capturing Synthesizing, Mixing,
and Mastering Designing Sound &
Integrating Delivering
Special effects
- Colorkinetics SmartJack3 (USB to DMX)- Colorkinetics JuiceBox2 / iColor MR Lights- Gilderfluke MP3-50/40- 4 Channel Dimmer Packs- Pneumatic / Smoke System- Sound Transducers (”Bass Shakers”)
- Colorkinetics SmartJack3 (USB to DMX)- Colorkinetics JuiceBox2 / iColor MR Lights- Gilderfluke MP3-50/40- 4 Channel Dimmer Packs- Pneumatic / Smoke System- Sound Transducers (”Bass Shakers”)
Tracking
Technologies Magnetic Optical Acoustical Inertial Vision (often with markers/features)
Hybrid (hardware and soft/hardware)
What we currently use
Wide field of view: 51 degree in horizontal direction, 37 degree in vertical direction-high
resolution: VGA (640 × 480) weight: 286 g standard camera I/O: NTSC
Coastar (Co-Optical Axis for AR)
InterTrax2 Wireless Tracker
• IS 900 Wide Area Tracker
Registration / Illumination
Virtual and real must be properly placed relative to each other
Inter-occlusion must be properly managed Mutual shadowing must occur, including shadows
from real caused by virtual light The effects of ambient light (real and virtual) must
be rendered
Story
Virtual characters must have appropriate behaviors, reactive and proactive
Virtual characters often need to mimic those of real characters
Capture, replay must be provided for entertainment, after-action review, etc.
Putting it together
Content is king
Create experiences that have long-term impact Experiential movie trailers Situational awareness training Creative collaboration Cognitive and physical rehabilitation Teacher screening and training Free-choice learning …
Imagination is the most important dimension
Siggraph ‘03 – experiential trailer
Situational awareness
MR MINI-MOUTMR MINI-MOUT
Peer Collaboration
Along the Continuum
Start in PR (look at current plant) Move to AR (add new equipment and new
windows) Individual jumps out to VR to privately
review designs Move to AV as all are surrounded by new
design, but still see each other
What We Must Support
Jointly visualized “what if?” scenarios Face-to-face interactions Alternative and often opposite POVs Personal and group creation
Enablers Tangible and tactile components Constructive distractions (sandbox) Shared display (1st & 3rd person views) Shared interfaces Scalability, interoperability & portability
Cognitive rehabilitation
Patient and therapist in context of patient’s kitchen but in safety of clinic
patient at home
reconfigurable kitchen
as seen thru HMD
movement for after action
review
real items to use
tethered for now
Pre vs. Post
Milk Milk
Spoon
Cereal
Bowls
Make Cereal
Bowls
Cereal
SpoonMake Cereal
Pre-testing Home Post-testing Home
Stuttering
Teacher screening and training
Cost to hire a teacher is $15,000 Cost to remove can be much higher Turnover in inner city is 50% in 3 years Many last less than a year Children are unintended guinea pigs Want to select those who can succeed Want to help those who want to succeed
Front room – early prototype
Back room -- puppeteers
Interaction with a Virtual Class
In front of class
At desk of student
Personal contact
AI for all but student being directly addressed
Vision-based tracking
Free-choice learning
Free-choice learning
Water’s journey
Coming to a science center near you Initial focus is on the Everglades But, water is central to all ecologies Move clock backwards or forwards
Real blends to virtual User-chosen policies affect what is experienced
Forest fire visualizations
Fire spread model from Farsite
Tree rendering from SpeedTree
Fire and smoke visualization is ours
Used in environmental economics & Everglades museum projects
Some basic research
Blending of real and virtual in changing illumination
Overcoming HMD camera limitations Real-time video chroma keying in presence of
noise and inconsistent lighting LOD management (forests) Neoroevolution for virtual character behaviors Painting in HDR
Lab people characteristics
Original director from theater and theme parks Lab manager from museum industry Chief programmer from Lego Content delivery people from digital media,
children’s literature, audio, theater, museum, … Content providers from all over campus PhD students have interests in graphics,
neuroevolution, vision, ecology, software engineering, simulation & modeling
Undergraduates are mainly from CS and DM
Charles E. Hughes
Professor & Associate DirectorSchool of Electrical Engineering & Computer Science
ProfessorDigital Media
Director, Media Convergence Lab
E-Mail: [email protected] Page: http://www.cs.ucf.edu/~ceh/Graphics Lab: http://graphics.cs.ucf.eduMedia Convergence Lab: http://mcl.ucf.edu
Contact information
J. Michael Moshell
ProfessorDigital Media
ProfessorSchool of Electrical Engineering & Computer Science
Co-founder, Media Convergence LabHome Page:
http://www.cs.ucf.edu/~jmmoshell/Media Convergence Lab: http://mcl.ucf.edu
Contact information