Beyond VR: Mixed Reality with Tango and HoloLens

Cyrus Lum, CTO Ant Hive GamesAaron Pulkka, CEO Rabbx Inc

Beyond VR: Mixed Reality with Tango and HoloLens

• Background• Mixed Reality• Tango vs HoloLens• ReFlow• Future

Cyrus Lum

AR Experience

Aaron Pulkka

VR Experience

Mixed Reality

Augmented Reality

The VOID

The VOID – Augmented Virtuality

Mixed Reality

Spatially Aware Devices

• Powerful Android Device• Motion Tracking• Area Learning• Depth Sensing• Walk-around VR Ready

Best Entertainment

WINNERProject Tango App

Contest 2015

Story-Driven Physical Hidden-Object Game

Motion Tracking driven design

“In this mystery sleuthing title, you play an imprisoned spirit, trying to find peace by solving the mystery of your own death.”

“It takes place in a spooky 3D mansion, and the entire game is controlled by walking around holding Tango.”

Development Kit and Lenovo Phone

Tango Development Environment

SDK Options• C, Java, Unity

UX Framework Active Community

Tango Motion Tracking

• Visual-Inertial Odometry (VIO)• Tracking image differences• Inertial motion sensors• Combined for greater accuracy• Limits:• Drift• No memory• Lighting

Tango Area Learning

• Simultaneous Location and Mapping (SLAM)

• Location memory • Drift correction• Limits:• Mapping time• Areas change over time

Tango Depth Sensing

• Point Cloud • Infrared light• Markerless AR• Limits:• Environmental• 0.5 to 4 meters

Environmental Issues

• “I can’t see.” – too dark• “I am lost.” – featureless • “I am dizzy.” – too fast

Tetherless Walk-Around VR

Mixed Reality with Tango

Google I/O 2016

• Wearable Windows PC• Spatial Mapping• Motion Tracking• Gesture Tracking• Speech Recognition

Wearable Windows 10 PC

HoloLens Development Environment

Rest API

Infrared Camera Spatial Mapping

Motion and Gesture Tracking

Speech Recognition

• System level commands

• User configurable commands

Best Spatial Mapping

WINNERSFVR HoloLens

Hackathon 2016

Behemoth for HoloLens• Enabling epic cinematic

experiences everywhere!• Discover mysterious dangers

lurking beyond your walls• Find critical resources under

and around furniture• Conquer beast to win an epic

battle

Mixed Reality UX

• Forget 2D UI design!• Integrate 3D UI elements• Physical movement and gaze• Use 3D audio and “offscreen” hints• ReFlow elements to environment

Mixed Reality ReFlow

Each player’s gameplay environment is unique• Artificial Intelligence• MxR layout engine• Surface reconstruction• Planar identification

vs

• Phablet• Touchscreen • Android• Consumer• Infrared mapping• Visual tracking

• Wearable• Gestures• Windows• Enterprise• Infrared mapping• Visual tracking

FUTURE

Smart Glasses Adoption Forecast

ODG R-7 Smartglasses

2017?

Questions?

Cyrus Lumcyrus@anthivegames.com

Aaron Pulkkaaaron@rabbx.com