1

Virtual RealityOverview & Update

Philip LelyveldUSC Entertainment Technology Center

PLelyveld@ETCenter.org

Founded 1993 by George Lucas tobringtogethertechnologyandentertainmentvisionariestocollaborateonthefutureofentertainmenttechnology

Adamantlyneutral• Discussions• Collaborations

• Research• Coverage

PLelyveld@ETCenter.org 2

CURRENT MEMBER COMPANIES

3

CURRENT MEMBER COMPANIES

4

In Augmented Reality, the world that you would normally see if you walked down the aisle without the screen is being augmented – enhanced, supplemented, added to - with additional data and experiences.

In Virtual Reality you are virtually – nearly, almost – but not really there in the grocery store aisle.

PLelyveld@ETCenter.org 5

RealEnvironment

AugmentedReality

VirtualReality

6

TransitionalMixed Reality

Blended Reality

TheAR– Reality– VRContinuum

PLelyveld@ETCenter.org

DifferentwaystohaveaVRexperience

MagicWindow

Cellphone-based viewer

HeadMountedDisplay(HMD)

GroupVR

7PLelyveld@ETCenter.org

8

VirtualRealityonCellphoneorTablet“MagicWindow”

Near$0costofentry

Swipe

Tilt and wave

PLelyveld@ETCenter.org

9

Cellphone-BasedViewerwithHeadphones

Near$0costofentryPLelyveld@ETCenter.org

Oculus Rift with Oculus Touch10

HeadMountedDisplayswithPeripherals

PLelyveld@ETCenter.org

TetheredversusUntethered

• BandwidthIssue• Compression• Latency

Sitting,Standing,WalkingHeadMountedDisplayswithPeripherals

PLelyveld@ETCenter.org 11

12

GroupVRExperiencesLocationBasedEntertainment

PLelyveld@ETCenter.org

TheVoidopensTimesSquareVRCenterFirstattraction:“Ghostbusters:Dimensions”OpensJuly1,2016,developedincooperationwiththemovie

TheLanguageofVR:Part1

Existingtoolsandskillsarebeingreworkedtocreatesomethingverydifferent

13PLelyveld@ETCenter.org

PLelyveld@ETCenter.org 14

“Virtual Reality is not a media experience. When it is done well, it is an actual experience.” Professor Jeremy BailensonStanford Virtual Human Interaction Lab

VideoPLelyveld@ETCenter.org 15

16

Senseof‘Presence’

1.Moresensesaccurately engaged=increasedpresence

2.FidelityContract

PLelyveld@ETCenter.org

17

HeightenedSenseofEmpathy

HenryOculusStoryStudio

7/28/15PLelyveld@ETCenter.org

“Ifeelhorribleaboutmakingthis,”…“Youactuallyfeelguilty.MymouthdroppedthefirsttimeIshotsomeonein

my GTA:V VRsetup.”

GrandTheftAutovideo-gamedesignerJosephDelgado,uponexperiencingtheVRversionofGTA:V

5/13/16PLelyveld@ETCenter.org 18

19

HeightenedSenseofEmpathy

3D facial expressions evoke stronger emotionsthan their 2D counterparts

due to the illusion of non-mediation

PLelyveld@ETCenter.orgSource: 11/24/15 research from Aalto Univ. & Univ. of Helsinki

“Hunger”(VirtualFoodBank)byNonnydelaPena20

Agency

PLelyveld@ETCenter.org

“My Mother’s Wing”By VRSE for UN

AgencyImmersiveJournalism

PLelyveld@ETCenter.org 21

TheultimateVRexperiencewillbeastorythatknowsyouareinthestory.

MarkBolas

PLelyveld@ETCenter.org 22

You are in a snow stormYou must get warm in 5 minutes or you will die

Audio is a key ‘presence’ componentPlayers ‘feel’ colder as they playYour body moves slower as time goes on

Languagestory telling + engagement design elements

From “Storm” by Andrick BregmanPLelyveld@ETCenter.org 23

Tools

PLelyveld@ETCenter.org 24

NextVR3603DRig6RedPro6Kcameras60fpsSpatialaudiocapture

JauntJ1-24REarly201632K(16Kpereye)120fps 25

VRCameras

Captured 3D

PLelyveld@ETCenter.orgAlgorithmic 3D

Nokia OZO

PLelyveld@ETCenter.org 26

Captures 360 stereoscopic 3D through:- 1950 capture with 50% overlap for 3D- 8 synchronized global shutter sensors- 8 integrated mikes for spatial audioJaunt committed to support Ozo for camera & production workflow

Surround 360 by Facebook

PLelyveld@ETCenter.org 27

Not a product; an open source design for others to build on2048x2048 60 fps genlocked cameras – 14 around, 1 up, 2 down

Orah 4i live-streaming Prosumer VR camera from VideoStitch

PLelyveld@ETCenter.org 284fisheyelenses,4mikes,4Koutput,30secdelay,~$3,600

VRCameras(consumermkt.)

Bublcam from Bubl ($799)

Kodak SP360 (2X$290)

Ricoh Theta ($350)

Giroptic ($499)PLelyveld@ETCenter.org 29

Samsung 360Two 15MP cameras360-degree video in 3,840-by-1,920 resolutionGalaxy S7 and S7 Edge, Gear VR

Nikon KeyMission 360 ($?)Waterproof to 100’

30

Wild– TheExperience,VRshortfromFox(ReeseWitherspoon, LauraDern)

Sound

PLelyveld@ETCenter.org

BinauralAudio

3Dio

Ambisonic Audio

TetraMic

PLelyveld@ETCenter.org 31

32

Wild– TheExperience,VRshortfromFox(ReeseWitherspoon, LauraDern)

SoundAudiobreakdown• Livespatialaudiocapture• Miked audiowithposition-tracking• Audiowithnoposition(musicscore)

PLelyveld@ETCenter.org

33

Stitching

PLelyveld@ETCenter.org

Stitching

FromtalkbyAnrick BregmanPLelyveld@ETCenter.org 34

Stitching

StandardVRcamerarigscanhavebuilt-in stitchingsoftwareFromtalkbyAnrick Bregman

PLelyveld@ETCenter.org 35

Editing tools

Skyboxiscreatinga‘directtoRift’ playerfortheAdobesuitethathangsthemenuswithintheHMDspace

PLelyveld@ETCenter.org 36

37

UserInterface

Tomaintainpresence,UIfornavigationdesign andinformationcommunicationmustbeconsistentwiththerestoftheexperience

PLelyveld@ETCenter.org

GameEngines

DigitalDistributionofVR

• YouTube360• AppleTV360• Hulu360• Amazon360• Netflix360

• Littlstar• Vrideo• WearVR• ….

PLelyveld@ETCenter.org 38

PLelyveld@ETCenter.org 39WearVR’s new home page 4/16/16

40

Technical ConsiderationsBandwidth and QoSCompressionLatencyProcessing powerLocal buffer/storage …

Option: Distribute a 4K frame, then client warps it into a sphere.

Capture at high-resolution, distribute at what client-processing, bandwidth, and display tech allow.

Google Fiber expressed interest in streaming VR

PLelyveld@ETCenter.org

HeadMountedDisplayTechnicalTargets

4K-8Kresolutionpereye

>90framespersecondrendering (vscapture)

<20msec latency

2200 fieldofview(FOV)

PLelyveld@ETCenter.org 41

Github VR standards discussions

Google put out RFComments for embedded spatial audio standards

FB camera released as open source to drive innovation and open standards

Google DayDream VR for Android N, including reference specs for phones and Gear-like HMDs

Standards?Standardization efforts.

PLelyveld@ETCenter.org 42

TheLanguageofVR:Part2

LiveActionorCGI?

PLelyveld@ETCenter.org 43

44

Camera-capturedVRLockedtocamerapositionTheDirector’sJourney

ComputerRenderedVRFreetowander(Storyworld)

Subtlecoercion

Decision

PLelyveld@ETCenter.org

MixedRealitySphericalcaptureliveandinsertintoCGI

“Volumetriccapturewillchangeeverything.”ChrisEdwards,TheVRCompanyPLelyveld@ETCenter.org 45

Hybrid- OutsidetoInside‘VolumetricVR’

Replay-technologies.com8i.comMS Hololens Holoport PLelyveld@ETCenter.org 46

Capture from many anglesCreate a model that can be explored

LytroCinemalight-fieldcameraCinemaframetoday,volumetricVRinthefuture

Everypixelhascolor,direction,x/y/zpositioninfoEveryobjectcanbeseparatedfromitssurroundings– greenscreen755RAWmegapixel40Kresolution,300FPS, 400Gbpsofdata

PLelyveld@ETCenter.org 47

Deluxe Launches VR Unit to Facilitate Industrial-Scale Virtual Reality Productions

Deluxeassembledateam ofspecialistsfromboththeworldofvisualeffectsandvideogameproduction.

Variety, 4/11/16PLelyveld@ETCenter.org 48

TheLanguageofVR:Part3

Thingstothinkabout

PLelyveld@ETCenter.org 49

TheCameraisYourAudience’sHead!!!

We don’t see shaky-cam when we run

The world doesn’t tilt when we tilt our heads

We see the world differently sitting and standing

PLelyveld@ETCenter.org 50

VRputsyouinthesceneThereisnothirdwalltobreak

Maintainingpresence dependsonmaintainingexpectationsintheexperience

HIDE(orbeabletoremove)EVERYTHINGTHATDOESN’TBELONGTHERE

CrewLightsEquipmentShadowsofequipment…NewLEDlightformfactorscanbehiddeninthesceneOrShootinmultipledirections(ex.180then180)andblendinpostIndoorscenes;sampletheceiling,putlightsthere,paintthemoutlater

Real-timeremotemonitoringoftheshootisbeingtested(ex.Teradek Sphere,Ozo system)ViewdailiesinHMDforsenseoftheexperienceANDlookatindividualsensorfeedsforproblems

PLelyveld@ETCenter.org 51

Solvingproblems

Greenscreenhelmetontoheadless torso.

Headbouncesdifferently frombody. FromtalkbyAnrick Bregman

PLelyveld@ETCenter.org 52

Thefinalimagemustbeerror-freeImperfectionsstandoutglaringly,andbreakpresence

Stitchingisa3-passprocess;- algorithmicreal-timefirstpass- quickpassforeditingwork- compositingheavyworkflowwithclean-upstitching

PLelyveld@ETCenter.org 53

54

Wild– TheExperience,VRshortfromFox(ReeseWitherspoon, LauraDern)

SoundAudioEngineering• Difficulttoproperlypositionfromaflatscreen• Toolsandroomsforspatialaudioarebeingdeveloped• Gameenginebasedaudioknowledge(Unity,Unreal)

PLelyveld@ETCenter.org

What’syourmessagetoSMPTEmembers?

Buyacheapcameraandexperiment!

PLelyveld@ETCenter.org 55

Resources

PLelyveld@ETCenter.org 56

LosAngelesVRandImmersiveTechnologiesMeetup

PLelyveld@ETCenter.org 57

August5th &6th 2016LosAngelesConvention

Center

SoCal VROctober 30, 2016Long Beach CC

New York

San Francisco

PLelyveld@ETCenter.org 59

For-Profitclasses

PLelyveld@ETCenter.org 60

MonthlySaturday,May28thVRLASchoolfocusesonthetoolsandprocessesforeditingstories

Websites

Road to VR – www.roadtovr.com - latest newshttps://www.facebook.com/groups/Virtual.Realities/ - latest newshttps://www.facebook.com/RoadtoVR/ - latest newsWearVR, Littlstar, Vrideo – latest contentReddit VR – community discussion groups

UploadVR - http://uploadvr.comVR Reporter - https://virtualrealityreporter.comVR Scout - http://vrscout.comWomen in VR – www.wivr.netWomen in VR Los Angeles - http://www.meetup.com/Women-in-VR/All Things VR – allthingsvr.coKzero - http://www.kzero.co.uk/blog/

PLelyveld@ETCenter.org 61

Podcasts

Voices of VR - dailyReal Virtual Show - weeklyhttp://a16z.com/?s=virtual+reality

http://www.apoip.org/home/events-calendar/http://allvirtualreality.com/events/list-virtual-reality-events-vr-expo-exhibitions-

conferences-2016.htmlhttps://www.jvrb.org/events

Lists of VR and AR conferences

PLelyveld@ETCenter.org 62

Modern era consumer VR is <3 years old

New art form and communication tool

WeneedtobringcinematicvisualandaudiocraftspeopleintothisfieldParaphrasedfromconversationwithJacquelineBosnjak,CEO,QDepartment

PLelyveld@ETCenter.org 63

Think about…

How to direct attention in an open space- Theme park and theme park ride design- Theatre in the round- Architecture and design

Existing skills and tools that port over to VR

Experimenting with a cheap VR camera

Learning Unreal, Unity, and other game engine UI tools

PLelyveld@ETCenter.org 64

ImagefromNotesonBlindnessVRexperience

Consumersbuyexperiences,nottechnologies

PLelyveld@ETCenter.org 65

66

Thank YouPhilip Lelyveld

USC Entertainment Technology Center

PLelyveld@ETCenter.org

www.etcentric.orgwww.PhilipLelyveld.com