Post on 20-May-2020
JonathanReeves
October9,2017
SeeWhatWeSee- SharingMixedRealityExperienceswithWebRTC
1©2017Arvizio,Inc.
Introduction
MixedRealityisagamechangingtechnologyforenterprisesincludingAEC,education,energy,engineering&healthcare
Keyrequirementsforenterpriseadoption:§ Mixedrealityplatformsmustscaletohandlecomplex3Dmodels§ Experiencesshouldbesharedbothlocallyandremotely§ IntegrationwithexistingITsystemsessentialforseamlessworkflow
Arvizio’s MRStudio platformgoal:§ Dynamicservingoflarge3DmodelswithrichLOD(levelofdetail)§ Sharingofexperiencesandcollaborationacrosslocations§ FlexibledataintegrationwithexistingITsystemsandIoTdata
2©2017Arvizio,Inc.
TheMRStudioPlatform
MRStudioImmerse
MRStudioRemoteASPENscalable3DmodelLODprocessingReal-timesession&sharingservicesOnPrem,edgeorclouddeployment
Large3DmodelvisualizationwithLODSharing&collaborationacrosslocationsSupportformultipleAR/MRdevicetypes
Viewmixedrealityscenesremotely2Wayvideo,audio&annotation
MRStudioDirector
IoTData&ITSystems
4G/5GorWiFi
3©2017Arvizio,Inc
IPNetwork
WebRTC
MRStudioDirector
Advanced3DVisualization§ Optimizeslarge3DmodelswithLOD(LevelofDetail)§ Mesh,CAD,Pointcloud,BIM,GIS&DICOM§ Workflowwizardsformanyspatialdataformats
Real-timeSharingServices§ SharedviewingformultipleHoloLensusers§ HoloLensusersorMRSDirectormayleadsessions§ WebRTCgatewayforsecureremoteparticipation
DataIntegration§ ConnectsMRStudiowithexternalITsystems§ ModularApacheNiFi dataflowbasedarchitecture§ ModulestoconnectwithIoTsensors,bigdataetc.
4©2017Arvizio,Inc.
LODDataPyramid
5©2017Arvizio,Inc..
Full 3D model
2nd Level of Detail
3rd Level of Detail
4th Level of Detail
ImmerseAppFramework
3DModelViewingwithLODLifeSizeModelWalkThroughSharingforMultipleHoloLens
RemoteVideoChatMixedRealityRTCFeed
DataintegrationwithITsystemsBigDataSystems,IoTStreams
MRStudioDirector
Annotation
MRStudioImmerse
6©2017Arvizio,Inc..
Local&RemoteCollaboration
7©2017Arvizio,Inc.
HoloSync allowsMRSDirectortointeractivelypublish3Dscenesandmodels
MRStudioRemoteActiveviewingandannotationoftheHoloLensfeedCreatetextualorgraphicalhologramsinthescene
WebRTC
LODDataServedonDemand
WebRTCAudio,Video&SpatialDataExchange
WebRTCRecap
VP8/9&H.264VideoCompression- AppleoffersonlyH.264inIOS11
STUN&TURNAllowNATRouterTraversal
ICE– InteractiveConnectivityEstablishmentmanagestheconnectivityprocess
8©2017Arvizio,Inc.
Remoteviewing&annotationoverWebRTC
9©2017Arvizio,Inc.
§ Remoteusersmayplaceholographicannotationsintothefieldofview§ Supportedonmanybrowsersacrossmultipleplatforms§ Bandwidthefficient- operatesoverbandwidthconstrainedwideareanetworks§ HoloLensfeedislimitedto720p,alsoheadmovementcanbeachallenge
MRDeviceView
RemoteAnnotationToolbar
CapturingtheSharedExperienceHoloLensSpectatorView
§ LocalviewingofmixedrealityscenesinHDbutnoremotestreaming§ Calibrationcheckerboardrequired,complexsoftwareinstall§ RequiresdedicatedHoloLens&digitalSLR,milledaluminumbracket§ Powerfulconcept,couldthisbeimprovedupon?
10©2017Arvizio,Inc.
MRStudioViewPointStreamingofMixedRealityinHD
§ Localviewing/recording&remoteWebRTCstreamingofmixedrealityscenes§ Headsetrequiredduringinitialsetuponly§ Usedepthcameraorwebcams(staticplacement)inplaceofDSLR§ Potentialtoworkacrossmultipletypesofdevice
11©2017Arvizio,Inc.
CameraCaptureUnity App
ComposerWebRTCGateway
Director Sharing Server
Depth SensingStereo Camera
MR Studio Director
WebRTCClient
LocalHD View(s)
RemoteView
PC
Audio
MRStudioViewPoint Architecture
12©2017Arvizio,Inc.
Shared Viewing
MRStudioUseCasesAEC(Architecture,Engineering&Construction)
§ Visualizationduringthedesign,engineeringandconstructionprocess§ Bringingprojectstolifeduringthesalesprocess§ Accelerateprojectapprovalprocess
IndustrialDesign&Engineering§ 3Dvisualizationofcomplexmachinery§ Assistedmaintenancetaskswithremotevideoexpertassistance
Training&Education§ Training&educationacrossindustries§ Classroom,lectures,onsitetrainingandremoteparticipants
Healthcare§ MRhospitalroomconcept,MRassistancefortelemedicine§ Presurgeryvisualization,prepareforcomplexsurgeries
13©2017Arvizio,Inc.
Demonstration