Overview and AR/VR Roadmap - Khronos Group · The Need for AR/VR Standards. Vertically Integrated...

© Khronos® Group Inc. 2018 -

Overview and AR/VR Roadmap

Neil TrevettKhronos President

NVIDIA VP Developer [email protected] | @neilt3d

mailto:[email protected]


Khronos Connects Software to Silicon

Software

Silicon

Khronos is an International Industry Consortium creating royalty-free, open standards to enable software to access hardware acceleration for

3D Graphics, Virtual and Augmented Reality, Parallel Computing, Neural Networks and Vision Processing


What is An Open Standard?

Bad Standards - Overprescribes implementation details

- Forces lowest common denominator - Stifles innovation

-> Commoditization

Good Standards- Prescribes ONLY interoperability- Enables implementation diversity

– Encourages innovation-> Differentiation

An INTEROPERABILITY STANDARD enables two entities to COMMUNICATE.

E.g. Software <-> Hardware

A Truly OPEN standardIs not controlled by a single company – but by the whole industry

Is freely available to use by any company without royalty paymentsHas a well defined IP Framework to protect standard AND member’s IP

Standards Grow Markets…By reducing consumer confusion and increasing capabilities and usability

… Reduce Costs …By sharing development between

many companies and driving volume

… Accelerate Time to MarketWith well-proven functionality,

testing and interoperability


The Need for AR/VR Standards

Vertically Integrated One-off Solutions in 1990s High-volume Consumer Platforms Today!

Consumer platforms have insatiable need for content to drive demand - can’t afford silo’d content!

Consumer confidence – will my device run the content I want? Avoid Betamax vs. VHS Syndrome!

VR/AR will build on a constellation of standards

Interoperability standards so content can run on all VR devices!

Compete on adding user value –not restricting user choice!

HMD Connectivity Verticals and Security

VR Video Formats Acceleration APIs XR in the Browser

Wireless and 5G


Standards for AR and VR

High-performance, low-latency 3D Graphics

Download 3D object and scene data

Vision and sensor processing -including neural network

inferencing for machine learning

Portable interaction with VR/AR sensor, haptic and

display devices


Vulkan and New Generation GPU APIs

Modern architecture | Low overhead| Multi-thread friendlyEXPLICIT GPU access for EFFICIENT, LOW-LATENCY,

PREDICTABLE performance

Vulkan Porting Tools

Non-proprietary, royalty-free open standard ‘By the industry for the industry’Portable across multiple platforms - desktop and mobile


Vulkan – Now Available Everywhere

VR-Related FeaturesNOW

Multi-GPU supportMultiview Rendering

Context priorityFront buffer rendering

IN DISCUSSIONVariable Rate Rendering

Tiled rendering (beam racing)


The Metaverse will be the Web!

https://xkcd.com/1367/

https://xkcd.com/1367/


ContentJavaScript, HTML, CSS, ...

WebGL Stack

JavaScript Middleware

CSS

JavaScript HTML5

Browser provides WebGL3D engine alongside other HTML5

technologies - no plug-in required

OS Provided Drivers WebGL uses native OpenGL

or OpenGL ES or Angle = OpenGL ES over DX

Content downloaded from the Web

Middleware provides accessibility for non-expert programmers

E.g. three.js library

Low-level WebGL API provides a powerful foundation for a rich

JavaScript middleware ecosystem

Reliable WebGL relies on work by

both GPU and Browser Vendors

->Khronos has the

right membership to enable that cooperation

http://code.google.com/p/angleproject/


WebGL Deployment – WebGL 2.0 is Here!

http://caniuse.com/#feat=webgl

93.26% Globally

65.77% Globally

WebGL 1.0

WebGL 2.0

WebGL 2.0 will reach WebGL 1.0 levels of availability

when Safari and Edge ship enabling a new class of

Web-based AR/VR 3D Apps!

Pervasive, portable access to OpenGL ES 2.0-class mobile graphics

OpenGL ES 3.0-based desktop-class graphics comes to the Web!

http://caniuse.com/#feat=webgl


Khronos Standards for AR and VR






display devices


glTF – The JPEG of 3D!

Efficient, reliable transmissionBring 3D assets into 1000s of

apps and engines

glTF 2.0 – June 2017Native AND Web APIsPhysically Based Rendering

Metallic-Roughness and Specular-Glossiness

glTF 1.0 – December 2015Primarily for WebGL

Uses GLSL for materials

glTF spec development on open GitHub – get involved!

https://github.com/KhronosGroup/glTF

Compact to TransmitSimple and Fast to LoadDescribes Full Scenes

Runtime NeutralOpen and Extensible

https://github.com/KhronosGroup/glTF


glTF Ecosystem

Apps and Engines

Creation Tools

Repositories

Sony 3D Creator

Oculus

Paint 3D

Collada2gltf FBX2glTFgltf-vscode

glTF-validator glTF-asset-generator

Experience

Discover

DriveDemand

Create Windows Mixed Reality Home

Users

Modo

Mixed Reality Viewer3D Builder

Prep for 3D printing


glTF Roadmap• glTF manages its roadmap very carefully – complexity is the enemy

- Mission #1: ensure widespread, consistent, reliable usage

• Optional Draco Mesh Compression Extension Shipping Now- Adoption enabled with open source encoders and decoders

• Extensions and projects in discussion…- Draco compressed Point clouds, Universal compressed textures, LODs and streaming- Nexgen PBR materials, Advanced animations (e.g. Avatars and Face emoji), Metadata

• Developing reference viewer for visual consistency across web and native engines

Mesh Compression

Ratios

Stable Core Spec

Domain-specificextensions stay as

extensions

New widely needed functionality ships first as

extensions

Integrate extensions into new core spec only when:1) Widespread need is confirmed by the industry

2) Widespread reliable implementation is enabled (e.g. open source)








display devices


Vision and Inferencing Runtimes

Machine Learning Acceleration

Desktop and Cloud GPU/TPU Acceleration

Diverse Inferencing Acceleration Hardware

Training on Desktop and Cloud

Neural Net Training Frameworks

Neural Net Training Frameworks

Neural Net Training FrameworksTraining

Frameworks

Deployment on Embedded Devices

OptimizationTrained

Networks

Vision and Inferencing Applications

Training Data Sets

ORLiveData


OpenVX – Vision and Inferencing Acceleration Wide range of vision and inferencing hardware architectures so

OpenVX provides a high-level Graph-based abstraction for portability->

Enables Graph-level optimizations!Can be implemented on almost any hardware or processor!

->

Portable, Efficient Vision and Inference Processing!

Shipping Implementations

VisionNode

VisionNode Vision

Node

Downstream ApplicationProcessing

NativeCamera Control CNN Nodes

NNEF Translator converts NNEF representation into OpenVX Node Graphs

An OpenVX graph mixing CNN nodes with traditional vision nodes


GPU

OpenCL – Unique Heterogeneous Runtime

FPGA DSP

Custom Hardware

GPUCPUCPUCPUGPU

Growing number of optimized OpenCLvision and inferencing librariesVision: OpenCV, Halide, VisioncppMachine Learning: Xiaomi MACE, Arm Compute LibraryLinear Algebra: clDNN, clBlast, ViennaCL

Application orInferencing Run-time

GPU APIs

OpenCL is the only industry standard for low-level heterogeneous computePortable control over memory and parallel task execution

“The closest you can be to your embedded accelerator and still be portable”

Application orInferencing Run-time








display devices


XR = AR + VR

V1.0 - focused on VR

V

AAfter 1.0 – equal focus on AR


OpenXR – Solving VR/AR Fragmentation

Proprietary Engine

VR App

1

AR App

2

ARApp

4

VRApp

3

VR Device

1

VR Device

2

VR Device

3

VR Device

5

VR Device

4

After OpenXR Wide interoperability of XR apps and devices

VRApp

1

ARApp

2

ARApp

4

VRApp

3

Proprietary Engine

VR Device

1

VR Device

3

VR Device

5

VR Device

2

VR Device

4

Before OpenXRXR Market Fragmentation

Device Integration Layer

Application Interface

WebXR WebXR


OpenXR Architecture

Portable AR/VRInput Device Discovery

Multiple Sensor TrackingDevice Events

Pose NormalizationHaptics Control

Optical Corrections

OpenXR doesn’t replace AR/VR runtimes!

It enables those runtimes to use PORTABLE APIs to

expose their functionality


Companies Publicly Supporting

OpenXR is a collaborative designIntegrating many lessons from proprietary ‘first-generation’ API designs


Epic ‘Showdown’ VR Demo at SIGGRAPH

Demo runs portably across StarVR and Microsoft Windows Mixed Reality

headsets through the OpenXR APIs via an Unreal Engine 4 plugin https://www.youtube.com/watch?v=FCAM-3aAzXg&t=17250s

https://www.youtube.com/watch?v=FCAM-3aAzXg&t=17250s


Mobile Augmented Reality Libraries

Pose Tracking Yes Yes

Plane Detection and Tracking (Hori. & Vert.) Yes Yes

Image Recognition and Tracking Yes Yes

3D Object Scanning, Recognition and Tracking Yes No

Ambient light level and temperature Yes Yes

Environment Light Probes Yes No

Environment Reflective Texturing Yes No

Link to Neural Net-based Object Detection Yes Yes

Access to Point Cloud Yes Yes

Camera Intrinsics Yes Yes

Multi-user and persistent cloud-based anchors Yes Yes

Multi-user Viewing Yes No

Face tracking (iPhone X) Yes No

OS Availability iOS Android and iOS

ARKit

Encapsulated Vision-based Functionality

Also leveraging motion sensors

3rd party AR Libraries typically use

ARKit/ARCore if available or

implement own tracking and

functionality if not


Bringing VR and AR to the WebWeb XR Apps

WebXR

Native XR Apps

System-exposed AR Capabilities

3D Engines 3D Engines

Close ongoing collaboration between

WebXR and OpenXR

Khronos providing the foundation for 3D and

XR in the Web and native stacks

Future versions of OpenXR will include

cross-platform extended AR functionality


Summary• Khronos is creating cutting-edge royalty-free open standards

- For 3D, VR/AR, Compute, Vision and machine learning

• Join Khronos – any company is welcome!- Influence the direction of industry standards- Get early access to draft specifications

• More Information- www.khronos.org- [email protected] @neilt3d

Overview and AR/VR Roadmap - Khronos Group · The Need for AR/VR Standards. Vertically Integrated...

Documents

Transcript of Overview and AR/VR Roadmap - Khronos Group · The Need for AR/VR Standards. Vertically Integrated...