Mark Kilgard, July 27, 2016

Virtual Reality Features ofNVIDIA GPUs

2

Mark Kilgard

Principal System Software EngineerOpenGL driver implementation & API evolutionOther interests

Programmable shading languages, original Cg GPU shading language

Web, vector graphics & path rendering, GPU rendering for 2D graphics

Also known for creating OpenGL Utility Toolkit (GLUT)

My Background

3

GPU Challenges for Virtual Reality Rendering

1. Must render unique view for each eyeTwo scene renders per frame

2. Wide field-of-view renderedMore pixels, distributed towards periphery

3. HMD lens magnifies rendered sceneNon-linear inverse lens warp

Allows the HMD lens magnification to provide convincing sense of presence

4

Magnification Effect of HMD Lens

Graph paper viewed & magnified through HMD lens

5

Head Mounted Display Lens Effect

Barrel Distorted Imageson HMD Screen

Magnified Overlapping Wide Field-of-viewBinocular View

6

Focus on Virtual Reality Features of Pascal GPUs

Virtual Reality rendering features in Pascal GPUsGeForce 1080/1070/1060 NEW Quadro P6000 & P5000

As exposed in standard 3D APIs

7

Background: NVIDA GPU Architecture Road Map

Our interest NVIDIA GPU architectures of interest: Pascal

What is Pascal mentioned on last slide?

8

Conventional 3D Single-eye Scene

9

Conventional 3D Single-eye Scene

10

Wide Stereo Views of Same Scene

Left Right

11 Left Right

12 Right Left

13

Left & Right Views AreSimilar but Different

Left and Right eye views have different occlusionWhile left & right eye views look similar, must be draw independently

Really only different in a small translation in X

NVIDIA Single Pass Stereo supports generating left & right eye positionsGPU automatically makes left & right eye versions of each primitiveThen renders each view to a different framebuffer sliceExposed in OpenGL by NV_stereo_view_rendering extensionExposed for Direct3D 11 by NVAPI SinglePassStereoMode feature

14

Image Difference of Two Views

− + 0.5 =

Left eye view Right eye viewClamped difference image

15

Left & Right Views Rendered Efficiently in One Pass

16

Lens Matched Shading

Left Right

17

Visualizing Lens Matched Shading

18

Lens Matched Shading

Left Right

19

Visualization of Lens Matched Shading Rendering

20

Still Needs Non-linear Warp

Lens Matched Shadingrendering

Warped image suitablefor HMD display

Non-linearimage

21

Still Needs Non-linear Warp

Lens Matched Shadingrendering

Warped image suitablefor HMD display

Non-linearimage

BUT substantiallyless pixel motion

and distortionthan conventional

VR re-warping

22

Example HMD Post-rendering Warp

23

Lens Matched Shadingwith Window Rectangle Testing

24

Lens Matched Shadingwith Window Rectangle Testing

Nothing in black corners is shaded or even rasterizedYellow lines show overlaid 8 inclusive window rectanglesSame 8 window rectangles “shared” by each view’s texture array layer

25

Lens Matched Shadingwith Window Rectangle Testing

26

What Didn’t Have to be Rendered

17% of left view pixels 17% of right view pixels

27

Window Rectangle TestingDuring Rasterization

New multi-vendor OpenGL extension

EXT_window_rectangles

Provides a set of window-space rectangles, 8 for NVIDIA GPUs

Rasterizer either includes or excludes rasterization from rectangles

Discards at speed-of-lightAs if free to rasterize to irregular rectangular regions

WindowRectangles

Test

NEWstage

OpenGL’s fragment processing pipeline

28

Warped Lens Matched Scene

Warped version of Lens Matched Shading to match HMD lens

29

Warped Lens Matched Shading

30

Visualizing Warp Window Rectangles

31

What Didn’t Have to be Warped

17% of left view pixels 17% of right view pixels

32

VR Rendering Pipeline in Images

LMS Right Eye View Warped Right Eye View

LMS Left Eye View Warped Left Eye View

Scene

LMS = Lens Matched Shading

33

VR Rendering Pipeline in Images

LMS Right Eye View Warped Right Eye View

LMS Left Eye View Warped Left Eye View

Scene

Displayed within HMD

Single Rendering PassSingle Pass Stereo +

Lens Matched Shading +Window Rectangle Testing

Drawn with Single TriangleFragment Shader WarpingWindow Rectangle Testing

Perception to user is linear rendering HMD lens “undoes” warping to provide

a perceived wide field-of-view

Pascal does all thisefficiently in a singlerendering pass!

8 viewports, 1 pass

34

Traditional 3D Rendering Pipeline

Transform3D View Downsample Display

FrameApp

Processing

if multisampled

uses pinhole camera projection,rasterizing to uniform rectilinear pixel grid

monitor directlydisplays uniform rectilinear pixel grid

Rasterize3D View

35

Virtual Reality Rendering PipelineRendering Left & Right Eye Views

TransformLeft View Downsample

DisplayFrame to HMD

AppProcessing

RasterizeLeft View

TransformRight View DownsampleRasterize

Right View

HMDWarp

HMDWarp

warps uniform rectilinear grid based on HMD’s device profile

TransformRight View

RasterizeRight View

TransformRight View DownsampleRasterize

Right ViewTransformRight View

HMDWarpDownsampleRasterize

Right ViewTransformRight View

second view rendering

36

Virtual Reality Rendering PipelineWith Lens Matched Shading

TransformLeft View Downsample

DisplayFrame to HMD

AppProcessing

RasterizeLeft View

TransformRight View DownsampleRasterize

Right View

HMDWarp

HMDWarp

LensMatchedShading

LensMatchedShading

replicates primitiveto 4 projected quadrants

less pixel resamplingeffort & error operating onLMS rasterized images

37

Virtual Reality Rendering PipelineWith Single Pass Stereo

TransformLeft & Right

View

Downsample

DisplayFrame to HMD

AppProcessing

RasterizeLeft View

DownsampleRasterizeRight View

HMDWarp

HMDWarp

one transform instance,outputs left & right position

38

Virtual Reality Rendering PipelineSingle Pass Stereo + Lens Matched Shading

Downsample

DisplayFrame to HMD

RasterizeLeft View

DownsampleRasterizeRight View

HMDWarp

HMDWarp

LensMatchedShading

LensMatchedShading

TransformLeft & Right

View

AppProcessing

+ Advantages1. Single transform for both views2. Closer match between rendering & lens images3. Fewer overall pixels for similar quality

use window rectangle testingto bound lens region, duringboth rasterization & warp

39

Pascal GPU Features for VRUnique to NVIDIA

Pascal renders left & right eyeIn single rendering pass

Halves CPU load for rendering

Better concurrent texture locality

OpenGL: NV_stereo_view_renderingDirect3D: NVAPI SinglePassStereoMode

Better matches rendering to HMD lensSingle pass

Four separately projected quadrants

Less rendering, less resampling error

OpenGL: NV_clip_space_w_scalingDirect3D: NVAPI ModifiedWSupport

40

Other NVIDIA Virtual Reality Initiatives

41

More Information

VRWORKS 2.0 Software Development KitAvailable NOW

Pascal OpenGL extensions for Virtual RealitySpecification now public

In official OpenGL.org extension registry

OpenGL Extension Wrangler 2.0 (GLEW) support

Example code in VRWORKS 2.0 SDK

VRWORKS also documents NVAPI usageFor Direct3D 11

Includes Examples code