Post on 17-Mar-2021
Ray Tracing In Unreal Engine 4.22
State of the art real-time cinematography
UNREAL SUMMIT 2019
Juan CañadaLead Rendering Programmer
UNREAL SUMMIT 2019
UNREAL SUMMIT 2019
Session Overview● Introduction
● Troll: A Real-Time Saga● Overview● Goodbye Kansas
● Ray Tracing in UE4● RHI Layer● Mesh Pipeline● Render Passes and Experiments
UNREAL SUMMIT 2019
How Ray Tracing InUnreal Has Progressed
UNREAL SUMMIT 2019
UNREAL SUMMIT 2019
UNREAL SUMMIT 2019
UNREAL SUMMIT 2019
UNREAL SUMMIT 2019
Goals of ‘Troll’● Make a high-end cinematic in UE4
● Not just a ray tracing demo!● Runs in real-time (24 fps)● Cinematographic tools● Digital humans
● Using tools available for everyone● Made by partner Goodbye Kansas● Code available in 4.22
UNREAL SUMMIT 2019
Rect area lightsStatic and skeletal meshesRay tracing reflections, AO62M triangles16 lightsFull res RT passesRay tracing translucencyNiagara spritesDigital humans (hair, skin) Alembic/Geometry cache...
2018 2019Rect area lights Static and skeletal meshesRaytraced reflections, AO5M triangles4 lightsHalf res RT passes
UNREAL SUMMIT 2019
Rect area lightsStatic and skeletal meshesRay tracing reflections, AO62M triangles16 lightsFull res RT passesRay tracing translucencyNiagara spritesDigital humans (hair, skin) Alembic/Geometry cache...
2018 2019Rect area lights Static and skeletal meshesRaytraced reflections, AO5M triangles4 lightsHalf res RT passes
24 fps 24 fps
UNREAL SUMMIT 2019
2018● Nvidia DGX station● 4 x Tesla V100 GPUs
UNREAL SUMMIT 2019
2019● 1 Nvidia 2080Ti
UNREAL SUMMIT 2019
Technical Challenges
● Performance● Digital Humans
UNREAL SUMMIT 2019
Technical Challenges
● Performance● Digital Humans
UNREAL SUMMIT 2019
Performance Challenges
UNREAL SUMMIT 2019
● Immediate vs Retained● Ray Navigation● Niagara particles
Performance Challenges
UNREAL SUMMIT 2019
● Immediate vs Retained● Ray Navigation● Niagara particles
Performance Challenges
UNREAL SUMMIT 2019
● Immediate vs Retained● Ray Navigation● Niagara particles
Performance Challenges
UNREAL SUMMIT 2019
● Shadows and AO are affordable● The most immediate use of ray tracing in games
● Other effects are expensive● Trace rays only when they are really needed ● Trace rays as short as possible
● Denoisers are critical
Ray Tracing Performance
UNREAL SUMMIT 2019
● Shadows and AO are affordable● The most immediate use of ray tracing in games
● Other effects are expensive● Trace rays only when they are really needed ● Trace rays as short as possible
● Denoisers are critical
Ray Tracing Performance
UNREAL SUMMIT 2019
UNREAL SUMMIT 2019
● Shadows and AO are affordable● The most immediate use of ray tracing in games
● Other effects are expensive● Trace rays only when they are really needed ● Trace rays as short as possible
● Denoisers are critical
Ray Tracing Performance
UNREAL SUMMIT 2019
UNREAL SUMMIT 2019
● Immediate vs Retained● Ray Navigation● Niagara Particles
Performance Challenges
UNREAL SUMMIT 2019
● Immediate vs Retained● Ray Navigation● Niagara particles
● DOF, Motion blur, translucency● Visible in ray tracing effects
Performance Challenges
UNREAL SUMMIT 2019
Initial Performance
UNREAL SUMMIT 2019
Final Performance
UNREAL SUMMIT 2019
Demo Challenges● Performance● Digital Humans
UNREAL SUMMIT 2019
Digital Humans● Ray tracing makes it much easier!● Barn doors (Available in 4.22)
UNREAL SUMMIT 2019
Anton PalmqvistHead of Real-time at Goodbye Kansas Studios
UNREAL SUMMIT 2019
● Game trailers, cinematics, feature films, TV shows, advertising
● HQ in Stockholm, Sweden● Real-time focused office in LA● Integrating Unreal in our VFX-pipeline
UNREAL SUMMIT 2019
What ray tracingbrings to UnrealReflections: Improvement over SSR & planar reflections
Translucency: Much better glass and liquids
Soft shadows: Area lights, textured lights
Global illumination: Don’t have to rely on lightmaps
A more cinematic way of lighting
UNREAL SUMMIT 2019
UNREAL SUMMIT 2019
Ray Traced Translucency
UNREAL SUMMIT 2019
● Ubershader● Light Rig● Alembic● Python
Fitting Unreal into a VFX Pipeline
UNREAL SUMMIT 2019
● Artist friendly● Saves time● Less prone to errors● UDIM
Goodbye Kansas Ubershader
UNREAL SUMMIT 2019
● Quick lookdev environment● Now with ray tracing● With runtime controls● Great for dailies!
Goodbye Kansas Light Rig
UNREAL SUMMIT 2019
● Improvements in 4.22, more efficient● Dress was imported as alembic
geometry cache
Alembic
UNREAL SUMMIT 2019
Cinematic lighting tools now better than ever
Changing the mindset(Linear vs. Parallel Production)
Takeaways
Patrick KellySr. Rendering Programmer
UNREAL SUMMIT 2019
UNREAL SUMMIT 2019
Technical Outline● RHI Layer● Mesh Pipeline● Render Passes
UNREAL SUMMIT 2019
Technical Outline● RHI Layer
● Geometry● Scene● Shaders● Pipeline
● Mesh Pipeline● Render Passes
#UE4 | @UNREALENGINE
virtual FRayTracingGeometryRHIRef RHICreateRayTracingGeometry(...);
virtual FRayTracingSceneRHIRef RHICreateRayTracingScene(...);
virtual FRayTracingShaderRHIRef RHICreateRayTracingShader(...);
virtual FRayTracingPipelineStateRHIRef RHICreateRayTracingPipelineState(...);
RHI Layer
UNREAL SUMMIT 2019
#UE4 | @UNREALENGINE
virtual FRayTracingGeometryRHIRef RHICreateRayTracingGeometry(...);
struct FRayTracingGeometryInitializer{
FVertexBufferRHIRef PositionVertexBuffer = nullptr;
FIndexBufferRHIRef IndexBuffer = nullptr;
TArrayView<FRayTracingGeometrySegment> Segments;
...};
RHI Layer: Geometry
#UE4 | @UNREALENGINE
#UE4 | @UNREALENGINE
struct FRayTracingGeometryInstance{
FRayTracingGeometryRHIParmRef GeometryRHI = nullptr;
FMatrix Transfom = FMatrix(EForceInit::ForceInitToZero);
uint32 UserData = 0;
uint8 Mask = 0;
...};
RHI Layer: Geometry
UNREAL SUMMIT 2019
#UE4 | @UNREALENGINE
virtual FRayTracingSceneRHIRef RHICreateRayTracingScene(...);
struct FRayTracingSceneInitializer{
TArrayView<FRayTracingGeometryInstance> Instances;
...};
RHI Layer: Scene
UNREAL SUMMIT 2019
UNREAL SUMMIT 2019
RHI Layer: Shaders● Closest-hit● Any-hit● Miss● Ray generation
UNREAL SUMMIT 2019
RHI Layer: Pipeline● Analog to graphics and compute PSOs● One RT PSO for all shaders
UNREAL SUMMIT 2019
RHI Layer: Shader Binding Tablevirtual void SetRayTracingHitGroup(...);
UNREAL SUMMIT 2019
Technical Outline● RHI Layer● Mesh Pipeline
● Supported Geometry● Extending Geometry● Special Considerations
● Render Passes● Path Tracing
UNREAL SUMMIT 2019
Mesh Pipeline
UNREAL SUMMIT 2019
Mesh Pipeline: Supported Geometry● Static Mesh● Skeletal Mesh● Geometry Cache● Niagara
UNREAL SUMMIT 2019
Mesh Pipeline: Extending Geometryvirtual void GetDynamicRayTracingInstances(...);
See NiagaraRendererSprites.cpp for a concrete example
UNREAL SUMMIT 2019
Mesh Pipeline: Special Considerations● Rebuild v. Refit● Level-of-Detail● Visible in Shadows
UNREAL SUMMIT 2019
Technical Outline● RHI Layer● Mesh Pipeline● Render Passes
● Shadows● Ambient Occlusion● Reflection / Transparency● Experimental Integrators
UNREAL SUMMIT 2019
Render Passes: Shadows● Replacement for shadow maps● Accurately computes visibility for area sources
UNREAL SUMMIT 2019
Render Passes: Shadows
UNREAL SUMMIT 2019
Render Passes: Shadows (1 spp Raw Input)
UNREAL SUMMIT 2019
Render Passes: Shadows (Denoised Result)
UNREAL SUMMIT 2019
Render Passes: Shadows (64 spp Reference)
UNREAL SUMMIT 2019
Render Passes: Ambient Occlusion● Replacement for SSAO● Cosine-weighted ray distribution
UNREAL SUMMIT 2019
Render Passes: Screen Space Ambient OcclusionUNREAL SUMMIT 2019
UNREAL SUMMIT 2019
Render Passes: Ray Tracing Ambient OcclusionUNREAL SUMMIT 2019
UNREAL SUMMIT 2019
Render Passes: Ray Tracing Ambient Occlusion (Denoised)UNREAL SUMMIT 2019
UNREAL SUMMIT 2019
Render Passes: Material Support● Vertex interpolants● Material graph● Closest-hit shader
UNREAL SUMMIT 2019
Render Passes:Reflection● GGX ray distribution● Direct lighting on secondary
surface
UNREAL SUMMIT 2019
Screen Space Reflections
UNREAL SUMMIT 2019
Ray Traced Reflections
UNREAL SUMMIT 2019
Render Passes: Reflection
UNREAL SUMMIT 2019
Render Passes: Translucency● Trace from camera● Direct lighting on secondary surface● Reflection and refraction
UNREAL SUMMIT 2019
Render Passes: Translucency
UNREAL SUMMIT 2019
Render Passes● Shadows● Ambient Occlusion● Reflection / Transparency● Experimental Integrators
● RectLight● SkyLight● Global illumination
UNREAL SUMMIT 2019
Experimental Integrators: Hierarchical sample warping
“Wavelet importance sampling: Efficiently evaluating products of complex functions” [Clarberg et al 2005]
UNREAL SUMMIT 2019
Experimental Integrators: RectLight
UNREAL SUMMIT 2019
Experimental Integrators: SkyLight
UNREAL SUMMIT 2019
UNREAL SUMMIT 2019
Experimental Integrators: Global Illumination
UNREAL SUMMIT 2019
UNREAL SUMMIT 2019
Experimental Integrators: Image-based Lighting
● Staple of environment lighting in VFX
UNREAL SUMMIT 2019
Rasterization
Content courtesy of Quixel
UNREAL SUMMIT 2019
Ray Tracing (Directional Light + RTAO)
Content courtesy of Quixel
UNREAL SUMMIT 2019
SkyLight + RTGI (24 fps)
Content courtesy of Quixel
UNREAL SUMMIT 2019
SkyLight + RTGI (1 fps)
Content courtesy of Quixel
UNREAL SUMMIT 2019
UNREAL SUMMIT 2019
Path Tracing● Reference● Unbiased● Foundation for other techniques
UNREAL SUMMIT 2019
UNREAL SUMMIT 2019
The Future● Performance● Stability● Geometry
● Intersection shader support● Landscape● Volumetrics
● Improvements ● Denoisers● Path tracing
● Interactive GPU light baking● Multi GPU● …
UNREAL SUMMIT 2019
Arne Schober
Charles De Rousiers
Guillaume Abadie
Patrick Kelly
Sebastien Hillaire
Yujiang Wang
Yuriy O’Donnell
Development Team
UNREAL SUMMIT 2019
● UE 4.22 includes ray tracing● Early Access ● Complete set of features to get cinematic quality pixels
● Can’t wait to see you using it!
Takeaways
UNREAL SUMMIT 2019
Thanks!