Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you...
Transcript of Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you...
Introduction to PowerVR Ray TracingTuesday 18th March, 2014 @ GDC
James A. McCombe
What are we launching today?
System Memory Bus
VertexData Master
PixelData Master
ComputeData Master
Course Grain
Scheduler Core Management
Memory Cache Unit
2D Core (PTLA)
Pixel Co-Processor
Tiling Co-processor
Control and Register BusHost CPU Interface
RayData Master
System Memory Interface
Frame Accumulator
Cache
Scene Hierarchy Generator
Intersection Processor Array Coherency Engine
Ray Tracing Unit
USC0 USC1Texture Unit
USCn-1 USCnTexture Unit
Unified Shading Cluster Array
...
What are the applications for this?
Augmented Reality
Image and Diagram CopyrightPeter Kán 2012
Virtual Reality a.k.a Multi-persective rendering
Lenticular Displays
Ultra-low latency rendering
Lens distortion and aberration correction
Hybrid rendering in games
• Shadows
• Reflections
• Transparency
• Better scaling with multiple dynamic lights
• Real-time light map updates for the rasteriser
• Easy to bolt on
Fully ray traced graphics
• Brute force path tracing
• Produces photo realism easily
• Pretty much requires all the 3D content to be ray traced
• Possible in todays technology in console / desktop using Wizard
• Probably not practical for fully real-time use in mobile for a couple more generations. Fine for sub-realtime use.
Non-graphics
• Better in-game AI
• Collision Detection
• 3D spatial search
Overview of PowerVR
Power and Bandwidth
0W
75W
150W
225W
300W
Workstation Mobile Workstation Laptop Light Laptop Tablet0Gb/s
30Gb/s60Gb/s90Gb/s
120Gb/s150Gb/s180Gb/s210Gb/s240Gb/s270Gb/s300Gb/s
Graphics Watts Graphics Bandwidth
Capability of our mobile graphics
0
37.5
75
112.5
150
2006 2007 2008 2009 2010 2011 2012 2013
Shad
er G
Flop
s
• Almost 5x compute capability
• 4x vec4 MAD → 16x Scalar 2xMAD
• 1.5 → 3.6 Gtexels/s
• OpenGLES 3.0 Support
System Memory Bus
VertexData Master
PixelData Master
ComputeData Master
Course Grain
Scheduler Core Management
Memory Cache Unit
2D Core (PTLA)
Pixel Co-Processor
Tiling Co-processor
Control and Register BusHost CPU Interface System
Memory Interface
USC0 USC1Texture Unit
USCn-1 USCnTexture Unit
Unified Shading Cluster Array
...
• An instance is a vertex, pixel or OpenCL thread
• All instances in the task share Program, Uniforms, Parameters, etc.
Compute Task
32 Instances
Vertex Task
32 Instances
CGS
USC0 USC1Texture Unit
USCn-1 USCnTexture Unit
Unified Shading Cluster Array
Pixel Task
32 Instances
VertexData
Master
PixelData
Master
ComputeData
Master
ALU
Common Store Registers (Shared across instances in the task)
ALU ALU ALU ALU ALU ALU ALU ALU ALU ALU ALU ALU ALU ALU ALU
Unified Store Registers (Private registers per instance)
CommonInstruction
Per Pixel / Vertex data and shader temps
Shader Uniforms and Constants
(not exact)
Output 1
Output 0
Source 0
Source 1
Source 2
Source 3
Source 4
Source 5
ABS, NEG, etc.
ABS, NEG, etc.
ABS, NEG, etc.
ABS, NEG, etc.
ABS, NEG, etc.
ABS, NEG, etc.
⩽ 2 Flops
Integer Math
Format Conversions
⩽ 2 Flops
Comparison Format Conversions
12 Cycles of Latency (not exact)
ALU
Common Store Registers (Shared across instances in the task)
ALU ALU ALU ALU ALU ALU ALU ALU ALU ALU ALU ALU ALU ALU ALU
Unified Store Registers (Private registers per instance)
CommonInstruction
varying vec2 texCoord;
void main() { if (texCoord.x > 0.5) { ... ... } else { ... ... }
Overall capability
• Series6 gives you over 100Gflops for shading on latest mobile devices
• OpenGL ES 3.0 support
• Massive capability gain allowing math heavy shaders
• Mobile graphics is where the actual innovation is happening
• Try it!
A Technical introduction to Wizard
System Memory Bus
VertexData Master
PixelData Master
ComputeData Master
Course Grain
Scheduler Core Management
Memory Cache Unit
2D Core (PTLA)
Pixel Co-Processor
Tiling Co-processor
Control and Register BusHost CPU Interface
RayData Master
System Memory Interface
Frame Accumulator
Cache
Scene Hierarchy Generator
Intersection Processor Array Coherency Engine
Ray Tracing Unit
USC0 USC1Texture Unit
USCn-1 USCnTexture Unit
Unified Shading Cluster Array
...
System Memory Bus
VertexData Master
PixelData Master
ComputeData Master
Course Grain
Scheduler Core Management
Memory Cache Unit
2D Core (PTLA)
Pixel Co-Processor
Tiling Co-processor
Control and Register BusHost CPU Interface
RayData Master
System Memory Interface
Frame Accumulator
Cache
Scene Hierarchy Generator
Intersection Processor Array Coherency Engine
Ray Tracing Unit
USC0 USC1Texture Unit
USCn-1 USCnTexture Unit
Unified Shading Cluster Array
...
Ray RAM
Ray RAM
Output 1
Output 0
Source 0
Source 1
Source 2
Source 3
Source 4
Source 5
ABS, NEG, etc.
ABS, NEG, etc.
ABS, NEG, etc.
ABS, NEG, etc.
ABS, NEG, etc.
ABS, NEG, etc.
⩽ 2 Flops
Integer Math
Format Conversions
⩽ 2 Flops
Comparison Format Conversions
12 Cycles of Latency (not exact)
New shader type
• Ray Shaders are invoked when a ray intersects a triangle
• Shaders types can emit any number of rays
• OpenRL has a frame shader to emit primary rays
• Existing fragment / pixel shaders can emit rays also!
• GLSL Programming model is the same
• Built-in functions, parameters, etc.
Primitive Objects
• Encapsulate the rendering state for one mesh
• Includes VBOs, uniforms, shader programs, texture bindings, etc.
• Ray tracing unit sorts rays into tasks with common primitive objects
• Persistent between frames
• Mutable objects by the client
Constraints / Limitations
• Shaders cannot wait for results of individual ray trace operations
• Shaders must provide a worst case estimate on number of child rays
• Per-ray user data payload must be carefully managed
Parallelism is on rays NOT pixelsTask 0
Task 3
Task 2
Task 1
ALU ALU ALU ALUCommonInstruction
(In this example, ALU width is 4 to make the diagram smaller)
Parallelism is on rays NOT pixels
Relief: Quick ray emission demo
Intersection Processor Arrayor
“Who needs Fixed Function?”
Ray:AABB Test
• “Fast Ray-Axis Aligned Bounding Box Overlap Tests with Plucker Coordinates.” – Jeffrey Mahovsky and Brian Wyvill
• 6 lines form the silhouette of the AABB
• 6 planes from the ray origin and each edge vector
• Dot product of plane normal and ray direction vector
• 6 signs must match and be negative.
3 MUL 6 SUB 6 ADD 12 MUL 6 bitwise OR
!voxel_center_x
origin_xcov_convertAX0min
BX0max
shift_center_x7+/!
ri(31)
XB
XA
ri(31)
XA
XB
>70
invert
ri(31)
>7
invert
ri(31)
ri
!voxel_center_y
origin_ycov_convertAY0min
BY0max
shift_center_y7+/!
rj(31)
YB
YA
rj(31)
YA
YB
>70
invert
rj(31)
>7
invert
rj(31)
rj
!voxel_center_z
origin_zcov_convertAZ0min
BZ0max
shift_center_z7+/!
rk(31)
ZB
ZA
rk(31)
ZA
ZB
>70
invert
rk(31)
>7
invert
rk(31)
rk
MISS
origin_x_miss
endpt_x_miss
origin_y_miss
endpt_y_miss
origin_z_miss
endpt_z_miss
Octant(0)
Octant(1)
Octant(2)
!
rj
rk
rixdirx
ray_distance
xdiry
ray_distance
xdirz
ray_distance
!
Ray distance setup Translation Box Plane Setup Plane tests Test Combine
Hitor
Miss
USC instruction group packingMULMUL
MULSUB
LDLD
LDLD
LDLD
SUBSUB
SUB–
SUBSUB
ADDADD
ADDADD
ADDADD
MULMUL
MULMUL
MULMUL
MULMUL
MULMUL
MULMUL
OR–
OR–
OR–
26 Instructions
(32 if using compressed data formats)OR
–
OR–
OR–
44x less area for this function
Tester and Rays
ChildAddress
Tester and Rays
Triangle vs. Ray
Boxvs.
Ray
Boxvs.
Ray
Boxvs.
Ray
Boxvs.
Ray
Boxvs.
Ray
Boxvs.
Ray
Boxvs.
Ray
Boxvs.
Ray
Tester and Rays Tester and Rays
Triangle vs. Ray
Boxvs.
Ray
Boxvs.
Ray
Boxvs.
Ray
Boxvs.
Ray
Boxvs.
Ray
Boxvs.
Ray
Boxvs.
Ray
Boxvs.
Ray
Triangle vs. Ray
Boxvs.
Ray
Boxvs.
Ray
Boxvs.
Ray
Boxvs.
Ray
Boxvs.
Ray
Boxvs.
Ray
Boxvs.
Ray
Boxvs.
Ray
Triangle vs. Ray
Boxvs.
Ray
Boxvs.
Ray
Boxvs.
Ray
Boxvs.
Ray
Boxvs.
Ray
Boxvs.
Ray
Boxvs.
Ray
Boxvs.
Ray
Coherence Queue RAM
Old Count Existing Queue
Update Existing Queue
Update New
Queue
Coherence Cache
2 7 2 4 2 2
2 2 1 5
5 2 3 2 1 2
5 3 1 2
5 2 7 7 2 5
Queue Packer
Inst Data Inst Data
BoxHits
FreeQueue
USC
USC
USC
USC
USC
USC
USC
USC
USC
USC
USC
USC
USC
USC
USC
USC
USC
USC
USC
USC
USC
USC
USC
USC
USC
USC
USC
USC
USC
USC
USC
USC
USC
USC
USC
USC
USC
USC
USC
USC
USC
USC
USC
USC
USC
USC
USC
USC
USC
USC
USC
USC
The Ray Tracing Unitand
Coherence Engine
System Memory Bus
System Memory Interface
Course Grain
Scheduler
RayData Master
USC0 USC1Texture Unit
USCn-1 USCnTexture Unit
Unified Shading Cluster Array
Intersection Processor Array Coherency Engine
Ray Tracing Unit
Rays in arbitrary orderRays grouped by PrimitiveObject
(Shader + Resources)
( around 100Mbytes for 1M triangles including varyings )
AABB Blocks Vertex Blocks Varyings
Ascending Virtual Memory Addresses
PrimitiveObject 0
PrimitiveObject 1
PrimitiveObject 2
node_22359node_22355
node_16991
node_6445
node_6450
node_16994
node_6452
node_6456
node_16999 node_6436
node_6438 node_6440
node_11744node_6442
node_22372node_22369
node_6428
node_11722node_6430 node_6432
node_11726node_6434
node_17005node_11732 node_6424
node_11730node_6426
node_22409node_22404
node_11714
node_16978node_11716 node_16981node_11718
node_22418
node_16984
node_16986
node_17121
node_22423
node_14127
node_13585
node_13629 node_17200node_13503
node_22510
node_22517 node_22528
node_22533
node_22543
node_22550
node_22558
node_13707
node_19064
node_7590
node_13788node_7608
node_7638
node_13868node_7676 node_7696
node_13977node_7714
node_22561
node_19307 node_8136
node_14009node_8290 node_8362
node_8419
node_19446 node_8774
node_9065 node_9116
node_9228
node_22490node_22487
node_13993
node_19252
node_13994
node_8261
node_8356
node_8412
node_19374 node_8725
node_9046 node_9110
node_9221
node_22497
node_22502
node_21187
node_18926 node_7854
node_8219 node_8344
node_8394
node_19067 node_8549
node_8998 node_9102
node_14080node_9213
node_21192
node_7502
node_13643node_7544 node_7586
node_13722node_7606
node_19311node_13887 node_7622
node_13802node_7662
node_21196
node_21202
node_21326
node_21331 node_21338
node_21343
node_21663node_21658
node_13393
node_16862node_13455 node_17007node_13523
node_21672
node_17123
node_17203
node_19703node_17454
node_21677
node_17503node_11809
node_11822
node_17555node_11835
node_21822
node_21828
node_11843
node_17816
node_6602
node_11863node_6604
node_6594
node_11883node_6596 node_6598
node_11903node_6600
node_21834
node_21839
node_17935 node_6620
node_11926node_6624 node_6630
node_6634
node_17998 node_6608
node_6610 node_6612
node_6616
node_21237
node_21242
node_21249
node_18928
node_18933
node_7722
node_7738
node_8174
node_8182
node_18937
node_8332
node_8336
node_8374
node_8378
node_18944
node_7766
node_8194
node_18948
node_8340
node_8390
node_19069
node_19071
node_19077
node_8439
node_8808
node_8447
node_8451
node_8895
node_8899
node_19082
node_9078
node_9082
node_9086
node_9126
node_14074
node_9159
node_9163
node_19091
node_8520
node_8919
node_19094
node_9098
node_14083node_9184
node_21253
node_19203
node_19206
node_7478
node_13633node_7520
node_13637 node_7482
node_7486 node_7524
node_7528
node_19210
node_7562
node_13712node_7594
node_13716node_7598 node_7566
node_7570
node_19217 node_13645
node_7498
node_13649node_7540
node_13724
node_7582
node_13728node_7602
node_19313node_13885
node_19317
node_7610
node_13792node_7650 node_7614
node_13796node_7654
node_19321
node_13879 node_13870
node_13874
node_19329
node_13804
node_7618
node_13808node_7658
node_19331
node_13890
node_13892
node_21260
node_21270 node_21278
node_21284
node_21685
node_21692
node_13389
node_16866node_13451
node_16869
node_13379
node_13383
node_16873
node_13441
node_13445
node_16879node_13395
node_16882node_13457
node_17010
node_13515
node_13517
node_17013node_13591
node_17017
node_13505
node_13509
node_17021node_13587
node_17026node_13525
node_17029node_13595
node_21702
node_11756
node_17262node_11760
node_17265
node_12258
node_12260
node_17269
node_12347
node_12349
node_17274node_12254
node_17277node_12345
node_11764
node_17308node_11768
node_17311node_12270
node_17315node_12266
node_12354
node_17324node_12358
node_21713
node_17255
node_11749
node_17244node_11752
node_17246node_12246
node_17249node_12242
node_19706
node_11654
node_17127node_11688
node_17136
node_17129node_11650
node_17132node_11684
node_17206node_11747
node_17210node_11745
node_17222node_17218node_12236
node_17229
node_17231
node_17234node_12336
node_12421
node_17458node_12500 node_17459
node_17463node_12417
node_17466node_12492
node_21730
node_21736
node_17507node_12594
node_17509node_12622
node_17514node_11803
node_17516node_12590
node_17521
node_11811
node_11814
node_17525
node_12616 node_12608
node_12611
node_17558node_12646
node_17561node_12676
node_17566node_11824
node_17569node_11837
node_17574
node_12636
node_12640
node_17579
node_12666
node_12670
node_21746
node_21751
node_21766node_21760
node_17388
node_17390
node_17396
node_6655
node_6657
node_6985
node_6987 node_6993
node_6995
node_17401
node_6650
node_6964
node_6966
node_6652
node_12320
node_6980 node_6972
node_6974
node_17411
node_7109
node_7111
node_17414
node_7105
node_12395node_7107
node_17422
node_17427
node_6673
node_6678
node_7009
node_7012
node_17434
node_17437
node_6661
node_6671
node_7003
node_7005
node_17445
node_7125
node_7131
node_17448
node_7115
node_7117
node_21782node_21777
node_17328
node_17331node_11772
node_17336
node_6638
node_12288node_6934 node_6640
node_12292node_6936
node_17340
node_12369
node_7093
node_12373node_7095
node_17345
node_12282 node_12274
node_12277
node_17352
node_12362
node_12365
node_17361
node_17366
node_6646
node_12308
node_6948
node_6950
node_6648
node_12312
node_6956
node_6958
node_17371
node_6642
node_12298
node_6938
node_6940
node_6644
node_12302node_6946
node_17377
node_12385
node_7101
node_12389node_7103
node_17380
node_12377
node_7097
node_12381node_7099
node_21890
node_21897
node_17818
node_11846
node_11848
node_17821
node_11869 node_6491
node_11867node_6493
node_17828node_17825node_12706
node_17833
node_12692
node_12696
node_12699
node_17840
node_7151
node_12724node_7153 node_6729
node_12728node_6731
node_17877
node_17883
node_11889 node_6495
node_11887node_6499
node_11909 node_6503
node_11907node_6507
node_17886
node_7155
node_12748node_7159 node_6733
node_12752
node_6743
node_6747
node_17890
node_7163
node_12771node_7167
node_12775 node_6759
node_6763 node_6775
node_6779
node_21904
node_21910
node_17939 node_6511
node_11929node_6517 node_6521
node_6525
node_17941
node_17944
node_17950
node_12795node_7171 node_6785
node_6789
node_7178 node_6804
node_6808
node_17957
node_7182
node_7186
node_6820
node_6824 node_6836
node_6840
node_18000
node_18003
node_18009
node_6530
node_6538
node_18012
node_6546
node_6558
node_18017
node_6871
node_7191
node_7202
node_6857
node_6861
node_18023
node_6877
node_6883
node_7220
node_7226
node_21917
node_21924 node_21933
node_21938
node_21354
node_21360 node_21369
node_21374
node_21489
node_21496
node_19161 node_19017
node_19020
node_21503
node_21509
node_19283
node_19413node_13947
node_21969
node_21977
node_18068
node_5839
node_5842
node_18071
node_5844
node_5846
node_18146 node_5814
node_5820 node_5824
node_11979node_5832
node_21983
node_21989
node_5852
node_11988node_5854 node_5856
node_12011node_5858
node_18275node_12049 node_5848
node_12031node_5850
node_22102
node_22108
node_12072
node_18344node_12080 node_18591node_12093
node_22115
node_13427
node_16946node_13489 node_13565
node_17092node_13617
node_22120
node_18640
node_19784node_18688
node_11674
node_11705
node_22041
node_22047
node_18074
node_18077
node_18082
node_5630
node_5644
node_18087
node_5650
node_5656
node_18092
node_5889
node_5904
node_6246
node_6259
node_18099
node_5910
node_5916
node_6265
node_6272
node_18149
node_5669
node_5674
node_18151
node_5678
node_11982node_5682
node_18155
node_18157
node_18164
node_6290
node_6295
node_5931
node_5935 node_5952
node_5956
node_18170
node_6299 node_5968
node_5972
node_12869node_6303 node_5984
node_5988
node_22053
node_18215
node_18221
node_11994 node_5687
node_11992node_5691
node_12017 node_5695
node_12015node_5699
node_18224
node_6307
node_12883node_6311
node_12887 node_6000
node_6004 node_6010
node_6014
node_18229
node_6315
node_12908node_6319
node_12912node_6042 node_6026
node_6030
node_18276
node_18279node_12964
node_18284
node_12037 node_5703
node_12035node_5705
node_18286
node_12060 node_12053
node_12056
node_18292
node_6323
node_12932node_6325 node_6052
node_12936node_6054
node_18297
node_12960 node_12953
node_12955
node_22073 node_22057
node_22066
node_21516
node_21521
node_21530
node_19024
node_19028
node_19034
node_8784
node_9071
node_19039
node_9122
node_9234
node_19044
node_8759
node_9059
node_9067
node_8776
node_8780
node_19052
node_9114
node_9118
node_9226
node_9230
node_19166
node_8150
node_8307
node_19168
node_8372
node_8434
node_14121node_8436
node_19172
node_19175
node_19180
node_8061
node_8288
node_8145node_8140
node_8294
node_8298
node_19186
node_8360
node_8364
node_8368
node_8416
node_14115
node_8429 node_8421
node_8425
node_21533
node_19285
node_19289
node_13703 node_7648
node_13701node_7684
node_13782 node_7702
node_13780node_7720
node_19292
node_13691 node_7640
node_7644 node_7683node_7678
node_7636
node_13695node_7674
node_19296
node_7698
node_13770node_7716 node_7694
node_13774node_7712
node_19416node_13949
node_19420
node_13862 node_7518
node_13860node_7560
node_19422
node_13971 node_13964
node_13967
node_19427
node_7514
node_13850node_7556 node_7512
node_13854node_7554
node_19431
node_13960
node_13955node_13953node_7592
node_21540
node_21548 node_21557
node_21562
node_22191
node_12982
node_18376node_13000
node_18379node_12074
node_18382node_12082
node_18387node_12978
node_18391node_12996
node_18595node_13024
node_18598node_13040
node_18602
node_12095
node_12097
node_18606
node_13017
node_13019
node_18612node_12106
node_18614node_13036
node_22194
node_22201
node_13241
node_18745node_13245
node_18748node_12177
node_18751
node_13282
node_13286
node_13289
node_18758node_12175
node_18760node_13278
node_13249
node_18778node_13253
node_18781node_12185
node_18784node_12181
node_18789
node_13305
node_13309
node_18794
node_13295
node_13299
node_22210
node_13437
node_16950node_13499
node_16953node_13429
node_16956node_13491
node_16961node_13433
node_16966node_13495
node_17094
node_13578
node_13580
node_17097node_13627
node_17101node_13567
node_17104node_13619
node_17108
node_13571
node_13574
node_17113node_13623
node_22218
node_13227
node_18727node_13233 node_18728
node_18732node_13270
node_18735node_13262
node_13064
node_18644node_13182 node_18645
node_18649node_13056
node_18652node_13175
node_17181
node_11676
node_17186node_11707
node_17188node_11680
node_17191node_11710
node_19786
node_13225
node_18690
node_18693node_13223
node_18699
node_18706node_13260
node_18720
node_18714node_12173
node_22236
node_22242
node_22251
node_18863
node_18868
node_5790
node_5792
node_18870
node_5788
node_5785
node_12226node_5883
node_18875
node_6403
node_6405
node_6208
node_6212 node_6221node_6216
node_18881
node_6401 node_6200
node_6204
node_13363node_6399 node_6192
node_6196
node_18894
node_18899
node_5801
node_5805
node_18902
node_5794
node_5799
node_18906
node_6232
node_6419
node_6421
node_6238
node_6242
node_18914
node_6222
node_6228
node_6407
node_6417
node_22256
node_18804
node_18807node_13257
node_18812
node_12204 node_5773
node_12202node_5775
node_18815
node_6166
node_13331node_6389 node_6162
node_13335node_6387
node_18821
node_12196 node_12189
node_12192
node_18825
node_13323 node_13315
node_13319node_6160
node_18836
node_18841
node_12220 node_5781
node_12218node_5783
node_18844
node_12212 node_5777
node_12210node_5779
node_18849
node_13351node_6397 node_6184
node_6188
node_13355node_6395node_6183node_6178
node_18854
node_6174
node_13341node_6393 node_6170
node_13345node_6391
node_22290
node_22294
node_22300
node_16820
node_16823node_10766
node_22307
node_22311 node_22317
node_22323
node_22431
node_22438
node_11100
node_16838node_11102 node_11104
node_16841node_11106
node_22446
node_10782
node_16827node_10804 node_10833
node_16831node_10853
node_22451
node_11094
node_19643
node_10965
node_11031
node_11096
node_11098
node_22459
node_22467
node_16854
node_16857
node_22474
node_22480
node_16850 node_16843
node_16846node_11108
node_21211
node_21216
node_21225
node_19254
node_19259
node_7692
node_13760node_7710 node_7688
node_13764node_7706
node_19264
node_7634
node_13681node_7672
node_13685node_7670 node_7626
node_7628
node_7671
node_7666
node_19272
node_13750
node_7686
node_13754node_7704
node_19275
node_13671
node_7624
node_13675node_7664
node_19380node_19377node_13945
node_19385
node_13933
node_13935
node_13939node_7588
node_19394
node_7510
node_13840node_7552 node_7506
node_13844node_7548
node_19403
node_13922
node_13924
node_13926
node_19406
node_13830
node_7504
node_13834node_7546
node_21229
node_18988
node_18993
node_9106
node_9112
node_9217
node_9224
node_18999
node_8744
node_9042
node_9053
node_8717
node_8719
node_19006
node_9104
node_9215
node_19009
node_8715
node_9040
node_19131
node_19136
node_14105node_8414 node_8399
node_8402
node_8404
node_8358 node_8348
node_8350
node_19145
node_7960
node_7995
node_8286
node_7961
node_7956
node_8251
node_8253
node_19157 node_7950
node_8245 node_8346
node_14101node_8396
node_21293
node_21300 node_21311
node_21316
node_21598
node_14145
node_14139 node_14131
node_14134
node_18661
node_13008
node_13011
node_18665
node_12086
node_12088
node_18671node_12990
node_18674node_12078
node_18680node_13004
node_18683node_12986
node_12102
node_12108
node_13032
node_13048
node_13028
node_13044
node_21608
node_18630
node_18623node_13274
node_18626node_13237
node_13230
node_18637node_13266
node_19762
node_17174
node_11670
node_17163node_11702
node_17165node_11698
node_17168node_11666
node_13185
node_18446node_18444node_13187
node_18457
node_18458
node_18453node_12112
node_18465node_18460node_13068
node_13171
node_18427node_13179
node_18436
node_18429node_13060
node_18432node_13052
node_21623
node_21630
node_13189
node_18348node_13193
node_18351
node_13080
node_13084
node_18356
node_13070
node_13074
node_18363node_12118
node_18366node_12114
node_13197
node_18402node_13201
node_18405node_13103
node_18408node_12126
node_18412
node_13097 node_13090
node_13093
node_18417node_12122
node_21641
node_21647
node_13423
node_16922node_13485
node_16925node_13481
node_16930node_13419
node_13415
node_16940node_13477
node_17066node_13613
node_17068
node_13557
node_13561
node_17073node_13609
node_17076
node_13550
node_13553
node_13546
node_17086node_13607
node_21846
node_18240
node_6301
node_6305
node_5976
node_5978 node_5992
node_5994
node_18244
node_6297 node_5960
node_5962
node_12843node_6292 node_5939
node_5942
node_5946
node_18254
node_18256
node_18265
node_5680
node_5685
node_18267
node_5676
node_5671
node_11961node_5837
node_18311
node_5912
node_5920
node_6267
node_6274
node_18318
node_5892
node_5906
node_6249
node_6261
node_18323
node_18327
node_18334
node_5652
node_5658
node_18337
node_5633
node_5646
node_21854
node_18111
node_6056
node_12918node_6058 node_6327
node_12922node_6329
node_18116
node_12894
node_12898
node_18125
node_12021
node_5707
node_12025node_5709
node_18128
node_12005 node_11998
node_12000
node_18135
node_18138node_12891
node_18207
node_18185
node_12968 node_6034
node_6036 node_6044
node_6046
node_6317
node_12972node_6321
node_18191
node_6008
node_12942
node_6018
node_6020
node_6309
node_12946node_6313
node_18198
node_12064
node_5697
node_12068node_5701
node_18201
node_12041
node_5689
node_12045node_5693
node_21862
node_21867
node_21876
node_21881
node_21571
node_21577
node_18531
node_18536
node_6090
node_13119node_6361 node_6086
node_13123node_6359
node_18540
node_6084
node_13109node_6357
node_6085
node_6080
node_13113node_6355 node_6072
node_6076
node_18547
node_12136
node_5721
node_12140node_5723
node_18550
node_12128
node_5717
node_12132node_5719
node_18561node_18559node_13205
node_18566
node_13137
node_13141
node_13145node_6070
node_18572
node_12152
node_12154
node_12156
node_18578
node_6066
node_13129node_6353 node_6062
node_13133node_6351
node_18582
node_12144
node_5713
node_12148node_5715
node_21581
node_21589
node_18476
node_18481
node_6134
node_6138
node_6383
node_6385
node_18486
node_6124
node_6130
node_6371
node_6373
node_18494 node_5734
node_5736 node_5741
node_5743
node_18500
node_18505
node_13165node_6369 node_6116
node_6120
node_6367 node_6108
node_6112
node_18513
node_6098
node_6363
node_6365
node_6099
node_6094
node_6100
node_6104
node_18521
node_5729
node_12171node_5731
node_18524
node_5725
node_5727
node_21793
node_21799 node_21808
node_21813
node_21383
node_21388 node_21396
node_21402
node_21447node_21442
node_19244
node_19224
node_13742node_7600 node_7574
node_7576
node_7584
node_13746node_7604
node_19228
node_13663 node_7490
node_7492 node_7532
node_7534
node_7500
node_13667node_7542
node_19235
node_13732
node_7564
node_13736node_7596
node_19238
node_13653
node_7480
node_13657node_7522
node_19340
node_13908
node_13911
node_13915
node_19346
node_7616
node_13822node_7656 node_7620
node_13826node_7660
node_19352
node_13897
node_13899
node_19356
node_13812
node_7612
node_13816node_7652
node_19366node_19363node_13904
node_21463node_21455
node_18982
node_18958
node_8338
node_8342
node_8392 node_8382
node_8384
node_18963
node_7752
node_7768
node_8188
node_8196
node_18972
node_8334
node_8376
node_18975
node_7724
node_8176
node_19103
node_14091node_9190 node_9167
node_9170
node_9172
node_9100 node_9090
node_9092
node_19110
node_8526
node_8929
node_8486
node_8488
node_8903
node_8905
node_19116
node_19119
node_19127 node_8441
node_8818 node_9080
node_14087node_9133
node_21475
node_21480
node_22002node_21997
node_17990
node_17970
node_12734node_6783 node_6767
node_6769
node_7165
node_12738node_7169
node_17974
node_12710 node_6735
node_6737 node_6751
node_6753
node_7157
node_12714node_7161
node_17981
node_11873
node_6505
node_11877node_6509
node_17984
node_11853
node_6497
node_11857node_6501
node_18034
node_12782
node_12784
node_12787
node_18041
node_11913
node_11915
node_11917
node_18047
node_6725
node_12758node_6727 node_7147
node_12762node_7149
node_18051
node_11893
node_6487
node_11897node_6489
node_18061node_18058node_12779
node_22018node_22011
node_17850
node_6879
node_7222
node_7228
node_6885
node_6887
node_17854
node_6865
node_6873
node_7193
node_7206
node_17859
node_17862
node_17871 node_6532
node_6540 node_6548
node_6560
node_17900
node_12829node_7188 node_6844
node_6847
node_6849
node_7184 node_6828
node_6830
node_17908
node_7176
node_7180
node_6796
node_6798 node_6812
node_6814
node_17914
node_17917
node_17925
node_6523 node_6527
node_11951node_6614
node_17928
node_6515
node_6519
node_22028
node_22032
node_22129
node_19744
node_17151
node_17144node_11694
node_17147node_11662
node_11658
node_17158node_11691
node_17653
node_17657node_12504
node_17664node_11783
node_17670node_11781
node_17684
node_17680node_17678node_12424
node_17543
node_17536node_12341
node_17539node_12250
node_12238
node_17550node_12338
node_17648
node_17636node_12496
node_17639node_12488
node_11774
node_17645node_11778
node_22143
node_16891
node_13469
node_13473
node_16895
node_13407
node_13411
node_16901node_13461
node_16904node_13399
node_16911node_13465
node_16914node_13403
node_17037node_13603
node_17040node_13599
node_17044
node_13537
node_13541
node_17049node_13533
node_17057node_13597
node_17059node_13529
node_22163node_22156
node_17590node_12510
node_17594node_12506
node_11785
node_17601node_11789
node_12426
node_17607node_12430
node_17612node_12522
node_17615node_12441
node_17619
node_12514
node_12518
node_17623
node_12434
node_12437
node_11793
node_17631node_11797
node_22180node_22174
node_17285node_12630
node_17289node_12602
node_11805
node_17296node_11818
node_12598
node_17302node_12626
node_17475node_12684
node_17478node_11841
node_17482
node_12658
node_12662
node_17487
node_11828
node_11831
node_17494node_12680
node_17496
node_12650
node_12654
node_21413
node_21419 node_21426
node_21431
node_21947
node_21950 node_21955
node_21960
node_22082node_22079
node_17774
node_17754
node_6692
node_12538node_7034 node_6694
node_12542
node_7036
node_7038
node_17758
node_6688
node_12528
node_7018
node_7020
node_6690
node_12532
node_7026
node_7028
node_17765
node_12451
node_6905
node_12455node_6907
node_17768
node_12443
node_6901
node_12447node_6903
node_17781
node_12556
node_12558
node_12561
node_17789
node_12467
node_12469
node_12471
node_17795
node_6684
node_12548node_7014 node_6686
node_12552node_7016
node_17799
node_12459
node_6897
node_12463node_6899
node_17808node_17806node_11801
node_22094node_22092
node_17711
node_17691
node_6719
node_6723
node_7085
node_7091
node_17696
node_6707
node_6709
node_7081
node_7083
node_17702
node_6928
node_6930
node_17705
node_6918
node_6920
node_17718
node_6700
node_12582
node_7060
node_7063
node_7065
node_6703
node_7073
node_7075
node_17728
node_6696
node_6698
node_7044
node_7046 node_7052
node_7054
node_17735
node_17738
node_17745
node_6913
node_12486node_6916
node_17748
node_6909
node_6911
node_22274 node_22263
node_22268
node_22279
node_22282
node_22332
node_22335 node_22340
node_22345
node_22381
node_22384 node_22389
node_22393
node_19955
node_19959
node_19964
node_19975node_19974
node_12822node_6345
node_19984node_19981
node_6335
node_12690node_6337 node_6339
node_12720node_6341
node_16792node_12766 node_6331
node_12744node_6333
node_20031
node_20037
node_16766
node_16773node_16771node_12403
node_20042
node_20047
node_16755
node_16759
node_16762
node_20056
node_12588
node_16748node_12606 node_16751node_12634
node_20061
node_16753
node_19624node_16786 node_12413
node_12415
node_20067
node_20073
node_12405
node_16779node_12407 node_12409
node_16783node_12411
node_19814node_19804
node_16554
node_9822
node_9826
node_9828
node_16559node_9756
node_16564node_9835
node_16567node_9760
node_9752
node_16574node_9818
node_16693
node_9962
node_9966
node_16698
node_9898
node_9902
node_16705node_9972
node_16708node_9908
node_9894
node_16716node_9958
node_19831
node_19838
node_19853
node_19858
node_19867
node_11473
node_15940node_11477
node_15943
node_10927
node_10931
node_15947
node_10919
node_10923
node_15953node_11012
node_15956node_11008
node_15987node_11489
node_15989node_10947
node_15994
node_11481
node_11485
node_15997
node_10942 node_10935
node_10938
node_16006node_11023
node_16008
node_11016
node_11019
node_16465
node_16458node_11638
node_16461node_11605
node_11602
node_16472node_11634
node_19518
node_16037node_16033node_11506
node_16042node_11504
node_16045node_10963
node_16060 node_16053
node_16056node_11029
node_16107
node_11515
node_16096node_11551
node_16098node_11090
node_16101node_11215
node_16027
node_11493
node_16016node_11501
node_16018node_10959
node_16021node_10955
node_16134node_11428
node_16136node_10778
node_16141node_11438
node_16144
node_10796
node_10800
node_16151node_10776
node_16153node_10792
node_16385node_11452
node_16388node_11463
node_16393
node_10825
node_10829
node_16397
node_10845
node_10849
node_16405node_10821
node_16408node_10841
node_20843
node_14528
node_14503
node_6142
node_6349
node_6144
node_6146
node_6375
node_6377
node_14508
node_5865
node_5867
node_5869
node_5745
node_9270
node_5747
node_5750
node_5754
node_14518
node_6236
node_6454
node_14521
node_5803
node_9276node_5887
node_14665
node_14646
node_6869
node_6875
node_7198
node_7218
node_14650
node_6536
node_6542
node_6622
node_6626
node_14656
node_6895
node_7232
node_14659
node_6564
node_6636
node_20850
node_14252
node_7298
node_9530node_7338 node_7296
node_9534node_7336
node_14257
node_9442 node_9433
node_9437
node_14267
node_9538
node_7304
node_9542node_7344
node_14269
node_9455 node_9448
node_9450
node_14276
node_14279node_9430
node_14398
node_14381
node_9690 node_7416
node_7418 node_7458
node_7460
node_7414
node_9694node_7456
node_14386
node_7354
node_9610
node_7374
node_7376
node_7352
node_9614node_7372
node_14394 node_9618
node_7360
node_9622node_7392
node_9698
node_7434
node_9702node_7476
node_20858
node_20864
node_20873
node_20877
node_20363
node_20368
node_20376
node_15593node_11417
node_15598
node_10644
node_10646 node_10653node_7934
node_15605
node_15608
node_8241
node_10685node_8243 node_7940
node_10689node_7946
node_15617
node_10637
node_10639
node_15621
node_10675
node_7938
node_10679node_7944
node_15666
node_15671
node_8223
node_10721node_8227 node_7874
node_10725node_7888
node_15676
node_8231
node_10758node_8235 node_7904
node_10762
node_7905
node_7900
node_7914
node_7916
node_15684
node_10711
node_7870
node_10715node_7884
node_15687
node_10748
node_7896
node_10752node_7910
node_20384
node_15729
node_15734
node_10569node_8204 node_7794
node_7796
node_8209 node_7814
node_7816
node_15743
node_7860
node_8213
node_8217
node_7834
node_7836
node_7861
node_7856
node_15751
node_7789
node_10565node_7810
node_15754
node_7830
node_7850
node_15793
node_15796
node_7734
node_7748
node_8180
node_8186
node_15802
node_7762
node_7779
node_8192
node_8200
node_15810
node_7730
node_7744
node_15813
node_7758
node_7775
node_20394
node_20399
node_20947
node_20953 node_20960
node_20965
node_20188
node_20193
node_20200
node_15882
node_15885
node_8047
node_10887node_8278
node_10891node_8280 node_8049
node_8051
node_15889
node_10877node_8274 node_8031
node_8033
node_10881node_8276 node_8039
node_8041
node_15896
node_10985
node_8166
node_10989node_8168
node_15899
node_10977
node_8162
node_10981node_8164
node_15908node_11469
node_15912
node_10905
node_10908
node_10912
node_15919
node_11001
node_11003
node_15926
node_8057
node_10897node_8282 node_8059
node_10901node_8284
node_15930
node_10993
node_8170
node_10997node_8172
node_20208
node_15825
node_15830
node_7991
node_7997
node_8259
node_8263
node_15835
node_7987
node_7989
node_8247
node_8249
node_15841
node_8148
node_8152
node_15844
node_8138
node_8146
node_15852
node_15855
node_15860
node_10869node_8269 node_8015
node_8017
node_8272 node_8023
node_8025
node_15868
node_8265
node_8267
node_7999
node_8001 node_8007
node_8009
node_15878 node_8154
node_8156 node_8158
node_10975node_8160
node_20216
node_20222
node_20334
node_20340 node_20348
node_20353
node_20813
node_20819 node_20828
node_20833
node_20540
node_20546
node_15320
node_15323
node_15328
node_8753
node_9050
node_9057
node_8734
node_8738
node_15336
node_8768
node_8797
node_9063
node_9076
node_15344
node_8727
node_8747
node_15347
node_8762
node_8787
node_15391
node_15396
node_8653
node_9026
node_9030
node_8654
node_8649
node_8667
node_8669
node_15403
node_9034 node_8687
node_8689
node_10553node_9038 node_8705
node_8707
node_15417 node_8639
node_8657 node_8677
node_10547node_8697
node_20554
node_20560
node_15461
node_15466
node_9010
node_10382node_9014
node_10386node_8605 node_8585
node_8587
node_8606
node_8601
node_15471
node_9018
node_10418node_9022 node_8617
node_10422node_8633
node_15481 node_10372
node_8579
node_10376node_8595
node_10408
node_8609
node_10412node_8623
node_15526
node_15529node_11415
node_15534
node_9000
node_10454node_9002 node_8553
node_10458node_8561
node_15539
node_10491node_10488node_8569
node_15549
node_10444
node_8551
node_10448node_8557
node_15551
node_10480
node_10482
node_20567
node_20573
node_20681
node_20688
node_11374
node_14939node_11384
node_14942node_10278
node_14946node_10304
node_14952node_10274
node_14955node_10300
node_14982node_11394
node_14985
node_10324
node_10327
node_14992node_11404
node_14994node_10345
node_15001node_10320
node_15004node_10343
node_20697
node_9740
node_16533node_9804
node_16536node_9814
node_16541node_9748
node_9746
node_16549node_9810
node_9880
node_16670node_9946
node_16673node_9954
node_16678node_9890
node_9886
node_16688node_9950
node_20709
node_15052
node_11287
node_15041node_11293
node_15043node_10092
node_15046node_10127
node_16452
node_11564
node_16441node_11626
node_16443node_11630
node_16446node_11598
node_19497
node_11301
node_15248node_15246node_10161
node_15260node_15255node_11366 node_15264
node_15270node_10266
node_10150
node_15229node_10158 node_15230
node_15234node_10262
node_15237node_10254
node_20726
node_20732
node_11348
node_15175node_11352
node_15178node_10138
node_15181node_10134
node_15186
node_10234
node_10238
node_15191
node_10226
node_10230
node_11356
node_15205node_11360
node_15208node_10146
node_15210node_10250
node_15215node_10142
node_15218
node_10242
node_10246
node_21003
node_21010 node_21018
node_21023
node_21082
node_21087
node_21095
node_14465
node_14467
node_14474
node_6682
node_7089
node_7123
node_6711
node_6713
node_7124
node_7119
node_14480
node_6580
node_6584
node_6586
node_6472
node_9262
node_6477
node_6479
node_14493
node_6705
node_7113
node_14496
node_6475
node_9256node_6582
node_14615
node_14620
node_5914
node_5925
node_6269
node_6280
node_6284
node_14626
node_5654
node_5663
node_5822
node_5828
node_14634
node_5923
node_6277
node_14637
node_5661
node_5826
node_21104
node_14218
node_14221node_9409
node_14226
node_7402
node_9520node_7444 node_7406
node_9524node_7448
node_14231
node_9422
node_9426
node_9423
node_7364
node_14240
node_9510
node_7404
node_9514node_7446
node_14243
node_9415
node_9417
node_14351
node_14356
node_7280
node_9680node_7318 node_7288
node_9684
node_7289
node_7284
node_7322
node_7324
node_14362
node_7242
node_9600node_7260 node_7246
node_9604node_7264
node_14369
node_9670
node_7282
node_9674node_7320
node_14372
node_9590
node_7244
node_9594node_7262
node_21114
node_21117
node_20507 node_20493
node_20499
node_20628
node_20636
node_15119
node_15125
node_11321 node_9201
node_11319node_9203
node_11329 node_9205
node_11327node_9207
node_15128
node_8510
node_10195node_8984 node_8508
node_10199node_8980
node_15132
node_8506
node_10185
node_8979node_8974
node_8504
node_10189
node_8966
node_8970
node_15142
node_11341
node_11343
node_15145
node_11337 node_9209
node_11335node_9211
node_15149node_10131
node_15153
node_10213
node_10216
node_10220node_8996
node_15161
node_15164
node_8514
node_10205node_8992 node_8512
node_10209node_8988
node_20642
node_20647
node_15060
node_15066 node_9180
node_9182 node_9186
node_9188
node_15069
node_8484
node_8494
node_8921
node_8925
node_15075
node_8480
node_8482
node_8911
node_8915
node_15086
node_15089
node_15094
node_11311
node_9197
node_11312
node_9199
node_15097
node_9193
node_9195
node_15101
node_8502
node_10177
node_8953
node_8956
node_8960
node_8500
node_8945
node_8949
node_15108
node_8496
node_8498
node_8936node_8931
node_8937
node_8941
node_21032
node_21037 node_21045
node_21050
node_19995
node_20002
node_7135
node_12879node_7137 node_7139
node_12904node_7141
node_16797node_12950 node_7143
node_12928node_7145
node_20008
node_12864node_7133
node_20026 node_20012
node_20019
node_20082
node_16816node_13217
node_13219
node_13221
node_20086
node_16805
node_16811
node_16813
node_20091
node_20099
node_12976
node_16800node_12994 node_16803node_13015
node_20141node_20137
node_13208node_7200
node_7204
node_7208
node_7210
node_7212 node_7214
node_7216
node_20154node_20152node_13215
node_20238
node_11378
node_15059node_11388 node_11398
node_15284node_11408
node_20243
node_11296
node_19475 node_11303
node_11368 node_11562
node_11624
node_20249
node_20256
node_9738
node_16528node_9801 node_9878
node_16640node_9944
node_20409
node_15485
node_15487 node_15557
node_15560
node_20413
node_15421 node_15355
node_15358node_11413
node_20418
node_20423
node_20886
node_20891
node_20901
node_14345node_14212node_9407
node_20904
node_14459
node_14608
node_20914
node_20920 node_20927
node_20936
node_20265
node_20271
node_14881
node_10102
node_10104
node_14884node_10098
node_14888
node_10057
node_10060
node_14893
node_11270
node_11272
node_14899node_10053
node_14902node_11268
node_10110
node_14911node_10114
node_14914
node_10074
node_10078
node_14919
node_10066
node_10070
node_14926node_11281
node_14929node_11277
node_20280
node_14152node_10339
node_14155node_10316
node_14160
node_10331
node_10335
node_14164
node_10308
node_10312
node_11390
node_14173node_11400
node_15012
node_10293
node_10295
node_15016
node_10289 node_10282
node_10284
node_15022node_10272
node_15024node_10268
node_15031node_11380
node_15034node_11370
node_20287
node_10120
node_14864node_10123
node_14873
node_14866node_10088
node_14869node_10084
node_14963
node_14967node_10258
node_14970node_10154
node_11298
node_14976node_11362
node_19458
node_11291 node_14829
node_14832node_10118
node_14843node_14839node_10082 node_14849
node_14852node_11285
node_16416
node_11559
node_16421node_11620
node_16423node_11555
node_16426node_11616
node_20303
node_16496node_9732
node_16498
node_9786
node_9788
node_16504node_9734
node_16506node_9728
node_16511
node_9793
node_9797
node_16515
node_9780 node_9772
node_9775
node_16607node_9870
node_16610node_9936
node_16615node_9874
node_16618node_9940
node_16623
node_9860
node_9864
node_16628
node_9926
node_9930
node_20308
node_20315
node_14765
node_14767
node_14773
node_8464
node_8466
node_8861 node_8849
node_8853
node_8862
node_8857
node_14779
node_8462
node_8841
node_8845
node_8460
node_9994
node_8836 node_8828
node_8832
node_14788
node_9143
node_9145
node_14790
node_9138
node_11225node_9141
node_14798
node_14803
node_8455
node_8457
node_8814
node_8820
node_8824
node_14811
node_8443
node_8445
node_8804
node_8810
node_14819
node_9131
node_9135
node_14822
node_9124
node_9129
node_20321
node_20327
node_14709
node_14712node_10096
node_14718
node_8478
node_10025node_8891 node_8476
node_10029node_8887
node_14722
node_11244
node_9155
node_11248node_9157
node_14726
node_10018 node_10010
node_10014node_8885
node_14734
node_11240 node_11233
node_11235
node_14743
node_14748
node_8474
node_10045
node_8877
node_8881
node_8472
node_10049node_8875
node_8876
node_8871
node_14752
node_8470
node_10035node_8867 node_8468
node_10039node_8863
node_14759 node_11252
node_9147
node_11256node_9149
node_11260
node_9151
node_11264node_9153
node_20432
node_20440
node_20448
node_15427 node_8675
node_10503
node_10504
node_8655
node_8695
node_8713
node_15430
node_9032
node_9036
node_8679
node_8683 node_8704node_8699
node_15437
node_9028 node_8659
node_8663
node_10499node_9024 node_8641
node_8645
node_15448
node_15451
node_15494
node_8772
node_8801
node_15497
node_8742
node_8757
node_15502
node_8764
node_9061
node_9073
node_8789
node_8793
node_15507
node_8730
node_8749
node_9048
node_9055
node_15514
node_15517
node_20453
node_15287
node_10404 node_8571
node_10402node_8577
node_15290
node_10368 node_10361
node_10364
node_15296
node_8565
node_10392node_8573 node_9004
node_10396node_9006
node_15301
node_10356
node_10352node_10350node_8559
node_15311
node_15314node_11410
node_15385
node_15363
node_10476 node_8621
node_10474node_8637
node_15366
node_10440 node_8593
node_10438node_8607
node_15371
node_10464 node_8616node_8611
node_8625
node_8629
node_9016
node_10468node_9020
node_15377
node_8581
node_10428node_8597 node_9008
node_10432node_9012
node_20460
node_20468 node_20478
node_20484
node_14178node_9405
node_14182
node_9398
node_9401
node_14185
node_9502 node_7398
node_9500node_7440
node_14189
node_9384
node_9391node_9389node_7362
node_14195
node_7394
node_9490node_7436 node_7400
node_9494node_7442
node_14321
node_14325
node_9582 node_7238
node_9580node_7256
node_9662 node_7276
node_9660node_7314
node_14328
node_7234
node_9570node_7252 node_7240
node_9574node_7258
node_14332
node_9650 node_7275node_7270
node_7306
node_7310
node_7278
node_9654node_7316
node_14435 node_9244
node_6576
node_9245
node_6468
node_6676
node_6932
node_14437
node_14442
node_9238node_6470 node_6460
node_6464
node_6578 node_6568
node_6572
node_14448
node_6680
node_7001
node_6663
node_6667
node_6927node_6922
node_14574
node_14576
node_14582
node_5642
node_5816
node_14585
node_5902
node_6257
node_14590
node_5638
node_5648
node_5810
node_5818
node_14596
node_5898
node_5908
node_6253
node_6263
node_20514
node_15762
node_15822node_11422
node_20519
node_15627
node_15695
node_15698
node_20524
node_20530
node_20655
node_16414
node_19560node_16493
node_11508
node_11513
node_20660
node_16603node_9846
node_9916
node_16744node_9980
node_20672node_20668
node_11432
node_16068node_11442 node_11456
node_16114node_11467
node_20974
node_20979 node_20987
node_20993
node_21059
node_9470
node_14320
node_7292
node_9556node_7332
node_7348
node_9636node_7368 node_7410
node_9716node_7452
node_21062
node_14572 node_5711
node_9284node_5860 node_6060
node_6343
node_14707 node_6513
node_6606 node_6793
node_7174
node_21066
node_21071
node_20750node_20744
node_16332
node_11179
node_11183
node_16336
node_11524
node_11526
node_16341node_11175
node_16344node_11520
node_16350node_11076
node_16353node_11074
node_16359
node_11199
node_11203
node_16363
node_11189
node_11193
node_16369node_11084
node_16372node_11080
node_11532
node_16380node_11536
node_20758
node_16326
node_16314node_11219
node_16317node_11211
node_11544
node_16323node_11547
node_19543
node_16277node_11209
node_16283node_11542
node_16288
node_16292node_11540
node_16300
node_16305node_16303node_11088
node_11613
node_16477node_11646
node_16486
node_16479node_11609
node_16482node_11642
node_16115
node_16119node_11025
node_16122node_10951
node_11497
node_16128node_11510
node_20770
node_9768
node_16580node_9856
node_16583node_9764
node_16586
node_9848
node_9852
node_16592node_9762
node_16594
node_9839
node_9842
node_9922
node_16722node_9986
node_16725node_9918
node_16729node_9982
node_9912
node_16736node_9976
node_20781
node_10772
node_15966node_10788
node_15969node_10784
node_15973node_10768
node_11424
node_15980node_11434
node_16072node_10839 node_10814
node_10816
node_16075node_10835
node_16078node_11459
node_16082
node_10806
node_10810
node_16086
node_11444
node_11448
node_20791node_20787
node_16209
node_8316
node_8318
node_8090
node_8092 node_8098
node_8100
node_16214
node_8311 node_8074
node_8076
node_11114node_8313 node_8082
node_8084
node_16223
node_7971
node_7973
node_16226
node_7967
node_11035node_7969
node_16232
node_16234
node_16242
node_8069
node_8072
node_8304
node_8309
node_16248
node_8063
node_8065
node_8292
node_8302
node_16255
node_7962
node_7965
node_16259
node_7952
node_7954
node_16266
node_16270
node_20804node_20800
node_16162
node_8132
node_11147node_8328 node_8134
node_11151node_8330
node_16166
node_11139 node_11130
node_11134
node_16174
node_11050
node_7983
node_11054node_7985
node_16176
node_11043
node_11046
node_16183node_11518
node_16204
node_16189
node_11167node_8324 node_8116
node_8118
node_11171node_8326 node_8124
node_8126
node_16193
node_11157node_8320 node_8106
node_8108
node_8114
node_11161node_8322
node_16200 node_11058
node_7975
node_11062node_7977
node_11066
node_7979
node_11070node_7981
node_21127
node_21132
node_21142
node_14288node_14286node_9486
node_14293
node_9474
node_9476
node_9479
node_14300
node_7302
node_9560node_7342 node_7300
node_9564node_7340
node_14306
node_9459
node_9461
node_9463
node_14310
node_9546
node_7294
node_9550node_7334
node_14404
node_14409
node_7358
node_9640node_7390 node_7356
node_9644
node_7382
node_7386
node_14413
node_7432
node_9720node_7474
node_9724 node_7424
node_7428 node_7466
node_7470
node_14420
node_9626
node_7350
node_9630node_7370
node_14423
node_9706
node_7412
node_9710node_7454
node_21146
node_14534
node_14537
node_14542
node_9287node_5796
node_5768 node_5760
node_5762
node_5885 node_5875
node_5879
node_14552
node_6226
node_6448
node_6152
node_6156
node_6409
node_6413
node_14560
node_5738
node_9280node_5863
node_14563
node_6128
node_6347
node_14671
node_14673
node_14680
node_6562
node_6628
node_6632
node_6550
node_6554
node_14685
node_6881
node_6893
node_7224
node_7230
node_14694
node_6534
node_6618
node_14697
node_6867
node_7195
node_21153
node_21159 node_21171
node_21176
node_20579
node_20584
node_20592
node_15725
node_15703
node_10671 node_7898
node_10669node_7912
node_15706
node_10633 node_7872
node_10631node_7886
node_15711
node_7892
node_10659node_7906 node_8229
node_10663node_8233
node_15716
node_10621 node_7862
node_7866 node_7883node_7878
node_8221
node_10625node_8225
node_15765
node_10741
node_10744
node_15768
node_10707 node_7926
node_10705node_7932
node_15773
node_10729
node_10734node_10732node_7936
node_15779
node_7922
node_10695node_7928 node_8237
node_10699node_8239
node_15788node_11420
node_20595
node_15586
node_15565
node_7760
node_7777
node_15568
node_7732
node_7746
node_15573
node_7754
node_7771
node_8190
node_8198
node_15578
node_7726
node_7740
node_8178
node_8184
node_15632
node_7832node_10608node_7852
node_15635
node_7792
node_7812
node_15640
node_10601node_8215 node_7842
node_7846
node_8211 node_7822
node_7826
node_15649
node_8202
node_8207
node_7788node_7783
node_7802
node_7806
node_15657
node_15660
node_20601
node_20605 node_20612
node_20618
node_19906
node_7250
node_9413node_7268 node_7290
node_9508node_7330
node_14614node_9666 node_7346
node_9588node_7366
node_19909
node_14216 node_5667
node_5835 node_5929
node_6288
node_14349 node_6485
node_6592 node_6721
node_9363node_7128
node_19914
node_19919
node_19928
node_19933 node_19941
node_19946
node_20109
node_16529
node_16655
node_16659
node_20115
node_14129
node_9744
node_9808 node_16437node_9884
node_20127node_20123
node_16662node_11592
node_11594
node_16666node_11596
node_20167node_20164
node_11576
node_16644
node_8545
node_11582node_8547
node_8537
node_11586node_8539 node_8541
node_11590node_8543
node_20179node_20175
node_16649 node_8528
node_11569node_8531 node_8533
node_8535
node_16653 node_8516
node_8518 node_8522
node_8524
node_19879
node_19887 node_19895
node_19900
node_14205
node_5635
node_5808
node_14208
node_5896
node_6251
node_14343 node_6458
node_6566 node_9360node_6659
PrimitiveObject 0
PrimitiveObject 1
PrimitiveObject 2
PrimitiveObject 0
PrimitiveObject 1
PrimitiveObject 2
(coherence queues)
Automatically finds coherence paths
The Scene Hierarchy Generator
System Memory Bus
System Memory Interface
Scene Hierarchy Generator
Course Grain
SchedulerVertex
Data Master
USC0 USC1Texture Unit
USCn-1 USCnTexture Unit
Unified Shading Cluster Array
World-space Trianglesand associated varyings
Limitations
• Scene is represented by triangles – same as today
• BVH is in a defined format optimised for construction and traversal
• Triangle order must generally follow a spatially coherent flow
• An approximate scene scale estimation is needed
• Geometry shaders are not inline with the ray tracing pipeline
Strengths• Shading cluster workload is no higher than a vertex shader
• Only needs to process geometry that actually moved in world space
• Unique algorithm constrains working set to internal registers only
• Single pass operation: in-line with vertex shader execution
• Handles the “long skinny triangles” problem well
• Streaming writes to external memory
• Losslessly compressed output formats due to build algorithm
• Compact logic
Built on sparse log2 oct-tree scaffolding
LOD = 1 LOD = 2 LOD = 3Level = 1 Level = 2 Level = 3
ProcessTriangle() AssembleParents()
Leaf VoxelCache
VBExtents
On
evic
tion Tree
VoxelCache
Output Scene Acceleration
Structure
Select LOD
For each triangle
Foreach VBNode in
LOD
Compute Parent Voxel
On VoxelCache HIT
VBNode Pool
Per LOD VBNode linked list
headpointers
GenerateVBNode()
v0
v1
v2
On
evic
tion
Add VBNode
After some triangles...
LOD16
LOD15
LOD10
LOD14
LeafCount=8TreeCount=0
LeafCount=3TreeCount=0
LeafCount=0TreeCount=0
LeafCount=3TreeCount=0
Leaf Head
Tree Head
Leaf Head
Tree Head
Leaf Head
Tree Head
Leaf Head
Tree Head
Assembling parents...
LOD16
LOD15
LOD10
LOD14
LeafCount=8TreeCount=0
LeafCount=3TreeCount=0
LeafCount=0TreeCount=0
LeafCount=3TreeCount=0
Leaf Head
Tree Head
Leaf Head
Tree Head
Leaf Head
Tree Head
Leaf Head
Tree Head
Tree VoxelCache
On eviction
Flush Leaf Nodes
VBNode Pool
GenerateVBNode()
Add VBNode to linked list of
VBNodes for its LOD
After parents for one level have been assembled...
LOD16
LOD15
LOD10
LOD14
A0 A1 B0 C0 C1 B1 B2 C2LeafCount=8TreeCount=0
LeafCount=3TreeCount=0
LeafCount=0TreeCount=0
LeafCount=3TreeCount=0
Leaf Head
Tree Head
Leaf Head
Tree Head
Leaf Head
Tree Head
Leaf Head
Tree Head
A B C
Performance Expectations
System Memory Bus
VertexData Master
PixelData Master
ComputeData Master
Course Grain
Scheduler Core Management
Memory Cache Unit
2D Core (PTLA)
Pixel Co-Processor
Tiling Co-processor
Control and Register BusHost CPU Interface
RayData Master
System Memory Interface
Frame Accumulator
Cache
Scene Hierarchy Generator
Intersection Processor Array Coherency Engine
Ray Tracing Unit
USC0 USC1Texture Unit
USCn-1 USCnTexture Unit
Unified Shading Cluster Array
...
6 clk/tri1-2 clk/ray
64 AABB/ray16 Tri/ray
1-2clk/accum
128 Inst/ray(512flops peak)
Get early access to the programming concepts at:http://tinyurl.com/9ddlsv2
Talk immediately after this on Hybrid rendering
See the demos at Imagination Booth #402 South Hall
QA
Thank you