Illustrative Volume Rendering

onConsumer Graphics

HardwareRoy van Pelt

Exam Committee:

dr. A. Vilanova (BMT)

dr. ir. H.M.M. van de Wetering (CSE)

dr. ir. M. Chaudron (CSE)

2/40

Project motivation

Improve comprehensibility Useful for context visualisations Similar to illustrations in physiology

books

3/40

Introduction: Illustrative Volume Rendering Volume Rendering

3D dataset, i.e.: CT / MRI

Images by kind courtesy of R. Brecheisen (2007) (MVR framework)

4/40

Introduction: Illustrative Volume Rendering Illustrative Volume Rendering

Depict dataset as a drawing/sketch

Stippling Hatching Contours

5/40

Introduction: Illustrative Volume Rendering Existing framework: VolumeFlies

Traditional illustration techniques Based on particle systems

6/40

Project objectives

Design and implement VolumeFlies, using consumer graphics hardware, such that real-time interaction with the data is possible.

Integrate VolumeFlies with a direct volume rendering approach.

7/40

Background Info: GPU pipeline

• Massively parallel– Adjust algorithms to use parallelism– Unified shader design

• Programmability– Stream programming model using shaders

RasterizationRasterizationGeometry Geometry

ProcessingProcessing

FragmentFragment

OperationsOperations

VertexVertex

ShaderShader

GeometryGeometry

ShaderShader

FragmentFragment

ShaderShader

8/40

Background Info: GPU pipeline

• Massively parallel– Adjust algorithms to use parallelism– Unified shader design

• Programmability– Stream programming model using shaders

RasterizationRasterizationGeometry Geometry

ProcessingProcessing

FragmentFragment

OperationsOperations

VertexVertex

ShaderShader

GeometryGeometry

ShaderShader

FragmentFragment

ShaderShader

9/40

Background Info: General GPU approach

RasterizationRasterizationGeometry Geometry

ProcessingProcessing

FragmentFragment

OperationsOperations

VertexVertex

ShaderShader

GeometryGeometry

ShaderShader

FragmentFragment

ShaderShader

Proxy geometry = “Array index”

Source buffer = “Input array” Destination buffer = “Output array”

Transform feedbackDiscard fragment shader

10/40

Background Info: Particle system Particle: an element described by

propertiesFor example particle position in the volume

p

Movie source: http://www.runevision.com

11/40

Framework modules

Initialiser1

2

3

4

Behaviour

Filter

Visualiser

Initialises particle set

Alters particle set

Removes unwanted particles

Illustratively depicts particles

12/40

Framework modules

Initialiser1

Initialize particles near the ISO-surface

Related to marching cubes

13/40

Framework modules

Initialiser1

VertexVertex

ShaderShader

GeometryGeometry

ShaderShader

FragmentFragment

ShaderShader

3D TextureVolume

Proxy geometry

Particle positions (x,y,z) = (r,g,b)

14/40

Framework modules

Initialiser1

2

3

4

Behaviour

Filter

Visualiser

15/40

Framework modules

2 Behaviour

0.2 0.4 0.6 0.8 1

20

40

60

ij i jr p p

( , , )i i ii

i i i

E E Ev

x y z

Redistribute particles evenly on the surface

Energy minimisation

ip1,

(| |)m

i ij ijj j i

E E r

jp

ijr| |ijriv

(| |)ij ijE r

| |ijr

“Robust particle systems for curvature dependent sampling of implicit surfaces” – Meyer et al. (SMI 2005)

Minimise the total energy within the particle system

1

m

i

i

E E

16/40

iv in

Framework modules

2 Behaviour

Redistribute particles evenly on the surface

Two step particle displacement

iso-surface

tangent plane

“Robust particle systems for curvature dependent sampling of implicit surfaces” – Meyer et al. (SMI 2005)

17/40

Framework modules

2 Behaviour

1. Sort particles by their bin numberOdd-even merge sort

2. Create a lookup table for the binsBinary search0 4 7 10

3. Redistribute particles by repulsionEnergy minimisation approach

X X X X X X X X X X X X

Y Y Y Y Y Y Y Y Y Y Y Y

Z Z Z Z Z Z Z Z Z Z Z Z

Bin 0 Bin 1 Bin 0 Bin 0 Bin 0 Bin 1 Bin 2 Bin 1 Bin 3 Bin 2 Bin 3 Bin 2

1

1

1

2

2

2

3

3

3

4

4

4

5

5

5

6

6

6

7

7

7

8

8

8

9

9

9

10

10

10

11

11

11

12

12

12

X X X X X X X X X X X X

Y Y Y Y Y Y Y Y Y Y Y Y

Z Z Z Z Z Z Z Z Z Z Z Z

Bin 0 Bin 0 Bin 0 Bin 0 Bin 1 Bin 1 Bin 1 Bin 2 Bin 2 Bin 2 Bin 3 Bin 3

1

1

1

3

3

3

4

4

4

5

5

5

2

2

2

6

6

6

8

8

8

7

7

7

10

10

10

12

12

12

9

9

9

11

11

11

Redistribution hard to port to GPU:

Addressing all neighbours is

computationally expensive

Solution:

Create a bin-structure to compute

energy minimisation locally.

18/40

Framework modules

2 Behaviour

VolumeFlies applied a fixed number of iterations

A new stop-criterion is introduced

Verify the change of the total energy

Apply reduction operation on the GPU

8E7E6E5E4E3E2E1E

'4E

'3E

'2E

'1E

'4E

'3E

'2E

'1E

''2E

''1E

'''1E Total system energy

+

+

+

1

m

i

i

E E

19/40

Framework modules

2 Behaviour

Odd-Even Merge Sort

Binary Search

Energy Minimization

Verify stop-criterion

GPU approach more elaborate.

Still much faster than CPU approach.

Particle system is generic.

20/40

Framework modules

Initializer12

3

4

Behaviour

Filter

Visualizer

21/40

Framework modules

3 Filter

a) Create the ISO surface

Cone splatting:

Cones directed to the viewing plane

Cones are scaled to prevent edge overlap

22/40

Framework modules

3 Filter

b) Hidden surface removal

Off-screen buffer:

Splat rgb = Particle xyz

Visible when particle position occurs as a colour

23/40

Framework modules

Initialiser1

2

3

4

Behaviour

Filter

Visualiser

24/40

Density-based stippling

4 Visualiser

Vary particle density to change tone

Initial density resembles darkest tone

Per particle random threshold

Based on basic diffuse lighting

Executed by vertex shader

User can control contrast

25/40

Scale-based stippling

4 Visualiser

Vary point size to change tone

Larger points create darker areas

Scale point representation

Based on basic diffuse lighting

Executed by vertex shader

User can control contrast

26/40

Direction-based hatching

4 Visualiser

Hatch tracing in a single direction

Single and cross hatching

Single hatches

Cross hatches

27/40

Direction-based hatching

4 Visualiser

iso-surface

Hatch tracing in a single direction

Segments projected in tangent plane

28/40

Direction-based hatching

4 Visualiser

VertexVertex

ShaderShader

GeometryGeometry

ShaderShader

FragmentFragment

ShaderShader

Particle positions

Generate hatches in fixed direction

3D Texture: Volume

Proxy geometry

Hatch segments

1

VertexVertex

ShaderShader

GeometryGeometry

ShaderShader

FragmentFragment

ShaderShader

Visualise generated hatchesProxy geometry

Hatch segments

2

29/40

Direction-based hatching

4 Visualiser

Again apply basic diffuse lighting

Two-level threshold

2nd level

1st level

30/40

Curvature-based hatching

4 Visualiser

Emphasize curvature with hatches

Fast curvature calculation approach

2. Optimize for GPU by using interpolation capabilities

Differential structure in volume required

“Fast Third-Order Texture Filtering” – Sigg, Hadwiger (GPU Gems 2)

1. Filtering approach for derivative reconstruction

3. Compute principal curvature

Re-implemented in high-level shading language

Based on general GPU-approach

31/40

Curvature-based hatching

4 Visualiser

Emphasize curvature with hatches

Defining the principal curvature

Image by Eric Gaba

Variation of the normal vector when moving a small distance

iso-surface

tangent plane

in

Maximum and minimum change describe the principal

curvature

1ik

2ik

32/40

Curvature-based hatching

4 Visualiser

Emphasize curvature with curvature

Calculate principal curvature

Measure first partial derivates: gradient

/ | |n g g<<<<<<<<<<<<<<<<<<<<<<<<<< <<

“Curvature-Based Transfer Functions for Direct Volume Rendering Methods and Applications” – Kindlmann et al. (IEEE Visualization 2003)

g<<<<<<<<<<<<<<

g<<<<<<<<<<<<<<

n

iso-surface

tangent plane

Measure second partial derivates: Hessian H Extract curvatures by eigen analysis on the shape operatorS

/ | |

TP I nn

S PHP g

<<<<<<<<<<<<<<

1k

33/40

Curvature-based hatching

4 Visualiser

1D Transfer function

-1 0 1

1 2

1k

2k

34/40

Curvature-based hatching

4 Visualiser

Messy hatch results using curvature directions

Smooth the field directions and weight the trace

Fixed direction when

surface is unreliable

35/40

Curvature-based hatching

4 Visualiser

Principal curvature can be calculated in real-time

Demonstrated by real-time curvature colour mapping

1

2

36/40

Contours

4 Visualiser

Draw ‘hatches’ for particles near the contour

Segments trace the direction of the contour

Contour: locations where normal

is perpendicular to the view

37/40

Results: Performance

0 2 4 6 8 10 12 14 16 18 20

Profiling time (seconds)

VolFliesGPU VolumeFlies

253.34

52.65

53.05

Load Volume

Brute-force initialiser

Redistribution

Smooth Field

Hatch generation (Direction)

Hatch generation (Scale)

Hatch visualisation

Contours

38/40

Results: Demonstration movie

39/40

Conclusions & Future work Completely GPU-based General GPU approach Two generic components:

Particle system Real-time curvature estimator

Integration with direct volume rendering More elaborate memory management Improvement of modules / Zooming New applications and styles: DTI, animation, ghosting,

exploded view…

40/40

Thank you for your attention!

Questions

?

41/40

Particle repulsion

Energy minimisation

Two step particle displacement

ij i jr p p

( , , )i i ii

i i i

E E Ev

x y z

1,

(| |)m

i ij ijj j i

E E r

1

m

i

i

E E

2

( )

( )

( )| |

Ti i i i i

i i

ii i i

i

p p I n n v

g f p

gp p f p

g

<<<<<<<<<<<<<<

<<<<<<<<<<<<<<

<<<<<<<<<<<<<<

42/40

Stippling

Density-based

Scale-based

( ) max( ,0)d i iL i n e

( ( ))ci dv L i

( ) max( ,0)d i iL i n e

3( (1 ( )))2

ci dS L i

43/40

Hatching Smooth field

2 1

2 1

2arctans

1 2

1 2

0, | | | |( , ) 1

1 | 2(| | ) |,2

if and

s otherwise

1( ) (1 )

( )

jji TT T

j

jT

w ks w w s

w

wb

44/40

Hatching Tracing

0

1 1 21

2 1 2

(1 ) , | | | |

(1 ) , | | | |

j j j

j j j

i

j j jj j jj

j j jj j j

j

h s

w h w k k h k hh

w h w k k h k h

jw

s

45/40

Contours

2 2 1 1 1 2( ) ( )d k e k k e k <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

Image by D. DeCarlo

46/40

Curvature estimation 1/3

Measure partial derivatives by convolution filtering with cubic B-spline

47/40

Curvature estimation 2/3

Optimise for the GPU by using interpolation

xi i+1i-1 i+2

Weights lookup texture

Colour texture

48/40

Curvature estimation 3/3

Compute principal curvature

Eigen analysis on S

/ | |

/ | |

T

n g g

P I nn

S PHP g

<<<<<<<<<<<<<<<<<<<<<<<<<< <<

<<<<<<<<<<<<<<

1 11

2 22

Sk k

Sk k