Image Synthesis

computer graphics & visualization

Image Synthesis

DirectX/OpenGL


Image Synthesis – WS 07/08Dr. Jens Krüger – Computer Graphics and Visualization Group computer graphics &

visualization

Graphics APIs

Graphics systems and libraries

• Low-level APIs enable direct control• Imperative – define how but not what • e.g. OpenGL (Open Graphics Library), DirectX

• High-level APIs support the non-expert• Declarative – define what but not how• Scene-Graph APIs: Inventor, Optimizer, Java3D, Jupiter• Scene Description Languages: RenderMan



visualization

Graphics APIs

Graphics programming using OpenGL

• Evolving and portable standard (OpenGL 2.0)• Now controlled by the Kronos Group• Still compatible to the very first version 1.0• Low-level library enables direct access• Efficiently exploits underlying hardware (if available)• Simple and orthogonal feature set• Building advanced algorithms on top of it



visualization

Graphics APIs

Graphics programming using Direct3D

• Evolving standard (currently D3D 10.1)• Controlled by Microsoft• Only available for Windows (Vista required for D3D 10 and higher)• Major DirectX versions are incompatible to older releases• “Adjusted” to hardware/pipeline changes



visualization

Graphics-APIs• DX/GL concepts:

– Implements the graphics-pipeline– Geometric primitives:

• Points, lines, polygons– Image primitives:

• Images and bitmaps– State machine:

• Colors, materials,...– Low-level and immediate mode API

• Higher modeling and animation conceptsnot available, no scenegraph

– Phong-Lighting, Gouraud-Shading, Textures, Mip-Mapping, Alpha-Blending, Z-Buffer, Stencil-Buffer, Accumulation-Buffer, ...

Application

Command

Geometry

Rasterization

Texture

Fragment

Display



visualization

The OpenGL Structure

GLUT

GLU

GL

GLX, AGLor WGL

X, Win32, Mac O/S

software and/or hardware

application program

OpenGL Motifwidget or similar



visualization

Introduction to OpenGLOverview (“A trip down the rendering pipeline”)



visualization

The “fixed function” Rendering Pipeline

Framebuffer

GeometryModelviewTransform Lighting Perspective

Transform Clipping

ScanConversion Texturing Fragment

Tests Blending

Rasterization Fragment Processing

Geometry Processing



visualization

OpenGL: Geometric primitivesEvery primitive defined by Vertices (in homogeneous coordinates)

GL_QUAD_STRIP

GL_POLYGON

GL_TRIANGLE_STRIP GL_TRIANGLE_FAN

GL_POINTS

GL_LINES

GL_LINE_LOOPGL_LINE_STRIP

GL_TRIANGLES

GL_QUADS



visualization

Geometric primitivesNote:DX/GL break down everything to triangles

• Triangles are flat• A linear interpolation function exists within triangles



visualization

OpenGL: Geometric primitivesglVertex3fv( v )

Number ofcomponents2 - (x,y) 3 - (x,y,z)4 - (x,y,z,w)

Data Typeb - byteub - unsigned bytes - shortus - unsigned shorti - intui - unsigned intf - floatd - double

Vectoromit “v” forscalar form

glVertex2f( x, y )



visualization

OpenGL: Geometric primitivesOpenGL commands

– Commands and parameters are send to the graphics system

– High performance applications use a different mode to send DX/GL primitives

glBegin(GL_TRIANGLES); glVertex3f(x1, y1, z1); glVertex3f(x2, y2, z2); glVertex3f(x3, y3, z3); glVertex3f(x4, y4, z4); glVertex3f(x5, y5, z5); glVertex3f(x6, y6, z6);glEnd();

glBegin(GL_TRIANGLES); glVertex3fv(p1); glVertex3fv(p2); glVertex3fv(p3); glVertex3fv(p4); glVertex3fv(p5); glVertex3fv(p6);glEnd();



visualization

Primitives and vertex-attributes• How to group primitives

glBegin( primType ); ...; glEnd();

• How to render primitives?Vertex-attributesglColor*(); glNormal*(); glTexCoord();

Global “fixed-function” statesglPointSize(size); glShadeModel(GL_SMOOTH);glEnable(GL_LIGHTING);...



visualization

States• DX/GL Systems are state machines

– State variables define the current state• Can be set and requested individually

– Current state defines how things get rendered

glShadeModel(GL_SMOOTH); glEnable(GL_LIGHTING);

glBegin(GL_TRIANGLES); glColor3v(c1); glNormal3v(n1); glVertex3v(t1); glColor3v(c2); glVertex3v(t2); glVertex3v(t3);glEnd();



visualization

The OpenGL Rendering PipelineOpenGL rendering pipeline – Geometry data

Vertexdata • •

•

Per-vertex operations and

primitive assembly

RasterizationTexture assembly

Fragment processing

Framebuffer



visualization

Geometry processing• Vertex operations and primitve assembly

– Coordinate/normal transformation– Lighting calculations– Texture coordinate generation & transformation

– Perspective projection– Front/back face culling– Clipping

Result is geometric primitives with color, depth, texCoords etc.



visualization

Geometry processing• Polygonal models (triangle meshes)

Per-vertex:Coordinate (x,y,z) Color (RGB) Normal (Nx,Ny,Nz) TexCoords(u,v,w)

Light and material properties:Light source type Color Reflectivity Tansparency ...



visualization

Geometry processingTransformation – camera analogy

– Modeling: move and scale model– Viewing: position and orientation of camera– Projection: adjust camera lens– Viewport: photograph

camera

tripot Model

viewing volume



visualization

Geometry processing

The transformation pipeline

ViewportTransform

PerspectiveDivision

ProjectionMatrix

ModelviewMatrix

vertex

Modelview

Modelview

Projection

lll

object eye clip normalizeddevice

window

lll

Matrix-Stacks:• Modelview• Projection• Texture



visualization

Geometry processing• Transformation matrices

– Choose matrix stackglMatrixMode(GL_MODELVIEW or

GL_PROJECTION)– Load or multiply matrix (from right)

glLoadMatrix(), glMultMatrix()– Manage stack

glLoadIdentity(),glPushMatrix(), glPopMatrix()

– Spezial Modelview-TransformationsglRotate(), glTranslate(), glScale()



visualization

Geometry processing• Transformation matrices



visualization

Geometry processingTransformations

– Affine transformations in homogeneous coordinates

1zyx

1......

...

...

...RotationShear

Scaling

ProjectionHomogeneous part

Translation

PMP 44



visualization

Geometry processingViewing and modeling

– Camera movement (foward) is equivalent to movement of objects towards the camera

– Viewing transformation equivalent to multiple modeling transformations

glTranslatef( 0.0, 0.0, -5.0 );gluLookAt( 0.0,0.0,5.0, 0.0,0.0,0.0, 0.0,1.0,0.0);



visualization

Geometry processing• Projective transformations

glFrustum(left,right,bottom,top,zNear,zFar) gluPerspective(fovy,aspect,zNear,zFar)glOrtho(left,right,bottom,top,zNear,zFar)

• glFrustum allows for non-symmetric viewing pyramids



visualization

Geometry processing• Lighting and shading

– Two ways to determine color of a vertex• Vertex attribut glColor()• Material color and illuminationglColorMaterial, glEnable(GL_LIGHTING)

RGBA-Color

Lighting

Texture

Shadingflat/smooth

ColorClamping



visualization

Geometry processingLighting and shading – Phong lighting

ambientdiffuse

specularattenuation

ModelviewTransformation

Matrix MVertexcolor

Material-colors

globalambient

Light

wzyx

z

y

x

nnn

Normal N

Vertex P

L

L

L

L

wzyx

PM

NM T1

Light source-position

color



visualization

Geometry processing• Phong lighting

– cos() = N * L– cos() = N * H (Blinn-Phong)

L + EL + EH =

HNL E

R



visualization

Geometry processingExtended Phong lighting

sis

ni

didia

ia

i

i

i

aaE

OIHN

OILNOIspotf

OIOI

0,max

0,max

exp2 0,max , 1 spoti

qlc

i DVspotdkdkk

f



visualization

Geometry processing• Turn on light

glEnable( GL_LIGHTING )glEnable( GL_LIGHTn )

• Light source propertiesglLightf{v}( light, property, value )

• Material propertiesglMaterialfv( face, property, value )

• Light source types– Spotlight: – Local (point light): into all direction– At infinity (directional light; directed): w=0 in position

• Front and back faces illuminatedglEnable( GL_LIGHT_MODEL_TWO_SIDE)



visualization

RasterizationThe graphics pipeline

Framebuffer


Transform Clipping


Tests Blending


Geometry Processing



visualization

Rasterization• Conversion of geometric data into fragments

• Fragment:– Represents a point of geometry

• Has color, texCoord, depth etc.– Associated with a pixel– But not yet a pixel (not stored in the framebuffer)

• Pixel:– Color of a pixel in the framebuffer



visualization

Rasterization• Scan conversion of geometric primitives

– For each covered pixel a fragment is generated– Attributes like color, depth, texture coordinates etc.

are generated from projected vertex attributes– During fragment generation OpenGL states, e.g.

shading model, line width, polygon mode etc. are taken into account



visualization

Rasterization• Rasterization of geometric data

Geometry processingVertex transformation Lighting Clipping ...

Rasterization

Screen coord (X,Y) Depth (z) Color (RGB) TexCoords (u,v,w) ...

Fragment ProcessingTexturing Per-fragment tests Per-fragment comparison Blending ...



visualization

Rasterization Once a fragment is generated, per-vertex attributes need to be assigned properly

– Determination of color and other properties for all pixels along a scanline is called Shading

– This is done via interpolation (see exercise for more information)

Rasterization

scanlinev0

v1

v2



visualization

Rasterization – wire frame



visualization

Rasterization – flat shading



visualization

Rasterization Smooth or Gouraud shading (1971)Idea: better quality by respecting color variations on the

polygon– Per vertex lighting– Linear interpolation of color using barycentric interpolation

in triangles• In screen space during scan conversion



visualization

Rasterization – Gouraud shading



visualization

Rasterization Phong shading (1975): higher quality shading by interpolating normals rather than colors

•Evaluates lighting model for every pixel

Þ Not supported by OpenGL



visualization

Phong shading



visualization

Rasterization - OpenGL shadingPhong lightingGouraud shading

Phong lightingand shading



visualization

Rasterization• Scan conversion of primitives

Rasterization

scanlinev0

v1

v2

Interpolation of per-vertex attributes along scanlines

Depth values:non-linear in [Znear, Zfar] due to perspective projection

z

Znear Zfar



visualization

Fragment processingGraphics APIs and the graphics pipeline

Framebuffer


Transform Clipping


Tests Blending


Geometry Processing



visualization

Fragment Processing• Fragment operations

(apply before fragments beome pixels)

– Texturing– Fog calculation– Tests and comparisons– Blending– Dithering– Logical operations and masking



visualization

Texture MappingTexture mapping

– For each vertex specify vertex position, normal, color and texture coordinate

glBegin(GL_TRIANGLES);

glColor4f(R0,G0,B0,0); glColor4f(R1,G1,B1,1); glColor4f(R2,G2,B2,2);glNormal3f(N0x,N0y,N0z); glNormal3f(N1x,N1y,N1z); glNormal3f(N2x,N2y,N2z);glTexCoord3f(T0x,T0y,T0z); glTexCoord3f(T1x,T1y,T1z); glTexCoord3f(T2x,T2y,T2z);glVertex3f(V0x,V0y,V0z)); glVertex3f(V1x,V1y,V1z)); glVertex3f(V2x,V2y,V2z));

glEnd();



visualization

Texture MappingHardware supported texture interpolation

– For each fragment, the appropriate texture value is fetched from the texture array

– Interpolation of texture values is supported by the hardware

Texture elements

Per-fragment sample point



visualization

Texture MappingTexture mapping

– Shape (geometry) & appearance (texture)



visualization

Texture MappingWarping (texture coordinates outside (0,1))

clamp/clamp

clamp/repeat

repeat/repeat

repeat/clamp



visualization

Texture MappingTexture filtering

– Combining texels to determine pixel color• Minification: pixels are larger than one texel• Magnification: pixels are smaller than one texel

– Filtering methods• Nearest: choose the texel closest to the pixel center• Linear: (bi/tri)-linear texture interpolation• MipMap: (bi/tri)-linear interpolation in a stack of textures

of decreasing resolution



visualization

Texture MappingTexture filtering

Textur

Pixel

Textur

Pixel

magnification minification



visualization

Texture Mapping - magnification

Nearest Neighbor linear



visualization

Texture MappingMip-Mapping



visualization

Texture Mapping

without MipMapping with MipMapping



visualization

Texture MappingMip-Mapping

– Only used for minification– Copies of the texture are generated that contain the data at

ever coarser resolution• Match between texture resolution and pixel resolution can be

achieved

Sample 0Sample 1

interpolate

assignSelect resolution

Bilinear in texture + linear between levels = trilinear



visualization

Texture MappingTexture functions

– How does texture color act on fragment color• Replace: replaces fragment color• Modulate: scales fragment color with texture

color• Blend: interpolates between fragment color

andtexture color

• Decal: interpolates between fragment- and texel-color

using alpha-blending (alpha-value from texture)



visualization

Buffers and Fragment TestsFrom fragments to pixels

– Fragments (X,Y,z,RGB,..) undergo operations and tests before they become pixels

• Fragment tests and blending– Several buffers can be used and accessed (read/write)– Frame buffer consists of:

• Color buffer• Depth buffer• Stencil buffer• Pixel buffer ...



visualization

Buffers and Fragment TestsBuffers and their use

– Color buffers: RGB(A)/ColorIndex• Front/Back buffers for interleaved rendering• Off-screen pixel buffers

– Depth buffer: high precision depth values• Stores distance to the eye point for each pixel• Resolves occlusions on a per-fragment basis

– Stencil buffer: per-pixel bit mask• Used to restrict drawing to particular pixels

– …



visualization

Buffers and Fragment TestsFragment tests and operations

Logicaloperations

Scissor Test Alpha Test Stencil Test Depth Test

Blending Dithering

Scissor: restrict drawing to particular screen rectangleDithering: increase color resolution by decreasing the spatial resolutionLogical Ops: bit-wise boolean operations between src and dst



visualization

Buffers and Fragment Tests• Alpha test

– Compare fragment alpha with reference alpha– Often used to discard specific data

FragmentAlpha

ReferenceAlpha

Alpha Function(Always, Equal,Less, Greater ..)

Draw fragment or

discard it



visualization

Buffers and Fragment TestsStencil buffer

– For each pixel the stencil buffer contains an implementation dependent number of bits

– It can be used to restrict drawing on a per-pixel basis• Often used in multipass rendering techniques

PIXEL

Red Green Blue Alpha Depth

Stencil



visualization

Buffers and Fragment TestsStencil test

– Compare pixel stencil bits with reference value– Depending on stencil & depth test fragments are drawn and

stencil bits are updated

Depth test

Pixel stencil value Global reference value

Stencil test(Always, Equal,Less, Greater ..)

FalseTrue

Update pixelsstencil value

and write pixel



visualization

Buffers and Fragment TestsStencil test – applications

– Rendering in non-rectangular regions



visualization

Blending OperationsBlending

– How do incoming fragments modulate pixels in the framebuffer

• Different blend functions exist• Fading in/out, replacing, -compositing



visualization

Blending Operations• Combination of fragments (src) and pixels in the

color buffer (dst)– Different blend factors for src and dst– Blend equations determine the combination C = CsSc o CdDc

C[s/d] = src/dst color[S/D]c = src/dst blend factorso = operator (Min,Max,Add,Sub,LogicOp)



visualization

Blending Operations• -compositing

– -values correspond to opacity [0,1]• How much light is blocked by a pixel/fragment • Consider pixel and fragment -value during compositing

CP = P CP + (1-P)FCF

P = P + (1-P)F

– OpenGL provides different blending operations• glBlendFunc(SRC, DEST)



visualization

Example (masking)



visualization

Example (masking)Masking with -compositing

– Draw the monkey image into the color buffer– Draw the text as binary image into the

alpha buffer– Draw the brick image with -blending

• Use blend factors GL_DST_ALPHA for incoming fragments and GL_ONE_MINUS_DST_ALPHA for pixels already in the frame buffer



visualization

Example (masking)Masking with alpha test & stencil test

– Draw the text as binary image into the alpha buffer• Use the alpha test to discard the background• Set the stencil bits where pixels are affected

– Draw the monkey image where the stencil bits are set

– Reverse the stencil test and draw the brick image



visualization

Coming up

The programmableD3D 10 Pipeline

Image Synthesis

Documents

Transcript of Image Synthesis