Post on 17-Aug-2020
R l Ti R d iReal-Time Rendering(Echtzeitgraphik)(Echtzeitgraphik)
Dr. Michael Wimmerwimmer@cg tuwien ac atwimmer@cg.tuwien.ac.at
Texturing
Overview
OpenGL lighting refresherTexture SpacesTexture SpacesTexture Aliasing and FilteringMultitexturing
LightmappingTexture Coordinate GenerationProjective TexturingProjective TexturingMultipass Rendering
Vienna University of Technology 3
But Before We Start: Shading
Flat shadingcompute light interaction per polygoncompute light interaction per polygonthe whole polygon has the same color
G d h diGouraud shadingcompute light interaction per vertexinterpolate the colors
Phong shadingg ginterpolate normals per pixel
Remember: difference betweenRemember: difference betweenPhong Light ModelPhong Shading
Vienna University of Technology 4
Phong Shading
But Before We Start: OpenGL Lighting
Phong light model at each vertex (glLight, …)Local model only (no shadows radiosity )Local model only (no shadows, radiosity, …)ambient + diffuse + specular (glMaterial!)
Fi d f ti G d h diFixed function: Gouraud shadingNote: need to interpolate specular separately!
Phong shading: calculate Phong model in fragment shader
Vienna University of Technology 5
ag e s ade
Why Texturing?
Idea: enhance visual appearance of plain surfaces by applying fine structured detailssurfaces by applying fine structured details
Eduard Gröller, Stefan Jeschke 6
OpenGL Texture Mapping
Basis for most real-time rendering effectsL k d f l f fLook and feel of a surfaceDefinition:
A regularly sampled function that is mappedonto every fragment of a surfaceonto every fragment of a surfaceTraditionally an image, but…
Can hold arbitrary informationTextures become general data structuresTextures become general data structuresWill be interpreted by fragment programs
Vienna University of Technology 7
Can be rendered into important!
Types of Textures
Spatial Layout1D 2D 3D1D, 2D, 3DCube Maps
Formats (too many), e.g. OpenGLLUMINANCE16 ALPHA16: 32bit = 2 x 16 bitLUMINANCE16_ALPHA16: 32bit = 2 x 16 bit bump mapRGBA4: 16bit = 4 x 4 colorsRGBA FLOAT32: 128 bit = 4 x 32 bit floatG _ O 3 8 b 3 b oacompressed formats, high dynamic range formats
Vienna University of Technology 8
formats, …
Texturing: General Approach
Texels
Texture space (u,v) Object space (xO,yO,zO) Image Space (xI,yI)
Parametrization Rendering(Projection etc.)
Eduard Gröller, Stefan Jeschke 9
Texture Spaces
RenderingRenderingModelingModeling
Object spaceObject space(x y z w)(x y z w)(x,y,z,w)(x,y,z,w)
Texture Texture projectionprojection
Parameter SpaceParameter Space(s,t,r,q)(s,t,r,q)
projectionprojection
( , , ,q)( , , ,q)
SS
Texture Texture functionfunctionTexture SpaceTexture Space
(u,v)(u,v)functionfunction
Vienna University of Technology 10
Texture Projectors
Where do texture coordinates come from?O li t t t i /t d tiOnline: texture matrix/texcoord generationOffline: manually (or by modeling prog)y ( y g p g)spherical cylindrical planar natural
Vienna University of Technology 11
Texture ProjectorsWhere do texture coordinates come from?
Offline: manual UV coordinates by DCC programy p gNote: a modeling Problem!
Vienna University of Technology 12
Texture Functions
How to extend texture beyond the border?B d d t/ l dBorder and repeat/clamp modesArbitrary (s,t,…) [0,1] [0,255]x[0,255]y ( , , ) [ , ] [ , ] [ , ]
repeat mirror/repeat clamp borderp p p
Vienna University of Technology 13
Texture Aliasing
Problem: One pixel in image space covers many texels
Eduard Gröller, Stefan Jeschke 14
Texture Aliasing
Caused by undersampling: texture information is lost
Texture spacep
Eduard Gröller, Stefan Jeschke 15
Image space
Texture Anti-Aliasing
A good pixel value is the weighted mean of the pixel area projected into texture spacearea projected into texture space
vPixelPixel
XX
Eduard Gröller, Stefan Jeschke 16Texture space u Image space
Texture Anti-Aliasing: MIP Mapping
MIP Mapping (“Multum In Parvo”)Texture size is reduced by factors of 2Texture size is reduced by factors of 2(downsampling = "much info on a small area")Simple (4 pixel average) and memory efficientSimple (4 pixel average) and memory efficientLast image is only ONE texel
Eduard Gröller, Stefan Jeschke 17
Texture Anti-Aliasing: MIP Mapping
MIP Mapping AlgorithmD := ld(max(d d )) d
d2"Mip Map level"D := ld(max(d1,d2))
T0 := value from texture D0= trunc (D)U bili i t l ti
d1X
p p
Use bilinear interpolation
Eduard Gröller, Stefan Jeschke 18Bilinear interpolation Trilinear interpolation
Texture Anti-Aliasing: MIP Mapping
Trilinear interpolation:T1 := value from texture D1 = D0+1 (bilin interpolation)T1 : value from texture D1 D0+1 (bilin.interpolation)Pixel value := (D1–D)·T0 + (D–D0)·T1
Linear interpolation between successive MIP MapsLinear interpolation between successive MIP MapsAvoids "Mip banding" (but doubles texture lookups)
Eduard Gröller, Stefan Jeschke 19
Texture Anti-Aliasing: Mip Mapping
Other example for bilinear vs. trilinear filtering
Eduard Gröller, Stefan Jeschke 20
Texture Anti-Aliasing
Bilinear reconstruction for texture magnification (D<0)("upsampling")( upsampling )
Weight adjacent texels by distance to pixel position
-v
T(u+du v+dv)(u,v) (u+1,v)
X
T(u+du,v+dv) = du·dv·T(u+1,v+1)
d (1 d ) T( 1 )d
dv( , )
+ du·(1–dv)·T(u+1,v)+ (1-du)·dv·T(u,v+1)
du
+ (1-du)·(1-dv)·T(u,v)(u+1,v+1)(u,v+1)
Eduard Gröller, Stefan Jeschke 21Texture space u
Anti-Aliasing (Bilinear Filtering Example)
Original imageg g
Eduard Gröller, Stefan Jeschke 22Nearest neighbor Bilinear filtering
Anti-Aliasing: Anisotropic Filtering
Anisotropic FilteringView dependent filter kernelView dependent filter kernelImplementation: summed area table, "RIP Mapping", "footprint assembly" “sampling”footprint assembly , sampling
Texture space
Eduard Gröller, Stefan Jeschke 23
Texture space
Anti-Aliasing: Anisotropic Filtering
Example
Eduard Gröller, Stefan Jeschke 24
Texture Anti-aliasing
Everything is done in hardware, nothing much to do!to do!gluBuild2DMipmaps()generates MIPmapsSet parameters in glTexParameter()
GL LINEAR MIPMAP NEARESTGL_LINEAR_MIPMAP_NEARESTGL_TEXTURE_MAG_FILTER
Anisotropic filtering is an extension:Anisotropic filtering is an extension:GL_EXT_texture_filter_anisotropic
Number of samples can be varied (4x,8x,16x)Vendor specific support and extensions
Vienna University of Technology 25
Vendor specific support and extensions
Multitexturing
Apply multiple textures in one passI t l t f bl h diIntegral part of programmable shading
e.g. diffuse texture map + gloss mape.g. diffuse texture map + light map
Performance issuesPerformance issuesHow many textures are free?How many are available
Vienna University of Technology 26
Multitexture – How?
Simple(!) texture environment example:glActiveTexture(GL_TEXTURE1);glTexEnvi(GL_TEXTURE_ENV, …)
GL TEXTURE ENV MODE GL COMBINE);… GL_TEXTURE_ENV_MODE, GL_COMBINE);… GL_COMBINE_RGB, GL_MODULATE);… GL SOURCE1 RGB, GL TEXTURE);_ _ , _ );… GL_OPERAND1_RGB, GL_SRC_COLOR);… GL_SOURCE2_RGB, GL_PREVIOUS);… GL_OPERAND2_RGB, GL_SRC_COLOR);
Programmable shading makes this easier!C = CT1 · CT0
Vienna University of Technology 27
Programmable shading makes this easier!
Example: Light Mapping
Used in virtually every commercial gameP l l t diff li hti t ti bj tPrecalculate diffuse lighting on static objects
Only low resolution necessaryDiffuse lighting is view independent!
Advantages:Advantages:No runtime lighting necessary
VERY fast!Can take global effects (shadows colorCan take global effects (shadows, color bleeds) into account
Vienna University of Technology 28
Light Mapping
Original LM texels Bilinear Filtering
Vienna University of Technology 29
Light Mapping
Original scene Light-mapped
Vienna University of Technology 30
Example: Light Mapping
Precomputation based on non-realtime methodsmethods
RadiosityRaytracing
Monte Carlo IntegrationMonte Carlo IntegrationPathtracingPh t iPhotonmapping
Vienna University of Technology 31
Light Mapping
Lightmap mapped
Vienna University of Technology 32
Light Mapping
Original scene Light-mapped
Vienna University of Technology 33
Ambient Occlusion
Special case of light mappingC i ht d i ibilit t i tCos-weighted visibility to environment modulates intensity:
Darker where more occludedDarker where more occludedOption: “per object” lightmap
Vienna University of Technology 34
p p j g pAllows to move object
Ambient Occlusion
Vienna University of Technology 35
Model/Texture: Rendermonkey
Light Mapping Issues
Map generation:Use single map for group of coplanar polysUse single map for group of coplanar polys
Lightmap UV coordinates need to be in (0..1)x(0..1)
Map application:Premultiply textures by light mapsPremultiply textures by light maps
Why is this not appealing?M lti ith f b ff bl dMultipass with framebuffer blend
Problems with specularMultitexture
Fast flexible
Vienna University of Technology 36
Fast, flexible
Light Mapping Issues
Why premultiplication is bad…
vs.
+
Full Size Texture(with Lightmap)
Tiled Surface Texture plus Lightmap
use tileable surface textures and low
(with Lightmap) plus Lightmap
Vienna University of Technology 37
use tileable surface textures and low resolution lightmaps
Light Mapping/AO Toolset
DCC programs (Blender, Maya…)G E i (I li ht)Game Engines (Irrlicht)Light Map Maker (free)g p ( )
A bi t O l iAmbient Occlusion:xNormal
Vienna University of Technology 38
Texture Coordinates
Specified manually (glMultiTexCoord())Using classical OpenGL texture coordinateUsing classical OpenGL texture coordinate generation
Li f bj t tLinear: from object or eye space vertex coordsS i l i d ( )Special texturing modes (env-maps)Can be further modified with texture matrix
E.g., to add texture animationCan use 3rd or 4th texture coordinate forCan use 3rd or 4th texture coordinate for projective texturing!
Shader allows complex texture lookups!Vienna University of Technology 39
Shader allows complex texture lookups!
Texture Coordinate Generation
Specify a “plane” (i.e., a 4D-vector) for each coordinate (s t r q)coordinate (s,t,r,q)Example: s = p1 x + p2 y + p3 z + p4 w
GLfloat Splane[4] = { p1, p2, p3, p4 };GLfloat Splane[4] = { p1, p2, p3, p4 };glTexGenfv(GL_S, GL_EYE_PLANE, Splane);glTexGenfv(GL_S, GL_EYE_PLANE, Splane);glEnable(GL_TEXTURE_GEN_S);glEnable(GL_TEXTURE_GEN_S);
Think of this as a matrix T with plane parameters p pas row vectors
Vienna University of Technology 40
Texture Coordinate GenerationObject-linear:
yx
ts
zzy
rt
T
Eye-linear: Te = T · M-1 object
wq
e(M…Modelview matrix at time of specification!) xs
Effect: uses coordinate space at time of specification!
E M id tit zy
rt
TeEye: M=identityWorld: M=view-matrix
eyewz
qr
Vienna University of Technology 41
eye
Texture Animation
Classic OpenGLCan specify an arbitrary 4x4 MatrixCan specify an arbitrary 4x4 Matrix,each frame!glMatrixMode(GL_TEXTURE);
There is also a texture matrix stack!There is also a texture matrix stack!Shaders allow arbitrary dynamic calculations
ith di twith uv-coordinatesMany effects possible:y pFlowing water, conveyor belts, distortions etc.
Vienna University of Technology 42
Projective TexturingProjective Texturing
Projective Texture Mapping
Want to simulate a beameror a flashlight or a slide projector… or a flashlight, or a slide projector
Precursor to shadowsInteresting mathematics: 2 perspective2 perspective projections involved!Easy to program!
Vienna University of Technology 44
Projective Texture Mapping
Vienna University of Technology 45
Projective Texture Mapping: Vertex Stage
Map vertices to light frustumOption 1: from object spaceOption 1: from object spaceOption 2: from eye space
Projection (perspective transform)(perspective transform)
Vienna University of Technology 46
Spaces
Vienna University of Technology 47
Projective Texture Mapping
OpenGL does not store Modeling MatrixN ti f ld !No notion of world space!
ModelviewModelview
CameraCameraxxooxxee
viewview(look at)(look at)matrixmatrix
ModelingModelingmatrixmatrix
yyoozzooww
yyeezzeeww
==matrixmatrix wwoowwee
C SC S Object SpaceObject Space
Vienna University of Technology 48
Camera SpaceCamera Space Object SpaceObject Space
Projective Texture Mapping
Version 1: transforming object space coordinatescoordinates
Disadvantage: need to provide model matrix ffor each object in shader!Classic OpenGL: even more difficult!p
1/21/2
1/21/2 1/21/2
1/21/2LightLight LightLight
ii ModelingModelingxxooyy
sstt 1/21/2
1/21/2 1/21/2
1/21/2(projection)(projection)
matrixmatrix
viewview(look at)(look at)matrixmatrix
ModelingModeling
matrixmatrix
yyoozzooww
ttrrqq
==
11wwooqq
M [ 1 1]
Vienna University of Technology 49
TMap [-1..1] to [0..1]
Projective Texture Mapping
Version 2: transforming eye space coordinatescoordinates
Advantage: matrix works for all objects!
1/21/2
1/21/2 1/21/2
1/21/2LightLight LightLight
ii
InverseInverseeyeeye
xxeeyy
sstt 1/21/2
1/21/2 1/21/2
1/21/2(projection)(projection)
matrixmatrix
viewview(look at)(look at)matrixmatrix
eyeeyeviewview
(look at)(look at)matrixmatrix
yyeezzeeww
ttrrqq
==
11matrixmatrix wweeqq
Vienna University of Technology 50
T
Classic OpenGL TexGen TransformM d l iM d l i
xxxxAutomatically Automatically
applied by applied by TexGenTexGen
ModelviewModelview
EyeEyeviewview
(look at)(look at)ModelingModeling
matrixmatrix
xxooyyoozzoo
xxeeyyeezzee
==
pp ypp y(set Modeling (set Modeling
matrix to matrix to eyevieweyeview))( )( )matrixmatrix
zzoowwoo
zzeewwee
1/21/2
1/21/2 1/21/2
1/21/2LightLight
f tf tLightLight
ii
InverseInverseeyeeye
xxeeyy
sstt 1/21/2
1/21/2 1/21/2
1/21/2 frustumfrustum(projection)(projection)
matrixmatrix
viewview(look at)(look at)matrixmatrix
eyeeyeviewview
(look at)(look at)matrixmatrix
yyeezzeeww
ttrrqq
==
11matrixmatrix wweeqq
Vienna University of Technology 51
Supply this combined transform to Supply this combined transform to glTexGenglTexGen
Projective Texture Mapping: Rasterization
Problem: texture coordinate interpolationTexture coordinates are homogeneous!Texture coordinates are homogeneous!
Look at perspective correct texturing first!
Vienna University of Technology 52
Perspective Texture Mapping
Problem: linear interpolation in rasterization?b
2
2
1
1
21
21
wxb
wxa
bwawbxax
screenspace screenspace interpolationinterpolation
objectspace objectspace interpolationinterpolation 2121 interpolationinterpolationinterpolationinterpolation
Vienna University of Technology 53a = b = 0,5
Perspective Texture Mapping
Solution: interpolate (s/w, t/w, 1/w)( / ) / (1/ ) t t f t(s/w) / (1/w) = s etc. at every fragment
h f th f teach fragmenteach fragment
Vienna University of Technology 54each vertexeach vertex
Projective Texturing
What about homogeneous texture coords?N d t d ti di id l fNeed to do perspective divide also for projector!
(s, t, q) (s/q, t/q) for every fragmentHow does OpenGL do that?How does OpenGL do that?
Needs to be perspective correct as well!Trick: interpolate (s/w, t/w, r/w, q/w)(s/w) / (q/w) = s/q etc at every fragment(s/w) / (q/w) s/q etc. at every fragment
Remember: s,t,r,q are equivalent to x,y,z,w in j t ! / j t d th!
Vienna University of Technology 55
projector space! r/q = projector depth!
Homogeneous Perspective Correct Interpolation
[x,y,z,1,r,g,b,a]texcoord generation [x y z 1 r g b a s t r q]texcoord generation [x,y,z,1, r,g,b,a, s,t,r,q] Modelviewprojection [x’,y’,z’,w,1, r,g,b,a, s,t,r,q]Project ( /w ) [x’/w, y’/w, z’/w, 1/w, r,g,b,a, s/w, t/w, r/w, q/w ]vert
Rasterize and interpolate [x’/w, y’/w, z’/w, 1/w, r,g,b,a, s/w, t/w, r/w, q/w ]frag[x /w, y /w, z /w, 1/w, r,g,b,a, s/w, t/w, r/w, q/w ]Homogeneous: texture project (/ q/w) [x’/w y’/w z’/w 1/w r g b a s/q t/q r/q 1][x /w,y /w,z /w,1/w, r,g,b,a, s/q,t/q,r/q,1]Or non-homogeneous: standard project (/ 1/w) [x’/w y’/w z’/w 1/w r g b a s t r q] (for normals)
Vienna University of Technology 56
[x /w, y /w, z /w, 1/w, r,g,b,a, s,t,r,q] (for normals)
Projective Texture Mapping
Problemreverse projectionreverse projection
SolutionsC ll bj tCull objectsbehind projectorU li l t li i t bj t b hi d j tUse clip planes to eliminate objects behind projectorFold the back-projection factor into a 3D attenuation t ttextureUse to fragment program to check q < 0
Vienna University of Technology 57
Projective Texture Mapping
ProblemsResolution problemsResolution problemsProjection behindshadow casters
Shadow Mapping! Shadow Mapping!
Vienna University of Technology 58
Projective Texture Mapping Example
Example shown in CG Shading LanguageCG is proprietary to NVIDIACG is proprietary to NVIDIAC-like synthaxHLSL (DirectX shading language) nearlythe same synthaxthe same synthax
Shading languages have specialized callsf j ti t t ifor projective texturing:
CG/HLSL: tex2DprojGLSL: texture2DProj
They include perspective divisionVienna University of Technology 59
They include perspective division
CG Vertex Program
Input: float4 position, float3 normalfloat3 normal
Output: float4 oPosition, float4 texCoordProj, fl t4 diff Li htifloat4 diffuseLighting
Uniform:float Kd, ,float4x4 modelViewProj,float3 lightPosition, float4x4 textureMatrix
Vienna University of Technology 60
CG Vertex Program
oPosition = m l(modelVie Proj position)mul(modelViewProj, position);
texCoordProj =mul(textureMatrix, position);
float3 N = normalize(normal);float3 N = normalize(normal);float3 L = normalize(lightPosition
– position.xyz);diffuseLighting =diffuseLighting =
Kd * max(dot(N, L),0);
Vienna University of Technology 61
CG Fragment Program
Input: float4 texCoordProj, float4 diff seLightingfloat4 diffuseLighting
Output: float4 colorUniform:sampler2D projectiveMapfl t4 t t C lfloat4 textureColor = tex2Dproj(projectiveMap,
texCoordProj);color = textureColor *color = textureColor
diffuseLighting;
Vienna University of Technology 62
CG vs. Classic OpenGL
Classic OpenGL:Just supply correct matrix to glTexGenJust supply correct matrix to glTexGen
Projective texturing is easy to program and very effective method.
Combinable with shadows Combinable with shadows
Vienna University of Technology 63
Projective Shadow in Doom 3
Vienna University of Technology 64
Texture Compression
S3TC texture compression (DXTn)Represent 4x4 texel block by two 16bit colorsRepresent 4x4 texel block by two 16bit colors (5 red, 6 green, 5 blue)St 2 bit t lStore 2 bits per texelUncompressp
Create 2 additionalColors between c1 and c2use 2 bits to indexuse 2 bits to indexwhich color
4:1 or 6:1 compressionVienna University of Technology 65
4:1 or 6:1 compression
Multipass RenderingMultipass Rendering
Multipass Rendering
Recall 80 million triangle sceneG NOT i 0 5Games are NOT using a = 0.5
at least not yetAssume a = 32, I = 1024x768, d=4
Typical for last generation gamesTypical for last generation gamesF = I * d = 3,1 MF/frame, T = F / a = 98304 T/frame60 Hz ~189 MF/s ~5 6 MT/s60 Hz ~189 MF/s, ~5,6 MT/s
Vienna University of Technology 67
Do More!
Hardware underused with standard OpenGL lighting and texturinglighting and texturing
What can we do with this power?R d ftRender scene more often: multipass renderingRender more complex pixels:multitexturingg
2 textures are usually for freeRender more complex pixels and triangles:Render more complex pixels and triangles:programmable shading
Vienna University of Technology 68
Note
Conventional OpenGL allows for many effectsusing multipassusing multipass
Still in use for mobile devices and lastgen consolesModern form: render to texture
Much more flexible but same principle
Programmable shading makes things easierSpecialized calls in shading languages
Vienna University of Technology 69
Multipass Rendering: Why?
OpenGL lighting model onlylocallocallimited in complexity
Many effects possible with multiple passes:Dynamic environment mapsDynamic environment mapsDynamic shadow mapsReflections/mirrorsDynamic impostorsDynamic impostors(Light maps)
Vienna University of Technology 70
Multipass Rendering: How?
Render to auxiliary buffers, use result as texturetexture
E.g.: environment maps, shadow mapsRequires pbuffer/fbo-support
Redraw scene using fragment operationsRedraw scene using fragment operationsE.g.: reflections, mirrorsUses depth, stencil, alpha, … tests
“Multitexture emulation mode”: redrawMultitexture emulation mode : redrawUses framebuffer blending
Vienna University of Technology 71
(light mapping)
Multipass Rendering: How?
(assume redraw scene…)Fi tFirst pass
Establishes z-buffer (and maybe stencil) glDepthFunc(GL_LEQUAL);
Usually diffuse lightingUsually diffuse lightingSecond pass
Z-Testing onlyglDepthFunc(GL_LEQUAL);_
Render special effect using (examples):Blending
Vienna University of Technology 72
BlendingglStencilFunc(GL_EQUAL, 1, 1);
Multipass – Framebuffer BlendingglEnable(GL_BLEND);glBlendEquation(GL_FUNC_ADD);
weighting factorsweighting factors
lt llt l C = Cs S + Cd Dresult colorresult color
incoming (source) incoming (source) fragment colorfragment color framebuffer colorframebuffer color
Other equations: SUBTRACT, MIN, MAX
gg
Vienna University of Technology 73
Other equations: SUBTRACT, MIN, MAX
Multipass – Blending - WeightsglBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA);
C = Cs · + Cd · (1- )Example: transparency blending (window)W i ht b d fi d l t bit il
C Cs Cd (1 )
Weights can be defined almost arbitrarilyAlpha and color weights can be defined p gseparatelyGL ONE GL ZERO GL DST COLORGL_ONE, GL_ZERO, GL_DST_COLOR, GL_SRC_COLOR, GL_ONE_MINUS_xxx
Vienna University of Technology 74