Image Synthesis
description
Transcript of Image Synthesis
computer graphics & visualization
Image Synthesis
The programmable(D3D 10) Pipeline
computer graphics & visualization
Image Synthesis – WS 07/08Dr. Jens Krüger – Computer Graphics and Visualization Group
Pixel Stage
Vertex Stage
User / Driver
First: the fixed function pipeline
Transform & Lighting Rasterizer
TexturingBlending/Ops
Texture 3Texture 2Texture 1Texture 0
Vertex Stream
Fragment Stream
computer graphics & visualization
Image Synthesis – WS 07/08Dr. Jens Krüger – Computer Graphics and Visualization Group
Fixed Function Pipeline
• Properties
• States can be changed• States out of a fixed number of alternatives• Same principal operations always performed on stream elements• Can only discard certain operations• Action on stream elements (vertices/polygons/fragments) can not be programmed
computer graphics & visualization
Image Synthesis – WS 07/08Dr. Jens Krüger – Computer Graphics and Visualization Group
Fixed Function Pipeline• To In the following:
– Non-OpenGL effects using fixed function pipeline– Multipass
• Combine pixel results of multiple rendering passes– Pre-computed texture maps
• Pre-compute results of certain non-standard operations and store in texture maps
computer graphics & visualization
Image Synthesis – WS 07/08Dr. Jens Krüger – Computer Graphics and Visualization Group
Advanced shadingThe Phong lighting model
– – N: surface normal– L: light source direction– R: reflection vector– V: direction to the viewpoint– n: materials specular reflection exponent
• Defines the highlights falloff from the direction of reflection– Specular reflections show the color of the light source
• Not valid for all materials
nsdpattaa RVkLNkIfkII
computer graphics & visualization
Image Synthesis – WS 07/08Dr. Jens Krüger – Computer Graphics and Visualization Group
Advanced shadingThe Phong/Blinn lighting model
– – The Halfway vector H points into the direction of a
surfaces´ normal that reflects light into V– If light source and view point are considered to be
at infinity, H is a constant– The angle between R and V is twice the angle
between N and H if all above vectors are complanar• Highlights appear sharper in the Phong model
nsdpattaa HNkLNkIfkII
computer graphics & visualization
Image Synthesis – WS 07/08Dr. Jens Krüger – Computer Graphics and Visualization Group
Advanced shading• In OpenGL, per-vertex Phong lighting is
employed to simulate specular reflections• But, during scan-conversion vertex color is
interpolated – OpenGL uses Gouraud shading instead of Phong
shading– Texture color affects highlight color– Sharp highlights might be missed if the surface is
tesselated coarsely
computer graphics & visualization
Image Synthesis – WS 07/08Dr. Jens Krüger – Computer Graphics and Visualization Group
Advanced shading• To get separate highlight color highlights have
to be added to the diffusely lit textured surface– OpenGL computes Ct(Cd + Cs)– Instead
• Compute Cd and Cs separately • Modulate Cd with Ct • Add result to Cs
– Solution can be obtained by -blending
computer graphics & visualization
Image Synthesis – WS 07/08Dr. Jens Krüger – Computer Graphics and Visualization Group
Advanced shading• Multipass technique to get separate highlight
color– Render the textured object without specular light
CtCd
– Render the untextured object with specular light only Cs
– Combine the results in the frame buffer by additive blend
• glBlendFunc(GL_ONE, GL_ONE)• glBlendEquation(GL_ADD)
computer graphics & visualization
Image Synthesis – WS 07/08Dr. Jens Krüger – Computer Graphics and Visualization Group
Advanced shading
Textured with diffuse light
Untextured with specular
light
Combined images
In the original image the
highlight can hardly be seen
computer graphics & visualization
Image Synthesis – WS 07/08Dr. Jens Krüger – Computer Graphics and Visualization Group
Advanced shading• Using OpenGL 1.2
glLightModel(GL_LIGHT_MODEL_COLOR_CONTROL, GL_SEPARATE_SPECULAR_COLOR);
computer graphics & visualization
Image Synthesis – WS 07/08Dr. Jens Krüger – Computer Graphics and Visualization Group
A few words on OpenGL Extensions• The concept of extensions
– Naming conventions• ARB vs. cross vendor vs. vendor specific extensions
– Querying:• compile time vs. run time
computer graphics & visualization
Image Synthesis – WS 07/08Dr. Jens Krüger – Computer Graphics and Visualization Group
OpenGL Extensions• Mechanism for providing access to non-
standard features of the hardware• Every vendor may define own extensions
without having to ask anybody else• Multiple vendors may cooperate to define more
common extensions to ensure compatibility across platforms
computer graphics & visualization
Image Synthesis – WS 07/08Dr. Jens Krüger – Computer Graphics and Visualization Group
Naming Conventions for Extensions• ARB: Architecture Review Board
– Ext. that is likely to become part of core OpenGL in the future
• EXT: Multi-vendor extensions– Ext. that multiple vendors have agreed upon
• NV, ATI, SGI, SUN...: vendor-specific ext.– Ext. supported only by one vendor, sometimes
experimental and not available on all systems of that vendor (e.g. NVX, SGIS, SGIX)
computer graphics & visualization
Image Synthesis – WS 07/08Dr. Jens Krüger – Computer Graphics and Visualization Group
Dealing with Extensions• Since all extensions are optional, applications
cannot rely on their presence• Have to check both at compile time and at run
time!• Compile time check:
#ifdef GL_ARB_multitexture// use multitexture ext. in here...#endif
computer graphics & visualization
Image Synthesis – WS 07/08Dr. Jens Krüger – Computer Graphics and Visualization Group
Dealing with Extensions (2)Run time check:
const Glubyte *extstring = glGetString(GL_EXTENSIONS);strcpy(exts, extstring);extsupported= 0;next= strtok(exts, “ “);while(next) {
if(!strcmp(next,“GL_ARB_multitexture”))extsupported= 1; break;
next= strtok(exts,NULL);}
computer graphics & visualization
Image Synthesis – WS 07/08Dr. Jens Krüger – Computer Graphics and Visualization Group
Advanced shadingHow to get sharp highlights without adaptive refinement (tesselation)
– Using the Blinn/Phong model specular highlights depend on L,N and V
– By fixing L and V highlights depend only on N– N can be used to index into a texture map in which
the reflection for R = f(N,L) are stored– Texture map looks like reflection from chrome
hemisphere
computer graphics & visualization
Image Synthesis – WS 07/08Dr. Jens Krüger – Computer Graphics and Visualization Group
Texture coordinates computed from the
normal vector
Texture coordinates map into sphere map
texture
Texel contain pre-computed Phong
lighting
Advanced shadingSpherical texture maps
computer graphics & visualization
Image Synthesis – WS 07/08Dr. Jens Krüger – Computer Graphics and Visualization Group
Advanced shadingPhong shading with highlight texture
– Render the object with diffuse light CtCd
– Render the unlit, untextured white object and modulate it with the highlight texture Cht
– Combine the results in the frame buffer by additive blend
– … or use multi-textures in one rendering pass
computer graphics & visualization
Image Synthesis – WS 07/08Dr. Jens Krüger – Computer Graphics and Visualization Group
Advanced shadingPhong shading with highlight texture
Textured with diffuse light
White object modulated by
highlight texture
Blended images
computer graphics & visualization
Image Synthesis – WS 07/08Dr. Jens Krüger – Computer Graphics and Visualization Group
No more “Hacks” …The Programmable pipeline• Direct3D 9 / OpenGL 2.0 Model
– Still used in many existing games– Supported by any graphics card of the last 5 years– Supported by every mayor OS
(DX only on Windows)• Direct3D 10 / OpenGL 2.0 + Ext / OpenGL 3.0
– Supported only by the latest GPUsNVIDIA 8xxx or ATI Radeon 2xxxx
– OpenGL 3.0 not out yet must be used via extensions– Direct3D 10.0 only supported by Windows Vista
(Direct3D 10.1 requires Vista SP1)
computer graphics & visualization
Image Synthesis – WS 07/08Dr. Jens Krüger – Computer Graphics and Visualization Group
The Direct3D 9.0c PipelineInput Data
Input AssemblerStage (IA)
Vertex ShaderStage (VS)
Rasterizer Stage(RS)
Pixel ShaderStage (PS)
Output MergerStage (OM)
Output Data
MemoryResources:
Buffers, TexturesBuffer
Texture, Constants
Texture , Constants
States
Texture
computer graphics & visualization
Image Synthesis – WS 07/08Dr. Jens Krüger – Computer Graphics and Visualization Group
Features of the D3D 9.0c pipeline• Programmable vertex and pixel shader
with limited shader program length (and many other limitations)
• Vertex and pixel shader texture fetch capabilities (some limitations apply)
• Configurable output merger (Blending/Ops/Tests)• Fixed function pipeline vertex and/or pixel shader
can be used• Floating point through the entire pipeline (except
for the blending)
computer graphics & visualization
Image Synthesis – WS 07/08Dr. Jens Krüger – Computer Graphics and Visualization Group
Benefits of the programmable Pipeline• No more “hacks” more sophisticated “hacks”• Straight forward program flow without the
necessity of cryptic configuration changes• Decent knowledge of the transformation and
lighting required to program the pipeline everything has to be done “by hand”
• Shaders can be programmed in a high level language HLSL/GLSL
computer graphics & visualization
Image Synthesis – WS 07/08Dr. Jens Krüger – Computer Graphics and Visualization Group
Shaders: a first look (HLSL)“Basic HLSL”-Sample from the DirectX SDK://--------------------------------------------------------------------------------------// Global variables//--------------------------------------------------------------------------------------float4 g_MaterialAmbientColor; // Material's ambient colorfloat4 g_MaterialDiffuseColor; // Material's diffuse colorint g_nNumLights;
float3 g_LightDir[3]; // Light's direction in world spacefloat4 g_LightDiffuse[3]; // Light's diffuse colorfloat4 g_LightAmbient; // Light's ambient color
texture g_MeshTexture; // Color texture for mesh
float g_fTime; // App's time in secondsfloat4x4 g_mWorld; // World matrix for objectfloat4x4 g_mWorldViewProjection; // World * View * Projection matrix
//--------------------------------------------------------------------------------------// Texture samplers//--------------------------------------------------------------------------------------sampler MeshTextureSampler = sampler_state{ Texture = <g_MeshTexture>; MipFilter = LINEAR; MinFilter = LINEAR; MagFilter = LINEAR;};
//--------------------------------------------------------------------------------------// Vertex shader output structure//--------------------------------------------------------------------------------------struct VS_OUTPUT{ float4 Position : POSITION; // vertex position float4 Diffuse : COLOR0; // vertex diffuse color (note that COLOR0 is clamped from 0..1) float2 TextureUV : TEXCOORD0; // vertex texture coords };
//--------------------------------------------------------------------------------------// This shader computes standard transform and lighting//--------------------------------------------------------------------------------------VS_OUTPUT RenderSceneVS( float4 vPos : POSITION, float3 vNormal : NORMAL, float2 vTexCoord0 : TEXCOORD0, uniform int nNumLights, uniform bool bTexture, uniform bool bAnimate ){ VS_OUTPUT Output; float3 vNormalWorldSpace; float4 vAnimatedPos = vPos; // Animation the vertex based on time and the vertex's object space
position if( bAnimate )
vAnimatedPos += float4(vNormal, 0) * (sin(g_fTime+5.5)+0.5)*5;
// Transform the position from object space to homogeneous
projection space
Output.Position = mul(vAnimatedPos, g_mWorldViewProjection); // Transform the normal from object space to world space vNormalWorldSpace = normalize(mul(vNormal,
(float3x3)g_mWorld)); // normal (world space) // Compute simple directional lighting equation float3 vTotalLightDiffuse = float3(0,0,0); for(int i=0; i<nNumLights; i++ ) vTotalLightDiffuse += g_LightDiffuse[i] *
max(0,dot(vNormalWorldSpace, g_LightDir[i])); Output.Diffuse.rgb = g_MaterialDiffuseColor * vTotalLightDiffuse + g_MaterialAmbientColor * g_LightAmbient; Output.Diffuse.a = 1.0f; // Just copy the texture coordinate through if( bTexture ) Output.TextureUV = vTexCoord0; else Output.TextureUV = 0; return Output; }
struct PS_OUTPUT{ float4 RGBColor : COLOR0; // Pixel color };
PS_OUTPUT RenderScenePS( VS_OUTPUT In, uniform bool bTexture ) { PS_OUTPUT Output;
// Lookup mesh texture and modulate it with diffuse if( bTexture ) Output.RGBColor = tex2D(MeshTextureSampler, In.TextureUV) * In.Diffuse; else Output.RGBColor = In.Diffuse;
return Output;}
technique RenderSceneWithTexture1Light{ pass P0 { VertexShader = compile vs_2_0 RenderSceneVS( 1, true, true ); PixelShader = compile ps_2_0 RenderScenePS( true ); // trivial pixel shader (could use FF instead if desired) }}technique RenderSceneWithTexture2Light{ pass P0 { VertexShader = compile vs_2_0 RenderSceneVS( 2, true, true ); PixelShader = compile ps_2_0 RenderScenePS( true ); // trivial pixel shader (could use FF instead if desired) }}technique RenderSceneWithTexture3Light{ pass P0 { VertexShader = compile vs_2_0 RenderSceneVS( 3, true, true ); PixelShader = compile ps_2_0 RenderScenePS( true ); // trivial pixel shader (could use FF instead if desired) }}technique RenderSceneNoTexture{ pass P0 { VertexShader = compile vs_2_0 RenderSceneVS( 1, false, false ); PixelShader = compile ps_2_0 RenderScenePS( false ); // trivial pixel shader (could use FF instead if desired) }}
computer graphics & visualization
Image Synthesis – WS 07/08Dr. Jens Krüger – Computer Graphics and Visualization Group
Vertex Shaders: a closer lookVS_OUTPUT RenderSceneVS( float4 vPos : POSITION, float3 vNormal : NORMAL, float2 vTexCoord0 : TEXCOORD0, uniform int nNumLights, uniform bool bTexture, uniform bool bAnimate ){ VS_OUTPUT Output; float3 vNormalWorldSpace; float4 vAnimatedPos = vPos; // Animation the vertex based on time and the vertex's object space position if( bAnimate ) vAnimatedPos += float4(vNormal, 0) * (sin(g_fTime+5.5)+0.5)*5; // Transform the position from object space to homogeneous projection space Output.Position = mul(vAnimatedPos, g_mWorldViewProjection); // Transform the normal from object space to world space vNormalWorldSpace = normalize(mul(vNormal, (float3x3)g_mWorld)); // normal (world space) // Compute simple directional lighting equation float3 vTotalLightDiffuse = float3(0,0,0); for(int i=0; i<nNumLights; i++ ) vTotalLightDiffuse += g_LightDiffuse[i] * max(0,dot(vNormalWorldSpace, g_LightDir[i])); Output.Diffuse.rgb = g_MaterialDiffuseColor * vTotalLightDiffuse + g_MaterialAmbientColor * g_LightAmbient; Output.Diffuse.a = 1.0f; // Just copy the texture coordinate through if( bTexture ) Output.TextureUV = vTexCoord0; else Output.TextureUV = 0; return Output; }
computer graphics & visualization
Image Synthesis – WS 07/08Dr. Jens Krüger – Computer Graphics and Visualization Group
Pixel Shaders: a closer lookPS_OUTPUT RenderScenePS( VS_OUTPUT In, uniform bool bTexture ) { PS_OUTPUT Output;
// Lookup mesh texture and modulate it with diffuse if( bTexture ) Output.RGBColor = tex2D(MeshTextureSampler, In.TextureUV) *
In.Diffuse; else Output.RGBColor = In.Diffuse;
return Output;}
computer graphics & visualization
Image Synthesis – WS 07/08Dr. Jens Krüger – Computer Graphics and Visualization Group
Putting it all together: Effect Files• Store many vertex and pixel shaders as well as
pipeline stages in a single file• Also Store variables, textures, and samplers• Allow for grouping of shaders into techniques and
passes• Can be accessed easily from the main program• Can be created/modified easily from a content
creation tool(AMD RenderMonkey/Nvidia FX Composer)
computer graphics & visualization
Image Synthesis – WS 07/08Dr. Jens Krüger – Computer Graphics and Visualization Group
FX Composer Showcase
computer graphics & visualization
Image Synthesis – WS 07/08Dr. Jens Krüger – Computer Graphics and Visualization Group
Effects in the “Basic HLSL”-Sample //--------------------------------------------------------------------------------------// Global variables//--------------------------------------------------------------------------------------float4 g_MaterialAmbientColor; // Material's ambient colorfloat4 g_MaterialDiffuseColor; // Material's diffuse colorint g_nNumLights;
float3 g_LightDir[3]; // Light's direction in world spacefloat4 g_LightDiffuse[3]; // Light's diffuse colorfloat4 g_LightAmbient; // Light's ambient color
float g_fTime; // App's time in secondsfloat4x4 g_mWorld; // World matrix for objectfloat4x4 g_mWorldViewProjection; // World * View * Projection matrix
computer graphics & visualization
Image Synthesis – WS 07/08Dr. Jens Krüger – Computer Graphics and Visualization Group
Effects in the “Basic HLSL”-Sample texture g_MeshTexture; // Color texture for mesh
//--------------------------------------------------------------------------------------// Texture samplers//--------------------------------------------------------------------------------------sampler MeshTextureSampler = sampler_state{ Texture = <g_MeshTexture>; MipFilter = LINEAR; MinFilter = LINEAR; MagFilter = LINEAR;};
computer graphics & visualization
Image Synthesis – WS 07/08Dr. Jens Krüger – Computer Graphics and Visualization Group
Effects in the “Basic HLSL”-Sample //--------------------------------------------------------------------------------------// Renders scene to render target//--------------------------------------------------------------------------------------technique RenderSceneWithTexture1Light{ pass P0 { VertexShader = compile vs_2_0 RenderSceneVS( 1, true, true ); PixelShader = compile ps_2_0 RenderScenePS( true );
// trivial pixel shader (could use FF instead if desired) }}
computer graphics & visualization
Image Synthesis – WS 07/08Dr. Jens Krüger – Computer Graphics and Visualization Group
Effects in the “Basic HLSL”-Sample technique RenderSceneWithTexture2Light{ pass P0 { VertexShader = compile vs_2_0 RenderSceneVS( 2, true, true ); PixelShader = compile ps_2_0 RenderScenePS( true );
// trivial pixel shader (could use FF instead if desired) }}
computer graphics & visualization
Image Synthesis – WS 07/08Dr. Jens Krüger – Computer Graphics and Visualization Group
The DX 10.0 PipelineInput Data
Input AssemblerStage (IA)
Vertex ShaderStage (VS)
Geometry ShaderStage (GS)
Stream OutputStage (SO)
Rasterizer Stage(RS)
Pixel ShaderStage (PS)
Output MergerStage (OM)
Output Data
Memory Resources:
Buffers, Textures,
Constant Buffers
Buffer
Texture, Constant Buffer
Texture, Constant Buffer
Texture, Constant Buffer
Buffer
States
Buffer, Texture, Constant Buffer
computer graphics & visualization
Image Synthesis – WS 07/08Dr. Jens Krüger – Computer Graphics and Visualization Group
Features of the D3D 10.1 pipeline• Programmable vertex, pixel, and geometry shader
with practically unlimited shader program length• Vertex, pixel, and geometry shader texture fetch
capabilities• Configurable output merger (Blending/Ops/Tests)• Fixed function pipeline vertex and/or pixel shader can
NOT be used anymore• Floating point through the entire pipeline including
blending• API Support for double precision (first hardware
supporting double expected to ship in late 2007)
computer graphics & visualization
Image Synthesis – WS 07/08Dr. Jens Krüger – Computer Graphics and Visualization Group
“Basic HLSL” D3D 10/--------------------------------------------------------------------------------------// This shader outputs the pixel's color by modulating the texture's// color with diffuse material color//--------------------------------------------------------------------------------------PS_OUTPUT RenderScenePS( VS_OUTPUT In, uniform bool bTexture ) { PS_OUTPUT Output;
// Lookup mesh texture and modulate it with diffuse if( bTexture ) Output.RGBColor = g_MeshTexture.Sample(MeshTextureSampler, In.TextureUV) * In.Diffuse; else Output.RGBColor = In.Diffuse;
return Output;}
//--------------------------------------------------------------------------------------// Renders scene to render target using D3D10 Techniques//--------------------------------------------------------------------------------------technique10 RenderSceneWithTexture1Light{ pass P0 { SetVertexShader( CompileShader( vs_4_0, RenderSceneVS( 1, true, true ) ) ); SetGeometryShader( NULL ); SetPixelShader( CompileShader( ps_4_0, RenderScenePS( true ) ) );
SetDepthStencilState( EnableDepth, 0 ); }}
computer graphics & visualization
Image Synthesis – WS 07/08Dr. Jens Krüger – Computer Graphics and Visualization Group
“Basic HLSL” D3D 10/--------------------------------------------------------------------------------------// This shader outputs the pixel's color by modulating the texture's// color with diffuse material color//--------------------------------------------------------------------------------------PS_OUTPUT RenderScenePS( VS_OUTPUT In, uniform bool bTexture ) { PS_OUTPUT Output;
// Lookup mesh texture and modulate it with diffuse if( bTexture ) Output.RGBColor = g_MeshTexture.Sample(MeshTextureSampler, In.TextureUV) * In.Diffuse; else Output.RGBColor = In.Diffuse;
return Output;}
//--------------------------------------------------------------------------------------// Renders scene to render target using D3D10 Techniques//--------------------------------------------------------------------------------------technique10 RenderSceneWithTexture1Light{ pass P0 { SetVertexShader( CompileShader( vs_4_0, RenderSceneVS( 1, true, true ) ) ); SetGeometryShader( NULL ); SetPixelShader( CompileShader( ps_4_0, RenderScenePS( true ) ) );
SetDepthStencilState( EnableDepth, 0 ); }}
computer graphics & visualization
Image Synthesis – WS 07/08Dr. Jens Krüger – Computer Graphics and Visualization Group
Coming up
GPU Effects