Image Synthesis

computer graphics & visualization

Image Synthesis

What is left to do?„The Real-World Problems!!!“

Image Synthesis – WS 07/08Dr. Jens Krüger – Computer Graphics and Visualization Group

So far …• Realtime

– OpenGL / DirectX basics– Programmable pipeline– Effects

• Non-Realtime– Raytracing– Radiosity– Photon Mapping

What‘s left?• „Optimizations“

• „Problem size“

TerabyteTo

ExabyteDatasets!

Image Synthesis

Scenegraph APIs

• Rather cumbersome to use for complex applications• Slow development cycle

• Non-reusable code• Non-structured data often difficult to optimize

• Expert knowledge often necessary• Low-level und hardware orientiert• Imperative instead of descriptive• No support for interactions• Minimal support for picking• No data formats• Immediate mode not always desired

Low-level APIs like GL/DX as well as self writen raytracing/radiosity implementations have several disadvantages:

Goals of a high-level API • Easy to use, easy to extend

• Multi processing• Build-in methods for efficient scene handling and rendering

• Plattform and Hardware Independent

• Management of complex scenes

• Hiding implementation details

• High performance - optimization

• Reliability

• Object oriented model

• Descriptive scene description

• Interaction and picking

• Data formats

What is a scenegraph?• An ordered data structure that stores a description of the scene to be

rendered

• The scenegraph comes with a traversal mechanism used to process the scene description in a pre-determined order

• It is accompanied with a data base that provides objects needed in graphics applications

API provides the interface to the user

• Library calls to

• Create the scenegraph• Access the data base• Use auxiliar objects and methods

• Usually, the API provides additional components

• Viewer necessary to view the scene• Editors• Manipulators

Scenegraph APIs:Rendering modes• Immediate mode (low-level)

• Render everything as soon as specified• No intermediate representation

• Retained mode (high-level)

• Store the scene it in an intermediate representation• Optimize (re-organize/simplify) this representation for rendering purposes

• Compile-immediate mode

• Caching strategy used to optimize the rendering of static parts of the scene (display lists)

Scenegraph APIsA historic overview

– 1992 OpenInventor (SGI) ? – 1992 Iris Performer (SGI) – 1995 Cosmo3D (SGI) – 1996 Optimizer (SGI) – 1996 DirectModel/Jupiter (HP) – 1997 OpenGL++ (ARB-Sun,SGI,Intel,...) – 1997 Java3D (Sun) – 1998 Fahrenheit (SGI,MS) – 2000 OpenSG (OpenSource - Germany) – 2000 OpenScenegraph – 2003 NVSG NVidia Scenegraph ?

Scenegraph APIsFeatures of a scenegraph API

– The scene is stored in a hierarchically organized tree-like data structure

Nodes are the objects describing the sceneEdges are references that define the relationship between objects

Appearance Geometr

Transform

Scenegraph APIsThe scenegraph is implemented as a directed acyclic graph

– It organizes and controls the rendering of the scene– It provides fixed or application dependent traversal

order (Depth-first, breath-first, arbitrary)– It determines the way in which attributes are

propagated to/associated with objects

Scenegraph APIsEach node belongs to a specific class

– Shape nodes -> geometry– Appearance nodes -> material properties– Transform nodes -> local object transforms– Environment nodes -> lights, space properties– Camera nodes -> viewing parameters– Etc.

OpenInventorFirst widely used scenegraph from SGI

– IRIS Inventor based on IrisGL– 1994: OpenInventor based on OpenGL– Basis for many products and projects

• Iris Explorer Visualization Toolkit• Amira

– Since 2000 original in OpenSource– Multipipe, Stereo,

3D Textures, Kollisionen,...– Multiplatforms:

Linux, Windows– http://oss.sgi.com/projects/inventor/

OpenInventor - Concepts• Scenegraph (DAG) for 3D-Models with many node types (Shape, Camera, Light, ...)

• Traversal with many actions (render, pick, bounding box, callback, ...)

• Interaction with fast picking: Manipulators

• GUI-Elements: Components (z.B. Viewer)

• Animation: Engines

• 3D-data format

• Plattform independent via OpenGL

• User defined extensions (ToolMaker)

• C++-Class library

OpenInventor - Nodes• Geometry: Shape Nodes

• Cube, Cone, Cylinder, Sphere, 2D/3D-Text• NURBS, Triangle Strips, Quad Meshes, Face Set,...

• Properties: Property Nodes• Material, Drawstyle, Environment, Texture, Light Model,...• Transformationes

• Grouping: Group Nodes

• Group, Separator, Switch, ...

• Virtual camera: Camera Nodes

• Perspective, Orthographic

• Lighting: Light Nodes

• Point, Directional, Spot, ...

OpenInventor - Example

OpenInventor - ExampleTable-scene with path to one leg

Different nodes trigger different actions• Render nodes provide a renderAction that implements the algorithm used

to render the object

• The renderAction is called in the current traversal state whenever a render node is visited

• State nodes just change the current render state without rendering anything

• Traversing the graph means visiting each node

Scenegraph APIsScenegraph traversal

– OpenInventor• Depth-first traversal• Inheritence of states from left to right and top to bottom

– Performer• Depth-first traversal• Different traversal processes (app., cull, draw, etc.)• No inheritance from left to right

OpenInventor - Details• Traversal from top to bottom and left to right• Update of traversal state

– Current transformation, geometry, material, texture, ...– Separator: push/pop of traversal state

• Multiple references to nodes possible– Shared instancing– Explicit reference counting with automatic delete (important for memory

management under C++)• Data elements of node: Fields

– Basis types: Bool, Color, Vec3f, Matrix, List, Name, String, ...– Single Value Fields and Multiple Value Fields– setValue/getValue and defaults – Ignore/Overwrite Flags

OpenInventor - Interactions• Events

– Window independent (Event Translator)– Processed by application, viewer or nodes– HandleEventAction: propagates events through graph until

one node reacts (grab)• Picking

– Yields path through scenegraph• Manipulators

– Interaction nodes with own geometry (Pick, Feedback)– Supports 3D-Navigation with 2D-Mouse (Projector)– Consists of Draggers (one Transformation)

OpenInventor - ManipulatorsHandlebox

OpenInventor - Manipulatoren Transformer Spot Light

OpenInventor - Viewer & Editors

OpenInventor - Scenegraph-EditorDirect modifikation of nodes in graph

OpenInventor – Data format#Inventor V2.0 asciiSeparator {

PerspectiveCamera {position 0 0 4.18154nearDistance 2.44949farDistance 5.91359focalDistance 4.18154

}DirectionalLight {}Material {

diffuseColor [ 1 1 0.2 ]}Cone { }

OpenInventor – Data format• OpenInventor-2.0-Format

– Basis for VRML 1.0– Only slight differences, e.g. nodes for Web-Access

• VRML 2.0 and following (VRML 97, X3D)– 3D-Formats for Internet– Take some OpenInventor concepts– But: structural changes

Scenegraph APIsSG APIs provide additional features to achieve interactive frame rates for large scenes

– Handling and manipulation of huge data structures– Multiprocessor/Multipipe rendering– Build-in components for efficient rendering of large

geometries– Pre-defined components, e.g. viewer including

event handling

Scenegraph APIsBasic scenegraph optimization techniques can be split into different categories

– Storage – Generate Stage– Graph traversal – Traversal Stage– Geometry processing – Transform Stage– Rasterization

Generate stage optimization• Storage

– Links instead of instances– Hierarchical partitioning of the scenegraph

• On-demand rendering from disk• Paging of (less frequently used) objects

– Multiprocessor-control• Thread manager• Synchronization of multiprozessor-SG traversal

– Geometry compression• Quantization and topology encoding

– Geometry simplification and optimization• Mesh reduction• Efficient conversion in strips and fans

Generate stage optimizationGeometry simplification

– Objects are represented with less triangles– Relevant features and the original shape within a

certain error tolerance are retained– Automatic selection of triangles to be removed

based on local curvature or other criteria– Automatic triangle removal

• Collapse, split, swap

Generate stage optimizationMesh reduction criteria

Mean-curvature

Gauss-curvature 2-i

Difference between area before and after reduction

Mesh analysisCriteria to analyze mesh quality

– Distance to other mesh, e.g. mesh before modification • Hausdorff-distance (two-sided):

dHaus(A,B) = max{maxaA dist(a,B), maxbBdist(A,b)} dist(a,B) = minbB dist(a,b) (one-side Hausdorff) dist(A,b) = minaA dist(a,b) (one-side Hausdorff) • Maximum of all shortest vertex-to-mesh distances

Mesh analysisHausdorff-distance dhaus=d

Mesh analysisHausdorff-distance

– d1=maxqQ dist(P,q)– d2=maxpP dist(p,Q)

Mesh analysisHausdorff-distance dhaus=d1=d2

Generate stage optimizationTriangle removal by re-tesselation

– Use gluTesselate to tesselate arbitrary polys

Generate stage optimization• n practical applications

– User defines compression factor– Different Levels-of-Detail are precomputed– Manual or automatic LOD selection during

rendering– E.g, less details are shown during movement or if

the object is far away

Generate stage optimizationGeometry simplification (5,10,50,100 %)

progressive

nonprogressive

Generate stage optimization• Geometry simplification

#: 1.087716 286578 30392 3774

Original

Simplified

Generate stage optimization• Geometry optimization

– Large redundant structures are stored into vertex arrays• Sharing of multiple defined geometry and attributes

– Re-organization into strips and fans• Adjacency information is exploited for efficient coding

5(012)(123)(234)(345) ->(012345) one strip

2 34 (012)(023)(034) -> (01234) one

Traversal stage optimization• Improved traversal by re-organization of the graph

structure– Spatialization

• Nodes are re-arranged with respect to their spatial position or other attributes

• Can lead to a more efficient traversal order• Accelerated culling and picking

– Flattening the graph structure results in less recursion stages• Better memory efficiency• Less CPU load

– Group geometry with equal material– Minimize GPU state changes

Traversal stage optimization• Spatialization

- Shared Instancing: group nodes

- Build parallel SG of inner nodes for efficient culling

- Remove unnecessary hierarchy stages

- Resolve shared instances

- Allows for adaptive BV-Hierarchies

Transform stage optimization• Objects rendered multiple times are stored into OpenGL

display lists• Renderable primitives that won´t be displayed are

automatically determined and culled– Culling: discard invisible objects before they are transfered to

the GPU• backface culling

discard backfacing facets• detail culling

discard „small“ objects• view frustrum culling

discard objects outside view frustum• occlusion culling

discard occluded objects

Transform stage optimization• Backface culling

– Triangles with a normal pointing away from the viewer are discarded

– Efficient computation by means of the area of the polygon in window coordinates

– Might lead tostrange results

Transform stage optimizationView frustum culling

– Parts of the scene outside the view frustum are discarded– Not supported by OpenGL; clipping is performed after

transformation to screen space coordinates– Efficient intersection test between objects and frustum

necessary

Transform stage optimizationOcclusion culling

– Parts of the scene that are entirely hidden by others are discarded

Occluded parts

– Sorting and front-to-back rendering necessary• Sorting key is distance to viewer

– Occlusions are determined in two stages:• Specify less complex enclosing hull (BoundingVolume)

and compute its projected screen rectangle• Grab the depth values within the rectangle and compare

them to the minimal depth between convex hull and view plane

– If at least one depth value is larger, the object has to be rendered

– Hardware support exists in OpenGL 1.5• A specific render mode is provided that allows to render

objects without affecting the color/depth buffer• An OcclusionQuery can be used (with BoundingVolume)

to determine the number of fragments that have been affected

– If no fragment has been affected, the object is invisible and can be culled, i.e. has not to be rendered

Transform stage optimization• Detail culling

– Objects that are small compared to the viewport resolution are discarded or represented at a coarse level

• E.g. if the entire object is projected into a few pixel

Transform stage optimizationSpace partitioning and portal culling

– Most frequently used in game engines– Partition the scene into disjoined parts and classify which

parts can´t see each other• Define portals through which one part can see another one

– While in one part, discard everything that can´t be seen and clip against the portal frustum

Transform stage optimizationFor efficient occlusion culling the API needs to compute ´good´ enclosing hulls

– Should be as close as possible to the surrounding geometry

– Should be as simple as possible for effcient computation and rendering

– Most commonly, bounding volumes are used

Axis aligned

Oriented

Transform stage optimizationBounding volumes

– Minimizing the number of computations necessary ... • ... to check wether an object might be hit or not• ... to check wether an object is contained in a cell or not

Bounding spheres axis-aligned BB oriented BB more complex intersection test

Transform stage optimization• Bounding volume hierarchy

– Adaptive space partitioning– Search complexity is O(logn)

Transform stage optimization• Bounding volume hierarchies

– Manually define BVs for sets of objects – Bottom-up approach

• Merge BVs of objects• Consider size of sectional areas• Consider number of objects

– Top-down approach• Recursively build smaller BVs• Subdivide underlying space

– Consider load needed to traverse each sub-space

Transform stage optimizationBounding volume hierarchies for triangle meshes

– Interpret mesh as point cloud– Successively generate smaller BVs

• Consider number of elements in each BV• Consider size of bounding volumes

Transform stage optimization• Normal-Maps for correct highlights• Decoupling of geometry and illumination

Transform stage optimization• For efficient occlusion culling the API needs to

compute ´good´ space partitions• Should allow for efficient visibility ordering of

objects• Most commonly tree structures are used

– Octrees– BSP-Trees– KD-Trees

Transform stage optimization• Space partitioning

– Minimizing the number of objects to be rendered by visibility ordering

– Grids partition the space into equidistant cells– Cells store links to all objects they touch– Grid cells are ordered with respect to current

viewpoint and viewing direction

Transform stage optimization• Grids

– Structure – Traversal

Ordering of cells easy to determine

Transform stage optimizationOctrees

– Hierarchical space partitioning• Start with one cell that covers everything• Recursively subdivide it into 2x2x2 äquidistant sub-cells• Each sub-cell is going to be subdivided further on if it

meats a certain criterion – E.g. it still contains too many objects

Transform stage optimizationOctrees

– Complex traversal algorithm • Varying distances between sub-cell borders at different levels• Recursive traversal and finding of neighbors in non-balanced trees

Transform stage optimization• Grids vs. Octrees

– Octrees:• Arbitrary but adaptive granularity• Non-simple traversal algorithms

– Grids:• Simple and efficient traversal• High memory requirement • Adaptivity by hierachical grids

Transform stage optimizationOctrees vs. Grids

Transform stage optimizationkD-Trees

– A tree where each node corresponds to an axis parallel split of the domain

• Objects are located in the leaves– Splitting is performed in alternating directions

Transform stage optimizationkD-Trees

– If a node is completely contained in a region, then all of its children are

-> output this subtree– Otherwise, recursively traverse the intersected subtree

Transform stage optimizationBsp-Trees

– Partition the space into two half-spaces at each level

– Continue with the same procedure for each of the half-spaces

– Choose arbitrary positions and orientation of the dividing planes

• More efficient scene partitioning than with octrees • Process half-space that contains viewpoint first

Transform stage optimizationOctrees vs. kD vs. Bsp

– Less objects have to be stored multiple times for kD/Bsp-trees

– More flexible/efficient scene partitioning than with octrees

– Traversing kD/Bsp-trees is more complex• Test with arbitrarily oriented dividing planes

Transform stage optimizationMultiprocessor rendering

– Parallelization of the rendering process difficult in general

• Explicit sorting necessary, i.e. transparent objects• Occlusion culling almost impossible• Good partitioning in order to achieve load balancing• High memory bandwidth necessary in order to merge

partial results rendered to multiple screens• Semaphor concepts needed to render from multiple

processors into one screen

Transform stage optimizationHow to parallelize ?

A)• Store the entire scene on each processor• Assign disjoint screen regions to each processor• Each unit renders the scene onto its local screen region• Append the results

B)• Partition the scene into disjoint, contiguous parts and distribute• Each unit renders it´s partition onto its local screen• Blend the results together

C)• Divide the rendering process into different tasks (Iris Performer)

that are assigned to processors• Pipeline the computation

Transform stage optimization• Performer style parallelization

– Rendering is split into• APP: the application dependent tasks• CULL: culling of non-visible geometry• DRAW: rendering the scene

– After the third rendering pass all three processes are busy– This strategy might lead to bad load balancing– Only three processors can be used in parallel

App 1 App 2 App 3Cull 1 Cull 2 Cull 3

Draw 1 Draw 2 Draw 3

Transform stage optimization• Synchronization

– Rendering output of multipe processors/graphics pipelines have to be synchronized

• Application in Tiled displays– Frame-to-frame synchronization

• Synchronize glSwapBuffer() using proc-to-proc communication, i.e. TCP/IP, MPI etc.

– Genlock-synchronization • Workstation’s graphics output is locked (LOCK) to an externally

generated (GEN) signal• Some graphics hardware ensures subscanline-level synchronization

to an external sync source• Synchronize with vertical refresh rate

Rasterization stage optimization• Build-in components for improved rasterization

– ???– Texture compression already available in low-level APIs

• Multiple texture entries are coded by their difference to an average value

• Less bits spent to adress the difference – Use textures instead of geometry

• Billboards (an image of a part of the scene) might be useful to display static parts of complex geometry

• Lightmaps, cubemaps etc. – Per-pixel deferred shading to render complex geometry

• Shift shading calulations to the fragment stage

Image Synthesis

Documents

Transcript of Image Synthesis

Photographic Image Synthesis with Cascaded Reﬁnement Networksvladlen.info/papers/photographic-image-synthesis.pdf · Photographic Image Synthesis with Cascaded Reﬁnement Networks

IMAGE SYNTHESIS 1 Image Synthesis Image synthesis operations create images from other images or non-image data Used when a desired image is either physically.

Adaptive Image-space Stereo View Synthesis

Medical Image Synthesis for Digital Simulation and Image ...

Generative adversarial text to image synthesis

Deep Markov Random Field for Image Modeling arXiv:1609 ... · Markov Random Field Deep Neural Networks Applications Texture Synthesis Image Super-Resolution Image Synthesis Fig.1:

A Survey on Variational Image Inpainting, Texture Synthesis and Image Completion

Realistic Image Synthesis

Realistic Image Synthesis - Universität des SaarlandesRealistic Image Synthesis SS18 –Instant Global Illumination Philipp Slusallek Instant Radiosity [Siggraph97] • Trace few

Image Synthesis Rabie A. Ramadan, PhD 7. 2 Image Rasterization.

Scribbler: Controlling Deep Image Synthesis With Sketch ...openaccess.thecvf.com/.../papers/Sangkloy_Scribbler... · Scribbler: Controlling Deep Image Synthesis with Sketch and Color

Example-Guided Style-Consistent Image Synthesis …openaccess.thecvf.com › content_CVPR_2019 › papers › Wang...Example-Guided Style-Consistent Image Synthesis from Semantic Labeling

Self-organizing trees for Image Synthesis

Realistic Image Synthesis with Light Transport

Image Synthesis with a Single (Robust) Classifierpeople.csail.mit.edu/tsipras/synthesis_poster.pdf · Image Synthesis with a Single (Robust) Classifier Author Shibani Santurkar*,

Synthesis of Image Calculation of Visibility

Realistic Image Synthesis - Universiteit Utrecht

High-Resolution Image Synthesis and Semantic Manipulation ...openaccess.thecvf.com/content_cvpr_2018/CameraReady/0768.pdf · High-Resolution Image Synthesis and Semantic Manipulation

Principles of Digital Image Synthesis

Image Synthesis