Andy Wilson – Dissertation defense, 10/23/02 - Slide 1 Spatially Encoded Image-Space...

Andy Wilson – Dissertation defense, 10/23/02 - Slide 1

Spatially Encoded Image-Space Spatially Encoded Image-Space Simplifications for Interactive WalkthroughSimplifications for Interactive Walkthrough

Andy WilsonAndy Wilson


Introduction: Driving ProblemIntroduction: Driving Problem

• Interactive visualization of large Interactive visualization of large man-made structuresman-made structures

–airplane, power plant, aircraft airplane, power plant, aircraft carriercarrier

• Reported CAD database sizes Reported CAD database sizes 101077-10-1099 polygons polygons

–1 to 100GB raw geometry1 to 100GB raw geometry

–Auxiliary data can be much Auxiliary data can be much largerlarger


Sample Models: Brooks HouseSample Models: Brooks House

260,000 polygons (Source: UNC Walkthrough Group)


Sample Models: SubmarineSample Models: Submarine

800,000 polygons (Source: Electric Boat division, General Dynamics)


Sample Models: Power PlantSample Models: Power Plant

12,700,000 polygons (Source: Anonymous donor)


Sample Models: Double EagleSample Models: Double Eagle

81,000,000 polygons (Source: Newport News Shipbuilding)


What’s hard about this?What’s hard about this?

• Models can be bigger than main memoryModels can be bigger than main memory

• Rendering load can exceed hardware capacity by a factor of Rendering load can exceed hardware capacity by a factor of 100 or more100 or more

– This isn’t getting much betterThis isn’t getting much better

– Reported model sizes are increasing faster than rendering Reported model sizes are increasing faster than rendering capacitycapacity

• Auxiliary data for rendering acceleration exacerbate the Auxiliary data for rendering acceleration exacerbate the storage problemstorage problem

– Increase database size by a factor of 2-10 (or more)Increase database size by a factor of 2-10 (or more)


Common approachesCommon approaches

• Global visibilityGlobal visibility

–Enumerate all visibility events in spaceEnumerate all visibility events in space

• Occlusion cullingOcclusion culling

–Remove hidden objectsRemove hidden objects

• SimplificationSimplification

–Reduce polygon countReduce polygon count

• Sample-based renderingSample-based rendering

–Replace objects with inexpensive approximationsReplace objects with inexpensive approximations


Related Work: Exact global visibilityRelated Work: Exact global visibility

• 2D Visibility Complex2D Visibility Complex

–Pocchiola and Vegter 1996Pocchiola and Vegter 1996

• 3D Visibility Complex3D Visibility Complex

–Durand 1999Durand 1999

• Aspect graphsAspect graphs

–Koenderink and van Doorn 1976, 1979Koenderink and van Doorn 1976, 1979


Related Work: Occlusion (preprocessing)Related Work: Occlusion (preprocessing)

• Cells and portals Cells and portals – Airey and Teller 1990, Luebke and Georges 1995Airey and Teller 1990, Luebke and Georges 1995

» Also has runtime componentAlso has runtime component

• Volume visibility in urban environmentsVolume visibility in urban environments– Schaufler et al. 2000Schaufler et al. 2000

• Occlusion using extended projectionsOcclusion using extended projections– Durand et al. 2000Durand et al. 2000

• Exact region-based visibilityExact region-based visibility– Nirenstein et al. 2002Nirenstein et al. 2002


Related Work: Occlusion (runtime)Related Work: Occlusion (runtime)

• Hierarchical Z-bufferHierarchical Z-buffer

–Greene, Kass, and Miller 1993Greene, Kass, and Miller 1993

• Hierarchical occlusion mapsHierarchical occlusion maps

–Zhang et al. 1997Zhang et al. 1997

• Occlusion culling with multiple graphics pipesOcclusion culling with multiple graphics pipes

–Baxter et al. 2002Baxter et al. 2002


Related work: SimplificationRelated work: Simplification

• Level-of-detail generation/representationLevel-of-detail generation/representation

– Quadric error metrics (Garland and Heckbert 1997)Quadric error metrics (Garland and Heckbert 1997)

– GAPS (Erikson and Manocha 1999)GAPS (Erikson and Manocha 1999)

– Progressive meshes (Hoppe 1996)Progressive meshes (Hoppe 1996)

• View-dependent LOD renderingView-dependent LOD rendering

– View-dependent refinement of PMs (Hoppe 1997)View-dependent refinement of PMs (Hoppe 1997)

– Hierarchical view-dependent simplification (Luebke and Hierarchical view-dependent simplification (Luebke and Erikson 1997)Erikson 1997)

– Merge trees (Xia and Varshney 1996)Merge trees (Xia and Varshney 1996)


Issues with the state of the artIssues with the state of the art

• SimplificationSimplification

– Many objects, deep environmentMany objects, deep environment

• Occlusion cullingOcclusion culling

– Current work: conservative object-based methodsCurrent work: conservative object-based methods

– An order of magnitude away from optimal polygon countAn order of magnitude away from optimal polygon count

• Exact and global visibilityExact and global visibility

– Current implementations do not scale to very large Current implementations do not scale to very large environmentsenvironments


Sample-based renderingSample-based rendering

• Idea:Idea: Replace parts of the environment with Replace parts of the environment with simplified representations simplified representations

–Build from discrete set of image-based samplesBuild from discrete set of image-based samples

• Two major issues:Two major issues:

–What samples do we acquire from what points?What samples do we acquire from what points?

–How do we store, represent, and render them?How do we store, represent, and render them?


Related Work: Image-based renderingRelated Work: Image-based rendering

• Texture mapping (Catmull 1974, Blinn and Newell 1976)Texture mapping (Catmull 1974, Blinn and Newell 1976)

• Movie-Maps (Lippman 1980)Movie-Maps (Lippman 1980)

• Image warping (McMillan and Bishop 1995, Mark 1999)Image warping (McMillan and Bishop 1995, Mark 1999)

• Layered depth imagesLayered depth images

– Shade et al. 1998, Chang et al. 1999Shade et al. 1998, Chang et al. 1999

• Light fields (Levoy and Hanrahan 1997, Isaksen et al. 2000)Light fields (Levoy and Hanrahan 1997, Isaksen et al. 2000)

• Lumigraphs (Gortler et al. 1997, Buehler et al. 2001)Lumigraphs (Gortler et al. 1997, Buehler et al. 2001)


Related work: ImpostorsRelated work: Impostors

• Textured billboards in flight simulationTextured billboards in flight simulation

• Textured clusters (Maciel and Shirley 1995)Textured clusters (Maciel and Shirley 1995)

• Dynamic replacement of geometry with images (Aliaga and Dynamic replacement of geometry with images (Aliaga and Lastra 1997)Lastra 1997)

• Textured depth meshesTextured depth meshes

– Darsa, Costa, and Varshney 1997Darsa, Costa, and Varshney 1997

– Decoret, Darsa, and Sillion 1999Decoret, Darsa, and Sillion 1999

– Aliaga et al. 1999Aliaga et al. 1999

Jeschke and Wimmer 2002Jeschke and Wimmer 2002


Thesis StatementThesis Statement

Distant objects in complex environments can be well Distant objects in complex environments can be well approximated by image-based simplifications approximated by image-based simplifications constructed using an incremental search for the next constructed using an incremental search for the next best view. Encodings of these simplified best view. Encodings of these simplified representations that eliminate redundant information representations that eliminate redundant information by exploiting spatial relationships between samples by exploiting spatial relationships between samples enable both higher fidelity and faster rendering when enable both higher fidelity and faster rendering when compared with previous impostor-based approaches.compared with previous impostor-based approaches.


New ResultsNew Results

• Next-best-view algorithm guided by visibility errors Next-best-view algorithm guided by visibility errors and the Voronoi diagram of existing viewpointsand the Voronoi diagram of existing viewpoints

• Spatial video encoding for a database of images in 3-Spatial video encoding for a database of images in 3-spacespace

• Incremental textured depth meshes as simplified Incremental textured depth meshes as simplified geometric impostors for distant objectsgeometric impostors for distant objects


Outline Outline

• IntroductionIntroduction

• Cell-based walkthroughCell-based walkthrough

• Incremental Voronoi-based samplingIncremental Voronoi-based sampling

• Spatial video encodingSpatial video encoding

• Incremental textured depth meshesIncremental textured depth meshes

• Conclusions and Future WorkConclusions and Future Work


Cell-based walkthroughCell-based walkthrough• ObservationObservation: Nearby geometry tends to take up much of the : Nearby geometry tends to take up much of the

viewview– But most resources are devoted to faraway objects!But most resources are devoted to faraway objects!

• ApproachApproach: Partition the model into nearby and distant : Partition the model into nearby and distant geometrygeometry– Render nearby objects at high qualityRender nearby objects at high quality– Replace distant objects with some inexpensive Replace distant objects with some inexpensive

approximationapproximation• Basic ideaBasic idea: Partition environment into cells. : Partition environment into cells.

– For each cell, partition model into nearby and distant For each cell, partition model into nearby and distant geometry, geometry,

– Represent near and far field separatelyRepresent near and far field separately


Near and far fieldNear and far field

• Partition space into Partition space into cellscells

• Each cell has a Each cell has a cull boxcull box

– Divides model geometry:Divides model geometry:» nearby objects (the nearby objects (the near fieldnear field) )

» distant objects (the distant objects (the far fieldfar field))

• Objects that overlap the interior Objects that overlap the interior of the cull box are part of the of the cull box are part of the near fieldnear field

• All other objects belong to the All other objects belong to the far fieldfar field

Cull box

Cell

Near field Far field


Near and Far Field: ExampleNear and Far Field: Example

• Highlighted geometry falls Highlighted geometry falls in the near fieldin the near field

• All other objects in All other objects in environment fall into far environment fall into far fieldfield

• Each face of the cull box is Each face of the cull box is covered with an inexpensive covered with an inexpensive impostorimpostor


Impostors: Textured depth meshesImpostors: Textured depth meshes

• Stage analogyStage analogy– Build papier-machBuild papier-machéé model model – Paint it to resemble original objectsPaint it to resemble original objects– Looks about right from a distance Looks about right from a distance – Quite cheap to construct!Quite cheap to construct!

• Corresponding algorithmCorresponding algorithm– Capture frame buffer, depth buffer, cameraCapture frame buffer, depth buffer, camera– Build and simplify mesh using depth bufferBuild and simplify mesh using depth buffer– Paste frame buffer on as a texturePaste frame buffer on as a texture– Inexpensive to render, looks about right!Inexpensive to render, looks about right!


Artifacts in TDMs: Skins and CracksArtifacts in TDMs: Skins and Cracks

• Skins are false Skins are false surfaces surfaces introduced by introduced by connecting connecting points across points across depth depth discontinuitiesdiscontinuities

• Cracks are Cracks are “holes” in the “holes” in the image that image that appear when appear when skins are skins are removedremoved

Original objects

Sample pointReconstructed surface

Skin edge


Artifacts in TDMs: SkinsArtifacts in TDMs: Skins

Geometry only (no artifacts) Geometry plus impostors (skins)


Artifacts in TDMs: CracksArtifacts in TDMs: Cracks

Geometry only Geometry plus impostors (cracks)


Outline Outline


• Related workRelated work







Incremental Voronoi-based samplingIncremental Voronoi-based sampling

• Goal: Simplify parts of a geometric environmentGoal: Simplify parts of a geometric environment

• Means: Replace primitives with a group of image-Means: Replace primitives with a group of image-based samplesbased samples

–Problem: where do we place samples?Problem: where do we place samples?

• Approach: an incremental search for the Approach: an incremental search for the next best next best viewview coupled with a measure of visibility error coupled with a measure of visibility error


Voronoi-based sampling: OutlineVoronoi-based sampling: Outline

• The general problem: sampling an environmentThe general problem: sampling an environment

• The nature and behavior of invisible spaceThe nature and behavior of invisible space

• Finding possible next best viewsFinding possible next best views

• Evaluating candidate viewpointsEvaluating candidate viewpoints

• ResultsResults


Environment capture: OverviewEnvironment capture: Overview

• General problem: Where do we place a minimum number of cameras in General problem: Where do we place a minimum number of cameras in order to sample all surfaces in an environment?order to sample all surfaces in an environment?

– This is closely related to the classic art gallery problem.This is closely related to the classic art gallery problem.» What is the minimum number of cameras necessary to guard every point in What is the minimum number of cameras necessary to guard every point in

the interior of a 2D polygon and where should they be placed?the interior of a 2D polygon and where should they be placed?

» This is NP-hard!This is NP-hard!

• Other global visibility approaches are also expensiveOther global visibility approaches are also expensive

– 3D visibility complex (Durand 1999) has size 3D visibility complex (Durand 1999) has size O(nO(n44)) wrt number of wrt number of primitives, can be computed in primitives, can be computed in O((nO((n33+k)log n)+k)log n) time time

– Aspect graph (Koenderink & vanDoorn 1976) is Aspect graph (Koenderink & vanDoorn 1976) is O(nO(n99)) (!) under 3D (!) under 3D perspective projectionperspective projection

• We use a simpler approachWe use a simpler approach


Environment capture: simplifying the Environment capture: simplifying the problemproblem

• Rather than solve the hard global problem, we use an Rather than solve the hard global problem, we use an incremental strategyincremental strategy

• Given a set of Given a set of nn cameras, where should we place camera cameras, where should we place camera n+1n+1 to see into the largest fraction of un-sampled parts of to see into the largest fraction of un-sampled parts of the environment?the environment?

» This is the This is the next best viewnext best view problem. problem.

• This sort of incremental, greedy approach is common to many This sort of incremental, greedy approach is common to many simplification algorithms.simplification algorithms.


Related work: Next Best View problemRelated work: Next Best View problem

• Outside looking inOutside looking in

– Conolly 1985Conolly 1985

– Maver and Bajcsy 1993Maver and Bajcsy 1993

– Banta et al. 1995Banta et al. 1995

– Pito and Bajcsy 1995Pito and Bajcsy 1995

– Reed, Allen, and Stamos 1997Reed, Allen, and Stamos 1997

– Stamos and Allen 2000Stamos and Allen 2000

• Inside looking outInside looking out

– Sanchiz and Fisher 1999Sanchiz and Fisher 1999

– Klein and Sequeira 2000Klein and Sequeira 2000


Environment capture as greedy, Environment capture as greedy, incremental optimizationincremental optimization

• We can cast the next-best-view approach as a classic We can cast the next-best-view approach as a classic optimization problem:optimization problem:

– Initial conditions:Initial conditions: What data do we start with? What data do we start with?

–Objective function:Objective function: How do we measure quality? How do we measure quality?

–Update:Update: How do we choose points to measure? How do we choose points to measure?

–Termination:Termination: How do we know when to stop? How do we know when to stop?







• ResultsResults


Visible and invisible volumeVisible and invisible volume

Sample location

Void surface

Object volume

Void volume

Visible volume


Void surface (multiple samples)Void surface (multiple samples)

Sample location

Object volume

Void volume (one sample)

Visible volume

Global void volume is the intersection of the void volumes for each sample

Global void volume


The void surface and visibility errorThe void surface and visibility error

• Any point where the void surface is visible is an error in the Any point where the void surface is visible is an error in the reconstructionreconstruction

– User sees into space that isn’t sampledUser sees into space that isn’t sampled

• Practical analogue: skins & cracksPractical analogue: skins & cracks

– Skins are part of the void surface for one sampleSkins are part of the void surface for one sample

– Cracks occur where you see into or through the void Cracks occur where you see into or through the void volumevolume

• The behavior of the void surface motivates our search The behavior of the void surface motivates our search algorithmalgorithm


Simple environment for characterizing the Simple environment for characterizing the visibility of the void surfacevisibility of the void surface

• One sample locationOne sample location

• One new viewpointOne new viewpoint

• One One apertureaperture to introduce a to introduce a void surfacevoid surface

• How does the angle How does the angle θvoid behave as the viewpoint moves?

θs

θv

θvoid

Sample locationViewpoint

Aperture


Void surface case 1Void surface case 1

• S is within V’s view of S is within V’s view of apertureaperture

• Void surface not visible at Void surface not visible at all from Vall from V

θs

θv

Sample location

Viewpoint

S

V



• Void surface visible Void surface visible on both sides of viewon both sides of view

• V’s view through V’s view through aperture exceeds aperture exceeds space visible from Sspace visible from S

• Void surface does not Void surface does not subtend entire view of subtend entire view of aperture from Vaperture from V

θv

θs

Visible segments of the void surface

S

V



• Void surface Void surface visible from V on visible from V on one side onlyone side only

• Void surface does Void surface does not subtend entire not subtend entire aperture seen by V aperture seen by V

θsθv

S V



• Void surface Void surface visiblevisible

• If viewpoint If viewpoint crosses beyond crosses beyond critical line L, void critical line L, void surface subtends surface subtends entire aperture as entire aperture as seen from Vseen from V

θs

θv

L

SV


Void surface behavior overallVoid surface behavior overall

• Visibility depends on…Visibility depends on…

–Position of SPosition of S

–Width/depth of aperture AWidth/depth of aperture A

–Location of viewpoint VLocation of viewpoint V

• Black, unlabeled regions are Black, unlabeled regions are where void surface subtends where void surface subtends all of aperture seen from Vall of aperture seen from V

2

1

3344

S

A







• ResultsResults


Choosing new viewpointsChoosing new viewpoints

• Observation: Observation: Artifacts get worse as you move away Artifacts get worse as you move away from a sample locationfrom a sample location

• This suggests that the next best view is far away from This suggests that the next best view is far away from all existing sample locationsall existing sample locations

• We restrict viewpoints to a planeWe restrict viewpoints to a plane

• The 2D Voronoi diagram gives us a set of candidate The 2D Voronoi diagram gives us a set of candidate points.points.


Finding candidate viewpointsFinding candidate viewpoints

• Starting conditions:Starting conditions:

– Environment Environment

» gray objectsgray objects

– Existing sample locations Existing sample locations PPii

» red pointsred points

– Navigable region Navigable region NN

» green rectanglegreen rectangle

Pi

N



• Compute 2D Voronoi Compute 2D Voronoi diagram (white lines) diagram (white lines) of existing sample of existing sample locations locations PPi i (red points)(red points)

Pi

N



• Candidate viewpoints Candidate viewpoints (blue squares) are…(blue squares) are…

– All vertices of All vertices of Voronoi diagram Voronoi diagram within Nwithin N

– Intersections of Intersections of Voronoi edges Voronoi edges with boundary of with boundary of NN

Cj

Pi

N







• ResultsResults


Evaluating candidate viewpointsEvaluating candidate viewpoints

• Goal: Capture the environment with few samplesGoal: Capture the environment with few samples

• A “good” sample sees new parts of the environmentA “good” sample sees new parts of the environment

• These parts are obscured by the void surfaceThese parts are obscured by the void surface

• Good sample = sees large areas of void surfaceGood sample = sees large areas of void surface


Evaluating candidate viewpointsEvaluating candidate viewpoints

• Objective functionObjective function: solid angle subtended by the : solid angle subtended by the void surfacevoid surface

• ProblemProblem: Maintaining an explicit representation of : Maintaining an explicit representation of the global void surface/volume is hard the global void surface/volume is hard

–Involves clipping sets of nonconvex, non-simple Involves clipping sets of nonconvex, non-simple polyhedra against one anotherpolyhedra against one another

–Our approachOur approach: let the graphics hardware do that!: let the graphics hardware do that!

» Construct the void surface for each sampleConstruct the void surface for each sample

» Use the depth and stencil buffers to resolve visibilityUse the depth and stencil buffers to resolve visibility


Individual void surfacesIndividual void surfaces

• We can compute per-pixel adjacency in every sampleWe can compute per-pixel adjacency in every sample

–This lets us identify/avoid most skins This lets us identify/avoid most skins

–We can also render the skins separately!We can also render the skins separately!

–Those skins are the void surface for that sample.Those skins are the void surface for that sample.


Per-pixel adjacencyPer-pixel adjacency

Dark pixels in bottom panorama are endpoints of skins


Combining the void surfacesCombining the void surfaces

1.1. Render the surfaces in all current samplesRender the surfaces in all current samples

2.2. Render the void surface for one sampleRender the void surface for one sample

3.3. Record those pixels where it’s the closest surface to Record those pixels where it’s the closest surface to the camerathe camera

4.4. Repeat for all samples, one at a timeRepeat for all samples, one at a time

• Pixels where the void surface is visible for all Pixels where the void surface is visible for all samples belong to the global void surface.samples belong to the global void surface.


Finding the global void surfaceFinding the global void surface

Lighter pixels are covered by the void surface for more samples

Red pixels belong to the global void surface


Beginning and ending the searchBeginning and ending the search

• Begin with a few samples from the view regionBegin with a few samples from the view region

–We choose the 4 corners and the centerpoint We choose the 4 corners and the centerpoint

• Acquire new samples to satisfy user-specified error Acquire new samples to satisfy user-specified error boundbound

• Typically between 1% and 2% of the viewTypically between 1% and 2% of the view

–This is not guaranteed to be an absolute bound!This is not guaranteed to be an absolute bound!

–There exist environments that break this approachThere exist environments that break this approach







• ResultsResults


Results: example environmentResults: example environment

Low error High error


Results: finding new viewpointsResults: finding new viewpoints

House (living room), 5 scans


Results: Complexity of samplingResults: Complexity of sampling

• Sampling algorithm is output-sensitiveSampling algorithm is output-sensitive

–Time depends on number of samples acquiredTime depends on number of samples acquired

• Acquiring Acquiring kk samples can take up to O(k samples can take up to O(k22) time) time

–Process O(k) samples at each iterationProcess O(k) samples at each iteration

–Evaluate O(k) candidate viewpoints in an iterationEvaluate O(k) candidate viewpoints in an iteration

» Evaluating one viewpoint is O(1) (amortized)Evaluating one viewpoint is O(1) (amortized)

–kk iterations overall iterations overall


Spatial Encoding SchemesSpatial Encoding Schemes

• Result: Database of samples for a view regionResult: Database of samples for a view region

• Those samples are an image-based simplificationThose samples are an image-based simplification

• Now we need a representation!Now we need a representation!


Desired properties of spatial encodingsDesired properties of spatial encodings

• CompactCompact

–Fit more into memory / disk spaceFit more into memory / disk space

• Easy to renderEasy to render

–Increase frame rate!Increase frame rate!

• Approach:Approach: Exploit spatial relationships in samples Exploit spatial relationships in samples

–Find and remove redundant dataFind and remove redundant data


Outline Outline








Spatial Video EncodingSpatial Video Encoding

• ObservationObservation: Images with similar view directions : Images with similar view directions and viewpoints show coherenceand viewpoints show coherence

• HypothesisHypothesis: Such groups of images can be well : Such groups of images can be well compressed as video sequences.compressed as video sequences.

Northward views from adjacent samples in the house model


Related work: Video CompressionRelated work: Video Compression

• Video as a sequence of independent still imagesVideo as a sequence of independent still images

–Motion JPEGMotion JPEG

• Video compression via temporal coherenceVideo compression via temporal coherence

–H.261 [Turletti and Huitema 1996]H.261 [Turletti and Huitema 1996]

–MPEG, MPEG-2 [Mitchell et al. 1996]MPEG, MPEG-2 [Mitchell et al. 1996]

–MPEG-4 [Ebrahimi and Pereira 2002]MPEG-4 [Ebrahimi and Pereira 2002]

• Most algorithms work on 1D temporal sequences Most algorithms work on 1D temporal sequences

–Non-sequential (random) access is expensiveNon-sequential (random) access is expensive


MPEG Encoded ImpostorsMPEG Encoded Impostors

• Still-image methods (PNG, JPEG) neglect image-Still-image methods (PNG, JPEG) neglect image-space coherence. space coherence.

• Video compression schemes (MPEG, MPEG2) Video compression schemes (MPEG, MPEG2) handle thishandle this

• Approach:Approach: Compress impostors as MPEG-2 Compress impostors as MPEG-2 video. video.

–ProblemProblem: This only captures coherence in one : This only captures coherence in one direction!direction!

–Our data set has much more than that. Our data set has much more than that.


1D Encoding Isn’t Enough1D Encoding Isn’t Enough

• 1D encoding yields inefficient frame-access 1D encoding yields inefficient frame-access patternpattern

–Exactly one path of efficient frame accessesExactly one path of efficient frame accesses

–User navigation doesn’t follow that path!User navigation doesn’t follow that path!

• 1D encoding neglects much coherence1D encoding neglects much coherence

–Image coherence exists along multiple spatial Image coherence exists along multiple spatial axesaxes

A 2D (or 3D) encoding can fix both of these.A 2D (or 3D) encoding can fix both of these.


Traditional MPEG compressionTraditional MPEG compression

Time

Frames indexed along temporal dimension.



Time


I I



Time


I IP P



Time


I IP PB B B B B B


Spatial MPEG compressionSpatial MPEG compression

X

Y Z

Frames indexed along spatial dimensions



X

Y Z

I I




X

Y Z

I IB B B B B B

B B B BB B B B B B



Motion Vector EstimationMotion Vector Estimation

• Use model information to Use model information to estimate motion vectorsestimate motion vectors

– Conjecture: better image Conjecture: better image qualityquality

– Image encoded in 16x16-Image encoded in 16x16-pixel macroblockspixel macroblocks

– On average, check 4.8 On average, check 4.8 unique motion vectors for unique motion vectors for every macroblockevery macroblock

– Downside: we must load and Downside: we must load and handle per-pixel depthhandle per-pixel depth

» I/O is expensive!I/O is expensive!

Pworld

PB-frame PI-frame

Motion Vector = (PI-frame - PB-frame)

Cell N (B-cell) Cell N+1 (I-cell)


Spatial Video Encoding: BenefitsSpatial Video Encoding: Benefits

• More efficient compressionMore efficient compression

– Coherence exploited in multiple spatial dimensions Coherence exploited in multiple spatial dimensions

– Subjective observation: at similar image quality, SVE Subjective observation: at similar image quality, SVE databases are 8-32% smaller with than logarithmic databases are 8-32% smaller with than logarithmic searchsearch

– Comparable encoding time (SVE 6% slower than log Comparable encoding time (SVE 6% slower than log search) search)

• Efficient runtime accessEfficient runtime access

– Decoder can access any adjacent cell quicklyDecoder can access any adjacent cell quickly

• Better motion compensation by using model informationBetter motion compensation by using model information

– Search window is effectively the entire imageSearch window is effectively the entire image


Results: Database sizesResults: Database sizes(Brooks House model)(Brooks House model)

• 1-floor, 15-room house model with high-1-floor, 15-room house model with high-resolution texturesresolution textures

–29x16 grid = 464 cells = 2784 impostors29x16 grid = 464 cells = 2784 impostors

–Uncompressed impostor data 2138MbUncompressed impostor data 2138Mb

• Spatial MPEG database = 21,216KbSpatial MPEG database = 21,216Kb

–Compression ratio 101:1Compression ratio 101:1


Results: Database sizes Results: Database sizes (power plant model)(power plant model)

• 2 floors of power plant2 floors of power plant

– 64x59 grid = 3776 cells = 22656 impostors64x59 grid = 3776 cells = 22656 impostors

– Uncompressed impostor data 16,992 MBUncompressed impostor data 16,992 MB

• Spatial MPEG database = 392MBSpatial MPEG database = 392MB

– Compression ratio 43:1Compression ratio 43:1

– This is a difficult environment to compress!This is a difficult environment to compress!

» High-frequency information everywhereHigh-frequency information everywhere


Results: Encoding ArtifactsResults: Encoding Artifacts

With spatially encoded impostors Geometry only


Outline Outline









Incremental textured depth meshesIncremental textured depth meshes

• GoalGoal: Spatial encoding for the geometric portion of : Spatial encoding for the geometric portion of samplessamples

• ApproachApproach: Remove redundant data : Remove redundant data

–Similar to spatial video encodingSimilar to spatial video encoding

• ProblemProblem: A difference operation is not as simple for : A difference operation is not as simple for two surfaces as for two imagestwo surfaces as for two images

• Proposed solutionProposed solution: Perform difference operation on : Perform difference operation on points, then create surfaces from differencepoints, then create surfaces from difference


ITDM difference operationITDM difference operation

• A difference operation implies a hierarchyA difference operation implies a hierarchy

– How do we choose subtrahends and minuends?How do we choose subtrahends and minuends?

• Approach:Approach:

– Create a hierarchy (spanning tree) over a group of samplesCreate a hierarchy (spanning tree) over a group of samples

» One node for each sampleOne node for each sample

» Re-use the idea that nearby viewpoints see similar things to Re-use the idea that nearby viewpoints see similar things to build treebuild tree

– Each node S stores the difference between S and the sum Each node S stores the difference between S and the sum of its ancestors’ samplesof its ancestors’ samples


ITDM dependency treeITDM dependency tree

• Green points are sample Green points are sample locationslocations

• Blue lines are tree edgesBlue lines are tree edges

• Tree root at center of imageTree root at center of image


ITDM mesh constructionITDM mesh construction

• Result: Result: scan cubesscan cubes with many pixels removed with many pixels removed

–One One intra scan intra scan (root of tree) with all pixels intact(root of tree) with all pixels intact

–Many Many delta scans delta scans (child nodes) with 70-99% of (child nodes) with 70-99% of pixels removedpixels removed

• Build TDMs from remaining pointsBuild TDMs from remaining points

–Avoid introducing skinsAvoid introducing skins» Yes, this leaves cracks. We want that!Yes, this leaves cracks. We want that!

–Create a dense mesh, then simplify itCreate a dense mesh, then simplify it


ITDM mesh simplificationITDM mesh simplification

• View-dependent View-dependent simplification is simplification is appropriate here!appropriate here!

• Navigable region (cell) Navigable region (cell) restricts the possible restricts the possible viewpoints / directionsviewpoints / directions

• We combine GAPS We combine GAPS (Erikson 1999) with (Erikson 1999) with VDSlib (Luebke 2001)VDSlib (Luebke 2001)

Cull box

Cell

Far-field impostor (ITDM)


ITDM: renderingITDM: rendering

• Result: six TDMs for each scan cubeResult: six TDMs for each scan cube

–One mesh for each face of the cubeOne mesh for each face of the cube

• Spanning tree will guide renderingSpanning tree will guide rendering

–Always render intra scanAlways render intra scan

–Use delta scans to fill in cracksUse delta scans to fill in cracks

» Delta scans are small, so we can render severalDelta scans are small, so we can render several


ITDM Results: Increasing fidelity with ITDM Results: Increasing fidelity with more delta scansmore delta scans


ITDM: video footageITDM: video footage


ITDM Results: Frame ratesITDM Results: Frame rates

0

20

40

60

80

100

120

140

1 201 401 601 801 1001 1201 1401 1601

Textured depth meshes Incremental TDMs Static LODs


ITDM Results: BenefitsITDM Results: Benefits

• Better fidelity than standard TDMsBetter fidelity than standard TDMs

No skinsNo skins

No poppingNo popping

Lower frame rateLower frame rate

• Better frame rate than static LODsBetter frame rate than static LODs

Consistently interactive frame rateConsistently interactive frame rate

Lower image qualityLower image quality


ITDM Results: BenefitsITDM Results: Benefits

• Removing redundant points saves preprocessing time Removing redundant points saves preprocessing time and storage spaceand storage space

–2:1 savings in storage space2:1 savings in storage space

–6:1 savings in preprocessing time6:1 savings in preprocessing time


Outline Outline









Conclusions: Problems AddressedConclusions: Problems Addressed

• Where do you acquire samples for an image-based Where do you acquire samples for an image-based simplification / acceleration method?simplification / acceleration method?

–Incremental Voronoi-based samplingIncremental Voronoi-based sampling

• How do you represent those samples for fast How do you represent those samples for fast rendering and compact storage?rendering and compact storage?

–Spatial video encodingSpatial video encoding for images for images

–Incremental textured depth meshes for geometryIncremental textured depth meshes for geometry


Conclusions: New ResultsConclusions: New Results

• Incremental Voronoi-based sampling finds good next Incremental Voronoi-based sampling finds good next viewsviews

–Local error decreases as samples are addedLocal error decreases as samples are added

–Algorithm works well in practiceAlgorithm works well in practice

• Spatial video encoding captures image-space Spatial video encoding captures image-space coherencecoherence

–Model information allows estimation of motion Model information allows estimation of motion vectors without expensive, error-prone search stepvectors without expensive, error-prone search step


Conclusions: New ResultsConclusions: New Results

• Incremental textured depth meshes as a Incremental textured depth meshes as a representation for the far fieldrepresentation for the far field

–Better frame rates than static levels of detailBetter frame rates than static levels of detail

–Better fidelity than single-sample TDMsBetter fidelity than single-sample TDMs

• All these approaches have been demonstrated on two All these approaches have been demonstrated on two real-world data setsreal-world data sets


Conclusions: Future WorkConclusions: Future Work

• We have only explored a few possibilities in a broad We have only explored a few possibilities in a broad design spacedesign space

–Voronoi-based sampling can be applied to standard Voronoi-based sampling can be applied to standard TDMsTDMs

–Incremental TDMs can be created from uniform Incremental TDMs can be created from uniform samplessamples

–Many other combinationsMany other combinations


Conclusions: Future WorkConclusions: Future Work

• We have not addressed the creation of the cell grid!We have not addressed the creation of the cell grid!

–Previous approaches have left cells small (1 meter) Previous approaches have left cells small (1 meter) to minimize TDM artifacts and maintain frame rateto minimize TDM artifacts and maintain frame rate

» Graphics cards are somewhat faster nowGraphics cards are somewhat faster now

» ITDMs exhibit fewer artifacts than TDMsITDMs exhibit fewer artifacts than TDMs

• Build cell grids to account for the uneven distribution Build cell grids to account for the uneven distribution of primitives in the modelof primitives in the model


Conclusions: Future workConclusions: Future work

• These methods might work in real-world scenesThese methods might work in real-world scenes

–Laser range scans of interesting environmentsLaser range scans of interesting environments

–We’ve tried to avoid assumptions that are invalid We’ve tried to avoid assumptions that are invalid in the real worldin the real world


AcknowledgementsAcknowledgements

• Committee: Dinesh Manocha, Fred Brooks, Ketan Committee: Dinesh Manocha, Fred Brooks, Ketan Mayer-Patel, Anselmo Lastra, Gary BishopMayer-Patel, Anselmo Lastra, Gary Bishop

• Funding agencies: NIH, ARO, NSF, DOEFunding agencies: NIH, ARO, NSF, DOE

• Model donors: Walkthrough group, Electric Boat, Model donors: Walkthrough group, Electric Boat, Newport News Shipbuilding, power plant donorNewport News Shipbuilding, power plant donor

• Moral support: Cat, Kirsti, Tony, Bernadette, Moral support: Cat, Kirsti, Tony, Bernadette, Colleen, Tanner, AndreaColleen, Tanner, Andrea


Questions?Questions?


UNUSED SLIDES FOLLOWUNUSED SLIDES FOLLOW


Issues in cell-based walkthroughIssues in cell-based walkthrough

• Where do you acquire your samples?Where do you acquire your samples?

–Previous work uses uniform grid or hand-picked Previous work uses uniform grid or hand-picked cellscells

• What impostor representation do you choose?What impostor representation do you choose?

–Textured depth meshes (TDMs) are common but Textured depth meshes (TDMs) are common but show artifactsshow artifacts


Update: searching for the next best viewUpdate: searching for the next best view

• The search space is a 2D planeThe search space is a 2D plane– Infinitely many possibilitiesInfinitely many possibilities– Brute-force search on a discrete grid is possible but very slowBrute-force search on a discrete grid is possible but very slow

• We want to find a set of points that is We want to find a set of points that is likelylikely to contain something close to contain something close to the next best viewto the next best view

– That set should be small enough for exhaustive search to be That set should be small enough for exhaustive search to be practicalpractical

• By characterizing the visibility of un-sampled parts of an environment By characterizing the visibility of un-sampled parts of an environment we can develop heuristics for the placement of new viewpoints.we can develop heuristics for the placement of new viewpoints.


Implementation: VWALK Implementation: VWALK

• System written in OpenGL, C++System written in OpenGL, C++

• Load model geometry, impostor data on demandLoad model geometry, impostor data on demand

– Prefetch data for nearby regionsPrefetch data for nearby regions

– Cache data whenever possibleCache data whenever possible

• Exploits multiple processors when availableExploits multiple processors when available

– MPEG decoding, prefetching, and rendering are all MPEG decoding, prefetching, and rendering are all separate tasksseparate tasks

• Custom MPEG encoder/decoderCustom MPEG encoder/decoder

– Based on Dali toolkitBased on Dali toolkit


Results: Frame rates (TDM vs cached Results: Frame rates (TDM vs cached images, power plant model)images, power plant model)

0

10

20

30

40

50

60

700 80 160

240

320

400

480

560

640

720

800

880

960

1040

1120

1200

1280

1360

1440

1520

1600

1680

1760

1840

1920

2000

2080

2160

2240

2320

2400

2480

Time (Frame Number)

Fra

me

Pe

r S

ec

on

d

SMPEG Flat SMPEG TDM


Visible, invisible, and void volumesVisible, invisible, and void volumes

• Divide space in a sampled environment as follows (after Pito Divide space in a sampled environment as follows (after Pito and Bajcsy):and Bajcsy):

– Visible volumeVisible volume contains everything we’ve seen so far. contains everything we’ve seen so far.

– Invisible volumeInvisible volume contains everything else: contains everything else:

» Object volumeObject volume, the interior of objects in the scene, the interior of objects in the scene

» Void volumeVoid volume, possibly empty space about which nothing is , possibly empty space about which nothing is knownknown

• We call the boundary between the visible and void volumes We call the boundary between the visible and void volumes the the void surfacevoid surface. .


Virtual CellsVirtual Cells

• Define a cell as an Define a cell as an arbitrary rectangular arbitrary rectangular regionregion

• Each cell has its own Each cell has its own cull boxcull box

• Cull box determines Cull box determines potentially visible setpotentially visible set

• Cull box boundaries Cull box boundaries covered by impostorscovered by impostors

viewpointCurrent

cell

cull box


MPEG Encoded ImpostorsMPEG Encoded Impostors

• Previous work: Wilson et al. 2000Previous work: Wilson et al. 2000

–Map 2D array of impostors to 1D streamMap 2D array of impostors to 1D stream

–Encode as standard MPEG-2 videoEncode as standard MPEG-2 video

–At runtime, use custom decoder with support At runtime, use custom decoder with support for random frame accessfor random frame access


Why is visibility hard?Why is visibility hard?

• There are apertures between There are apertures between nearly every pair of objects nearly every pair of objects herehere

• Objects occlude one anotherObjects occlude one another

• Visibility of any one Visibility of any one aperture is piecewise aperture is piecewise smooth but their smooth but their combination is difficultcombination is difficult

• Robustness problems when Robustness problems when edges coincide in screen edges coincide in screen spacespace


ITDM dependency tree propertiesITDM dependency tree properties

• Root node at centrally located sampleRoot node at centrally located sample

– Minimize the delta from root to any other sampleMinimize the delta from root to any other sample

• Child nodes are close to their parentsChild nodes are close to their parents

– Minimize the delta from a parent node to its childrenMinimize the delta from a parent node to its children

• Construct a minimum spanning tree over a complete graph Construct a minimum spanning tree over a complete graph on sample pointson sample points

– Use cubed distance as edge weight to get nearby parentsUse cubed distance as edge weight to get nearby parents

– Root node at center of cellRoot node at center of cell

– More complex schemes are certainly possibleMore complex schemes are certainly possible


ITDM: renderingITDM: rendering

• Start with a scan budget (# scans or # triangles)Start with a scan budget (# scans or # triangles)

• Identify the nearest scan cube SIdentify the nearest scan cube S

• Start at the root of the treeStart at the root of the tree

– Walk down toward S Walk down toward S

– Render TDMs at each nodeRender TDMs at each node

– Stop when you reach S or exhaust scan budgetStop when you reach S or exhaust scan budget

• While the budget is not exhausted, identify the next nearest While the budget is not exhausted, identify the next nearest scan and repeat the processscan and repeat the process


Related work: WalkthroughRelated work: Walkthrough

• BRUSHBRUSH

– Schneider et al. 1994Schneider et al. 1994

• Berkeley walkthrough system Berkeley walkthrough system

– Funkhouser et al. 1996Funkhouser et al. 1996

• MMRMMR

– Aliaga et al. 1999Aliaga et al. 1999

• Video-based systemsVideo-based systems

– Wilson et al. 2000, 2001Wilson et al. 2000, 2001

Andy Wilson – Dissertation defense, 10/23/02 - Slide 1 Spatially Encoded Image-Space...

Documents

Transcript of Andy Wilson – Dissertation defense, 10/23/02 - Slide 1 Spatially Encoded Image-Space...