INFORMATIKINFORMATIK

Hendrik LenschHendrik Lensch

Hardware-AcceleratedHardware-AcceleratedSilhouette MatchingSilhouette Matching

Hendrik Lensch, Wolfgang Heidrich,Hendrik Lensch, Wolfgang Heidrich, and Hans-Peter Seideland Hans-Peter Seidel

Max-Planck-Institut fMax-Planck-Institut für Informatik,ür Informatik,Saarbrücken (Germany)Saarbrücken (Germany)

Hendrik Lensch, Wolfgang Heidrich,Hendrik Lensch, Wolfgang Heidrich, and Hans-Peter Seideland Hans-Peter Seidel

Max-Planck-Institut fMax-Planck-Institut für Informatik,ür Informatik,Saarbrücken (Germany)Saarbrücken (Germany)

INFORMATIKINFORMATIK

Hendrik LenschHendrik Lensch

OverviewOverview

• MotivationMotivation• Comparing SilhouettesComparing Silhouettes• Stitching and Combining TexturesStitching and Combining Textures• Results and ConclusionsResults and Conclusions

• MotivationMotivation• Comparing SilhouettesComparing Silhouettes• Stitching and Combining TexturesStitching and Combining Textures• Results and ConclusionsResults and Conclusions

INFORMATIKINFORMATIK

Hendrik LenschHendrik Lensch

Acquiring Real World ModelsAcquiring Real World Models

GeometryGeometry• 3D scanner3D scanner

GeometryGeometry• 3D scanner3D scanner

Texture dataTexture data• digital cameradigital camera

Texture dataTexture data• digital cameradigital camera

single sensor vs. multiple sensorssingle sensor vs. multiple sensors

INFORMATIKINFORMATIK

Hendrik LenschHendrik Lensch

3D – 2D Registration3D – 2D Registration

Find the camera setting for each 2D image.Find the camera setting for each 2D image.Find the camera setting for each 2D image.Find the camera setting for each 2D image.

INFORMATIKINFORMATIK

Hendrik LenschHendrik Lensch

TransformationsTransformations• to camera coordinates (extrinsic):to camera coordinates (extrinsic):

• to 2D image space (intrinsic):to 2D image space (intrinsic):

determine determine R, tR, t and and ff (6+1 dimensions) (6+1 dimensions)

TransformationsTransformations• to camera coordinates (extrinsic):to camera coordinates (extrinsic):

• to 2D image space (intrinsic):to 2D image space (intrinsic):

determine determine R, tR, t and and ff (6+1 dimensions) (6+1 dimensions)

tgxR

z

y

x

w

c

c

c

Camera ModelCamera Model

c

c

c y

x

z

f

v

u

v

u1

0

0

INFORMATIKINFORMATIK

Hendrik LenschHendrik Lensch

Similarity MeasureSimilarity Measure

Which features to investigate?Which features to investigate?• no color information on the modelno color information on the model• correspondence of geometric features hard to findcorrespondence of geometric features hard to find

Which features to investigate?Which features to investigate?• no color information on the modelno color information on the model• correspondence of geometric features hard to findcorrespondence of geometric features hard to find

INFORMATIKINFORMATIK

Hendrik LenschHendrik Lensch

Similarity MeasureSimilarity Measure

Compare silhouettes Compare silhouettes [Etienne de Silhouette 1709-1767][Etienne de Silhouette 1709-1767]

• model: render monochromemodel: render monochrome• photo: automatic histogram-based segmentationphoto: automatic histogram-based segmentation

Compare silhouettes Compare silhouettes [Etienne de Silhouette 1709-1767][Etienne de Silhouette 1709-1767]

• model: render monochromemodel: render monochrome• photo: automatic histogram-based segmentationphoto: automatic histogram-based segmentation

INFORMATIKINFORMATIK

Hendrik LenschHendrik Lensch

Similarity MeasureSimilarity Measure

Compare silhouettes Compare silhouettes [Etienne de Silhouette 1709-1767][Etienne de Silhouette 1709-1767]

• model: render monochromemodel: render monochrome• photo: automatic histogram-based segmentationphoto: automatic histogram-based segmentation

Compare silhouettes Compare silhouettes [Etienne de Silhouette 1709-1767][Etienne de Silhouette 1709-1767]

• model: render monochromemodel: render monochrome• photo: automatic histogram-based segmentationphoto: automatic histogram-based segmentation

INFORMATIKINFORMATIK

Hendrik LenschHendrik Lensch

Distance Measure for Distance Measure for SilhouettesSilhouettes

Point-to-outline Point-to-outline distancesdistances

• slow because points on slow because points on the outline must be the outline must be determineddetermined

• speedup by distance speedup by distance mapsmaps

Point-to-outline Point-to-outline distancesdistances

• slow because points on slow because points on the outline must be the outline must be determineddetermined

• speedup by distance speedup by distance mapsmaps

INFORMATIKINFORMATIK

Hendrik LenschHendrik Lensch

11

00

intensityintensity

xx

differencedifference

11

00xx

Pixel-based Pixel-based Distance MeasureDistance Measure

Count the number of Count the number of pixels covered by pixels covered by just one silhouette.just one silhouette.• XOR the imagesXOR the images

• compute histogram compute histogram (hardware)(hardware)

• gives linear response to gives linear response to the displacementthe displacement

Count the number of Count the number of pixels covered by pixels covered by just one silhouette.just one silhouette.• XOR the imagesXOR the images

• compute histogram compute histogram (hardware)(hardware)

• gives linear response to gives linear response to the displacementthe displacement

displacementdisplacement

INFORMATIKINFORMATIK

Hendrik LenschHendrik Lensch

1111

00

intensityintensity

xx

11

00

differencedifference

xx

Pixel-based Pixel-based Distance MeasureDistance Measure

Count the number of Count the number of pixels covered by pixels covered by just one silhouette.just one silhouette.• XOR the imagesXOR the images

• compute histogram compute histogram (hardware)(hardware)

• gives linear response to gives linear response to the displacementthe displacement

Count the number of Count the number of pixels covered by pixels covered by just one silhouette.just one silhouette.• XOR the imagesXOR the images

• compute histogram compute histogram (hardware)(hardware)

• gives linear response to gives linear response to the displacementthe displacement

displacementdisplacement

INFORMATIKINFORMATIK

Hendrik LenschHendrik Lensch

Approximation ofApproximation ofSquared DistancesSquared Distances

Use smooth transitionsUse smooth transitions• blur imagesblur images

• integrate squared integrate squared differencesdifferences

• faster convergencefaster convergence• reduced variance reduced variance • higher evaluation costhigher evaluation cost

Use smooth transitionsUse smooth transitions• blur imagesblur images

• integrate squared integrate squared differencesdifferences

• faster convergencefaster convergence• reduced variance reduced variance • higher evaluation costhigher evaluation cost

INFORMATIKINFORMATIK

Hendrik LenschHendrik Lensch

Approximation ofApproximation ofSquared DistancesSquared Distances

Use smooth transitionsUse smooth transitions• blur imagesblur images

• integrate squared integrate squared differencesdifferences

• faster convergencefaster convergence• reduced variance reduced variance • higher evaluation costhigher evaluation cost

Use smooth transitionsUse smooth transitions• blur imagesblur images

• integrate squared integrate squared differencesdifferences

• faster convergencefaster convergence• reduced variance reduced variance • higher evaluation costhigher evaluation cost

1111

00

intensityintensity

xx

1111

00

intensityintensity

xx

11

00

differencedifference

xx

differencedifference

11

00xx

filtersizefiltersize

INFORMATIKINFORMATIK

Hendrik LenschHendrik Lensch

Non-linear Optimization Non-linear Optimization

Downhill Simplex Method [Press 1992] Downhill Simplex Method [Press 1992] • works for works for NN dimensions dimensions• no derivatives no derivatives • easy to controleasy to control

Downhill Simplex Method [Press 1992] Downhill Simplex Method [Press 1992] • works for works for NN dimensions dimensions• no derivatives no derivatives • easy to controleasy to control

INFORMATIKINFORMATIK

Hendrik LenschHendrik Lensch

Simplex Method in 3D Simplex Method in 3D

original simplexoriginal simplexoriginal simplexoriginal simplex

reflection and/orreflection and/orexpansionexpansionreflection and/orreflection and/orexpansionexpansion

shrinkingshrinkingshrinkingshrinking

random random perturbationperturbationrandom random perturbationperturbation

INFORMATIKINFORMATIK

Hendrik LenschHendrik Lensch

Hierarchical OptimizationHierarchical Optimization

• optimize on low resolution firstoptimize on low resolution first• restart optimization to avoid local minimarestart optimization to avoid local minima• switch to higher resolutionswitch to higher resolution• mesh resolution can be adaptedmesh resolution can be adapted

• optimize on low resolution firstoptimize on low resolution first• restart optimization to avoid local minimarestart optimization to avoid local minima• switch to higher resolutionswitch to higher resolution• mesh resolution can be adaptedmesh resolution can be adapted

INFORMATIKINFORMATIK

Hendrik LenschHendrik Lensch

Starting Point GenerationStarting Point Generation

• set camera distance set camera distance ttzz depending on object size depending on object size

• setset ttxx and and ttyy to zero to zero• select 48 sample rotationsselect 48 sample rotations• run optimization for each of the samplesrun optimization for each of the samples

(40 evaluations)(40 evaluations)• select top 5 resultsselect top 5 results• restart optimization (200 evaluations)restart optimization (200 evaluations)• take best result as starting pointtake best result as starting point

• set camera distance set camera distance ttzz depending on object size depending on object size

• setset ttxx and and ttyy to zero to zero• select 48 sample rotationsselect 48 sample rotations• run optimization for each of the samplesrun optimization for each of the samples

(40 evaluations)(40 evaluations)• select top 5 resultsselect top 5 results• restart optimization (200 evaluations)restart optimization (200 evaluations)• take best result as starting pointtake best result as starting point

INFORMATIKINFORMATIK

Hendrik LenschHendrik Lensch

Texture StitchingTexture Stitching

projective texture mappingprojective texture mapping assign one image to each triangle assign one image to each triangle • triangle visible in image? (test every vertex)triangle visible in image? (test every vertex)• select best viewing angleselect best viewing angle• discard data near depth discontinuities discard data near depth discontinuities

projective texture mappingprojective texture mapping assign one image to each triangle assign one image to each triangle • triangle visible in image? (test every vertex)triangle visible in image? (test every vertex)• select best viewing angleselect best viewing angle• discard data near depth discontinuities discard data near depth discontinuities

INFORMATIKINFORMATIK

Hendrik LenschHendrik Lensch

Blending Across Blending Across Assignment BordersAssignment Borders

• find border verticesfind border vertices

• release all triangles release all triangles around them around them

• assign boundary assign boundary vertices to best regionvertices to best region

• assign alpha-values for assign alpha-values for each region each region

– 1 to vertices 1 to vertices included in the regionincluded in the region

– 0 to all others.0 to all others.

• find border verticesfind border vertices

• release all triangles release all triangles around them around them

• assign boundary assign boundary vertices to best regionvertices to best region

• assign alpha-values for assign alpha-values for each region each region

– 1 to vertices 1 to vertices included in the regionincluded in the region

– 0 to all others.0 to all others.

INFORMATIKINFORMATIK

Hendrik LenschHendrik Lensch

Entire TextureEntire Texture

INFORMATIKINFORMATIK

Hendrik LenschHendrik Lensch

Results and ConclusionsResults and Conclusions

Problems solved:Problems solved:• automatic texture registration (automatic texture registration (R, t, fR, t, f))• view-independent texture stitchingview-independent texture stitching• blending across assignment boundariesblending across assignment boundaries• rough manual alignment helps (speedup, failures) rough manual alignment helps (speedup, failures)

Further problems:Further problems:• extract purely diffuse part of textureextract purely diffuse part of texture• generate texture where data is missinggenerate texture where data is missing

Problems solved:Problems solved:• automatic texture registration (automatic texture registration (R, t, fR, t, f))• view-independent texture stitchingview-independent texture stitching• blending across assignment boundariesblending across assignment boundaries• rough manual alignment helps (speedup, failures) rough manual alignment helps (speedup, failures)

Further problems:Further problems:• extract purely diffuse part of textureextract purely diffuse part of texture• generate texture where data is missinggenerate texture where data is missing

INFORMATIKINFORMATIK

Hendrik LenschHendrik Lensch

Questions?Questions?

visit us at visit us at

www.mpi-sb.mpg.dewww.mpi-sb.mpg.de

visit us at visit us at

www.mpi-sb.mpg.dewww.mpi-sb.mpg.de