WIEN
Stereo-based Image and Video Stereo-based Image and Video Analysis for Multimedia ApplicationsAnalysis for Multimedia Applications
Application:Application: “Computer-generated Stereoscopic Paintings“ “Computer-generated Stereoscopic Paintings“
M. Gelautz, E. Stavrakis, and M. BleyerM. Gelautz, E. Stavrakis, and M. Bleyer
Interactive Media Systems GroupInteractive Media Systems Group
Technical University Vienna, A-1040 Vienna, AustriaTechnical University Vienna, A-1040 Vienna, Austria
Email: [email protected]: [email protected]
ISPRS 2004
WIEN
Stereo Analysis for Multimedia Applications
Artistic Rendering of Stereo ViewsArtistic Rendering of Stereo Views
Our goal is to provide stereo views of Our goal is to provide stereo views of real scenes a hand-painted appearance.real scenes a hand-painted appearance.
The work combines photogrammetric The work combines photogrammetric techniques (stereo) with computer techniques (stereo) with computer graphic algorithms.graphic algorithms.
Available painterly rendering algorithms Available painterly rendering algorithms have been designed for single images – have been designed for single images – painterly rendering of stereo views painterly rendering of stereo views presents a new topic of research! presents a new topic of research!
ISPRS 2004
WIEN
Stereo Analysis for Multimedia Applications
Traditional Stereoscopic PaintingTraditional Stereoscopic Painting
Fig1. Example of a stereoscopic painting by Salvador DalFig1. Example of a stereoscopic painting by Salvador Dalíí (“The sleeping smoker”, 1972/73)..
ISPRS 2004
WIEN
Stereo Analysis for Multimedia Applications
Motivation for Computer-Motivation for Computer-generated Stereo Paintinggenerated Stereo Painting The manual creation of stereo paintings is a The manual creation of stereo paintings is a
labor intensive task.labor intensive task. The artist needs to reproduce the same The artist needs to reproduce the same
composition twice from different viewpoints.composition twice from different viewpoints. Some painters used stereo photography to Some painters used stereo photography to
base their compositions on.base their compositions on. The excessive effort associated with the The excessive effort associated with the
manual creation of stereoscopic paintings can manual creation of stereoscopic paintings can be reduced by computer vision techniques.be reduced by computer vision techniques.
ISPRS 2004
WIEN
Stereo Analysis for Multimedia Applications
Single Image Painterly RenderingSingle Image Painterly RenderingHertzmann, A., 1998: “Painterly rendering with Hertzmann, A., 1998: “Painterly rendering with curved brush strokes of multiple sizes”curved brush strokes of multiple sizes”
(a) Original image(a) Original image (b) Rendered image (b) Rendered image (impressionist style)(impressionist style)
paint spillingpaint spilling
ISPRS 2004
WIEN
Stereo Analysis for Multimedia Applications
Requirements of Stereoscopic Requirements of Stereoscopic Painterly RenderingPainterly Rendering Observation:Observation: Individual painting of the left and Individual painting of the left and
right stereo image will usually not produce right stereo image will usually not produce satisfactory results.satisfactory results.
Requirements:Requirements: To preserve coherence between the brush To preserve coherence between the brush
strokes of the two images.strokes of the two images. To preserve the depth discontinuities in To preserve the depth discontinuities in
order to prevent „paint spilling“.order to prevent „paint spilling“. To deal with occlusions.To deal with occlusions.
Solution:Solution: We incorporate a stereo-derived We incorporate a stereo-derived depth map into the painting algorithm.depth map into the painting algorithm.
ISPRS 2004
WIEN
Stereo Analysis for Multimedia Applications
Stereo Matching AlgorithmStereo Matching Algorithm ““A Layered Stereo Matching Algorithm Using Color A Layered Stereo Matching Algorithm Using Color
Segmentation and Global Visibility Constraints“Segmentation and Global Visibility Constraints“ Key features:Key features:
Color segmentation of the reference image.Color segmentation of the reference image. Color segments are clustered into more robust Color segments are clustered into more robust
depth layers (mean shift clustering).depth layers (mean shift clustering). Iterative refinement of the solution based on a cost Iterative refinement of the solution based on a cost
function (greedy search algorithm).function (greedy search algorithm). Results:Results: 2nd rank (among 30 algorithms) on the 2nd rank (among 30 algorithms) on the
Middlebury Stereo Evaluation Website Middlebury Stereo Evaluation Website (http://www.middlebury.edu/stereo)(http://www.middlebury.edu/stereo)
ISPRS 2004
WIEN
Stereo Analysis for Multimedia Applications
Clustering of Color SegmentsClustering of Color Segmentsinto Layersinto Layers
Fitting of planar Fitting of planar models (depth layers) models (depth layers) to segments (top) and to segments (top) and layers (bottom) ->layers (bottom) ->layer fitting is more layer fitting is more robust than segment robust than segment fitting!fitting!
ISPRS 2004
WIEN
Stereo Analysis for Multimedia Applications
Stereo Matching ResultsStereo Matching Results
Evaluation: 0.2% Evaluation: 0.2% „bad“ pixels (i.e., „bad“ pixels (i.e., unoccluded pixels unoccluded pixels whose absolute whose absolute disparity error is disparity error is greater than 1)greater than 1)
SawtoothSawtooth stereo pair stereo pair (Middlebury website)(Middlebury website)
Left imageLeft image Right imageRight image
Stereo disparitiesStereo disparities Ground truthGround truth
ISPRS 2004
WIEN
Stereo Analysis for Multimedia Applications
Stereo Painting Algorithm (1)Stereo Painting Algorithm (1) Modify Hertzmann‘s single image painting Modify Hertzmann‘s single image painting
algorithm to prevent brush strokes from algorithm to prevent brush strokes from crossing depth discontinuities.crossing depth discontinuities.
Paint the reference image using the modified Paint the reference image using the modified single image painting algorithm.single image painting algorithm.
Paint (dilated) occluded regions of the second Paint (dilated) occluded regions of the second image.image.
Project paint from the reference image to non-Project paint from the reference image to non-occluded regions of the second image. As a occluded regions of the second image. As a consequence, we avoid repainting large parts consequence, we avoid repainting large parts of the second image.of the second image.
ISPRS 2004
WIEN
Stereo Analysis for Multimedia Applications
Stereo Painting Algorithm (2)Stereo Painting Algorithm (2)
Original imageOriginal image Stereo disparitiesStereo disparities
Occlusion mapOcclusion map Occlusion paintOcclusion paint
ISPRS 2004
WIEN
Stereo Analysis for Multimedia Applications
Stereo Painting Result (1)Stereo Painting Result (1)
Single image Single image painting resultspainting results
Stereo image Stereo image painting resultspainting results
Left imageLeft image
Left imageLeft image Right imageRight image
Right imageRight image
ISPRS 2004
WIEN
Stereo Analysis for Multimedia Applications
Stereo Painting Result (2)Stereo Painting Result (2)
Original Original stereo pairstereo pair
Painted Painted stereo pairstereo pair
ISPRS 2004
WIEN
Stereo Analysis for Multimedia Applications
Stereo Painting Result (3)Stereo Painting Result (3)
Creating the Creating the paintings…paintings…
ISPRS 2004
WIEN
Stereo Analysis for Multimedia Applications
Summary and OutlookSummary and Outlook We have presentedWe have presented
a new stereo matching algorithm that a new stereo matching algorithm that relies on clustering color-segmented relies on clustering color-segmented regions into depth layers.regions into depth layers.
an algorithm for stereoscopic painterly an algorithm for stereoscopic painterly rendering based on real image pairs.rendering based on real image pairs.
We plan to use the stereo-derived depth We plan to use the stereo-derived depth maps to enrich the computer-generated maps to enrich the computer-generated paintings with further depth cues (e.g., paintings with further depth cues (e.g., different levels of detail).different levels of detail).
ISPRS 2004
WIEN
Stereo Analysis for Multimedia Applications
AcknowledgementsAcknowledgements
This work was supported by the Austrian This work was supported by the Austrian Science Fund (FWF) under project Science Fund (FWF) under project P15663.P15663.
We wish to thank “Verwertungsgesell-We wish to thank “Verwertungsgesell-schaft Bildender Künstler (VBK)“, Austria, schaft Bildender Künstler (VBK)“, Austria, for permission to reproduce „The for permission to reproduce „The sleeping smoker“.sleeping smoker“.
Top Related