Image Matting and Its Applications

Chen-Yu TsengAdvisor: Sheng-Jyh Wang

2012-10-29

Image Matting

• A process to extract foreground objects from an image, along with an alpha matte (the opacity of the foreground color)

Input Image Alpha Matte Extracted Foreground

Two Approaches of Image Matting

• Supervised Matting• With User’s Guidance

• Unsupervised Matting• Without User’s Guidance

Input Image User’s Guidance

e.g. Trimap:White ForegroundBlack BackgroundUnknown Gray

Two Schemes of Supervised Matting

Propagation-based Scheme• Infer Alpha Matte with

Propagation through a Graphical Model

• A Global-based Approach

Sampling-based Scheme• Infer Alpha Matte with

Some Color Samples• A Local-based Approach

Foreground Pixel

Background Pixel

Unknown Pixel

Foreground Color Set

Background Color Set

Unknown Pixel

Propagation-based scheme -Matting Laplacian Approach

• A Graphical Model with Connectivity between Pixels• The Connectivity Is Learned from the Image Structure

• Capability for Dealing with Both • Supervised Matting (Inference Problem)• Unsupervised Matting (Decomposition Problem)

Foreground Pixel

Background Pixel

Unknown Pixel

Reference of Matting Laplacian Approach

• First proposed by Levin et al. for supervised matting (closed-form matting)• A. Levin, D. Lischinski, Y. Weiss. “A Closed Form Solution to Natural

Image Matting,” IEEE T. PAMI, vol. 30, no. 2, pp. 228-242, Feb. 2008.• Extended to unsupervised matting (spectral matting)

• A. Levin, A. Rav-Acha, D. Lischinski. “Spectral Matting,” IEEE T. PAMI, vol. 30, no. 10, pp. 1699-1712, Oct. 2008.

• Extended to learning-based matting• Y. Zheng and C. Kambhamettu. “Learning based digital matting,” In

ICCV, pages 889–896, 2009.• Extended to multi-layer matting

• D. Singaraju, R. Vidal. “Estimation of Alpha Mattes for Multiple Image Layers,” IEEE T. PAMI, vol. 33, no. 7, pp. 1295-1309, July 2011.

Matting Laplacian

Input Image

EstimatingPair-wise Affinity

Graphical ModelNode: Image PixelsEdge: Affinity

Supervised Matting

Background

Foreground

Matting Laplacian Matrix:Recording the Connectivity between Pair of Pixels

Introduction of Graph Laplacian

2

3

1

4

5

A Graph with Five Vertexes

: Adjacency Matrix

𝐿=𝐷−𝑊: Laplacian Matrix

𝑊=(𝑤𝑖𝑗)𝑖 , 𝑗=1 ,… ,𝑛

: Degree Matrix

𝑑𝑖𝑖=∑𝑗=1

𝑛

𝑤𝑖𝑗

0 1 1 0 01 0 1 0 01 1 0 0 00 0 0 0 10 0 0 1 0

12345

1 2 3 4 5

: Adjacency Matrix

Vertex Index

Cutting Cost Function with Graph Laplacian

𝜶𝑇 𝐿𝜶=12 ∑𝑖 , 𝑗=1

𝑛

𝑤𝑖𝑗 (𝛼 𝑖−𝛼 𝑗 )2

Cost Function for Cutting Criterion

Low-costAssignment

High-costAssignment2

3

1

4

5

2

3

1

4

5

Construction of Matting Laplacian

• Color-model-based Approach (Original)• Estimating Affinity Based on Relative Color Distance

• Learning-based Approach (Extended)• Learning Affinity Based on Image Structure

Construction of Matting LaplacianColor-model-based Approach

Color Distribution

𝐼 𝑖

𝐼 𝑗

𝜇𝑘

Input Image

A. Levin, D. Lischinski, Y. Weiss. “A Closed Form Solution to Natural Image Matting,” IEEE T. PAMI, vol. 30, no. 2, pp. 228-242, Feb. 2008.

gr

b

Construction of Matting LaplacianLearning-based Approach

• Learning Affinity among Local Pixels

¿ [𝐱 𝑖𝑇 1 ] [ 𝜷𝛽0 ]

Linear Alpha-color Model for Single Pixel:

: Alpha Value for Pixel i: Feature Vector (): Linear Coefficient

Extending to a Local Patch qAssuming all Pixels Sharing the Same Linear Coefficient 𝛼 𝑖=𝐱 𝑖

𝑇 𝜷+𝛽0�⃗�𝑞=𝐗𝒒

𝑇 [ 𝜷𝛽0]: Alpha Vector for Patch q: Feature Matrix: Linear Coefficient

Construction of Matting LaplacianLearning-based Approach

[ 𝜷𝛽0]=argmin𝜷 ,𝛽0‖�⃗�𝑞−𝐗𝒒𝑇 [ 𝜷𝛽0]‖

2

+𝜆𝑟 𝜷𝑇 𝜷

¿ (𝐗𝒒𝑇𝐗𝒒+𝜆𝑟 𝐈 )−𝟏𝐗𝒒 �⃗�𝑞

�⃗�𝑞=𝐗𝒒𝑇 [ 𝜷𝛽0]

¿𝐗𝒒𝑇 (𝐗𝒒

𝑇𝐗𝒒+𝜆𝑟 𝐈 )−𝟏𝐗𝒒 �⃗�𝑞

Derived Linear Coefficient

Rewritten Linear Model

Construction of Matting LaplacianLocal Cost Function

�⃗�𝑞=𝐗𝒒𝑇 (𝐗𝒒

𝑇𝐗𝒒+𝜆𝑟 𝐈 )−𝟏𝐗𝒒 �⃗�𝑞

Local Cost Function

¿ �⃗�𝑞𝑇 𝑳𝑞 �⃗�𝑞

: Local Laplacian Matrix for Patch qInput Image

Patch q

Local Linear Model

Construction of Matting LaplacianLocal Global

Local Cost Function

¿ �⃗�𝑞𝑇 𝑳𝑞 �⃗�𝑞

: Local Laplacian Matrix for Patch q

Input Image

Patch q

Global Cost Function

¿ �⃗�𝑇 𝑳 �⃗�

Supervised Matting (Closed-form Matting)

Foreground Pixel

Background Pixel

Unknown Pixel

Input Image User’s Guidance,

𝐸 ( �⃗� )=�⃗�𝑇 𝑳 �⃗�+(�⃗�− �⃗�)𝑇𝚲 (�⃗�− �⃗�)

Foreground

Background

Unknown

1 0 -1 1 0

Cost Function for Supervised Matting

Affinity Cost Data Cost

�⃗�∗=(𝑳+𝚲 )−1𝚲 �⃗�Optimal Solution

Experimental Results

Input Image Alpha Matte Synthesized Result

Unsupervised Matting (Spectral Matting)

• Solving Alpha Matte without User’s Guidance• Procedures

• Decomposing Image into Several Matting Components• Combining Matting Components into Alpha Matte

Spectral Clustering

s.t. =1 𝐿 𝒇 =λ 𝒇1. L is symmetric and positive semi-definite.2. The smallest eigenvalue of L is 0, the

corresponding eigenvector is the constant one vector 1.

3. L has n non-negative, real-valued eigenvalues

0= λ 1 ≦ λ 2 ≦ . . . ≦ λ n.

: Eigenvector: Eigenvalue

2

3

1

4

5

A Graph Example

2 -1 -1 0 0-1 2 -1 0 0-1 -1 2 0 00 0 0 1 -10 0 0 -1 1

12345

1 2 3 4 5

: Laplacian Matrix

0.0470.0470.0470.0470.047

0.5770.5770.57700𝒇 1 𝒇 2

Spectral Clustering & Matting Components

2 -1

-1

0 0 0 0

-1

2 -1

0 0 0 0

-1

-1

2 0 0 0 0

0 0 0 1 1 0 00 0 0 -

1-1

0 0

0 0 0 0 0 1 10 0 0 0 0 -

1-1

: Laplacian Matrix

1110000

0001100

0000011

Zero-Eigenvectors Binary Indicating Vectors

×𝑹3×3Linear

Transformation

Overview of Spectral Matting

Input Image

Smallest Eigenvectors

Matting Components

K-means Clustering

&Linear

TransformationMatting

Laplacian

Spectral Clustering & K-means

Input Image

s-smallest Eigenvectors

…

Pixel i

s-dimensional

Space

K-means Clustering

Generating Matting Components

Smallest Eigenvectors

Projection into Eigen Space

..K-means .… … …

𝑬=[𝒆𝟏 … 𝒆𝒔 ] 𝒎𝒌 𝜶𝒌=𝑬 𝑬𝑻𝒎𝒌

Reconstructing Alpha Matte from Matting Components

=+ +

Input Image

Matting Components

Selected Matting Components

Alpha Matte

Reconstructing Alpha Matte by Grouping Matting Components

Matting cost function

𝐽 ( �⃗� )=�⃗�𝑇 𝑳 �⃗�

�⃗�=[ �⃗�1 … �⃗�𝑘 ] �⃗�Alpha Matte Generation

: Combination Vector

¿ �⃗�𝑇 [ �⃗�1 … �⃗�𝑘 ]𝑇 𝑳 [�⃗�1 … �⃗�𝑘 ] �⃗�¿ �⃗�𝑇𝜱�⃗�

Evaluating All Grouping Hypothesis to Derive the Optimal Alpha Matte

Results by Levin et al.

Summary

• Constructing Matting Laplacian• Solving Supervised Matting Problem• Solving Unsupervised Matting Problem

Proposed Approaches

• Efficient Cell-based Framework for Reducing Computations• Multi-scale Analysis• Extended Applications (Depth Image Reconstruction)

Input Image Reconstructed Depth

Depth Reconstruction from Single Image

Depth Reconstruction in Shape From Focus (SFF)

Input Image Reconstructed Depth

Cell-based Framework

Image

Pixel-wise Data Distribution

Cell-wise Data Distribution

ConventionalMatting Laplacian

Cell-basedMatting Laplacian

Pixel-wise Affinity

Cell-wise Affinity

Multi-scale Affinity Learning

Image & Computation Patterns

Pixel-based Approach

Cell-based Approach

Multi-scale Affinity Learning

…Finest Level Coarsest

Level …

Cell-based Graph

Results of Reconstructed Alpha Matte

1st Rank 2nd Rank

(a) Grouping Results by Levin et al.

(b) Grouping Results by Levin et al. with Coarse-to-fine Scheme.

(c) Ours

Input

Results

(a) Input images

(b) Levin’s result (c) Our result

Proposed Approaches

• Efficient Cell-based Framework for Reducing Computations• Multi-scale Analysis• Extended Applications (Depth Image Reconstruction)

Input Image Reconstructed Depth

Depth Reconstruction from Single Image

Depth Reconstruction in Shape From Focus (SFF)

Input Image Reconstructed Depth

Depth Reconstruction in Shape From Focus (SFF)

Optical Direction

Multi-focus Image Sequence

Optical Direction

FocusValueW1

W2

W2

W1

Low-SNR Problem

• Spatially Varying Precision• Low-texture Low-SNR• Leading Noisy Result

Input Image Observation

High-precision

Low-precision

Proposed Maximum-a-posteriori Estimation

Multi-focus Image Sequence

Learning-based Graph

Local Learning

Inference

Reconstructed Depth

Proposed Maximum-a-posteriori Estimation

𝐷∗=max (𝑝 (𝐷|𝑌 , 𝐼 ) ): Optimal Result: Depth Image: Observation: Input Image

Posterior Likelihood Prior

Local Observation with Spatial-varying Precision

Learned from Image

Likelihood Model

Input Observation

Precision Result

High-precision

Low-precision

𝑰 𝒀

𝚲 𝐷∗

Posterior Likelihood Prior

Local Observation with Spatial-varying Precision

Prior Model Posterior Likelihood Prior

Learning from Input Image

Learning-based Graph

Local Learning

Multi-focus Image Sequence

Maximum-a-posteriori Estimation for Depth Reconstruction

𝐷∗=max (𝑝 (𝐷|𝑌 , 𝐼 ) )

− log𝑝 (𝐷|𝑌 , 𝐼 )∝ ( �⃗�−𝒚 )𝑇𝚲 ( �⃗�− �⃗� )+ �⃗�𝑇 𝑳 �⃗�

𝐷∗= (𝑳+𝚲 )−1𝚲 �⃗�

Input Image Observation Reconstructed Depth

Results of Shape from Focus

Input Image M. Mahmood, 2012 T. Aydin, 2008 OursS. Nayar, 1994

Conclusions

• Construction of Matting Laplacian• Conventional Approach• Multi-scale Cell-based Approach

• Supervised Matting• Spectral Matting• Depth Reconstruction