Natural Video Matting using Camera Arrays

Neel S. Joshi Wojciech Matusik Shai Avidan

Mitsubishi ElectricResearch Laboratory

University of California,San Diego

Related Work

• Studio Matting– Vlahos 71, Wallace 82, Smith and Blinn 96, Ben-Ezra 00,

Debevec et. al 02, McGuire et al. 06

• User Assisted Natural Matting– Ruzon and Tomasi 00, Chuang et al. 01, Chuang et al. 02,

Sun et al. 04, Li et al. 04, Rother et al 04, Li et al. 05, Wang et al. 05, Wang and Cohen 05, Levin et al. 06

• Automated Natural Matting– Wexler et al. 02, Zitnick et al. 04, McGuire et al. 05, Kolmogrov

et al. 05

– Our Method

Contributions

• Extending the matting equation to use the variance of pixel measurements

• Using a camera array for alpha matting

• An automatic method for computing alpha mattes that is linear in the number of pixels

• A near real-time system for natural video matting

– 2 to 5 FPS

Algorithm Overview

Foreground Object

Background

Camera Array

Using a Camera Array

Foreground and Background Parallax

Aligning to a Foreground Plane

Foreground Plane

I0

In

I0 InIi

Ii

Aligned Images

Variance across cameras

Taking the Variance

Background

Background

Foreground

Foreground

Transparent(Mixture)

Transparent

I0 InIi

Key Idea

alpha is a function of: variance across cameras images

Traditional Alpha Matting [Smith and Blinn 96]

I = F + (1 - )B

I = observed image

= transparency

F = foreground

B = background

Alpha Matting with Multiple Images

I = F + (1 - )B

I, F, and B are continuous random variables

is a scalar

I = F + (1 - )B

Solving for alpha

2var(F) + (1 - )2var(B) - var(I) = 0

var(I) = var[F + (1 - )B]

Solving for alpha and F

2var(F) + (1 - )2var(B) - var(I) = 0

)(mean1)(meanF BI

a2 + b + c = 0

a = var(F) + var(B) b = -2var(B) c = var(B) - var(I)

Triangulation Matting: A Special Case of our Method

I1 I2

21

211BBII

[Smith and Blinn 96]

2

)var(2

21 III

0)var( F

2

)var(2

21 BBB

)var()var(

1BI

F is a function of: [mean(I) ,mean(B)]

Variance and Mean Statistics

• We observe I

• F and B can’t be directly measured

is a function of: [var(I) ,var(F) ,var(B)]?

?

?

Foreground = Low var(I)

Background = High var(I)

Threshold var(I) to get Trimap

Propagating Variance and Mean

For each pixel in the unknown (gray) region:

– var(F) = var(Inf)

where Inf is the nearest pixel in the foreground

– var(B) = var(Inb), mean(B) = mean(Inb)

where Inb is the nearest pixel in the background

Algorithm Steps

1. Calibrate camera array and film scene

2. Find the foreground depth

3. Compute var(I) and mean(I)

4. Threshold var(I) to get trimap

5. Estimate var(F), var(B) , mean(B)

6. Compute and F

System and Implementation Details

• 8 640x480 (Bayer pattern) Basler Video Cameras

• 1 Desktop PC (P4 3.0 Ghz)

• Calibration:

– Color w/ Macbeth chart

– Bundle Adjustment

• Average Running Time:

2 to 5 FPS at 320 x 240

500

2500

4500

Our method

Using a known background

500

2500

4500

Our method

Using a known background

Limitations

• Self-occlusion: alpha is not view independent

Known Backgroun

d

Our Method

• Need background variation, i.e., var(B) should be large

X

Conclusions

• Standard matting equation extended to work with variance and mean statistics

• Running time is linearly proportional to the number of pixels

• Automatic, near real-time alpha matting using a camera array

Acknowledgments

• SIGGRAPH Reviewers

• Joe Marks, John Barnwell, Frédo Durand, Matthias Zwicker, Bennett Wilburn, Dan Morris, and Merrie Morris

• Our willing and patient “Actors”: Lavanya Sharan, Chien-i Tu, Bernd Bickel, Yuanzhen Li

• MERL (for my internship)

• NSF IGERT Grant (for additional funding)

http://graphics.ucsd.edu/papers/camera_array_matting/