1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.
-
Upload
jeffrey-mckenzie -
Category
Documents
-
view
212 -
download
0
Transcript of 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.
1
Roey Izkovsky
Yuval Kaminka
Matting
Helping Superman fly since 1978
2
Outline
• The matting problem• Previous work• New approaches:
– The iterative approach
Jue Wang, Michael F.Cohen
– Closed form solutionAnat Levin, Dani Lischinski,Yair Weiss
• Comparison and summary• Bonus?
3
Outline
• The matting problem• Previous work• New approaches:
– The iterative approach
Jue Wang, Michael F.Cohen
– Closed form solutionAnat Levin, Dani Lischinski,Yair Weiss
• Comparison and summary• Bonus?
4
The matting problem - Motivation
Image and video editing
New backgroundComposite image
5
The matting problem - Motivation
Image and video editing
Input image New image
6
The matting problem
iiiii BFI )1(
– The separation of an image I intoI. Foreground object image F
II. Background image B
III.Alpha matte α – the opacity )10(
– Problem: extract F, B, α from image
hair fur
7
Why is matting challenging?
• Under constrained problem:One equation, 3 unknowns
iiiii BFI )1(
We need to constrain the problem!
8
Outline
• The matting problem• Previous work• New approaches:
– The iterative approach
Jue Wang, Michael F.Cohen
– Closed form solutionAnat Levin, Dani Lischinski,Yair Weiss
• Comparison and summary• Bonus?
9
Previous work
Two types:
Known background Natural image
matting matting
10
Known Background
• Blue screen Matting• Still under-constrained
– Solution: make more assumptions• “Foreground contains no blue”• Other foreground distribution assumption…
• Use two different backgrounds
• Main flaw: need to know the background…
iiiii BFI )1( Blue background Composite image
11
Natural Image Matting
• The assumptions:– Smoothness of the alpha matte– GMM for the Background and Foreground
colors
• Initial estimate: trimap provided by the user
Input image Trimap
• Background
• Foreground
• Unknown
12
Natural Image Matting
• The algorithms framework:– Estimate F, B distributions from close pixels– Find best α by some method
13
Knockout
– Extrapolate F,B from close neighborhood
– Estimate α from calculated F, B values
14
Bayesian
– Estimate F, B distributions in area
– Find best α matching distributions
CBFPBF
|,,maxarg,,
)(/)()()(,,maxarg,,
CPPBPFPBFCPBF
)()(,,maxarg,,
BPFPBFCPBF
15
Bayesian
– P(F), P(B) from image samples
– P(C|F,B,α) using a distribution for C BFC )1(
)()(,,maxarg,,
BPFPBFCPBF
16
Natural Image Matting
• Main flaw: Accurate trimap required• Tedious to provide manually
• Hard to extract automatically
In particular, not feasible to videos
Binary segmentation
Adding unknown region
Input imageTrimap
17
Great.So let’s get started…
18
Outline
• The matting problem• Previous work• New approaches:
– The iterative approach
Jue Wang, Michael F.Cohen
– Closed form solutionAnat Levin, Dani Lischinski,Yair Weiss
• Comparison and summary• Bonus?
19
New Approach to Matting
Trimap reduces to scribbles
Two new methods– Iterative optimization approach
• Heuristic algorithmic optimization
– A closed form solution• Mathematical approach
Trimap Scribbles
20
Iterative optimization approach
Jue Wang
Michael F. Cohen
21
Iterative approach
22
Iterative approach
• Score: fit to image data
+alpha matte smoothness
• Iteratively propagating estimated results.
23
Iterative optimization - outline
• Initialize “work pixels” from scribbles
• Repeatedly:• Expand work pixels • Find best alpha matte
• Stop when finished
24
Initialization
• Introducing:– ui - uncertainty variable
– Uc – work pixels
ui = 0
α = 0
Uc = {user scribbles}
ui = 0
α = 1
ui = 1
α = 0.5
)10( iu
25
Optimization
Uc = {user scribbles + 15 pixel radius}
Our goal:
find α matte for Uc that minimizes the energy -
),()(
~,
qp
qpUqp
sUp
pd
cc
VVV
DataSmoothness
26
Vd
Score for αp = α
),()(
~,
qp
qpUqp
sUp
pd
cc
VVV
N Possible values for F N Possible values for B
BF )1(
2))1(,( BFId p
Fw2
Fw3
Fw4
Fw1
Bw1
Bw2
Bw3
Bw4
Image color Ip
27
Vd
• Fit measure of αp to Ip
• Score for αp = α :
),()(
~,
qp
qpUqp
sUp
pd
cc
VVV
2
2
1 12 2
))1(,(exp
1
jip
N
i
N
j
Bj
Fi
BFIdww
N
Fi , Bj – possible values for F, B in the pixel
wFi, wB
j – corresponding weights
28
Vd
F Samples
B Samples
2
2
1 12 2
))1(,(exp
1
jip
N
i
N
j
Bj
Fi
BFIdww
N
Fi , Bj – possible values for F, B in the pixel
wFi, wB
j – corresponding weights
)2/(
),(exp))(1(
2
2
r
ppspuw iii
),()(
~,
qp
qpUqp
sUp
pd
cc
VVV
p
α = 0.9
u = 0.2
α = 0.8
u = 0.3
α = 0.4
u = 0.5
α = 0.4
u = 0.4
α = 0.2
u = 0.3
α = 0.3
u = 0.3
α = 0.5
u = 1.0
What happens when there are
not enough F/B samples?
29
Vd
• Score for αp = α :
),()(
~,
qp
qpUqp
sUp
pd
cc
VVV
2
2
1 12 2
))1(,(exp
1)(
jipN
i
N
j
Bj
Fip
BFIdww
NL
• Discretize
• and normalize
},...,,{ 2521
j
jp
kpk
pd L
LV
)(
)(1)(
30
Vs
• Matte smoothness :
),()(
~,
qp
qpUqp
sUp
pd
cc
VVV
)/)(exp(1),( 222121 ssV
31
Iterative optimization – step 2
Uc = {user scribbles + 15 pixel radius}
Our goal: find α matte for Uc that minimizes the energy -
),()(
~,
qp
qpUqp
sUp
pd
cc
VVV
Uc Graph
Nodes = Pixels, Edges by 4-connectivity
32
Iterative optimization – step 2
),()(
~,
qp
qpUqp
sUp
pd
cc
VVV
GOAL: Minimize
BELIEF PROPAGATION
33
Iterative optimization – step 2
),()(
~,
qp
qpUqp
sUp
pd
cc
VVV
GOAL: Minimize
BELIEF PROPAGATION
αlog p
0-2
0.04-1.7
……
12.3
p q
mpq – message from p to q
t=0
y
0tpqm
)(),(min)(0 pqp
p
q kpd
kq
kps
k
kq
tpq VVm
Vector: p’s “opinion” for eachpossible α for q
34
Iterative optimization – step 2
),()(
~,
qp
qpUqp
sUp
pd
cc
VVV
GOAL: Minimize
αlog p
0-1.6
0.04-1.2
……
12.2
p q
t=1
y
1tpqm
αlog p
01
0.040.7
……
1-2.1
0typm
BELIEF PROPAGATION
mpq – new message pq
myp – previous message yp
qrpr
kp
trp
kpd
kq
kps
k
kq
tpq
ppqp
p
q mVVm~
01 )()(),(min)(
35
Iterative optimization – step 2
),()(
~,
qp
qpUqp
sUp
pd
cc
VVV
GOAL: Minimize
BELIEF PROPAGATION
p q
t=2,3,4…
y
tpqm
1typm
qrpr
kp
trp
kpd
kq
kps
k
kq
tpq
ppqp
p
q mVVm~
1 )()(),(min)(
36
Iterative optimization – step 2
),()(
~,
qp
qpUqp
sUp
pd
cc
VVV
GOAL: Minimize
BELIEF PROPAGATION
p q
t=T (stopping time)
y
αLog p
0-1.7
0.04-1.3
……
11
αLog p
0-1.6
0.04-1.1
……
11.3
αLog p
0-1.7
0.04-1.3
……
11
αLog p
0-1.7
0.04-1.3
……
11
qr
qtrq
kqd kmV q
~
1 )()(
37
Iterative optimization – step 2
),()(
~,
qp
qpUqp
sUp
pd
cc
VVV
GOAL: Minimize
BELIEF PROPAGATION
p q
t=T (stopping time)
y
αLog p
0-1.7
0.04-1.3
……
11
αLog p
0-1.4
0.04-1.5
……
11.3
αLog p
0-1.7
0.04-1.3
……
11
αLog p
0-1.7
0.04-1.3
……
11
Best state calculated for each node:
prp
trp
kpd
kp kmV p
p ~
1 )()(minarg
38
Iterative optimization – step 3
),()(
~,
qp
qpUqp
sUp
pd
cc
VVV
Found α matte for Uc that minimizes the energy -
Update F, B and uncertainty:
**
1
))1((minarg,
*
2
,
**
Bj
Fi
jipBF
wwu
BFIBFji
39
Iterative optimization - algorithm
•Initialize Uc, F, B, u and alpha matte from scribbles
•Repeatedly:•Expand Uc by another 15 pixel radius•Find best alpha matte (BP)•Update F,B,u for new matte
•Stop when total uncertainty is minimal
Initial matte Propagation of α matte Final matte
40
Iterative optimization - Results
Input image Extracted matte
41
Iterative optimization - Results
Input image
Extracted matte
Composite image
42
Iterative optimization - Results
The ambiguity bunny
43
Ambiguity bunny with trimap
Iterative optimization - Results
Scribbles result Trimap resultAmbiguity bunny with scribbles
44
Iterative optimization - Summary
• Minimal user input• Applicable to video
• Sensitive to ambiguity in F, B• Uses simple color-model
• Performance:– 15-20 min. on a 640x480 image– Factor 50 reported by better implementation
45
Fantastic.Let’s go on…
46
Outline
• The matting problem• Previous work• New approaches:
– The iterative approach
Jue Wang, Michael F.Cohen
– Closed form solutionAnat Levin, Dani Lischinski,Yair Weiss
• Comparison and summary• Bonus?
47
Closed form solution
Anat Levin
Dani Lischinski
Yair Weiss
48
Closed form solution
• Assumption: local smoothness in F, B
cancel out unknowns from the matte equs.
• Solve for F,B and alphausing algebraic tricks.
49
Closed form solution
Assumptions:– F,B locally smooth.
treat F,B as constant in a small window w
ww
ww
www
wiwiwiwii
BF
Bb
BFa
bIawiBFI
1
)1(
50
Closed form solution
GOAL:
Minimize -
Ij wiwwiwi
j
jjjabIabaJ 22)(),,(
-Numerical stability
-Bias to smoother matte
wj
wiwi bIawi
ww
ww
www BF
Bb
BFa
1
51
Closed form solution
• GOAL:– Minimize:
Ij wiwwiwi
j
jjjabIabaJ 22)(),,(
211 )( bIa pp
299 )( bIa pp
29 ia
52
Closed form solution
• Minimize:
Ij wiwwiwi
j
jjjabIabaJ 22)(),,(
3N Variables (N = image size)
We can rid a, b by algebraic manipulation
53
Closed form solution
• Minimize:
Ij wiwwiwi
j
jjjabIabaJ 22)(),,(
Theorem: for we have
),,(min)(,
baJJba
k kwjik
kjki
kwkijij II
wL
),|(2
||
))((1
1||
1
LJ T)(
Intuitively, L is some covariance matrix
kwjik k
kjki
kij
II
wL
),|(2
))((
||
1~
54
Closed form solution
• Minimize:
Ij wiwwiwi
j
jjjabIabaJ 22)(),,(
Proof: Rewrite in matrix form:
k ww
w
w kk
k
kb
a
I
I
baJ
2
||
1
2/1||
1
00
1
1
),,(
55
Closed form solution
• Minimize:
Ij wiwwiwi
j
jjjabIabaJ 22)(),,(
Proof: Rewrite in matrix form:
0
1
1
2/1||
1
k
k
ww I
I
G
2
),,(
k
w
w
w
w k
k
k
k b
aGbaJ
0||
1
k
k
ww
By mean-least-squares, best a,b pair
for each window is:
kkkkkkw
Tww
Twww GGGba 1** )(),(
k
wwTww
Tww kkkkkk
GGGG2
1)(
),,(min)(,
baJJba
56
Closed form solution
• Some more manipulation give the required result
LJ T)(
k kwjik
kjki
kwkijij II
wL
),|(2
||
))((1
1||
1
EXCITED?
GET YOUR I LOVE MATHT-SHIRT, NOW FOR ONLY $1999
k
wTww
Tww kkkkk
IGGGGJ2
1 ))(()(
57
Closed form solution
• For color images:– Simple: Do each channel separately– Smart: Assume one alpha for R,G,B.
Use redundancy to allow a “color-line” model per window
21
21
)1(
)1(
wiwii
wiwii
BBB
FFF
Color line model:
OUT: F, B Constant within a window
IN: F, B are on some line
BGRc
wci
cwi bIawi
,,
R
GF1
F2
58
Closed form solution
c
wci
cwi bIawi
• For color images:– Simple: Do each channel separately– Smart: Assume one alpha for R,G,B.
Use redundancy to allow a “color-line” model per window
59
Closed form solution
c
wci
cwi bIawi
Ij wi c
cww
c
ci
cwi
j
jjjabIabaJ 22 )()(),,(
Now, as before, cost is:
And a,b can be cancelled out.
• For color images:– Simple: Do each channel separately– Smart: Assume one alpha for R,G,B.
Use redundancy to allow a “color-line” model per window
60
Closed form solution
Now problem reduced to finding best α for:
LJ T)(
k kwjik
kjki
kwkijij II
wL
),|(2
||
))((1
1||
1
L is Huge size NxN (N = # image pixels)
But Sparse…
61
Closed form solution
• The algorithm:– Compute L– Solve for given the scribbles.
• Solving a sparse set of bilinear equationswith constraints (Lagrange multipliers)
– Find F, B given the matte• Adding smoothness assumptions on F, B
• Improvements:– Use larger environment in low cost by “pyramids”
LTminarg
62
Closed form solution - Results
Input image Extracted matte
63
Closed form solution - Results
Input image with scribbles
Problematic matte
64
Eigenvectors as guides
Small eigenvectors of L are
correlated with minimal matte
L is positive definite.
Eigenbasis: v1,…,vN
Eigenvalues: λ1 > λ2 > … > λN > 0
2211
211 ...)( NNNN
t
ii
LJ
v
65
Eigenvectors as guides
Small eigenvectors of L are
correlated with minimal matte
66
Eigenvectors as guides
Small eigenvectors of L are
correlated with minimal matte
can guide user scribbles
Eigenvectors matching smallest eigenvalues
Guided scribbles Resulting matte
67
Closed form solution - Summary
• Minimal user input• Provable optimality (under assumptions)• Assumes only smooth F,B (no color model)
• Applicable to video (as we speak…)
• Problematic with textures
• Performance:– 20-40 seconds for a 200x300 image– Expensive in memory
68
Superb.Let’s sum up…
69
Outline
• The matting problem• Previous work• New approaches:
– The iterative approach
Jue Wang, Michael F.Cohen
– Closed form solutionAnat Levin, Dani Lischinski,Yair Weiss
• Comparison and summary• Bonus?
70
Comparison
Iterative approachPoisson Closed form solution
Input image
Matte ground truth
71
Main improvements
Trimap based approaches
New approaches
User inputTrimapScribbles
Complex foreground
Poor results. Exact trimap required
Good results
VideoNot easily applicable.Applicable
72
Comparison
Color ambiguity
Iterative approach Closed form Sensitive Sensitive
Solvable by adding more scribbles
73
Comparison
Improving results…
Iterative approachBayesian Closed form solution
Ambiguity bunny
74
Comparison
Optimality?
Iterative approach Closed form
Uses heuristics
to optimize
Provably optimal
But for the specific
(simplified) cost
75
Comparison
Textures
Iterative approach Closed form
Assumes only
Alpha matte smooth
F,B must satisfy
color-line model
76
Comparison
Rough edges
Iterative approach Closed form
Assumes
Alpha matte smooth
Can handle rough
edges
Input image with scribbles
matte results
77
Comparison
Running time
Iterative approach Closed form
~20 sec. 20/40 seconds
Costly in memory
(For medium size image)
78
Comparison
Tests
Iterative approach Closed form
No quantitative
results reported
Extensively tested
quantitative results
79
Outline
• The matting problem• Previous work• New approaches:
– The iterative approach
Jue Wang, Michael F.Cohen
– Closed form solutionAnat Levin, Dani Lischinski,Yair Weiss
• Comparison and summary• Bonus?
80
Environment Matting and Compositing
Douglas E. Zongker ~ Dawn M. Werner ~ Brian Curless ~ David H. Salsin
81
Environment Matting
C = F + (1- )B + ~ Contribution of light from Environment
that travels through the object
R – reflectance imageT – Texture image
82
Environment Matting?
Alpha Matte Environment Matte Photograph
83
Environment Mattin
Alpha Matte Environment Matte Photograph
84
Summary
• The matting problem• Old methods: require trimap• Two new methods from scribbles:
– Iterative optimization• Assume: matte smooth, F,B locally similar• Use heuristic optimization for alpha
– Close form solution• Assume: F, B locally smooth (color-line model)• Solve linear equations for alpha
85
ANY LAST
86