Objective - CMU · 1. Mean Curvature Flow of Level Lines 2. Perona‐Malik Scheme 3. Perona‐Malik...
Transcript of Objective - CMU · 1. Mean Curvature Flow of Level Lines 2. Perona‐Malik Scheme 3. Perona‐Malik...
Objective
Variety of variational models in image processing.
Gradient descent: Curve / surface evolution:
Algorithms for finding global minimizers.
6/7/2010 2 Computing Interfacial Motions in Imaging and Vision
Example:
Outline
Monday: Problems, Models, Basic Facts.
Tuesday: Diffuse Interface Methods:
Phase field
Threshold dynamics
Distance function dynamics
Thursday:
Level sets
Redistancing: Fast marching
Friday: Finding global minima via PDEs.
Saturday: Network flows and graph cuts.
6/7/2010 3 Computing Interfacial Motions in Imaging and Vision
Monday
1. Image Segmentation 1. Snakes: Active Contours
2. Mumford‐Shah Functional
3. Piecewise Constant Mumford‐Shah
4. Segmentation with Depth: Nitzberg‐Mumford‐Shiota Model
2. Image Denoising 1. Mean Curvature Flow of Level Lines
2. Perona‐Malik Scheme
3. Perona‐Malik and Mumford‐Shah
4. Rudin‐Osher‐Fatemi’s Total Variation Model
3. Inpainting.
4 6/7/2010 Computing Interfacial Motions in Imaging and Vision
Representing Images
Represent a gray‐scale image by a function :
:Ω → 0,1
Ω is the image domain = Computer screen (a rectangle).
Value of = Gray‐scale intensity of pixel at location .
6/7/2010 5 Computing Interfacial Motions in Imaging and Vision
Image Segmentation
Assumption: Image depicts a scene containing several objects.
Segmentation: Divide image into distinct regions.
Mathematically: Partition image domain Ω.
Ω Σ
Each Σ contains an object, and
Σ ∩ Σ Σ ∩ Σ
Edges in the image: Σ
6 6/7/2010 Computing Interfacial Motions in Imaging and Vision
Image Segmentation
Edges in the image: Boundaries of distinct objects.
Assume: Different objects have different:
Color
Grayscale intensity
Texture
etc.
Expect:
is discontinuous, or | | is very large
at edges.
6/7/2010 7 Computing Interfacial Motions in Imaging and Vision
Image Segmentation
6/7/2010 8 Computing Interfacial Motions in Imaging and Vision
Hand segmented image from Berkeley database.
Image Segmentation
6/7/2010 9 Computing Interfacial Motions in Imaging and Vision
Segmentation and edge detection:
Hand segmented image from Berkeley database.
Segmentation is an ill‐posed task.
It is necessary to specify the level of detail desired. 10 6/7/2010 Computing Interfacial Motions in Imaging and Vision
Hand segmented image from Berkeley image database. 11 6/7/2010 Computing Interfacial Motions in Imaging and Vision
Hand segmented image from Berkeley image database. 12 6/7/2010 Computing Interfacial Motions in Imaging and Vision
Snakes: Active Contours
Kaas, Witkin, Terzopoulos (1992); Kichenassamy, Sapiro, Tannenbaum (1996)
IDEA: Initialize a curve (snake) on the image domain Ω.
, : 0,1 → Ω.
Prescribe a normal speed to drive it towards edges.
Edge detection:
large ⇒ 1
1 ∗ 0
where 14
Gradient descent for:
11 ∗
6/7/2010 13 Computing Interfacial Motions in Imaging and Vision
Snakes: Active Contours
Global minimizer = A point. ⇒ Look for a local minimizer.
Start near, containing object of interest. Let curve shrink‐wrap it.
Alternatively, start small, in the interior of object of interest.
Provide an “expansion force”:
Area in Γ ⋅ ⋅
where
,12 ,
6/7/2010 14 Computing Interfacial Motions in Imaging and Vision
Snakes: Active Contours
6/7/2010 15 Computing Interfacial Motions in Imaging and Vision
Image Edge map ∗
Snakes: Active Contours
6/7/2010 16 Computing Interfacial Motions in Imaging and Vision
Snakes: Active Contours
6/7/2010 17 Computing Interfacial Motions in Imaging and Vision
Gradient Descent: Perimeter
arclength parametrization of Σ, oriented counterclockwise.
Let:
and Outward unit normal.
We have:
so that 0 for convex shapes by our convention.
Consider the perturbation of :
where : Σ → .
6/7/2010 18 Computing Interfacial Motions in Imaging and Vision
Gradient Descent: Perimeter
Compute the length:
,
,
1 2
Differentiate w.r.t. t:
Thus, we see that
6/7/2010 19 Computing Interfacial Motions in Imaging and Vision
Gradient Descent: Weighted Perimeter
Let be a given positive weight function on Ω.
Define the weighted length
for a curve parametrized by arclength.
As before:
1 2
We find
⋅
6/7/2010 20 Computing Interfacial Motions in Imaging and Vision
Gradient Descent: Weighted Perimeter
Thus, we get:
⋅
Example: Geodesic active contours:
⋅ .
6/7/2010 21 Computing Interfacial Motions in Imaging and Vision
Mumford‐Shah Segmentation Model
Find the best piecewise smooth approximation, in least squares sense, to the given image:
min⊂Ω
LengthΩ∖
Ω
Unknowns of the problem:
u(x): The piecewise smooth approximation. Smooth except across “edges” .
: The set of edges across which u(x) is allowed to be discontinuous.
: Acts as a scale parameter.
22 6/7/2010 Computing Interfacial Motions in Imaging and Vision
Mumford‐Shah Segmentation Model
Ω∖ : Ensures smoothness away from edges.
Length : Prevents oversegmentation. Allows selection of scale.
Ω : Fidelity terms. Ensures approximation of given image.
Advantages:
No explicit edge detection needed.
Does not require presence of prominent transitions .
Robust w.r.t. noise.
6/7/2010 23 Computing Interfacial Motions in Imaging and Vision
Mumford‐Shah Segmentation Model
6/7/2010 24 Computing Interfacial Motions in Imaging and Vision
Without the length term, the Mumford‐Shah function can be trivially minimized:
inf,
Ω∖
Ω
0
Mumford‐Shah Segmentation Model
6/7/2010 25 Computing Interfacial Motions in Imaging and Vision
Example:
An image and its piecewise smooth approximation found by minimizing the Mumford‐Shah functional.
Mumford‐Shah Segmentation Model
min⊂Ω
LengthΩ∖
Ω
Consider the limit:
, , → 0 .
0 in Ω ∖ .
⇒ Piecewise constant .
Model becomes:
min∐ Ω
Length Σ
No a priori restriction on number of regions Σ .
However, # of regions bdd. in terms of .
6/7/2010 26 Computing Interfacial Motions in Imaging and Vision
Piecewise Constant Mumford‐Shah Model
Approximate the image by a piecewise constant function.
Simplest example: A function of two values.
Any such function can be written as:
Ω∖
Ω ∖ Σ
Σ
27 6/7/2010 Computing Interfacial Motions in Imaging and Vision
Chan‐Vese (2000):
Ω∖ Then, we have: 1. K Σ,
2. u dxΩ∖K 0,
3. Length K Per Σ ,
4. f u dxΩ f c dx f c dxΩ∖
Piecewise Constant Mumford‐Shah Model
6/7/2010 28 Computing Interfacial Motions in Imaging and Vision
Piecewise Constant Mumford‐Shah Model
The energy becomes:
Σ, , Per Σ Ω∖
How can we minimize such energies?
Make an intial guess for Σ.
Update Σ so that energy decreases as fast as possible.
⇒ SOLVE a PDE describing the motion of Σ.
29 6/7/2010 Computing Interfacial Motions in Imaging and Vision
Piecewise Constant Mumford‐Shah Model
6/7/2010 30 Computing Interfacial Motions in Imaging and Vision
Piecewise Constant Mumford‐Shah Model
6/7/2010 31 Computing Interfacial Motions in Imaging and Vision
Piecewise Constant Mumford‐Shah Model
6/7/2010 32 Computing Interfacial Motions in Imaging and Vision
Piecewise Constant Mumford‐Shah Model
6/7/2010 33 Computing Interfacial Motions in Imaging and Vision
Piecewise Constant Mumford‐Shah Model
6/7/2010 34 Computing Interfacial Motions in Imaging and Vision
Piecewise Constant Mumford‐Shah Model
6/7/2010 35 Computing Interfacial Motions in Imaging and Vision
Gradient Descent: Bulk Energy
Take variation of terms of the form:
Σ
where is a given function.
Again, perturb as . Assume 0 ∀ .
6/7/2010 36 Computing Interfacial Motions in Imaging and Vision
Σ
ΔΣ
A parametrization for ΔΣ:
, where ∈ 0, and ∈ 0, .
Gradient Descent: Bulk Energy
We have:
Also,
| |
Furthermore,
and
That gives:
.
6/7/2010 37 Computing Interfacial Motions in Imaging and Vision
Gradient Descent: Bulk Energy
1
1
.
We thus see that
6/7/2010 38 Computing Interfacial Motions in Imaging and Vision
Gradient Descent: Mumford‐Shah
P.C. Mumford‐Shah:
Σ, , Per Σ Ω∖
Rewrite as:
Σ, , Per Σ
Ω
Normal speed:
6/7/2010 39 Computing Interfacial Motions in Imaging and Vision
Segmentation with Depth
Given: A 2D picture of a scene with various objects in it:
:Ω → 0,1
Goal: Determine automatically the shapes and relative nearness of the objects to the observer through their occlusion relations:
We have prejudices: Prefer low curvature, connect T‐junctions.
Find an algorithm that mimics our prejudices:
⇒ CURVATURE DEPENDENT FUNCTIONALS
40 6/7/2010 Computing Interfacial Motions in Imaging and Vision
Nitzberg‐Mumford‐Shiota Model
Each object lives in a plane perp. to line of sight:
No self occlusions
No entanglements.
Each object may occlude parts of objects behind it:
41 6/7/2010 Computing Interfacial Motions in Imaging and Vision
Nitzberg‐Mumford‐Shiota Model
Strategy: Exhibit the solution as minimizer of an energy.
Unknowns:
1. Number of objects n,
2. Regions Σ ,… , Σ that they occupy.
3. Approximate “color” of each object: , … , .
Notation:
1. j i means Σ is in front of Σ .
2. Σ is the visible part of Σ .
Σ ≔ Σ Σ
3. = Curvature of Σ .
42 6/7/2010 Computing Interfacial Motions in Imaging and Vision
Nitzberg‐Mumford‐Shiota Model
The Energy:
. 1
where the function is:
1. ξ for |ξ| small,
2. |ξ| for |ξ| large,
3. +ve, even, C , and convex.
43 6/7/2010 Computing Interfacial Motions in Imaging and Vision
Explanation of Terms
Length term: : Regions should be simple.
Curvature term: : Edge contours should have a tendency to
continue straight, not make sharp turns.
Fidelity term: Approximate scene (assembled from regions Σ and the constants ), given by
should be faithful to the original image :
44 6/7/2010 Computing Interfacial Motions in Imaging and Vision
Role of Curvature Dependence
6/7/2010 45 Computing Interfacial Motions in Imaging and Vision
Top object
Bottom object
Role of Curvature Dependence
6/7/2010 46 Computing Interfacial Motions in Imaging and Vision
Fidelity term inactive in this region.
Minimization w.r.t. Σ :
1 1
1 Σ
Role of Curvature Dependence
6/7/2010 47 Computing Interfacial Motions in Imaging and Vision
Σ Σ
Curvature term will prefer the completion on the right; fidelity term is indifferent between the two.
Nitzberg‐Mumford‐Shiota Model
6/7/2010 48 Computing Interfacial Motions in Imaging and Vision
Nitzberg‐Mumford‐Shiota Model
6/7/2010 49 Computing Interfacial Motions in Imaging and Vision
Image Denoising: Heat Equation
GOAL: Remove oscillations from a noisy image.
Simplest method: Filtering.
→ ∗
Equivalent to solving the heat equation:
Δ , 0
Oscillations are suppressed: Good.
Edges are blurred: Bad.
Generates a one‐parameter family of gradually simplifying images.
⇒ Scale space
6/7/2010 50 Computing Interfacial Motions in Imaging and Vision
Image Denoising: Heat Equation
6/7/2010 51 Computing Interfacial Motions in Imaging and Vision
Heat equation:
Image Denoising: Heat Equation
6/7/2010 52 Computing Interfacial Motions in Imaging and Vision
Image Denoising: Mean Curvature Motion
IDEA: Suppress diffusion across edges:
Δ , ⟨ , ⟩
where , 1 and .
Choose:
and
Suppress diffusion in direction:
, ⋅
⇒ Motion by mean curvature of level sets of .
6/7/2010 53 Computing Interfacial Motions in Imaging and Vision
Image Denoising: Mean Curvature Motion
Curvature: Let be an arc‐length parametrization of the 0‐level set:
: ∈ : 0
Since 0 for all , we have
0 ⋅
and
0 ,
,
,
so that 1
,
Image Denoising: Curvature Motion
In addition,
, , , , ⋅ 0
But, ⋅
so that
Thus, combining: ,
which means .
6/7/2010 55 Computing Interfacial Motions in Imaging and Vision
Image Denoising: Mean Curvature Motion
6/7/2010 56 Computing Interfacial Motions in Imaging and Vision
Mean curvature motion:
Image Denoising: Mean Curvature Motion
6/7/2010 57 Computing Interfacial Motions in Imaging and Vision
Image Denoising: Ill‐Posed Equations
Go even further: Reverse diffusion in direction.
⋅
where
e.g. 1
1
Expand:
⋅ , 2 ′ ⟨ , ⟩
6/7/2010 58 Computing Interfacial Motions in Imaging and Vision
+ve for 1, ‐ve for 1
Image Denoising: Perona‐Malik Model
Perona‐Malik Model:
⋅
Typically implemented as
where
e.g. 1
6/7/2010 59 Computing Interfacial Motions in Imaging and Vision
Image Denoising: Perona‐Malik Model
6/7/2010 60 Computing Interfacial Motions in Imaging and Vision
Perona‐Malik Evolution
Image Denoising: Perona‐Malik Model
Perona‐Malik Scheme
is gradient descent for the energy
6/7/2010 61 Computing Interfacial Motions in Imaging and Vision
where density is:
Convex for small, Concave for | | large.
E.g. for ,
log 1 .
Image Denoising: Perona‐Malik Model
Perona‐Malik is intimately related to Mumford‐Shah.
Consider energy densities of the form
if 1
1 if
1
6/7/2010 62 Computing Interfacial Motions in Imaging and Vision
Image Denoising: Perona‐Malik Model
Corresponding discrete energies:
,
Weak spring model of Geman & Geman (Blake & Zisserman):
6/7/2010 63 Computing Interfacial Motions in Imaging and Vision
Image Denoising: Perona‐Malik Model Send → 0 :
lim→
Ω∖
Mumford Shah
Explanation:
1. If is differentiable near , , , then
, 1 and , 1 as → 0 .
Hence,
→ as → 0 .
2. If has a vertical edge at , , , then
,1
as → 0 .
Hence,
→1 as → 0 .
6/7/2010 64 Computing Interfacial Motions in Imaging and Vision
Rigorous result: A. Chambolle, SIAM J. Appl. Math (1996)
Image Denoising: Perona‐Malik Model
For energy densities of the form
1 1 where ∈ 0,1
If scaled correctly w.r.t. :
lim→
Ω∖
6/7/2010 65 Computing Interfacial Motions in Imaging and Vision
Image Denoising: Perona‐Malik Model
An old method for minimizing Mumford‐Shah.
Graduated non‐convexity of Blake & Zisserman.
Gradient descent (Perona‐Malik) for energies:
,
with gradually less convex :
6/7/2010 66 Computing Interfacial Motions in Imaging and Vision
Image Denoising: Total Variation Model
Rudin, Osher, and Fatemi (1992):
Preserves sharp edges.
Many advantages over Perona‐Malik:
Strictly convex functional: Existence, uniqueness of solutions.
Continuous dependence on data (i.e. ), and parameters (i.e. ).
Only one parameter to be chosen by user:
Still some difficulties:
Non‐differentiable.
Naïve numerical methods slow to converge.
Intimately related to: Piecewise constant Mumford‐Shah.
6/7/2010 67 Computing Interfacial Motions in Imaging and Vision
Total Variation: Basic Facts
To define total variation for possibly discontinuous functions:
Given a vector ∈ , we can write:
max ⋅
Apply this to
max ⋅
When is smooth, can take to be smooth and compactly supported, and move “max” outside:
sup
∈
⋅
6/7/2010 68 Computing Interfacial Motions in Imaging and Vision
Total Variation: Basic Facts
Integrate by parts:
sup
∈
⋅
Right hand side can be finite even for discontinuous .
It is taken to be the definition of total variation:
| | sup
∈
⋅
6/7/2010 69 Computing Interfacial Motions in Imaging and Vision
Total Variation: Basic Facts
Example:
where Σ is a compact set with smooth boundary Σ.
First of all,
⋅ ⋅ ⋅ Length Σ
Second: There exists a vector field s.t.
1. is smooth, compactly supported,
2. 1 for all ,
3. for all ∈ Σ.
We have:
⋅ ⋅ Length Σ
6/7/2010 70 Computing Interfacial Motions in Imaging and Vision
Total Variation: Basic Facts
Hence, we see that
| | Length Σ ≔ Per Σ
when Σ is smooth.
If is piecewise constant:
with 0, Σ ⊂ Σ , and Σ smooth for all j, then:
6/7/2010 71 Computing Interfacial Motions in Imaging and Vision
| | Per Σ
Total Variation: Basic Facts
Given any function ∈ , approximate by such piecewise constant functions.
Use our formula.
Pass to the limit.
You get the co‐area formula
| | Per ∶
6/7/2010 72 Computing Interfacial Motions in Imaging and Vision
Image Denoising: Total Variation Model
6/7/2010 73 Computing Interfacial Motions in Imaging and Vision
Image Denoising: Total Variation Model
6/7/2010 74 Computing Interfacial Motions in Imaging and Vision
Image Inpainting
Image information missing (scratches, holes) in ⊂ Ω.
Interpolate image into .
Nonstandard requirement: Propagate sharp edges into .
6/7/2010 75 Computing Interfacial Motions in Imaging and Vision
D
Ω
Image Inpainting
PDE approach: Originates in the work of Bertalmio, Caselles, Sapiro, et. al.
6/7/2010 76 Computing Interfacial Motions in Imaging and Vision
Image Inpainting
More examples (from Bertalmio, Caselles, Sapiro, Ballester):
6/7/2010 77 Computing Interfacial Motions in Imaging and Vision
Image Inpainting
Inpainting via the Total Variation Model: Work of T. F. Chan and J. Shen.
Very robust, variational model.
Ω∖Ω
6/7/2010 78 Computing Interfacial Motions in Imaging and Vision
Image Inpainting
Caveats of 2nd order inpainting models:
No long distance connections between contours:
Non‐smooth, unnatural contour extensions:
6/7/2010 79 Computing Interfacial Motions in Imaging and Vision
D
D