Intelligent Scissors for Image Composition

Anthony Dotterer

01/17/2006

Citation

• Title– Intelligent Scissors for Image Composition

• Author– Eric N. Mortensen– William A. Barrett

• Publication– 1995

Intelligent Scissors Tool

• Interactive image segmentation and composition tool– Easy to use– Quick– Quality output

• Features– Best path along image edges– Cooling – On-the-fly training– Source to destination warping and composition– Destination matching

Intelligent Scissors

• Need– Optimal path along edges starting at a ‘seed’

point– Optimal path creation must be quick

• Solution – Use dynamic programming to create path

reference• Local cost definition• Path reference creation

Local Cost Definition

• Define l(p,q) as the cost for going from pixel p to pixel q

• Incorporate the several edge functions into the cost– Laplacian Zero-Crossing, fZ(q)– Gradient Magnitude, fG(q)– Gradient Direction, fD (p,q)

• Relate edge functions to the cost function– Use ωZ, ωD, ωG as constants to weight features

l(p,q) = ωZ · fZ(q) + ωD · fD (p,q) + ωG · fG(q)

Laplacian Zero-Crossing

• Properties– Approximate 2nd partial derivative of Image– Zero-crossings represent maxima and minima

• Good image edges

• Cost– Define IL(q) as Laplacian at pixel q– Get low cost by defining Laplacian as a binary

fZ(q) = { 0; if IL(q) = 0, 1; if IL(q) ≠ 0

Laplacian Zero-Crossing (cont.)

• Issue– Zeros rarely occur

• Solution– Use pixel closest to

zero

• Examples– Image (top)– Laplacian (bottom)

Gradient Magnitude

• Properties– Magnitude of 1st partial derivatives of an image– Direct correlation between edge strength and local

cost• Cost

– Define G as gradient magnitude G = √(Ix² + Iy²)

– Get low cost by inverting and scaling fG = 1 – G / (max(G))

– Also factor in Euclidean distance• Scale adjacent pixels cost by 1• Scale diagonal pixels cost by 1/√2

Gradient Magnitude (cont.)

• Examples– Original (top left)– Gradient Magnitude

(top right)– Inverted & Scaled

Gradient Magnitude (bottom)

Gradient Direction

• Properties– Vectors created by the 1st derivatives of an image– High cost for shape changes

• Adds smoothing constraint

• Cost– Give low costs to gradients in the same direction– Define D(p) as the unit vector perpendicular to the

gradient vector at point p

D(p) = norm(Iy(p), -Ix(p))

Gradient Direction (cont.)

• Cost– Define L(p, q) to be the link between point q

and p, such thatL(p, q) = { q – p; if D(p) · (q – p) ≥ 0, p – q; if D(p) · (q – p) < 0

– Let dp(p, q) and dq(p, q) as followsdp(p, q) = D(p) · L(p, q)

dq(p, q) = L(p, q) · D(q)

– Finally, the cost functionfD(p, q) = 1/π ( cos-1(dp(p, q)) + cos-1(dq(p, q)) )

Gradient Direction (cont.)• Let

– p = (3, 3)– q = (3, 4)– D(p) = (0, 1)– D(q) = (0, 1)

• Calculate L(p, q)– L(p, q) = ((3, 4) – (3, 3)) = (0,

1)• Determine d(p, q)

– dp(p, q) = (0, 1) · (0, 1) = 1 – dq(p, q) = (0, 1) · (0, 1) = 1

• Finally fD(p, q)– fD(p, q) = 1/π ( 0 + 0 ) = 0– Low Cost!

• Let – p = (3, 3)– q = (4, 3)– D(p) = (0, 1)– D(q) = (0, 1)

• Calculate L(p, q)– L(p, q) = ((4, 3) – (3, 3)) = (1,

0)

• Determine d(p, q)– dp(p, q) = (0, 1) · (1, 0) = 0 – dq(p, q) = (1, 0) · (0, 1) = 0

• Finally fD(p, q)– fD(p, q) = 1/π ( π/2 + π/2 ) = 1– High Cost!

Path Reference Creation

• Differs from method studied in class– No stages– Link cost between nodes changes– No destination

• Inputs– Seed point, s– Local cost function, l(q, r)

Path Reference Creation (cont.)

• Data structures– Sorted list of active pixels, L– Neighborhood of pixel q, N(q)– Flag map of expanded pixels, e(q)– Cumulative cost from seed point, g(q)

• Output– Path reference map, p

Path Reference Creation (cont.)

• Start at seed point– Cost is adjusted for

Euclidean distance

• Put all neighbor pixels into the active list– No other pixel has yet

to be expanded

• Set pointers for all neighbors to the seed point

1

2

3

4

5

6

7

8

9

1 2 3 4 5 6 7 8 9 10 11

1

112

4

713 7

7

L = (4,8), (3,7), …

Path Reference Creation (cont.)

• Expand to least cost node– Remove that node from

active list• Calculate cumulative cost

of all neighbor pixels– Excludes seed point

• Change pointers of neighbor pixels– Only if new cost is smaller

and pixel is not expanded• Add or replace neighbor

pixels into active list– Do nothing if pointer was

not updated

1

2

3

4

5

6

7

8

9

1 2 3 4 5 6 7 8 9 10 11

1

92

4

713 6

7

L = (3,7), (2,8), (5,7) …

14

6

5

Path Reference Creation (cont.)

• Expand to least cost node– Remove that node from

active list• Calculate cumulative cost

of all neighbor pixels– Excludes expanded pixels

• Change pointers of neighbor pixels– Only if new cost is smaller

and pixel is not expanded• Add or replace neighbor

pixels into active list– Do nothing if pointer was

not updated

1

2

3

4

5

6

7

8

9

1 2 3 4 5 6 7 8 9 10 11

1

92

4

713 6

7

L = (2,6), (3,6), (4,9), …

14

6

5

18

16

20

6 6 12 14 23

9

13

Path Reference Creation (cont.)

• Finished– No more pixels to

expand– No more pixels on

active list

‘Live-Wire’ Action

• Mouse will constantly redraw optimal path– A wire will ‘snap’ to

objects with an image

• New seed points– New seed points must

be defined to surround an object

– Points will ‘snap’ to nearest edge

Cooling

• Problem– All seeds must be manually selected

• Complex objects may require many seed points

• Solution– Apply automatic seed point– As the user wraps the object, a common path

is formed• Make common path ‘cool’ into a new seed point

Cooling (cont.)

• Examples– Manual seed points (bottom left)– Auto seed points via cooling (bottom right)

Interactive Dynamic Training

• Problem– Some objects have stronger edges then others

• If the desired edge is weaker than a nearby edge, then the path ‘jumps’ over to the stronger edge

• Solution– Train the gradient magnitude to desire the weaker

edge• Use a sample of good path to train gradient magnitude• Update sample as path moves along the desired edge

– Allow user to enable and disable training as needed

Interactive Dynamic Training

• Examples– Path segment jumps

without training (top)– Path segment follows

trained edge (middle)

• Cost fG

– Normal response without training (lower left)

– Trained response from edge sample (lower right)

Image Composition

• Need– Source objects need blend in with a new background– Background may need to be in front of objects

• Solution– Allow for 2-D transformations to occur on source

objects– Use low pass filters to blend the object into the

destination’s scene– Mask background objects to appear in front of source

object

Image Composition (cont.)

Critique

• Paper– Describes a tool

• Selects image objects quickly and easily• Provides the means manipulate and paste them into different

images

• Abstract– Brief mention of need– List of abilities for a tool called ‘Intelligent Scissors’

• Introduction – Defines need– Claims current methods are not enough– Claims this tool will help the problem– Gives a small background on similar segmentation tools and

their flaws

Critique (cont.)

• Algorithms– The paper does a good job on explaining how

dynamic programming is used– ‘Cooling’ was explained well, but no

suggested times were given– The section on ‘Dynamic Training’ could be

explained more to better understand it– Spatial Frequency and Contrast Matching

needs more explanation

Critique (cont.)

• Dynamic Programming– Used as the main driving force of this tool– The authors spend a lot of time on the

dynamic programming section but not gratuitously

– Cost must be correctly attributed to the different edge features to take advantage of dynamic programming

– Optimal path is ‘Optimal’, not just a local answer

Questions?