Anat Levin Dani Lischinski Yair Weiss

School of Computer Science and EngineeringThe Hebrew University of Jerusalem, Israel

Colorization Using Optimization

1Tuesday, June 19, 2007

Colorization: a computer-assisted process of adding color to a monochrome image or movie.

(Invented by Wilson Markle, 1970)

Colorization

2Tuesday, June 19, 2007

• Colorizing black and white movies?

• Maybe not...

• But what about black and white TV shows?

Motivation

3Tuesday, June 19, 2007

• Colorizing black and white movies / TV shows

• Recoloring color images for special effects

Motivation

4Tuesday, June 19, 2007

5Tuesday, June 19, 2007

Color 101• Typically represented by points in a three

dimensional color space:• (Red, Green, Blue)• (Cyan, Magenta, Yellow)

• Some color spaces separate between the grayscale and the chromatic components of a color:

6Tuesday, June 19, 2007

Typical Colorization Process

Images from: “Yet Another Colorization Tutorial”

http://www.worth1000.com/tutorial.asp?sid=161018

7Tuesday, June 19, 2007

Images from: “Yet Another Colorization Tutorial”

http://www.worth1000.com/tutorial.asp?sid=161018

Typical Colorization Process• Delineate region boundary

8Tuesday, June 19, 2007

Images from: “Yet Another Colorization Tutorial”

http://www.worth1000.com/tutorial.asp?sid=161018

Typical Colorization Process• Delineate region boundary• Choose region color from palette

9Tuesday, June 19, 2007

Images from: “Yet Another Colorization Tutorial”

http://www.worth1000.com/tutorial.asp?sid=161018

Typical Colorization Process• Delineate region boundary• Choose region color from palette

10Tuesday, June 19, 2007

Images from: “Yet Another Colorization Tutorial”

http://www.worth1000.com/tutorial.asp?sid=161018

Typical Colorization Process• Delineate region boundary• Choose region color from palette

11Tuesday, June 19, 2007

Video Colorization Process• Delineate region boundary• Choose region color from palette

• Track regions across video frames

12Tuesday, June 19, 2007

Process Limitations

• Time consuming and labor intensive

• Difficult to delineate detailed or fuzzy boundaries

• Failures in tracking

13Tuesday, June 19, 2007

Hertzmann et al. 2001, Welsh et al. 2002

A : A’

B : B’

?

Colorization by Analogy

14Tuesday, June 19, 2007

Colorization by Analogy - Discussion

• Indirect artistic control

• No spatial continuity constraint

15Tuesday, June 19, 2007

Our Approach

16Tuesday, June 19, 2007

Artist scribbles desired colors inside regions

Our Approach

17Tuesday, June 19, 2007

Colors are propagated to all pixels

“Nearby pixels with similar intensities should have the same color”

Our Approach

18Tuesday, June 19, 2007

Intensity channel Color channels

Assumption: Neighboring pixels with similar intensities should have similar colors

Propagation using Optimization

19Tuesday, June 19, 2007

Intensity channel Color channels

Propagation using Optimization

Minimize the sum of affinity-weighed squared distances between the colors of adjacent pixels:

J(U) = !r,s

wrs (U(r)!U(s))2

20Tuesday, June 19, 2007

Affinity Functions

proportional to local variancewrs ! e!(Y (r)!Y (s))2/2!2

r

21Tuesday, June 19, 2007

Space-time Affinity

wrs ! e!(Y (r)!Y (s))2/2!2r

22Tuesday, June 19, 2007

Since the cost function is quadratic, its minimum may be found by solving sparse system of linear equations.

Minimize the cost function:

Subject to labeling constraints

Optimization

J(U) = !r,s

wrs (U(r)!U(s))2

!U(rk) = Uk

"

23Tuesday, June 19, 2007

Optimization

Minimize by solving the equations:

!J(U) = 0!J(V ) = 0

U! = argminU

J(U)

V ! = argminV

J(V )

24Tuesday, June 19, 2007

DerivationJ(U) = !

r,swrs (Ur!Us)2

= !r,s

wrs!U2

r !2UrUs +U2s"

= !r,s

wrsU2r !2!

r,swrsUrUs +!

r,swrsU2

s

= !r

DrrU2r !2!

r,swrsUrUs +!

sDssU2

s

= 2UT (D!W )U

25Tuesday, June 19, 2007

Derivation

J(U) = 2UT (D!W )U

Unconstrained system:

(D!W )U = 0

Note: has a trivial solution (any constant vector U).

!J(U) = 4(D!W )U = 0

26Tuesday, June 19, 2007

DerivationAdd constraints:

J(U) = UT (D!W )U +! !k

!Uk!Uk

"2

J(U) = UT (D!W )U +!!U!U

"T C!U!U

"

!J(U) = 2(D!W )U +2!C!U!U

"= 0

Constrained system:

(D!W +!C)U = !CU

27Tuesday, June 19, 2007

Color Interpolation

28Tuesday, June 19, 2007

Colorizing Stills

29Tuesday, June 19, 2007

30Tuesday, June 19, 2007

Original Colorized

Colorizing Stills

31Tuesday, June 19, 2007

Colorizing Stills

32Tuesday, June 19, 2007

Colorizing Stills

33Tuesday, June 19, 2007

Challenging Case

34Tuesday, June 19, 2007

Our result

Segmentation aided colorization

NCuts Segmentation (Shi & Malik 97)

Segmentation?

35Tuesday, June 19, 2007

Affinity between pixels – based on intensity AND color similarities.

Recoloring

36Tuesday, June 19, 2007

Recoloring

37Tuesday, June 19, 2007

Recoloring

38Tuesday, June 19, 2007

13 out of 92 frames

Colorizing Video

39Tuesday, June 19, 2007

16 out of 101 frames

Colorizing Video

40Tuesday, June 19, 2007

Red channel<->matte

Matting

41Tuesday, June 19, 2007

Matting

42Tuesday, June 19, 2007

Future Work

• Import image segmentation advantages: affinity functions, optimization techniques.

• Alternative color spaces, propagating hue and saturation differently

43Tuesday, June 19, 2007

★ Interface: User scribbles color on a small number of pixels

★ Colors propagate in space-time volume respecting intensity boundaries

★ Convincing colorization with a small amount of user effort

Code & examples available: www.cs.huji.ac.i/~yweiss/Colorization/

Summary

44Tuesday, June 19, 2007