. . . . . .

. .Introduction

. . . . . .Formulation Experimental Results References

.

......Epitomic Image Colorization

Yingzhen Yang1, Xinqi Chu1, Tian Tsong Ng2, Alex Yong-SangChia2, Jianchao Yang3, Hailin Jin3, Thomas S. Huang1

1Department of Electrical and Computer Engineering, University of Illinois atUrbana-Champaign

2 Institute for Infocomm Research, A*STAR, Singapore 1386323 Adobe Research, San Jose, CA 95110, USA

1 / 22

. . . . . .

. .Introduction

. . . . . .Formulation Experimental Results References

Outline

...1 IntroductionRelated WorkContribution

...2 FormulationEpitomeRobust Patch Dissimilarity Measure via Epitome

...3 Experimental Results

2 / 22

. . . . . .

. .Introduction

. . . . . .Formulation Experimental Results References

Image Colorization

A process of adding color to grayscale images

Increasing the visual appeal of imagesInformation illustration in scientific images

Manual colorization is time consuming and tedious.

We focus on automatic image colorization that transfers colorfrom the reference image to the grayscale target image.

3 / 22

. . . . . .

. .Introduction

. . . . . .Formulation Experimental Results References

Image Colorization

Figure 1: Colorize the Nano Mushroom-like structure by our method.From left to right: the reference image, the target image, the colorizedtarget image.

4 / 22

. . . . . .

. .Introduction

. . . . . .Formulation Experimental Results References

Related Work

Transferring Color to Greyscale Images (Welsh et al., 2002)

Pixel-level matching by luminance value and neighborhoodstatisticsSuffers from spatial inconsistency

Image Colorization Using Similar Images (Gupta et al., 2012)

A cascade feature matching scheme for matching the targetsuperpixels to the reference superpixelsLacks robust to change in pose or orientation

5 / 22

. . . . . .

. .Introduction

. . . . . .Formulation Experimental Results References

Contribution

We propose a new automatic image colorization method byepitome, called Epitomic Image Colorization

Achieve feature matching robust to both noise and the largechange in the pose or orientation of the objectsEpitome is a generative model which summarizes raw imagepatches into a condensed representation.

A new robust patch dissimilarity measure by epitome and theMRF inference.

6 / 22

. . . . . .

. .Introduction

. . . . . .Formulation Experimental Results References

Introduction to Epitome

Epitome (Jojic, Frey, & Kannan, 2003) is a generative modelwhich summarizes raw image patches into a condensedrepresentation similar to Gaussian Mixture Models (GMMs).

In contrast to tradition GMMs, the Gaussian components ofepitome can be overlapping with each other.

Figure 2: Examples of the learned epitome

7 / 22

. . . . . .

. .Introduction

. . . . . .Formulation Experimental Results References

Introduction to Epitome

The epitome e is obtained by maximizing the log likelihoodfunction:

e = argmaxe

log p({Zk}Qk=1|e

), (1)

Epitome

Reference Image

eN

eM

M

N

......

cI

ekZ

jZkT

jT

QZ

QT

keT

Figure 3: Learn the epitome from the reference image. Zk : patch fromthe reference image; Tk: hidden mappings that maps the image patch Zk

to the epitome patch.

8 / 22

. . . . . .

. .Introduction

. . . . . .Formulation Experimental Results References

Heterogeneous Feature Epitome

We learn the pixel epitome eY IQ, the dense SIFT epitomeeSIFT and the LBP epitome eLBP jointly from the the rawpixel, the dense SIFT feature (Lazebnik, Schmid, & Ponce,2006) and the rotation invariant Local Binary Pattern (LBP)(Ojala, Pietikainen, & Maenpaa, 2002) of the reference image.

The heterogeneous feature epitome e = (eY IQ, eSIFT , eLBP )

9 / 22

. . . . . .

. .Introduction

. . . . . .Formulation Experimental Results References

Robust Patch Dissimilarity Measure via Epitome

In order to match the target patch to the reference patch forcolor transfer, we need a robust patch dissimilarity measure.

We propose a robust dissimilarity measure between the targetpatch Zi and the reference patch Zj with the heterogeneousfeature epitome e learned from the reference image:

De

(Zi,Zj

)= 1− p(T ∗

i |Zj , e) (2)

where T ∗i is the most probable hidden mapping for Zi:

T ∗i = argmax

Ti

p(Ti|Zi, e

)(3)

10 / 22

. . . . . .

. .Introduction

. . . . . .Formulation Experimental Results References

Robust Patch Dissimilarity Measure via Epitome

This dissimilarity measure is robust to noise and the largechange in the pose or orientation of the objects.

Figure 4: Colorize the cheetah

11 / 22

. . . . . .

. .Introduction

. . . . . .Formulation Experimental Results References

Epitomic Image Colorization

Use the robust patch dissimilarity measure via epitome to findsimilar reference patches for each target patchTransfer color from the similar reference patch to the targetpatchUse MRF inference to obtain a smooth colorization result

Figure 5: Comparison between colorizing the Nano image with MRFinference (left) or not (right).

12 / 22

. . . . . .

. .Introduction

. . . . . .Formulation Experimental Results References

Parameter Setting

The area of the heterogeneous feature epitome is no morethan 1

4 of that of the reference images.

The patch size is 9× 9 or 12× 12.

13 / 22

. . . . . .

. .Introduction

. . . . . .Formulation Experimental Results References

Colorization Results

learned epitome reference image target image

Welsh et al. Gupta et al. our result.

14 / 22

. . . . . .

. .Introduction

. . . . . .Formulation Experimental Results References

Colorization Results Cont.

learned epitome reference image target image

Welsh et al. Gupta et al. our result.

15 / 22

. . . . . .

. .Introduction

. . . . . .Formulation Experimental Results References

Colorization Results Cont.

learned epitome reference image target image

Welsh et al. Gupta et al. our result.

16 / 22

. . . . . .

. .Introduction

. . . . . .Formulation Experimental Results References

Colorization Results Cont.

learned epitome reference image target image

Welsh et al. Gupta et al. our result.

17 / 22

. . . . . .

. .Introduction

. . . . . .Formulation Experimental Results References

Colorization Results Cont.

learned epitome reference image target image

Welsh et al. Gupta et al. our result.

18 / 22

. . . . . .

. .Introduction

. . . . . .Formulation Experimental Results References

Colorization Results Cont.

19 / 22

. . . . . .

. .Introduction

. . . . . .Formulation Experimental Results References

End

Thank you!

20 / 22

. . . . . .

. .Introduction

. . . . . .Formulation Experimental Results References

Reference I

Gupta, R. K., Chia, A. Y. S., Rajan, D., Ng, E. S., & Huang, Z.(2012). Image colorization using similar images. In Acmmultimedia (p. 369-378).

Jojic, N., Frey, B. J., & Kannan, A. (2003). Epitomic analysis ofappearance and shape. In Iccv (p. 34-43).

Lazebnik, S., Schmid, C., & Ponce, J. (2006). Beyond bags offeatures: Spatial pyramid matching for recognizing naturalscene categories. In Computer vision and patternrecognition, 2006 ieee computer society conference on(Vol. 2, p. 2169 - 2178).

21 / 22

. . . . . .

. .Introduction

. . . . . .Formulation Experimental Results References

Reference II

Ojala, T., Pietikainen, M., & Maenpaa, T. (2002, jul).Multiresolution gray-scale and rotation invariant textureclassification with local binary patterns. Pattern Analysis andMachine Intelligence, IEEE Transactions on, 24(7), 971 -987.

Welsh, T., Ashikhmin, M., & Mueller, K. (2002). Transferringcolor to greyscale images. ACM Trans. Graph., 21(3),277-280.

22 / 22