Epitomic Image Colorization - Yingzhen...
Transcript of Epitomic Image Colorization - Yingzhen...
. . . . . .
. .Introduction
. . . . . .Formulation Experimental Results References
.
......Epitomic Image Colorization
Yingzhen Yang1, Xinqi Chu1, Tian Tsong Ng2, Alex Yong-SangChia2, Jianchao Yang3, Hailin Jin3, Thomas S. Huang1
1Department of Electrical and Computer Engineering, University of Illinois atUrbana-Champaign
2 Institute for Infocomm Research, A*STAR, Singapore 1386323 Adobe Research, San Jose, CA 95110, USA
1 / 22
. . . . . .
. .Introduction
. . . . . .Formulation Experimental Results References
Outline
...1 IntroductionRelated WorkContribution
...2 FormulationEpitomeRobust Patch Dissimilarity Measure via Epitome
...3 Experimental Results
2 / 22
. . . . . .
. .Introduction
. . . . . .Formulation Experimental Results References
Image Colorization
A process of adding color to grayscale images
Increasing the visual appeal of imagesInformation illustration in scientific images
Manual colorization is time consuming and tedious.
We focus on automatic image colorization that transfers colorfrom the reference image to the grayscale target image.
3 / 22
. . . . . .
. .Introduction
. . . . . .Formulation Experimental Results References
Image Colorization
Figure 1: Colorize the Nano Mushroom-like structure by our method.From left to right: the reference image, the target image, the colorizedtarget image.
4 / 22
. . . . . .
. .Introduction
. . . . . .Formulation Experimental Results References
Related Work
Transferring Color to Greyscale Images (Welsh et al., 2002)
Pixel-level matching by luminance value and neighborhoodstatisticsSuffers from spatial inconsistency
Image Colorization Using Similar Images (Gupta et al., 2012)
A cascade feature matching scheme for matching the targetsuperpixels to the reference superpixelsLacks robust to change in pose or orientation
5 / 22
. . . . . .
. .Introduction
. . . . . .Formulation Experimental Results References
Contribution
We propose a new automatic image colorization method byepitome, called Epitomic Image Colorization
Achieve feature matching robust to both noise and the largechange in the pose or orientation of the objectsEpitome is a generative model which summarizes raw imagepatches into a condensed representation.
A new robust patch dissimilarity measure by epitome and theMRF inference.
6 / 22
. . . . . .
. .Introduction
. . . . . .Formulation Experimental Results References
Introduction to Epitome
Epitome (Jojic, Frey, & Kannan, 2003) is a generative modelwhich summarizes raw image patches into a condensedrepresentation similar to Gaussian Mixture Models (GMMs).
In contrast to tradition GMMs, the Gaussian components ofepitome can be overlapping with each other.
Figure 2: Examples of the learned epitome
7 / 22
. . . . . .
. .Introduction
. . . . . .Formulation Experimental Results References
Introduction to Epitome
The epitome e is obtained by maximizing the log likelihoodfunction:
e = argmaxe
log p({Zk}Qk=1|e
), (1)
Epitome
Reference Image
eN
eM
M
N
......
cI
ekZ
jZkT
jT
QZ
QT
keT
Figure 3: Learn the epitome from the reference image. Zk : patch fromthe reference image; Tk: hidden mappings that maps the image patch Zk
to the epitome patch.
8 / 22
. . . . . .
. .Introduction
. . . . . .Formulation Experimental Results References
Heterogeneous Feature Epitome
We learn the pixel epitome eY IQ, the dense SIFT epitomeeSIFT and the LBP epitome eLBP jointly from the the rawpixel, the dense SIFT feature (Lazebnik, Schmid, & Ponce,2006) and the rotation invariant Local Binary Pattern (LBP)(Ojala, Pietikainen, & Maenpaa, 2002) of the reference image.
The heterogeneous feature epitome e = (eY IQ, eSIFT , eLBP )
9 / 22
. . . . . .
. .Introduction
. . . . . .Formulation Experimental Results References
Robust Patch Dissimilarity Measure via Epitome
In order to match the target patch to the reference patch forcolor transfer, we need a robust patch dissimilarity measure.
We propose a robust dissimilarity measure between the targetpatch Zi and the reference patch Zj with the heterogeneousfeature epitome e learned from the reference image:
De
(Zi,Zj
)= 1− p(T ∗
i |Zj , e) (2)
where T ∗i is the most probable hidden mapping for Zi:
T ∗i = argmax
Ti
p(Ti|Zi, e
)(3)
10 / 22
. . . . . .
. .Introduction
. . . . . .Formulation Experimental Results References
Robust Patch Dissimilarity Measure via Epitome
This dissimilarity measure is robust to noise and the largechange in the pose or orientation of the objects.
Figure 4: Colorize the cheetah
11 / 22
. . . . . .
. .Introduction
. . . . . .Formulation Experimental Results References
Epitomic Image Colorization
Use the robust patch dissimilarity measure via epitome to findsimilar reference patches for each target patchTransfer color from the similar reference patch to the targetpatchUse MRF inference to obtain a smooth colorization result
Figure 5: Comparison between colorizing the Nano image with MRFinference (left) or not (right).
12 / 22
. . . . . .
. .Introduction
. . . . . .Formulation Experimental Results References
Parameter Setting
The area of the heterogeneous feature epitome is no morethan 1
4 of that of the reference images.
The patch size is 9× 9 or 12× 12.
13 / 22
. . . . . .
. .Introduction
. . . . . .Formulation Experimental Results References
Colorization Results
learned epitome reference image target image
Welsh et al. Gupta et al. our result.
14 / 22
. . . . . .
. .Introduction
. . . . . .Formulation Experimental Results References
Colorization Results Cont.
learned epitome reference image target image
Welsh et al. Gupta et al. our result.
15 / 22
. . . . . .
. .Introduction
. . . . . .Formulation Experimental Results References
Colorization Results Cont.
learned epitome reference image target image
Welsh et al. Gupta et al. our result.
16 / 22
. . . . . .
. .Introduction
. . . . . .Formulation Experimental Results References
Colorization Results Cont.
learned epitome reference image target image
Welsh et al. Gupta et al. our result.
17 / 22
. . . . . .
. .Introduction
. . . . . .Formulation Experimental Results References
Colorization Results Cont.
learned epitome reference image target image
Welsh et al. Gupta et al. our result.
18 / 22
. . . . . .
. .Introduction
. . . . . .Formulation Experimental Results References
Colorization Results Cont.
19 / 22
. . . . . .
. .Introduction
. . . . . .Formulation Experimental Results References
End
Thank you!
20 / 22
. . . . . .
. .Introduction
. . . . . .Formulation Experimental Results References
Reference I
Gupta, R. K., Chia, A. Y. S., Rajan, D., Ng, E. S., & Huang, Z.(2012). Image colorization using similar images. In Acmmultimedia (p. 369-378).
Jojic, N., Frey, B. J., & Kannan, A. (2003). Epitomic analysis ofappearance and shape. In Iccv (p. 34-43).
Lazebnik, S., Schmid, C., & Ponce, J. (2006). Beyond bags offeatures: Spatial pyramid matching for recognizing naturalscene categories. In Computer vision and patternrecognition, 2006 ieee computer society conference on(Vol. 2, p. 2169 - 2178).
21 / 22
. . . . . .
. .Introduction
. . . . . .Formulation Experimental Results References
Reference II
Ojala, T., Pietikainen, M., & Maenpaa, T. (2002, jul).Multiresolution gray-scale and rotation invariant textureclassification with local binary patterns. Pattern Analysis andMachine Intelligence, IEEE Transactions on, 24(7), 971 -987.
Welsh, T., Ashikhmin, M., & Mueller, K. (2002). Transferringcolor to greyscale images. ACM Trans. Graph., 21(3),277-280.
22 / 22