Automatic Texture Guided Color Transfer and Colorization...Automatic Texture Guided Color Transfer...
Transcript of Automatic Texture Guided Color Transfer and Colorization...Automatic Texture Guided Color Transfer...
Automatic Texture Guided Color Transfer and Colorization
B. Arbelot1, R. Vergne1, T. Hurtut2 & J. Thollot1
1 Inria-LJK (UGA, CNRS), France2 Polytechnique Montréal, Canada
2
Color Transfer
Input & Reference Our Result
3
Colorization
Input & Reference Our Result
4
Applications
● Photography / Cinema – Colors = mood, atmosphere, emotion
© Sweeney Todd
© The Matrix © Le Fabuleux Destin d'Amelie Poulain
© The Grand Budapest Hotel
5
Applications
● Visualization / Imaging– Colors = information
© livescience.com© H. Zaidi
6
Previous Work
● Global histogram transfers[Reinhard01], [Xiao06], [Pitié07]
[Reinhard01] Input Reference Result
7
Previous Work
● Global histogram transfers[Reinhard01], [Xiao06], [Pitié07]
++ Simple, fast
++ Work well on similar images
[Reinhard01] Input Reference Result
8
Previous Work
● Global histogram transfers[Reinhard01], [Xiao06], [Pitié07]
- - Unreliable
[Xiao 2006] Input Reference Result
9
Previous Work
● Global histogram transfers[Reinhard01], [Xiao06], [Pitié07]
- - Unreliable
- - Can create new colors
Input Reference Result [Xiao 2006]
10
Previous Work
● Local Transfers
Images from [Hristova15]
11
Previous Work
● Local Transfers[Tai05], [Bonneel11], [Hristova15]
++ More accurate than global transfers
[Hristova15] Input Reference Result
12
Previous Work
● Local Transfers[Tai05], [Bonneel11], [Hristova15]
++ More accurate than global transfers
- - Fail to distinguish between textured regions
[Hristova15] Input Reference Result
13
Colorization
● No source color histogram
14
Colorization
● No source color histogram– Transfer is impossible
– Chrominance need to be added
15
Colorization
● Optimization based stroke diffusion [Levin04]
Input + user strokes Result
16
Colorization
● Exemplar based approaches– Use a reference image [Welsh02], [Charpiat08],
[Gupta12], [Bugeau14]
Input Reference Result
17
Colorization
● Exemplar based approaches
Images from [Gupta12]
Reference
Input
18
Colorization
● Exemplar based approaches
Images from [Gupta12]
Features extraction
Reference
Input
19
Colorization
● Exemplar based approaches
Images from [Gupta12]
Features extraction
Feature matching Reference
Input
20
Colorization
● Exemplar based approaches
Images from [Gupta12]
Features extraction
Feature matching
Result
Reference
Input
21
Colorization
● Exemplar based approaches
++ Very few user input
22
Colorization
● Exemplar based approaches
++ Very few user input
- - Color coherency is hard to preserve
23
Our Approach
24
Our Approach
25
Our Approach
26
Our Approach
27
Descriptors
28
Descriptors
● How to describe a texture ?
29
Descriptors
● How to describe a texture ?
Through the 1st and 2nd order luminance variations [Julesz70-80]
30
Descriptors
● How to describe a texture ?
Through the 1st and 2nd order luminance variations [Julesz70-80] – We chose the covariance matrices introduced by
[Tuzel06] as 2nd order descriptors
31
Descriptors
● How to describe a texture ?
Through the 1st and 2nd order luminance variations [Julesz70-80] – We chose the covariance matrices introduced by
[Tuzel06] as 2nd order descriptors
– Enriched by 1st order means similar to [Karacan13]
32
Descriptors
● How to describe a texture ?
Through the 1st and 2nd order luminance variations [Julesz70-80] – We chose the covariance matrices introduced by
[Tuzel06] as 2nd order descriptors
– Enriched by 1st order means similar to [Karacan13]
– Computed on the luminance and its derivatives
33
Descriptors
● Our feature vector
34
Descriptors
● Our feature vector
Luminance
35
Descriptors
● Our feature vector
First derivatives of theluminance
36
Descriptors
● Our feature vector
Second derivativesof the luminance
37
Descriptors
● Our descriptor
S(p) =Covariance matrix
Information(6x6)
Second orderInformation
Meanof
features(6x1)
First orderInformation
38
Descriptors
● Descriptor for each pixel
39
Descriptors
● Descriptor for each pixel
40
Descriptors
● Descriptor for each pixel
41
Descriptors
● Descriptor for each pixel
42
Descriptors
● Descriptor for each pixel
43
Descriptors
● Descriptor for each pixel
44
Descriptors
● Descriptor for each pixel
45
Descriptors
● Descriptor for each pixel
Descriptors
● Descriptor for each pixel
47
Descriptors
● Descriptor for each pixel
48
Descriptors
● Descriptor for each pixel
49
Descriptors
● Descriptor for each pixel
50
Descriptors
● Descriptor for each pixel
51
Descriptors
● Descriptor window size
Input window 13x13 window 43x43(512x512)
52
Descriptors
● We want homogeneous descriptors over textured regions
53
Descriptors
● We want homogeneous descriptors over textured regions➔ Compute them with a large window
54
Descriptors
● We want homogeneous descriptors over textured regions➔ Compute them with a large window
● We want to preserve image structure and texture transitions
55
Descriptors
● We want homogeneous descriptors over textured regions➔ Compute them with a large window
● We want to preserve image structure and texture transitions➔ Adapt descriptors around those transitions
56
Descriptors
● We want homogeneous descriptors over textured regions➔ Compute them with a large window
● We want to preserve image structure and texture transitions➔ Adapt descriptors around those transitions➔ The descriptors variance allows to detect those
transition areas !
57
Descriptors
● Gradient descent guided by the descriptors variance
58
Descriptors
● Gradient descent guided by the descriptors variance
Input
59
Descriptors
● Gradient descent guided by the descriptors variance
Input Descriptors
60
Descriptors
● Gradient descent guided by the descriptors variance
Input Descriptors Variance
61
Descriptors
● Gradient descent guided by the descriptors variance
Input Descriptors Variance
62
Descriptors
● Gradient descent guided by the descriptors variance
Input Descriptors Variance
63
Descriptors
● Gradient descent guided by the descriptors variance
Input Descriptors Variance
64
Descriptors
● Gradient descent guided by the descriptors variance
Input Descriptors Variance
65
Descriptors
● Gradient descent guided by the descriptors variance
Input Descriptors Variance Descriptors aftergradient descent
66
Descriptors
● Multiscale gradient descent
Descriptors
67
Descriptors
● Multiscale gradient descent
Descriptors Standard gradientdescent
68
Descriptors
● Multiscale gradient descent
Descriptors Standard gradientdescent
Multiscale gradient descent
69
Descriptors
● Multiscale gradient descent
Descriptors Standard gradientdescent
Multiscale gradient descent
!!!
70
Descriptors
● Unnormalized bilateral filter [Aubry14]
71
Descriptors
● Unnormalized bilateral filter [Aubry14]– Guided by the input image luminance
72
Descriptors
● Unnormalized bilateral filter [Aubry14]– Guided by the input image luminance
– Little to no halos, no gradient reversal
73
Descriptors
● Unnormalized bilateral filter [Aubry14]– Guided by the input image luminance
– Little to no halos, no gradient reversal
– Similar to an anisotropic diffusion
74
Descriptors
● Unnormalized bilateral filter [Aubry14]
Input Descriptors aftergradient descent
Descriptors aftergradient descent
and filtering
75
Similarity Maps
76
Similarity Maps
● Similarity computed as the L2 Euclidean distance between two descriptors, filtered by a Gaussian
77
Similarity Maps
● Similarity computed as the L2 Euclidean distance between two descriptors, filtered by a Gaussian
78
Similarity Maps
● Similarity computed as the L2 Euclidean distance between two descriptors, filtered by a Gaussian
Control parameter
79
Similarity Maps
80
Similarity Maps
81
Similarity Maps
82
Color Transfer
83
Color Transfer
● Global transfer from [Reinhard01]
84
Color Transfer
● Global transfer from [Reinhard01]– Transfer the mean and standard deviation of each
channel of the reference to the input
85
Color Transfer
● Global transfer from [Reinhard01]– Transfer the mean and standard deviation of each
channel of the reference to the input
– Translate and stretch the input histogram
86
Color Transfer
● Global transfer from [Reinhard01]– Transfer the mean and standard deviation of each
channel of the reference to the input
– Translate and stretch the input histogram
● We use weighted means and standard deviations using our similarity maps
87
Color Transfer
● Global transfer from [Reinhard01]– Transfer the mean and standard deviation of each
channel of the reference to the input
– Translate and stretch the input histogram
● We use weighted means and standard deviations using our similarity maps➔ Perform a local transfer between similar pixels
88
Color Transfer
● Our Result
Input & Reference Result with σd= 1
89
Color Transfer
● Effect of σd
Input & Reference Result with σd= 2
90
Color Transfer
● Effect of σd
Input & Reference Result with σd= 4
91
Color Transfer
● Effect of σd
= [Reinhard01]
92
Color Transfer
Input Reference Our Result
93
Transfer - Comparisons
Input Reference
Transfer - Comparisons
Input Reference [Reinhard01] [Xiao06]
[Pitié07] [Pouli11] [Hristova15] Our Result
Transfer - Comparisons
Input Reference [Reinhard01] [Xiao06]
[Pitié07] [Pouli11] [Hristova15] Our Result
Transfer - Comparisons
Input Reference [Reinhard01] [Xiao06]
[Pitié07] [Pouli11] [Hristova15] Our Result
Transfer - Comparisons
Input Reference [Reinhard01] [Xiao06]
[Pitié07] [Pouli11] [Hristova15] Our Result
98
Colorization
99
Colorization
● We cannot transfer missing colors
100
Colorization
● We cannot transfer missing colors● Each output pixel is given the mean
chromaticity of the reference
101
Colorization
● We cannot transfer missing colors● Each output pixel is given the mean
chromaticity of the reference, weighted by the similarity maps
102
Colorization
● Result
Input & Reference Result with σd= 0.2
103
Colorization
● Effect of σd
Input & Reference Result with σd= 1
104
Colorization
● Effect of σd
Input & Reference Result with σd= 2
105
Colorization
Input Reference Our Result
106
Colorization - Comparisons
Input Reference
Colorization - Comparisons
Input Reference [Welsh02] [Charpiat08]
[Gupta12] [Bugeau14] [Pierre15] Our Result
Colorization - Comparisons
Input Reference [Welsh02] [Charpiat08]
[Gupta12] [Bugeau14] [Pierre15] Our Result
Colorization - Comparisons
Input Reference [Welsh02] [Charpiat08]
[Gupta12] [Bugeau14] [Pierre15] Our Result
Colorization - Comparisons
Input Reference [Welsh02] [Charpiat08]
[Gupta12] [Bugeau14] [Pierre15] Our Result
Colorization - Comparisons
Input Reference [Welsh02] [Charpiat08]
[Gupta12] [Bugeau14] [Pierre15] Our Result
112
Limitations
● Texture based approach
113
Limitations
● Texture based approach– Not well suited for texture-less images
● Similar to applying a global transfer
114
Limitations
● Texture based approach– Not well suited for texture-less images
– Fine structures can be mistaken as textures
115
Limitations
● Texture based approach– Not well suited for texture-less images
– Fine structures can be mistaken as textures
Input Reference Our Result
116
Limitations
● Texture based approach– Not well suited for texture-less images
– Fine structures can be mistaken as textures
Input Reference Our Result
117
Future Works
118
Future Works
● Edge preserving texture descriptors
119
Future Works
● Edge preserving texture descriptors– Can be used in a lot of different applications
120
Future Works
● Edge preserving texture descriptors– Can be used in a lot of different applications
● Transfer and colorization
121
Future Works
● Edge preserving texture descriptors– Can be used in a lot of different applications
● Transfer and colorization– Several references could be used
122
Future Works
● Edge preserving texture descriptors– Can be used in a lot of different applications
● Transfer and colorization– Several references could be used
– Could be extended to video inputs
123
Thank you for your attention
124
Descriptors
● Effect of each step on a transfer result
Initial descriptors Only grad. descent Only filtering Both