Post on 19-Mar-2020
Automatic Texture Guided Color Transfer and Colorization
B. Arbelot1, R. Vergne1, T. Hurtut2 & J. Thollot1
1 Inria-LJK (UGA, CNRS), France2 Polytechnique Montréal, Canada
2
Color Transfer
Input & Reference Our Result
3
Colorization
Input & Reference Our Result
4
Applications
● Photography / Cinema – Colors = mood, atmosphere, emotion
© Sweeney Todd
© The Matrix © Le Fabuleux Destin d'Amelie Poulain
© The Grand Budapest Hotel
5
Applications
● Visualization / Imaging– Colors = information
© livescience.com© H. Zaidi
6
Previous Work
● Global histogram transfers[Reinhard01], [Xiao06], [Pitié07]
[Reinhard01] Input Reference Result
7
Previous Work
● Global histogram transfers[Reinhard01], [Xiao06], [Pitié07]
++ Simple, fast
++ Work well on similar images
[Reinhard01] Input Reference Result
8
Previous Work
● Global histogram transfers[Reinhard01], [Xiao06], [Pitié07]
- - Unreliable
[Xiao 2006] Input Reference Result
9
Previous Work
● Global histogram transfers[Reinhard01], [Xiao06], [Pitié07]
- - Unreliable
- - Can create new colors
Input Reference Result [Xiao 2006]
10
Previous Work
● Local Transfers
Images from [Hristova15]
11
Previous Work
● Local Transfers[Tai05], [Bonneel11], [Hristova15]
++ More accurate than global transfers
[Hristova15] Input Reference Result
12
Previous Work
● Local Transfers[Tai05], [Bonneel11], [Hristova15]
++ More accurate than global transfers
- - Fail to distinguish between textured regions
[Hristova15] Input Reference Result
13
Colorization
● No source color histogram
14
Colorization
● No source color histogram– Transfer is impossible
– Chrominance need to be added
15
Colorization
● Optimization based stroke diffusion [Levin04]
Input + user strokes Result
16
Colorization
● Exemplar based approaches– Use a reference image [Welsh02], [Charpiat08],
[Gupta12], [Bugeau14]
Input Reference Result
17
Colorization
● Exemplar based approaches
Images from [Gupta12]
Reference
Input
18
Colorization
● Exemplar based approaches
Images from [Gupta12]
Features extraction
Reference
Input
19
Colorization
● Exemplar based approaches
Images from [Gupta12]
Features extraction
Feature matching Reference
Input
20
Colorization
● Exemplar based approaches
Images from [Gupta12]
Features extraction
Feature matching
Result
Reference
Input
21
Colorization
● Exemplar based approaches
++ Very few user input
22
Colorization
● Exemplar based approaches
++ Very few user input
- - Color coherency is hard to preserve
23
Our Approach
24
Our Approach
25
Our Approach
26
Our Approach
27
Descriptors
28
Descriptors
● How to describe a texture ?
29
Descriptors
● How to describe a texture ?
Through the 1st and 2nd order luminance variations [Julesz70-80]
30
Descriptors
● How to describe a texture ?
Through the 1st and 2nd order luminance variations [Julesz70-80] – We chose the covariance matrices introduced by
[Tuzel06] as 2nd order descriptors
31
Descriptors
● How to describe a texture ?
Through the 1st and 2nd order luminance variations [Julesz70-80] – We chose the covariance matrices introduced by
[Tuzel06] as 2nd order descriptors
– Enriched by 1st order means similar to [Karacan13]
32
Descriptors
● How to describe a texture ?
Through the 1st and 2nd order luminance variations [Julesz70-80] – We chose the covariance matrices introduced by
[Tuzel06] as 2nd order descriptors
– Enriched by 1st order means similar to [Karacan13]
– Computed on the luminance and its derivatives
33
Descriptors
● Our feature vector
34
Descriptors
● Our feature vector
Luminance
35
Descriptors
● Our feature vector
First derivatives of theluminance
36
Descriptors
● Our feature vector
Second derivativesof the luminance
37
Descriptors
● Our descriptor
S(p) =Covariance matrix
Information(6x6)
Second orderInformation
Meanof
features(6x1)
First orderInformation
38
Descriptors
● Descriptor for each pixel
39
Descriptors
● Descriptor for each pixel
40
Descriptors
● Descriptor for each pixel
41
Descriptors
● Descriptor for each pixel
42
Descriptors
● Descriptor for each pixel
43
Descriptors
● Descriptor for each pixel
44
Descriptors
● Descriptor for each pixel
45
Descriptors
● Descriptor for each pixel
Descriptors
● Descriptor for each pixel
47
Descriptors
● Descriptor for each pixel
48
Descriptors
● Descriptor for each pixel
49
Descriptors
● Descriptor for each pixel
50
Descriptors
● Descriptor for each pixel
51
Descriptors
● Descriptor window size
Input window 13x13 window 43x43(512x512)
52
Descriptors
● We want homogeneous descriptors over textured regions
53
Descriptors
● We want homogeneous descriptors over textured regions➔ Compute them with a large window
54
Descriptors
● We want homogeneous descriptors over textured regions➔ Compute them with a large window
● We want to preserve image structure and texture transitions
55
Descriptors
● We want homogeneous descriptors over textured regions➔ Compute them with a large window
● We want to preserve image structure and texture transitions➔ Adapt descriptors around those transitions
56
Descriptors
● We want homogeneous descriptors over textured regions➔ Compute them with a large window
● We want to preserve image structure and texture transitions➔ Adapt descriptors around those transitions➔ The descriptors variance allows to detect those
transition areas !
57
Descriptors
● Gradient descent guided by the descriptors variance
58
Descriptors
● Gradient descent guided by the descriptors variance
Input
59
Descriptors
● Gradient descent guided by the descriptors variance
Input Descriptors
60
Descriptors
● Gradient descent guided by the descriptors variance
Input Descriptors Variance
61
Descriptors
● Gradient descent guided by the descriptors variance
Input Descriptors Variance
62
Descriptors
● Gradient descent guided by the descriptors variance
Input Descriptors Variance
63
Descriptors
● Gradient descent guided by the descriptors variance
Input Descriptors Variance
64
Descriptors
● Gradient descent guided by the descriptors variance
Input Descriptors Variance
65
Descriptors
● Gradient descent guided by the descriptors variance
Input Descriptors Variance Descriptors aftergradient descent
66
Descriptors
● Multiscale gradient descent
Descriptors
67
Descriptors
● Multiscale gradient descent
Descriptors Standard gradientdescent
68
Descriptors
● Multiscale gradient descent
Descriptors Standard gradientdescent
Multiscale gradient descent
69
Descriptors
● Multiscale gradient descent
Descriptors Standard gradientdescent
Multiscale gradient descent
!!!
70
Descriptors
● Unnormalized bilateral filter [Aubry14]
71
Descriptors
● Unnormalized bilateral filter [Aubry14]– Guided by the input image luminance
72
Descriptors
● Unnormalized bilateral filter [Aubry14]– Guided by the input image luminance
– Little to no halos, no gradient reversal
73
Descriptors
● Unnormalized bilateral filter [Aubry14]– Guided by the input image luminance
– Little to no halos, no gradient reversal
– Similar to an anisotropic diffusion
74
Descriptors
● Unnormalized bilateral filter [Aubry14]
Input Descriptors aftergradient descent
Descriptors aftergradient descent
and filtering
75
Similarity Maps
76
Similarity Maps
● Similarity computed as the L2 Euclidean distance between two descriptors, filtered by a Gaussian
77
Similarity Maps
● Similarity computed as the L2 Euclidean distance between two descriptors, filtered by a Gaussian
78
Similarity Maps
● Similarity computed as the L2 Euclidean distance between two descriptors, filtered by a Gaussian
Control parameter
79
Similarity Maps
80
Similarity Maps
81
Similarity Maps
82
Color Transfer
83
Color Transfer
● Global transfer from [Reinhard01]
84
Color Transfer
● Global transfer from [Reinhard01]– Transfer the mean and standard deviation of each
channel of the reference to the input
85
Color Transfer
● Global transfer from [Reinhard01]– Transfer the mean and standard deviation of each
channel of the reference to the input
– Translate and stretch the input histogram
86
Color Transfer
● Global transfer from [Reinhard01]– Transfer the mean and standard deviation of each
channel of the reference to the input
– Translate and stretch the input histogram
● We use weighted means and standard deviations using our similarity maps
87
Color Transfer
● Global transfer from [Reinhard01]– Transfer the mean and standard deviation of each
channel of the reference to the input
– Translate and stretch the input histogram
● We use weighted means and standard deviations using our similarity maps➔ Perform a local transfer between similar pixels
88
Color Transfer
● Our Result
Input & Reference Result with σd= 1
89
Color Transfer
● Effect of σd
Input & Reference Result with σd= 2
90
Color Transfer
● Effect of σd
Input & Reference Result with σd= 4
91
Color Transfer
● Effect of σd
= [Reinhard01]
92
Color Transfer
Input Reference Our Result
93
Transfer - Comparisons
Input Reference
Transfer - Comparisons
Input Reference [Reinhard01] [Xiao06]
[Pitié07] [Pouli11] [Hristova15] Our Result
Transfer - Comparisons
Input Reference [Reinhard01] [Xiao06]
[Pitié07] [Pouli11] [Hristova15] Our Result
Transfer - Comparisons
Input Reference [Reinhard01] [Xiao06]
[Pitié07] [Pouli11] [Hristova15] Our Result
Transfer - Comparisons
Input Reference [Reinhard01] [Xiao06]
[Pitié07] [Pouli11] [Hristova15] Our Result
98
Colorization
99
Colorization
● We cannot transfer missing colors
100
Colorization
● We cannot transfer missing colors● Each output pixel is given the mean
chromaticity of the reference
101
Colorization
● We cannot transfer missing colors● Each output pixel is given the mean
chromaticity of the reference, weighted by the similarity maps
102
Colorization
● Result
Input & Reference Result with σd= 0.2
103
Colorization
● Effect of σd
Input & Reference Result with σd= 1
104
Colorization
● Effect of σd
Input & Reference Result with σd= 2
105
Colorization
Input Reference Our Result
106
Colorization - Comparisons
Input Reference
Colorization - Comparisons
Input Reference [Welsh02] [Charpiat08]
[Gupta12] [Bugeau14] [Pierre15] Our Result
Colorization - Comparisons
Input Reference [Welsh02] [Charpiat08]
[Gupta12] [Bugeau14] [Pierre15] Our Result
Colorization - Comparisons
Input Reference [Welsh02] [Charpiat08]
[Gupta12] [Bugeau14] [Pierre15] Our Result
Colorization - Comparisons
Input Reference [Welsh02] [Charpiat08]
[Gupta12] [Bugeau14] [Pierre15] Our Result
Colorization - Comparisons
Input Reference [Welsh02] [Charpiat08]
[Gupta12] [Bugeau14] [Pierre15] Our Result
112
Limitations
● Texture based approach
113
Limitations
● Texture based approach– Not well suited for texture-less images
● Similar to applying a global transfer
114
Limitations
● Texture based approach– Not well suited for texture-less images
– Fine structures can be mistaken as textures
115
Limitations
● Texture based approach– Not well suited for texture-less images
– Fine structures can be mistaken as textures
Input Reference Our Result
116
Limitations
● Texture based approach– Not well suited for texture-less images
– Fine structures can be mistaken as textures
Input Reference Our Result
117
Future Works
118
Future Works
● Edge preserving texture descriptors
119
Future Works
● Edge preserving texture descriptors– Can be used in a lot of different applications
120
Future Works
● Edge preserving texture descriptors– Can be used in a lot of different applications
● Transfer and colorization
121
Future Works
● Edge preserving texture descriptors– Can be used in a lot of different applications
● Transfer and colorization– Several references could be used
122
Future Works
● Edge preserving texture descriptors– Can be used in a lot of different applications
● Transfer and colorization– Several references could be used
– Could be extended to video inputs
123
Thank you for your attention
124
Descriptors
● Effect of each step on a transfer result
Initial descriptors Only grad. descent Only filtering Both