Laplacian colormaps: a frameworkfor structure-preserving color transformations

Davide Eynard, Artiom Kovnatsky, Michael Bronstein

Institute of Computational Science, Faculty of InformaticsUniversity of Lugano, Switzerland

Eurographics, 8 April 2014

This research was supported by the ERC Starting Grant No. 307047 (COMET).

1 / 40

2 / 40

3 / 40

4 / 40

Color transformations

RGB source Luma

Standard color transformations may break image structure!

5 / 40

Color transformations

RGB source Luma

Standard color transformations may break image structure!

6 / 40

Color transformations

RGB source Luma Desired outcome

Standard color transformations may break image structure!

7 / 40

Image Laplacian

Input N ×M image with d colorchannels, column-stacked into anNM × d matrix X

Represented as graph with K vertices(e.g. superpixels) and weighted edges

K ×K adjacency matrix WX

wij = exp

(−δ2ij2σ2s

+‖x′ki − x′kj‖

22

2σ2r

)

K ×K Laplacian

LX = DX−WX, DX = diag(∑j 6=i

wij)

8 / 40

Image Laplacian

Input N ×M image with d colorchannels, column-stacked into anNM × d matrix X

Represented as graph with K vertices(e.g. superpixels) and weighted edges

K ×K adjacency matrix WX

wij = exp

(−δ2ij2σ2s

+‖x′ki − x′kj‖

22

2σ2r

)

K ×K Laplacian

LX = DX−WX, DX = diag(∑j 6=i

wij)

x′ki

x′kj

wij

9 / 40

Laplacians = structure descriptors

UTLXU = ΛX, VTLYV = ΛY

X u4 u5 u6 u7

Y v4 v5 v6 v7

Similar structure ⇐⇒ similar Laplacian eigenvectors

10 / 40

Laplacians = structure descriptors

UTLXU = ΛX, VTLYV = ΛY

X u4 u5 u6 u7

Y v4 v5 v6 v7

Similar structure ⇐⇒ similar Laplacian eigenvectors

Ideally, two Laplacians are jointly diagonalizable (iff theycommute): there exists a joint eigenbasis U = U = V

11 / 40

Laplacians = structure descriptors

X u2 u3 u4 u5

RGB source

12 / 40

Laplacians = structure descriptors

X u2 u3 u4 u5

RGB source

Y v2 v3 v4 v5

Luma (‘bad’ color conversion)

13 / 40

Laplacians = structure descriptors

X u2 u3 u4 u5

RGB source

Y v2 v3 v4 v5

Luma (‘bad’ color conversion)

Z t2 t3 t4 t5

‘Good’ color conversion

14 / 40

Laplacians = structure descriptors

X u2 u3 u4 u5 Clustering

RGB source

Y v2 v3 v4 v5 Clustering

Luma (‘bad’ color conversion)

Z t2 t3 t4 t5 Clustering

‘Good’ color conversion

15 / 40

Finding joint eigenbases

Joint approximate diagonalization

Find joint approximateeigenbasis U

minU

off(UTLXU) + off(U

TLYU)

s.t. UTU = I

where off(A) =∑

i 6=j a2ij .

These two problems are equivalent!(approx. joint diagonalizability ⇐⇒ approx. commutativity)

Cardoso 1995, Eynard et al. 2012, Kovnatsky et al. 2013

, Bronstein et al. 2013

16 / 40

Finding joint eigenbases

Joint approximate diagonalization

Find joint approximateeigenbasis U

minU

off(UTLXU) + off(U

TLYU)

s.t. UTU = I

where off(A) =∑

i 6=j a2ij .

Closest commuting Laplacians

Find closest commutingpair LX, LY

minLX,LY

‖LX − LX‖2F + ‖LY − LY‖2F

s.t. LXLY = LYLX

Since LX and LY commute, theyhave a joint eigenbasis U

These two problems are equivalent!(approx. joint diagonalizability ⇐⇒ approx. commutativity)

Cardoso 1995, Eynard et al. 2012, Kovnatsky et al. 2013, Bronstein et al. 201317 / 40

Finding joint eigenbases

Joint approximate diagonalization

Find joint approximateeigenbasis U

minU

off(UTLXU) + off(U

TLYU)

s.t. UTU = I

where off(A) =∑

i 6=j a2ij .

Closest commuting Laplacians

Find closest commutingpair LX, LY

minLX,LY

‖LX − LX‖2F + ‖LY − LY‖2F

s.t. LXLY = LYLX

Since LX and LY commute, theyhave a joint eigenbasis U

These two problems are equivalent!(approx. joint diagonalizability ⇐⇒ approx. commutativity)

Cardoso 1995, Eynard et al. 2012, Kovnatsky et al. 2013, Bronstein et al. 201318 / 40

Laplacian colormaps

X

19 / 40

Laplacian colormaps

X

−→Φθ

Y = Φθ(X)

Parametric colormap Φθ : RNM×d → RNM×d′ parametrizedby θ = (θ1, . . . , θn)

Global: each pixel x is transformed same way, y = Φθ(x)Local: different transformations in q regions,Φθ(X) =

∑qi=1 wiΦθi(X)

20 / 40

Laplacian colormaps

X

−→Φθ

Y = Φθ(X)

LX LY

LX = DX −WX LΦθ(X) = DΦθ(X) −WΦθ(X)

Find an optimal parametric color transformation

minθ∈Rn

‖LXLΦθ(X) − LΦθ(X)LX‖2F + regularization on θ

21 / 40

Color-to-gray conversion

Color mapping by a global color transformation of the form

Φθ(R,G,B) = θ1 + θ2Rθ3 + θ4G

θ5 + θ6Bθ7

Luma Col2Gray Rasche Decolorize Neumann Smith Lu Ours

2.18/-1.05 1.96/-0.10 1.43/-1.38 1.35/0.86 2.22/0.29 2.13/-0.29 1.47/0.82 1.19/1.15

Cadık 2008; Gooch et al. 2005; Rasche et al. 2005; Grundland, Dodgson 2007;

Neumann et al. 2007; Smith et al. 2008; Lu et al. 2012; Kuhn et al. 200822 / 40

Color-to-gray conversion

Color mapping by a global color transformation of the form

Φθ(R,G,B) = θ1 + θ2Rθ3 + θ4G

θ5 + θ6Bθ7

Luma Col2Gray Rasche Decolorize Neumann Smith Lu Ours

5.01/-0.55 3.42/-0.89 3.59/-0.48 3.44/1.41 5.44/-0.66 5.04/-0.19 2.90/0.50 1.28/0.86

9.27/-0.57 7.05/-0.53 7.20/-0.04 7.28/1.45 10.17/-1.05 9.13/-1.02 6.30/1.01 3.78/0.76

Cadık 2008; Gooch et al. 2005; Rasche et al. 2005; Grundland, Dodgson 2007;

Neumann et al. 2007; Smith et al. 2008; Lu et al. 2012; Kuhn et al. 200823 / 40

Color-to-gray conversion

Color mapping by a global color transformation of the form

Φθ(R,G,B) = θ1 + θ2Rθ3 + θ4G

θ5 + θ6Bθ7

Luma Col2Gray Rasche Decolorize Neumann Smith Lu Ours

0.97/0.27 1.24/-1.30 0.97/-0.08 1.02/0.61 1.66/-0.86 1.05/0.32 0.80/0.22 0.85/0.82

Cadık 2008; Gooch et al. 2005; Rasche et al. 2005; Grundland, Dodgson 2007;

Neumann et al. 2007; Smith et al. 2008; Lu et al. 2012; Kuhn et al. 200824 / 40

Color-to-gray conversion

Color mapping by a global color transformation of the form

Φθ(R,G,B) = θ1 + θ2Rθ3 + θ4G

θ5 + θ6Bθ7

Luma Col2Gray Rasche Decolorize Neumann Smith Lu OursRWMS 2.84 2.31 2.46 2.20 4.85 2.94 1.90 1.33z-score -0.17 -0.31 -0.63 0.55 -0.53 -0.09 0.34 0.84

Cadık 2008; Gooch et al. 2005; Rasche et al. 2005; Grundland, Dodgson 2007;

Neumann et al. 2007; Smith et al. 2008; Lu et al. 2012; Kuhn et al. 200825 / 40

Computational complexity: Color-to-gray example

10-1

100

101

102

103

Tim

e (sec)

#vertices 253 641 1130 22946 91784 367136

0.597

RW

MS e

rror

0.599

x10-3

Linear (n=3)

Non-linear (n=7)

Superpixels Scaling

Complexity O(K2)

Laplacian dimension K �MN (realtime performance withsmall K)

Optimization on θ is performed with small Laplacians. Then,Φθ is applied on full image

Superpixels: Ren, Malik 200326 / 40

Color-blind image optimization

RGB source

X

Ψ

Seen by color-blind

Ψ(X)

Vienot et al. 1999, Kim et al. 201227 / 40

Color-blind image optimization

RGB source

X Φθ(X)

Ψ

Seen by color-blind

Φθ

(Φθ ◦Ψ)(X)

︸ ︷︷ ︸‖LXLΦθ(X)−LΦθ(X)LX‖

‖LXL(Φθ◦Ψ)(X)−L(Φθ◦Ψ)(X)LX‖︷ ︸︸ ︷

28 / 40

Color-blind image optimization

RGB source

X Φθ(X)

Ψ

Seen by color-blind

Φθ

(Φθ ◦Ψ)(X)︸ ︷︷ ︸‖LXLΦθ(X)−LΦθ(X)LX‖

‖LXL(Φθ◦Ψ)(X)−L(Φθ◦Ψ)(X)LX‖︷ ︸︸ ︷

29 / 40

Color-blind image optimization: protanopia

RGB Lau

1.23

Optimized

0.50

Lau et al. 201130 / 40

Color-blind image optimization: tritanopia

RGB Lau

1.69

Optimized

0.53

Lau et al. 201131 / 40

Gamut mapping

Map image colors to a gamut G(convex polytope)

minθ∈Rn

‖LXLΦθ(X) − LΦθ(X)LX‖2F+ regularization on θ

s.t. Φθ(X) ⊆ G

sRGB

G

32 / 40

Gamut mapping

Original Lau et al. Ours HPMINDE (clip)

Lau et al. 201133 / 40

RGB+NIR fusion

NIR RGB Lau et al. Ours

Lau et al. 201134 / 40

Multiple image fusion

Morning

Day

Evening

Night

Fusion

35 / 40

Summary

Framework

theoretically grounded

versatile

global/local

realtime

Applications

color-to-grayscale

color-blind optimization

gamut mapping

multispectral image fusion

36 / 40

Summary

Framework

theoretically grounded

versatile

global/local

realtime

Applications

color-to-grayscale

color-blind optimization

gamut mapping

multispectral image fusion

37 / 40

Thank you!

38 / 40

Qualitative evaluation

Web survey

124 volunteers, 2884 pairwise evaluations

Thurstone’s law of comparative judgements → z-score

Consistent with Cadık’s results

39 / 40

Extension: local colormap

RGB Luma Lau et al.

Global Local Clusters

Φθ(X) =∑q

i=1 wiΦθi(X)

Lau et al. 201140 / 40