Post on 12-Apr-2022
Generalized Lifting for SparseImage Representation and Coding
Julio C. Rolón1,2, Philippe Salembier1
1Technical University of Catalonia (UPC), SpainDept. of Signal Theory and Telecommunications
2National Polytechnic Institute (IPN), MexicoCITEDI Research Center
2
Generalized Lifting for Sparse Image Representation and Coding
Outline
Introduction• Motivation• Previous work
Lifting structures• Classical lifting• Generalized lifting
Coding scheme
Experimental results• Energy and entropy• Rate-distortion• Perceptual quality
Conclusions
3
Generalized Lifting for Sparse Image Representation and Coding
Outline
Introduction• Motivation• Previous work
Lifting structures• Classical lifting• Generalized lifting
Coding scheme
Experimental results• Energy and entropy• Rate-distortion• Perceptual quality
Conclusions
4
Generalized Lifting for Sparse Image Representation and Coding
Introduction: Motivation
Limitations of wavelets in image representation• Excellent for decorrelation of impulsional-like events
• Limitations on images because wavelets are not able to fully decorrelate edges
DWT
LH subband zoom
5
Generalized Lifting for Sparse Image Representation and Coding
Introduction: Motivation
Limitations of wavelets in image representation• Excellent for decorrelation of impulsional-like events
• Limitations on images because wavelets are not able to fully decorrelate edges
DWT
LH subband zoom
Need of additional decorrelation for coding applications
6
Generalized Lifting for Sparse Image Representation and Coding
Outline
Introduction• Motivation• Previous work
Lifting structures• Classical lifting• Generalized lifting
Coding scheme
Experimental results• Energy and entropy• Rate-distortion• Perceptual quality
Conclusions
7
Generalized Lifting for Sparse Image Representation and Coding
Introduction: Previous work
Main approaches• Go beyond wavelets trying to overcome their limitations• Edges become 1-dimensional localizable singularities• Increased sparsity of the coefficients
Curvelets[Candès,Donoho 1999; Candès 2006]
Contourlets[Do,Vetterli 2005]
Frequency-domain methodsBandelets
[LePennec,Mallat 2005; Peyré,Mallat 2005]
Adaptive directional lifting[Chappelier et al 2006; Ding et al 2007;
Heijmans et al 2005; Mehrseresht,Taubman 2006; Piella et al 2002,2006; Wang et al 2006;
Zhang et al 2005 ]
Spatial-domain methods
Generalized liftingMay be highly non-linear,
preserving perfect reconstruction
8
Generalized Lifting for Sparse Image Representation and Coding
Outline
Introduction• Motivation• Previous work
Lifting structures• Classical lifting• Generalized lifting
Coding scheme
Experimental results• Energy and entropy• Rate-distortion• Perceptual quality
Conclusions
9
Generalized Lifting for Sparse Image Representation and Coding
Lifting structures: Classical lifting
Approximation signal: x'[n] = x[n] + U(y'[n])
s[n]
x[n]
y[n]
x[n]
y[n]
s[n]
x'[n]
y'[n]
y'[n] = y[n] – P(x[n])Detail signal:
[Sweldens 1996]
10
Generalized Lifting for Sparse Image Representation and Coding
LAZY WAVELET TRANSFORM
s[n]
nx[n]
y[n]
x'[n]
y'[n]
x[n]
n
y[n]
n
LWT
11
Generalized Lifting for Sparse Image Representation and Coding
x[n]
n
y[n]
n
x[n]
y[n]
x'[n]
y'[n]
y'[n] = y[n] – P(x[n])Detail signal:
12
Generalized Lifting for Sparse Image Representation and Coding
x[n]
n
y[n]
n
x[n]
y[n]
x'[n]
y'[n]
P(x[n])
n
n
P
y'[n]
y'[n] = y[n] – P(x[n])Detail signal:
13
Generalized Lifting for Sparse Image Representation and Coding
x[n]
n
y[n]
n
x[n]
y[n]
x'[n]
y'[n]
P(x[n])
n
n
P
y'[n]Details are the prediction error
y'[n] = y[n] – P(x[n])Detail signal:
14
Generalized Lifting for Sparse Image Representation and Coding
y'[n] = y[n] – P(x[n])Detail signal:
Lifting structures: Classical lifting
Approximation signal: x'[n] = x[n] + U(y'[n])
s[n]
x[n]
y[n]
x[n]
y[n]
s[n]
x'[n]
y'[n]
[Sweldens 1996]
+
+
Invertibility key:+/- operations
15
Generalized Lifting for Sparse Image Representation and Coding
Outline
Introduction• Motivation• Previous work
Lifting structures• Classical lifting• Generalized lifting
Coding scheme
Experimental results• Energy and entropy• Rate-distortion• Perceptual quality
Conclusions
16
Generalized Lifting for Sparse Image Representation and Coding
Lifting structures: Generalized lifting
Approximation signal: x'[n] = U(x[n],y'[n])
y'[n] = P(y[n],x[n – i])|i∈CDetail signal:
[Solé,Salembier 2004]
s[n]
x[n]
y[n]
x[n]
y[n]
s[n]
x'[n]
y'[n]
17
Generalized Lifting for Sparse Image Representation and Coding
LAZY WAVELET TRANSFORM
s[n]
nx[n]
y[n]
x'[n]
y'[n]
x[n]
n
y[n]
n
LWT
18
Generalized Lifting for Sparse Image Representation and Coding
x[n]
n
y[n]
n
x[n]
y[n]
0 1 2 3 4 5 6 7 8 9
P
y'[n] = P(y[n],x[n – i])|i∈CDetail signal:
19
Generalized Lifting for Sparse Image Representation and Coding
x[n]
n
y[n]
n
x[n]
y[n]
0 1 2 3 4 5 6 7 8 9
Py'[n]
n0 1 2 3 4 5 6 7 8 9
y'[n]
y'[n] = P(y[n],x[n – i])|i∈CDetail signal:
20
Generalized Lifting for Sparse Image Representation and Coding
x[n]
n
y[n]
n
x[n]
y[n]
0 1 2 3 4 5 6 7 8 9
Details produced by P operator
y[n] → y'[n]P
Py'[n]
n0 1 2 3 4 5 6 7 8 9
y'[n]
y'[n] = P(y[n],x[n – i])|i∈CDetail signal:
21
Generalized Lifting for Sparse Image Representation and Coding
x[n]
n
y[n]
n
x[n]
y[n]
0 1 2 3 4 5 6 7 8 9
Details produced by P operator
y[n] → y'[n]P
Py'[n]
n0 1 2 3 4 5 6 7 8 9
y'[n]
y'[n] = P(y[n],x[n – i])|i∈CDetail signal:
x0
x2
x4
x6
x8
y1
y3
y5
y7
y9
y'3
y'5
y'1
y'7
y'9
y'x y P(y,x)
22
Generalized Lifting for Sparse Image Representation and Coding
Lifting structures: Generalized lifting
Approximation signal: x'[n] = U(x[n],y'[n])
y'[n] = P(y[n],x[n – i])|i∈CDetail signal:
[Solé,Salembier 2004]
y[n] y[n]
s[n] s[n]
x'[n]
y'[n]
x[n] x[n]
BIJECTIVITY OF P, U IS MANDATORYTO ACHIEVE PERFECT RECONSTRUCTION
23
Generalized Lifting for Sparse Image Representation and Coding
Lifting structures: Generalized lifting
Previous work has been done in: [Solé,Salembier 2004,2007]
• Lossless coding• 1-dimensional separable context• Applied directly in the lmage domain
24
Generalized Lifting for Sparse Image Representation and Coding
Lifting structures: Generalized lifting
Previous work has been done in: [Solé,Salembier 2004,2007]
• Lossless coding• 1-dimensional separable context• Applied directly in the lmage domain
Our main goal is to evaluate the potential of the GL method for lossy coding• Quantization• Comparison between 1-dimensional context and 2-dimensional
non-separable context• Applied in the wavelet domain (as in bandelets)• Assumption: the pdf of the signal is known in the decoder
(as well as in the coder)
25
Generalized Lifting for Sparse Image Representation and Coding
Outline
Introduction• Motivation• Previous work
Lifting structures• Classical lifting• Generalized lifting
Coding scheme
Experimental results• Energy and entropy• Rate-distortion• Perceptual quality
Conclusions
26
Generalized Lifting for Sparse Image Representation and Coding
Coding scheme: The encoder
DWT Q
A 011101…
A 10001…
ORIGINAL
27
Generalized Lifting for Sparse Image Representation and Coding
Coding scheme: The encoder
DWT Q
A 011101…
A 10001…
ORIGINAL
2-D WAVELETTRANSFORMED
28
Generalized Lifting for Sparse Image Representation and Coding
Coding scheme: The encoder
DWT Q
A 011101…
A 10001…
ORIGINAL
2-D WAVELETTRANSFORMED
SCALARQUANTIZED
29
Generalized Lifting for Sparse Image Representation and Coding
Coding scheme: The encoder
DWT Q
A 011101…
A 10001…
ORIGINAL
LWT 1-D row-wisedownsampling
and 2-D quincunxdownsampling
2-D WAVELETTRANSFORMED
SCALARQUANTIZED
30
Generalized Lifting for Sparse Image Representation and Coding
Coding scheme: The encoder
DWT Q
A 011101…
A 10001…
ORIGINAL
LWT 1-D row-wisedownsampling
and 2-D quincunxdownsampling
2-D WAVELETTRANSFORMED
SCALARQUANTIZED
31
Generalized Lifting for Sparse Image Representation and Coding
Coding scheme: The encoder
DWT Q
A 011101…
A 10001…
ORIGINAL
LWT 1-D row-wisedownsampling
and 2-D quincunxdownsampling
2-D WAVELETTRANSFORMED
SCALARQUANTIZED
GL – MAPPEDDETAIL
32
Generalized Lifting for Sparse Image Representation and Coding
Coding scheme: The decoder
DWT-1Q-1
A-1011101…
A-110001…
33
Generalized Lifting for Sparse Image Representation and Coding
-4 -3 -2 -1 0 1 2 3 4
Coding scheme: Generalized predict design
-4 -3 -2 -1 0 1 2 3 4
Input signal pdf mapped pdf
GL
Criterion: Energy minimization of the mapped details
][ny ][' ny
34
Generalized Lifting for Sparse Image Representation and Coding
-4 -3 -2 -1 0 1 2 3 4
Coding scheme: Generalized predict design
-4 -3 -2 -1 0 1 2 3 4 ][ny
Input signal pdf mapped pdf
GL
][' ny
Criterion: Energy minimization of the mapped details
22
11
33
44
-2-2
-3-3
-1-1
00
22
11
33
44
-2-2
-3-3
-1-1
00
][' ny][ny
35
Generalized Lifting for Sparse Image Representation and Coding
-4 -3 -2 -1 0 1 2 3 4
Coding scheme: Generalized predict design
-4 -3 -2 -1 0 1 2 3 4 ][ny
Input signal pdf mapped pdf
GL
][' ny
Criterion: Energy minimization of the mapped details
22
11
33
44
-2-2
-3-3
-1-1
00
22
11
33
44
-2-2
-3-3
-1-1
00
][' ny][ny
36
Generalized Lifting for Sparse Image Representation and Coding
-4 -3 -2 -1 0 1 2 3 4
Coding scheme: Generalized predict design
-4 -3 -2 -1 0 1 2 3 4 ][ny
Input signal pdf mapped pdf
GL
][' ny
Criterion: Energy minimization of the mapped details
22
11
33
44
-2-2
-3-3
-1-1
00
22
11
33
44
-2-2
-3-3
-1-1
00
][' ny][ny
37
Generalized Lifting for Sparse Image Representation and Coding
-4 -3 -2 -1 0 1 2 3 4
Coding scheme: Generalized predict design
-4 -3 -2 -1 0 1 2 3 4 ][ny
Input signal pdf mapped pdf
GL
][' ny
Criterion: Energy minimization of the mapped details
22
11
33
44
-2-2
-3-3
-1-1
00
22
11
33
44
-2-2
-3-3
-1-1
00
][' ny][ny
38
Generalized Lifting for Sparse Image Representation and Coding
-4 -3 -2 -1 0 1 2 3 4
Coding scheme: Generalized predict design
-4 -3 -2 -1 0 1 2 3 4 ][ny
Input signal pdf mapped pdf
GL
][' ny
Criterion: Energy minimization of the mapped details
22
11
33
44
-2-2
-3-3
-1-1
00
22
11
33
44
-2-2
-3-3
-1-1
00
][' ny][ny
39
Generalized Lifting for Sparse Image Representation and Coding
-4 -3 -2 -1 0 1 2 3 4
Coding scheme: Generalized predict design
-4 -3 -2 -1 0 1 2 3 4 ][ny
Input signal pdf mapped pdf
GL
][' ny
Criterion: Energy minimization of the mapped details
22
11
33
44
-2-2
-3-3
-1-1
00
22
11
33
44
-2-2
-3-3
-1-1
00
][' ny][ny
40
Generalized Lifting for Sparse Image Representation and Coding
-4 -3 -2 -1 0 1 2 3 4
Coding scheme: Generalized predict design
-4 -3 -2 -1 0 1 2 3 4 ][ny
Input signal pdf mapped pdf
GL
][' ny
Criterion: Energy minimization of the mapped details
22
11
33
44
-2-2
-3-3
-1-1
00
22
11
33
44
-2-2
-3-3
-1-1
00
][' ny][ny
41
Generalized Lifting for Sparse Image Representation and Coding
-4 -3 -2 -1 0 1 2 3 4
Coding scheme: Generalized predict design
-4 -3 -2 -1 0 1 2 3 4 ][ny
Input signal pdf mapped pdf
GL
][' ny
Criterion: Energy minimization of the mapped details
22
11
33
44
-2-2
-3-3
-1-1
00
22
11
33
44
-2-2
-3-3
-1-1
00
][' ny][ny
42
Generalized Lifting for Sparse Image Representation and Coding
-4 -3 -2 -1 0 1 2 3 4
Coding scheme: Generalized predict design
-4 -3 -2 -1 0 1 2 3 4 ][ny
Input signal pdf mapped pdf
GL
][' ny
Criterion: Energy minimization of the mapped details
22
11
33
44
-2-2
-3-3
-1-1
00
22
11
33
44
-2-2
-3-3
-1-1
00
][' ny][ny
ASSUMPTION: Mapping is known by the decoder, i.e. the decoder knows the pdf of the signal
43
Generalized Lifting for Sparse Image Representation and Coding
Outline
Introduction• Motivation• Previous work
Lifting structures• Classical lifting• Generalized lifting
Coding scheme
Experimental results• Energy and entropy• Rate-distortion• Perceptual quality
Conclusions
44
Generalized Lifting for Sparse Image Representation and Coding
0
20
40
60
80
100
1 2 3 4 5
100% reference: DWT DWT + GL, 1-D context DWT + GL, 2-D context
energy entropy% %
DWTscale
DWTscale
Experimental results: Energy and entropy
0
20
40
60
80
100
1 2 3 4 5
45
Generalized Lifting for Sparse Image Representation and Coding
Outline
Introduction• Motivation• Previous work
Lifting structures• Classical lifting• Generalized lifting
Coding scheme
Experimental results• Energy and entropy• Rate-distortion• Perceptual quality
Conclusions
46
Generalized Lifting for Sparse Image Representation and Coding
Experimental results: Rate-distortion
0.5 0.6 0.7 0.8 0.9 1 1.1 1.2 1.3 1.4 1.530
32
34
36
38
40
42
Bitrate (bpp)
PSN
R (
dB)
Wavelet + GLBandeletsWavelets
0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 130
32
34
36
38
40
42
Bitrate (bpp)
PSN
R (
dB)
Wavelet + GLBandeletsWavelets
Generalized liftingPotential gain is ~1-3 db, for 2-D context
47
Generalized Lifting for Sparse Image Representation and Coding
Experimental results: Rate-distortion
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 124
26
28
30
32
34
36
38
40
Bitrate (bpp)
PSN
R (
dB)
Wavelet + GLWavelets
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 124
26
28
30
32
34
36
38
40
Bitrate (bpp)
PSN
R (
dB)
Wavelet + GLWavelets
Additional comparisons against wavelets
48
Generalized Lifting for Sparse Image Representation and Coding
Outline
Introduction• Motivation• Previous work
Lifting structures• Classical lifting• Generalized lifting
Coding scheme
Experimental results• Energy and entropy• Rate-distortion• Perceptual quality
Conclusions
49
Generalized Lifting for Sparse Image Representation and Coding
Experimental results: Perceptual quality @ 0.2bpp
original wavelets32.03 dB
Bandelets31.24 dB
GL33.33 dB
50
Generalized Lifting for Sparse Image Representation and Coding
Experimental results: Perceptual quality @ 0.11bpp
original wavelets29.47 dB
Bandelets28.52 dB
GL30.63 dB
51
Generalized Lifting for Sparse Image Representation and Coding
Outline
Introduction• Motivation• Previous work
Lifting structures• Classical lifting• Generalized lifting
Coding scheme
Experimental results• Energy and entropy• Rate-distortion• Perceptual quality
Conclusions
52
Generalized Lifting for Sparse Image Representation and Coding
Conclusions
The potential of GL has been evaluated for lossy coding
53
Generalized Lifting for Sparse Image Representation and Coding
Conclusions
The potential of GL has been evaluated for lossy coding
The use of GL approach proposed here follows the ideas of spatial-domain methods(bandelets and adaptive directional lifting)
54
Generalized Lifting for Sparse Image Representation and Coding
Conclusions
The potential of GL has been evaluated for lossy coding
The use of GL approach proposed here follows the ideas of spatial-domain methods(bandelets and adaptive directional lifting)
The GL method differentiates from these methods in thatis a non-linear framework
55
Generalized Lifting for Sparse Image Representation and Coding
Conclusions
The potential of GL has been evaluated for lossy coding
The use of GL approach proposed here follows the ideas of spatial-domain methods(bandelets and adaptive directional lifting)
The GL method differentiates from these methods in thatis a non-linear framework
Our predict design is based on the pdf of the image
56
Generalized Lifting for Sparse Image Representation and Coding
Conclusions
The potential of GL has been evaluated for lossy coding
The use of GL approach proposed here follows the ideas of spatial-domain methods(bandelets and adaptive directional lifting)
The GL method differentiates from these methods in thatis a non-linear framework
Our predict design is based on the pdf of the image
Generalized lifting approach gives promising results atreducing the energy while improving the R-D performance in 1-3 dB
57
Generalized Lifting for Sparse Image Representation and Coding
Conclusions: Future work
Remove the assumption that the decoder knows the pdf of the signal• This work: pdf is assumed to be known in coder and decoder• Future: pdf estimation with fixed as well as adaptive strategies
58
Generalized Lifting for Sparse Image Representation and Coding
Conclusions: Future work
Remove the assumption that the decoder knows the pdf of the signal• This work: pdf is assumed to be known in coder and decoder• Future: pdf estimation with fixed as well as adaptive strategies
Multiscale GL approach• This work: one scale of GL applied for 1-D and 2-D contexts• Future: multiscale GL in 2-D context
59
Generalized Lifting for Sparse Image Representation and Coding
Conclusions: Future work
Remove the assumption that the decoder knows the pdf of the signal• This work: pdf is assumed to be known in coder and decoder• Future: pdf estimation with fixed as well as adaptive strategies
Multiscale GL approach• This work: one scale of GL applied for 1-D and 2-D contexts• Future: multiscale GL in 2-D context
Quantization• This work: scalar pre-quantization• Future: include Q in the GL mapping, post-quantization
60
Generalized Lifting for Sparse Image Representation and Coding
Conclusions: Future work
Remove the assumption that the decoder knows the pdf of the signal• This work: pdf is assumed to be known in coder and decoder• Future: pdf estimation with fixed as well as adaptive strategies
Multiscale GL approach• This work: one scale of GL applied for 1-D and 2-D contexts• Future: multiscale GL in 2-D context
Quantization• This work: scalar pre-quantization• Future: include Q in the GL mapping, post-quantization
Efficient entropy coding• This work: arithmetic encoding• Future: embedded-block coders like EBCOT, SPECK, etc.
Questions
Generalized Lifting for SparseImage Representation and Coding
Julio C. Rolón, Philippe SalembierTechnical University of Catalonia (UPC), SpainDept. of Signal Theory and Telecommunications
National Polytechnic Institute (IPN), MexicoCITEDI Research Center
{jcrolon,philippe}@gps.tsc.upc.edu
Thank you !!!
Generalized Lifting for SparseImage Representation and Coding
Julio C. Rolón, Philippe SalembierTechnical University of Catalonia (UPC), SpainDept. of Signal Theory and Telecommunications
National Polytechnic Institute (IPN), MexicoCITEDI Research Center
{jcrolon,philippe}@gps.tsc.upc.edu