Application (I): Impulse Noise Removal Impulse noise.

Application (I): Impulse Noise Removal

Impulse noise

Application (II): Predictive Coding

var=4653, var=26.78

EE591b Advanced Image Processing Copyright Xin Li 2006 3

Lossless Image Compression

Review of MED used in JPEG-LS from EE465

GAP in CALIC scheme Least-Square based edge directed prediction Intra-coding scheme adopted by H.264/JVT

standard


LinearPrediction

entropycoding

discretesource X

binarybit stream

probabilityestimation

P(Y)

Y

Recall: Predictive Coding

Prediction residue sequence Y usually contains lessuncertainty (entropy) than the original sequence X


2D Predictive Coding

raster scanning order

Xm,n

causal half-plane

nmXaXK

kkknm ,,ˆ

1,

nmnmnm XXY ,,,ˆ


Median Edge Detection (MED) Prediction

xw

nnw

),,(ˆ nwwnwnmedianx xxy ˆ

Key:

MED use the median operator to adaptively select one from three candidates (Predictors #1,#2,#4 in slide 44) as the predicted value.


Another Way of Implementation

xw

nnw),,(ˆ nwwnwnmedianx

If ),max( wnnw ),min(ˆ wnx

else if ),min( wnnw

nwwnx ˆ

Q: which one is faster?You need to find itout using MATLAByourself


Numerical Examples

100 50100 50

100 100 50 50

V_edge H_edgen=50,w=100,nw=100

n+w-nw=50

50)50,100,50(

),,(ˆ

median

nwwnwnmedianx

n=100,w=50,nw=100

n+w-nw=50

50)50,50,100(

),,(ˆ

median

nwwnwnmedianx

0ˆ xxy 0ˆ xxy

Note how we can get zero prediction residues regardless of the edge direction


Image Example

Fixed vertical predictorH=4.67bpp

Adaptive (MED) predictorH=4.55bpp


JPEG-LS (the new standard for

lossless image compression)*


Lossless Image Compression Review of MED used in JPEG-LS from

EE465 GAP in CALIC scheme Least-Square based edge directed

prediction Intra-coding scheme adopted by H.264/JVT

standard


Gradient Adjusted Prediction


Image Example

MED predictorH=4.55bpp

GAP predictorH=4.39bpp


Context-based, Adaptive, Lossless Image Codec (CALIC)


Context QuantizationContext formulation

Without quantization, we will have 2568 different contexts(so-called “context dilution” problem)

After binary context quantization, we will reduce the numberof contexts to 28=256


Lossless Image Compression Review of MED used in JPEG-LS from

EE465 GAP in CALIC scheme Least-Square based edge directed

prediction (EDP) Intra-coding scheme adopted by H.264/JVT

standard


Motivation Behind EDP

GAP appears ad-hoc due the following reasons Only two directions are considered Difficult to justify the thresholds (32 and 80) used

to classify weak and strong edges Fundamental limitation with local gradients

Two potential improvements Truly direction adaptive From local to nonlocal prediction


Key Observation

No matter which classification strategy we adopt (e.g., strong vs. weak, horizontal vs. vertical), the prediction result can be viewed as a linear weighted average of the local neighborhood

Is there a systematic and provably optimal way of tuning the weighting coefficients? Again, the idea of localization will fly again


Recall: Autoregressive (AR) Model

N

nknknk wXaX

1

N

k

a

a

a

NMXNMXMX

NMXMX

NXXX

MX

kX

X

1

)()1()1(

)1(......)2(

......

)1(......)1()0(

)(

)(

)1(

M equations, N unknown variables11 NNMM aCy


Least-Square Estimation

N

nknknk wXaX

1

N

k

a

a

a

NMXNMXMX

NMXMX

NXXX

MX

kX

X

1

)()1()1(

)1(......)2(

......

)1(......)1()0(

)(

)(

)1(

M equations, N unknown variables11 NNMM aCy


LS-based Training of AR Model within a Local Causal Neighborhood

iX

T

T+1 T

T+1

iM

iN

iM double rectangular windowcontaining causal neighbors

iN the nearest N causal neighbors

)1(2 TTM

ij NX

jji XaX̂ iii XXe ˆ)()( 1 yCCCa TT


iX

10iii MMM

Edge Directed Property

1

iM

0

iM

0

iM

1

iM : edge pixels

: non-edge pixels

• The LS method provides a convenientway of finding solution for edge pixelswithout explicitly picking them out

• The weights derived by edge pixels work for Xi since it lives along the same edge

no preferred direction

unique preferred direction

edge pixels dominates the Least-Square process

direction adaptive prediction


Predictor Performance Comparison

MAP(4.56bpp) GAP(4.40bpp) 10th-order EDP(4.22bpp)

Comparison of prediction residue images by MAP, GAP and EDP


Efficient Implementation(I)

• Inclusion-and-Exclusion:

memory complexity

+

+++++

------

++

+

- -

(1)

(2)

straightforward implementation

2)1(2 NTT

11200)10,7( NT

arithmetic operations

(1)

(2)

5N2=500

2(T+1)N2=1600


Efficient Implementation(II)

• Switching strategy:

per pixel per edge

(speed-up ratio=262,144/25,87010)

activate LS to update aj when |ei|>T

- LS-based adaptation only enhances prediction performance around edges

- predictor optimized for an edge pixelalso works for its neighboring pixels along the same edge


Performance of EDP

CALIC TMW scheme I

airplane

baboon

lenna

peppers

barb

barb2

boats

goldhill

average

3.74 3.60 3.75

4.11

4.42

4.32

4.53

3.83

4.39

4.41

4.01

4.33

4.08

4.46

3.75

4.38

4.32

3.91

4.25

4.08

4.38

3.61

4.27

4.23

11.4 3.82 10.7

11.5

11.5

17.2

17.0

16.9

17.1

-

10.8

12.7

18.2

26.3

15.2

22.5

-

4.02

4.35

4.11

4.52

3.80

4.39

4. 35

time(seconds) scheme II

EDP(N=6,T=6)time

(seconds)

5.88 5.73 5.81 11.6 5.81 34.4

(seconds) (hours)

Performance (bpp) comparison among CALIC, TMW and EDP


Lossless Image Compression

Review of MED used in JPEG-LS from EE465

GAP in CALIC scheme Least-Square based edge directed prediction Intra-coding scheme adopted by H.264/JVT

standard


A Glimpse into H.264

It is a video (not image) coding standard However, there is so-called I (Intra-coded)

frame in video coding which does not involve any temporal prediction

Therefore, I-frame coding is conceptually identical to image coding

H.264 Intra-coding is a lossy scheme (though lossless extension is conceivable)


Intra-prediction Modes in H.264

The idea of directional prediction is obvious, but moreover,the prediction goes from local to nonlocal (a pixel can be usedto predict four pixels along the specified direction)


Patch-based Prediction (Open Research Problem)

pp

non-parametricsampling

Input image

Shouldn’t the prediction of P be based on nonlocal patches insteadof local neighborhood?


Preliminary Result

Patch-based, H=3.67bpp EDP, H=4.43bpp

It is going to be the next breakthrough in lossless image compression


Lossless Image Coding Summary A well-define objective: use as few bits as

possible (MSE=0) From ad-hoc prediction to more systematic

way of designing predictors which can exploit the fundamental dependency of image source

It still has a long way to go

Application (III): Super-resolution

SR

EE565 Advanced Image Processing Copyright Xin Li 2008 34

Heuristics: Edge Orientation

Can we do better? Yes! Gradient is only a first-order characteristics of

edge location ESI makes binary decision with two orthogonal

directions How to do better?

We need some mathematical tool that can work with arbitrary edge orientation


Motivation

x

y

Along the edge orientation,We observe repeated pattern

(0,0)

(-1,2)

(-2,4)

(1,-2)

:

:.

.

pattern


Geometric Duality

same pattern

downsampling


Bridge across the resolution

High-resolution

Low-resolution

2i

2j

2i+2

2i-2

2j-2 2j+2

Cov(X2i,2j,X2i+k,2j+l)≈Cov(X2i,2j,X2i+2k,2j+2l)

(k,l)={(0,1),(1,1),(1,0),(1,-1),(0,-1),(-1,-1),(-1,0),(-1,1)}


Least-Square (LS) Method

nnnnnnnnnnnn YAXXAY 1

Solve over-determined system

Solve square linear system

)()( 111 YAAAXXAY TT

nnmm


LS-based estimation

X1X2X3

X4

X5 X6 X7

X8

X

8

1iii XaX

For all pixels in 7x7 window,we can write an equation likeabove, which renders anover-determined systemwith 49 equations and 8 unknown variables

Use LS method to solve


Step 1: Interpolate diagonal pixels

0

1

2 3

40

1 4

2 3

-Formulate LS estimationproblem with pixels atlow resolution and solve{a1,a2,a3,a4}

-Use {a1,a2,a3,a4} tointerpolate the pixel0 at the high resolution

Implementation:


Step 2: Interpolate the Other Half

0

1

2

3

4

0

1

2

3

4

-Formulate LS estimationproblem with pixels atlow resolution and solve{a1,a2,a3,a4}

-Use {a1,a2,a3,a4} tointerpolate the pixel0 at the high resolution

Implementation:


Experiment Result

bilinear Edge directed interpolation


After Thoughts

Pro Improve visual quality dramatically

Con Computationally expensive

Further optimization Translation invariant derivation of interpolation

coefficients a’s

Application (IV): Image Denoising

Noisy denoised

Ref.: Hirakawa, K.; Parks, T.W., "Image denoising using total least squares," Image Processing, IEEE Transactions on , vol.15, no.9, pp.2730-2742, Sept. 2006

Unsettled Questions

Why AR is more effective on speech than image?

How to choose the order of AR model and the size of training window?

How to handle the interference of outliers? When does AR fail (as a tool of

dimensionality reduction)?

Application (I): Impulse Noise Removal Impulse noise.

Documents

Transcript of Application (I): Impulse Noise Removal Impulse noise.