An Introduction To Distributed

Source Coding

Chao YuIP Lab, University of Rochester

July 1, 2008

1

1 Distributed Source Coding (DSC)What is DSC ?Nested Quantizer: A Simple ApproachDSC using LDPC

2 DSC For Noisy ObservationA Separative StructureConnection With Transform Coding

3 Summary

2

Outline

1 Distributed Source Coding (DSC)What is DSC ?Nested Quantizer: A Simple ApproachDSC using LDPC

2 DSC For Noisy ObservationA Separative StructureConnection With Transform Coding

3 Summary

3

Distributed Source Coding

Compression of multiple correlated sources (e.g. sensornetworks)

sources do not communicate with each other.

Encoder

DecoderEncoder

JointJoint

DecoderEncoder

Joint

X

X, Y

Y R2

R1X

X, Y

R2Y

R1

Question: will the coding performance degrade due to mutualblindness

4

Theoretical Results

Encoder

DecoderEncoder

JointJoint

DecoderEncoder

Joint

X

X, Y

Y R2

R1X

X, Y

R2Y

R1

R1 + R2 ≥ H(X,Y)

R1 ≥ H(X | Y)R2 ≥ H(Y | X)

R1 + R2 ≥ H(X,Y)

Slepian-Wolf theorem [Slepian and Wolf, 1973]

lossless transmission of discrete, memoryless sources

Wyner-Ziv theorem [Wyner, 1975]

lossy transmission of continuous-valued sources under a fidelitycriterion.

5

Side-Informed Source Coding

Y available at the decoder as the side information

Encoder DecoderChannel

yjoint statistics of X,Y

xx

the encoder exploits the SI,

R ≥ H(X | Y).

A special case of DSC

Channel is rate constrained, however error-free

6

Outline

1 Distributed Source Coding (DSC)What is DSC ?Nested Quantizer: A Simple ApproachDSC using LDPC

2 DSC For Noisy ObservationA Separative StructureConnection With Transform Coding

3 Summary

7

DSC by Nested Quantizer [Pradhan and Ramchandran, 2003]

Consider y = x + n as the side information

x ∼ N (0, σx ),n ∼ N (0, σn), n ⊥ x, σn < σx (relevant side information)

8

DSC by Nested Quantizer [Pradhan and Ramchandran, 2003]

Consider y = x + n as the side information

x ∼ N (0, σx ),n ∼ N (0, σn), n ⊥ x, σn < σx (relevant side information)

8

DSC by Nested Quantizer [Pradhan and Ramchandran, 2003]

Consider y = x + n as the side information

x ∼ N (0, σx ),n ∼ N (0, σn), n ⊥ x, σn < σx (relevant side information)

8

DSC vs. Channel Coding

improvements to nested quantizer:

better quantization (TCQ)

channel coding in DSC

y viewed as a virtual channel output,

extra information generated at encoder performs errorcorrection

DSC resembles channel coding: Trellis code, Turbo code, LDPCcode can be properly applied.

9

Outline

1 Distributed Source Coding (DSC)What is DSC ?Nested Quantizer: A Simple ApproachDSC using LDPC

2 DSC For Noisy ObservationA Separative StructureConnection With Transform Coding

3 Summary

10

DSC using LDPC: binary case

Slepian-Wolf Code: Discrete Source, lossless transmission

The initialization step incorporates the side information:

qinit

i = log(Pr(v1

i= 0|wi )

Pr(v1i

= 1|wi ))

qinit

iis determined by statistical correlation between v and w.

11

DSC using LDPC: continuous case

Wyner-Ziv Code: Non-discrete Source, lossy transmission

Quantizer + Slepian-Wolf Code

Side information propagation

Decoded higher bit-planes serve as side-information for lowerbit-planes

12

DSC using LDPC: continuous case

Pr(v1i = 0|wi ) =

MX

m=2

1X

bm=0

Pr(v1i = 0, v

2i = b2, . . . , v

M

i = bM |wi)

Pr(vm

i = 0 | v1i , . . . , v

m−1i

,wi) =

MX

n=m+1

1X

bn=0

Pr(vm

i = 0, vn

i = bn, vM

i = bM |v1i , . . . , v

m−1i

,wi )

13

Outline

1 Distributed Source Coding (DSC)What is DSC ?Nested Quantizer: A Simple ApproachDSC using LDPC

2 DSC For Noisy ObservationA Separative StructureConnection With Transform Coding

3 Summary

14

Transmission of Noisy Observation with SI

The observation at the encoder is indirect observation

Constrained

ChannelDecoderEncoder

Side InformationStatistics of

m m

y0

Remote Node (RN) Central Processor (CP)

xm

(y0)y1

Utilization of DSC

Potential: The existence of y0 motivates DSC

Challenge: irrelevant information in y1

15

A Separative Structure

It can be shown:

arg minm

E[

‖X − xm(Y0)‖2 | y1

]

= arg minm

E[

‖v − vm(Y0)‖2 | y1

]

wherev = RxsR

−1s y1, s = y1 − y1(y0)

DecoderChannel

Encoder

Axs

Axs

y1(y0) x(y0)

m

Central Processor (CP)

vm

(w)

w = v + nv

y0

xm

(y0)

of V,W

joint statistics

Constrainedy1

Remote Node (RN)

mv = Axs

y1

of Y0,Y1

joint statistics

MMSE Estimator

for X from S

optimal transmission of v guarantees optimal estimation of x.

DSC techniques utilized for v as a practical scheme.

16

Outline

1 Distributed Source Coding (DSC)What is DSC ?Nested Quantizer: A Simple ApproachDSC using LDPC

2 DSC For Noisy ObservationA Separative StructureConnection With Transform Coding

3 Summary

17

Connection With Transform Coding

Transform coding as a practical scheme after the proposeddecomposition:

The proposed decomposition generalize [Sakrison, 1968] intoside-informed scenario. 18

Summary

The redundancy of correlated data motivates DSC

DSC resembles channel coding

LDPC/Turbo adapt well for DSC

SI propagation for continuous Symbols

DSC for noisy observations: decomposition

Connection between proposed decomposition with transformcoding

19

Reference

S.S. Pradhan and K. Ramchandran. Distributed source codingusing syndromes (DISCUS): design and construction. IEEE

Trans. Info. Theory, 49(3):626–643, 2003.

D.J. Sakrison. Source encoding in the presence of randomdisturbance. IEEE Trans. Info. Theory, 14:165–167, 1968.

David Slepian and Jack K. Wolf. Noiseless coding of correlatedinformation sources. IEEE Trans. Info. Theory, 19(4), January1973.

Aaron D. Wyner. On source coding with side information at thedecoder. IEEE Trans. Info. Theory, 21(3), May 1975.

20