Study on Frequency Domain Primary-Ambient Extraction (PAE) HE Jianjun PhD Candidate, DSP Lab, School...

Study on Frequency Domain Primary-Ambient Extraction (PAE)

HE Jianjun

PhD Candidate, DSP Lab, School of EEE, Nanyang Technological University, Singapore

Email: [email protected]

Introduction– PAE based Spatial Audio System

2

PAEInput

Primary components

Pre-processing

Ambient components

Primaryrendering

Ambient rendering

Spatial attributes

Post- processing

Output

Primary components highly correlated

Ambient components uncorrelated

Primary ambient components uncorrelated

Ambient power balanced

Stereo Signal Model

3

L L L

R R R

x p a

x p a

Signal = Primary + Ambient

: Left channel

: Right channel

L

R

Assumptions

R Lkp p

L Ra a

L RP Pa a

( ) ( )L R L Rp a

Stereo Signal Model

4

2

Primary panning factor PPF:

12 2

R

L

RR LL RR LL

LR LR

r r r

r

k

rk

r

p

p

Total primary powerPrimary power ratio PPR:

Total signal power

2, [0,1]LR RR LL

RR LL

r r r k

r r k

, : autocorrelation of the left, right channel; : cross correlation between the left and right channelLL RR LRr r r

k

1

Center

RightLeft

1/10 10

PAE in full band, time domain

5

0 0.5 1 1.5 2 2.5 3 3.5 4

x 105

-0.02

0

0.02

0 0.5 1 1.5 2 2.5 3 3.5 4

x 105

-0.02

0

0.02

200 400 600 800 1000 1200 1400 1600 1800 20000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Compute parameters: k, γ

22

1,2 2

LR RR LLRR LL RR LL

LR LR RR LL

r r r kr r r rk

r r r r k

22

2

2 2

2

22

2 2

111

1 1

111

1

ˆ

1

ˆ

ˆ

ˆ

L

R

L

R

L

R

k

kkk k

k kkkkk

k

k k

p

xp

a

xa

500 1000 1500 2000 2500 3000 3500 4000-2

0

2x 10

-3

500 1000 1500 2000 2500 3000 3500 4000-2

0

2x 10

-3

PAE in full band, frequency domain

6

0 0.5 1 1.5 2 2.5 3 3.5 4

x 105

-0.02

0

0.02

0 0.5 1 1.5 2 2.5 3 3.5 4

x 105

-0.02

0

0.02

200 400 600 800 1000 1200 1400 1600 1800 20000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Compute parameters: k, γ

500 1000 1500 2000 2500 3000 3500 4000-2

0

2x 10

-3

500 1000 1500 2000 2500 3000 3500 4000-2

0

2x 10

-3

5001000

15002000

25003000

35004000

0

0.05

5001000

15002000

25003000

35004000

0

0.05

FFT

PAE in subband, frequency domain

7

5001000

15002000

25003000

35004000

0

0.05

5001000

15002000

25003000

35004000

0

0.05

f

L R

X(f) k(f)

k

k

k

k

k

k

k

k

k

(1)

(2)

(3)

(4)

(5)

(6)

(7)

(8)

X(f) k represent the panning of the source

AssumptionIn each band, only one source is dominant. The overlapping among the spectrum of different sources shall be minimal.

Correlation computation and time shifting

8

, 44,44N

xyc x n y n

*

*

,0 44

, 44 0xy

IDFT X f Y fc

IDFT X f Y f

2 o

DFTj f N

o Nx n X f e

2

2

ˆ

ˆ

o

o

j f NLpL LpRL L

j f NRR RpL RpR

w f w f eP f X f

X fP f w f e w f

In time domain

In frequency domain

Find the inter-channel time difference (ICTD) arg maxo xyc

Apply ICTD in frequency domain

φ0

How to partition the bands?

9

5001000

15002000

25003000

35004000

0

0.05

5001000

15002000

25003000

35004000

0

0.05

f

L R

X(f) Ideally, the number of partitions = number of sources

Fixed partitioning: independent of input• Uniform (2, 4, 8, etc.)• Non-uniform (e.g. ERB)

Based on inter-channel cross- correlation coefficient (ICC) φ, Two thresholds: φL , φH

φ1

Adaptive partitioning: dependent of input • Top-down• Bottom-up

…Conditions for partition: •φ0 < φH

•max(φ1, φ2) > φ0

•min(φ1, φ2) > φL

Unknown

φ2

Multiple (2) sources

10

Three cases for the directions of two sources:1.At different sides (DS)2.One at the center (C)3.At the same side (SS)

Four ways to synthesize the source direction1.Amplitude panning (AP)2.Time shifting (TS)3.Amplitude panning and time shifting (APTS)4.HRTF filtering (HRTF)

Simulation testing: setup

11

Primary components: Speech, musicAmbient components: white Gaussian noisePrimary power ratio = 0.9Frame length: 4096Hanning window, 50% overlapping

We test PCA and SPCA with different frequency partitioning•Time domain, full band (Reference)•Uniform partitioning with [1, 2, 4, 8, 16, 32, 64] partitions•Non-uniform partitioning with 20 partitions (Faller, BCC [6])•Top-down (TD) partitioning, with φL = 0.1; φH =0.8

[6] C. Faller, and F. Baumgarte, “Binaural cue coding-part II: schemes and applications,” IEEE Tran. Speech and Audio Processing, vol. 11, no. 6, Nov. 2003.

Performance measure: Error-to-Signal Ratio

ˆ ˆ

10

,

( ) 10log .2

L L R R

L R

P P P PL R

P P

L R

P PESR ESR

P P

ESR ESRESR dB

Simulation Results: 1 source

12

Primary component: speech shifted by 20 lags, panned by k=3.

T 1 2 4 8 16 32 64 20non TDPCA -3.69 -3.72 -3.38 -3.45 -3.34 -3.32 -3.16 -3.19 -3.33 -3.72

SPCA -14.78 -14.85 -12.34 -12.05 -11.52 -11.35 -10.63 -9.30 -10.34 -14.38

1. Generally, SPCA better than PCA.2. The time domain PCA (SPCA) is very close to the frequency domain PCA (SPCA)

when there is only one partition.3. Significant worse performance is found in the frequency domain approaches with

fixed partitioning.4. The performance of the top down partitioning is acceptable.

0 2 4 6 8 10 12 14 16

-40

-30

-20

-10

0

10

20

30

40

Frequency partition k

ICT

D

estimated

true

0 2 4 6 8 10 12 14 160.5

1

1.5

2

2.5

3

3.5

4

Frequency partition k

PP

F

estimated

true

Primary panning factor

ICTD

Simulation Results: 2 sources

13

Four ways to synthesize the source direction1.Amplitude panning (AP)2.Time shifting (TS)3.Amplitude panning and time shifting (APTS)4.HRTF filtering (HRTF)

Three cases for the directions of two sources:1.At different sides (DS)2.One at the center (C)3.At the same side (SS)

Simulation Results: 2 sources-AP

14

1. Generally, the performance of SPCA and PCA is similar.

DS T 1 2 4 8 16 32 64 20non TDPCA -7.95 -8.10 -8.13 -8.22 -8.34 -8.56 -8.56 -9.80 -10.86

SPCA -7.94 -8.15 -8.18 -8.26 -8.36 -8.61 -8.39 -9.33 -9.95 -8.36

C T 1 2 4 8 16 32 64 20non TDPCA -10.15 -10.25 -10.14 -10.22 -10.27 -10.38 -10.34 -11.16 -11.99

SPCA -10.14 -10.33 -10.22 -10.24 -10.30 -10.43 -10.04 -10.29 -10.12 -10.38

SS T 1 2 4 8 16 32 64 20non TDPCA -13.04 -13.10 -11.82 -11.88 -11.75 -11.53 -11.31 -11.40 -11.93

SPCA -13.02 -13.23 -12.04 -11.81 -11.65 -11.56 -10.30 -10.24 -10.52 -13.21

2. The performance is better when the two directions become closer.3. The frequency domain approaches with fixed partitioning show some advantage when the primary components are not in the same side.

4. The frequency domain approach with top down partitioning yields a good performance.

Simulation Results: 2 sources-TS

15

1. Clearly, SPCA perform better than PCA.

DS T 1 2 4 8 16 32 64 20non TDPCA -5.16 -5.21 -5.23 -5.24 -5.23 -5.23 -5.26 -5.38 -5.47 -5.21

SPCA -7.98 -8.44 -8.43 -8.59 -8.69 -8.73 -8.62 -9.12 -8.58 -8.91

C T 1 2 4 8 16 32 64 20non TDPCA -9.10 -9.14 -9.18 -9.20 -9.18 -9.17 -9.14 -9.18 -9.27 -9.14

SPCA -9.13 -9.60 -9.70 -9.85 -9.97 -9.85 -9.54 -9.91 -9.35 -10.05

SS T 1 2 4 8 16 32 64 20non TDPCA -5.37 -5.38 -5.40 -5.42 -5.41 -5.42 -5.41 -5.44 -5.49 -5.38

SPCA -11.15 -11.65 -11.71 -11.78 -11.94 -11.86 -11.02 -11.20 -9.42 -11.68

2. The performance of SPCA is better when no directions in the center.3. The frequency domain approaches with fixed partitioning show some slightly advantage and does not vary too much in different partitioning.

4. The frequency domain approach with top down partitioning yields the best overall performance.

Simulation Results: 2 sources-APTS

16


DS T 1 2 4 8 16 32 64 20non TDPCA -5.16 -5.21 -5.23 -5.23 -5.23 -5.23 -5.26 -5.38 -5.47

SPCA -7.99 -8.44 -8.43 -8.59 -8.69 -8.73 -8.62 -9.12 -8.58 -8.91

C T 1 2 4 8 16 32 64 20non TDPCA -8.06 -8.28 -8.19 -8.27 -8.34 -8.46 -8.44 -9.04 -9.55 -8.28

SPCA -8.07 -8.43 -8.38 -8.40 -8.57 -8.63 -8.44 -8.70 -9.07 -8.68

SS T 1 2 4 8 16 32 64 20non TDPCA -4.18 -4.18 -3.95 -3.97 -3.91 -3.92 -3.89 -3.87 -3.98 -4.19

SPCA -10.16 -10.60 -9.89 -9.75 -9.80 -9.77 -9.07 -8.68 -7.29 -10.82

2. The performance is better when two directions are closer.3. The frequency domain approaches with fixed partitioning perform better when the two directions are not in the same side.

4. The frequency domain approach with top down partitioning yields the best overall performance for all cases.


17


DS T 1 2 8 3220no

nTD18

0.05,0.8

0.2,0.8

0.2,0.7

0.1,0.7

0.05,0.7

0.05,0.9

0.1,0.9

0.2,0.9

PCA -4.74 -5.04 -5.04 -5.22 -5.48 -6.85 -5.03 -5.03 -5.03 -5.03 -5.03 -5.03 -5.03 -5.03 -5.03

SPCA -6.45 -6.85 -6.85 -7.11 -7.25 -7.73 -6.85 -8.36 -6.85 -6.85 -6.85 -7.93 -8.58 -6.85 -6.85

C T 1 2 8 3220no

nTD18

0.05,0.8

0.2,0.8

0.2,0.7

0.1,0.7

0.05,0.7

0.05,0.9

0.1,0.9

0.2,0.9

PCA -8.06 -8.28 -8.19 -8.34 -8.44 -9.55 -8.28 -8.44 -8.28 -8.28 -8.28 -8.44 -8.44 -8.27 -8.27

SPCA -8.07 -8.43 -8.38 -8.57 -8.44 -9.07 -8.68 -9.06 -8.44 -8.44 -8.68 -8.58 -9.93 -8.6 -8.44

SS T 1 2 8 3220no

nTD18

0.05,0.8

0.2,0.8

0.2,0.7

0.1,0.7

0.05,0.7

0.05,0.9

0.1,0.9

0.2,0.9

PCA -4.18 -4.18 -3.95 -3.91 -3.89 -3.98 -4.19 -4.19 -4.19 -4.19 -4.19 -4.19 -4.19 -4.19 -4.19

SPCA-

10.16-

10.60-9.89 -9.80 -9.07 -7.29

-10.82

-10.11

-10.6 -10.6 -10.6-

10.41-8.53

-10.27

-10.6




18


DS T 2 8 3220no

n0.05,0.7

TD180.05,0.8

0.2,0.8

0.2,0.7

0.1,0.7

0.05,0.7

0.05,0.9

0.1,0.9

0.2,0.9

PCA -4.74 -5.04 -5.22 -5.48 -6.85 -5.03 -5.03 -5.03 -5.03 -5.03 -5.03 -5.03 -5.03 -5.03 -5.03

SPCA -6.45 -6.85 -7.11 -7.25 -7.73 -7.93 -6.85 -8.36 -6.85 -6.85 -6.85 -7.93 -8.58 -6.85 -6.85

C T 2 8 3220no

n0.05,0.7

TD180.05,0.8

0.2,0.8

0.2,0.7

0.1,0.7

0.05,0.7

0.05,0.9

0.1,0.9

0.2,0.9

PCA -8.06 -8.19 -8.34 -8.44 -9.55 -8.44 -8.28 -8.44 -8.28 -8.28 -8.28 -8.44 -8.44 -8.27 -8.27

SPCA -8.07 -8.38 -8.57 -8.44 -9.07 -8.58 -8.68 -9.06 -8.44 -8.44 -8.68 -8.58 -9.93 -8.6 -8.44

SS T 2 8 3220no

n0.05,0.7

TD180.05,0.8

0.2,0.8

0.2,0.7

0.1,0.7

0.05,0.7

0.05,0.9

0.1,0.9

0.2,0.9

PCA -4.18 -3.95 -3.91 -3.89 -3.98 -4.19 -4.19 -4.19 -4.19 -4.19 -4.19 -4.19 -4.19 -4.19 -4.19

SPCA-

10.16-9.89 -9.80 -9.07 -7.29

-10.41

-10.82

-10.11

-10.6 -10.6 -10.6-

10.41-8.53

-10.27

-10.6



Simulation Results: 2 sources-HRTF

19


DS T 1 2 4 8 16 32 64 20non TDPCA -3.28 -3.46 -3.46 -3.44 -3.50 -3.74 -3.77 -3.89 -3.85 -2.47

SPCA -6.49 -6.07 -6.08 -6.13 -6.42 -7.33 -5.70 -5.56 -5.71 -6.52

C T 1 2 4 8 16 32 64 20non TDPCA -6.96 -7.16 -7.09 -7.12 -7.21 -7.37 -7.50 -7.60 -7.65 -7.16

SPCA -7.41 -7.97 -7.89 -7.92 -7.87 -8.33 -6.97 -6.15 -7.17 -8

SS T 1 2 4 8 16 32 64 20non TDPCA -1.14 -1.26 -1.14 -1.15 -1.12 -1.06 -1.04 -1.07 -1.19 -1.88

SPCA -6.58 -6.81 -6.48 -6.48 -6.70 -7.44 -2.51 -2.44 -2.90 -6.98

2. The performance of SPCA is better when one direction is in the center.3. The frequency domain approaches with fixed partitioning show better performance only in some partitionings.


Summary of Simulation Results: 2 sources

20

2. Generally, SPCA perform better than PCA in almost all cases.

3. The performance varies as the directions of the sources change.

4. The frequency domain approaches with fixed partitioning cannot always give a better performance.

5. The frequency domain approach with top down partitioning approach yields the best overall performance in most of the cases.

1. The overall performance of PAE: AP > TS > APTS > HRTF.

How about perceptual testing?Usually only one source (speech) is better extracted. Because the spectrum of speech (as compared to music) is more focused in certain bands.

Conclusions and thoughts

21

2. Many considerations should go to the partitioning of the frequency bands.

4. The performance of PAE in frequency domain with fixed partitioning is not consistent as the directions of the sources change and the number of partitions changes.

3. Generally, SPCA outperforms PCA in almost all cases.

5. The frequency domain approach with top down partitioning approach yields some promising results in the performance in most of the cases.

1. A study of frequency domain PAE with different partitioning is conducted. It is targeted for multiple primary components that come from different directions concurrently. Two PAE approaches tested are PCA and shifted PCA.

Thoughts:A more robust partitioning is required! How to determine the threshold for other input signals.Need more accurate estimation of primary panning factor, and ICTD/ICC.How about other PAE approaches such as least squares ?

Study on Frequency Domain Primary-Ambient Extraction (PAE) HE Jianjun PhD Candidate, DSP Lab, School...

Documents

Transcript of Study on Frequency Domain Primary-Ambient Extraction (PAE) HE Jianjun PhD Candidate, DSP Lab, School...