IEEE TRANSACT10NS ON ACOUSTICS, SPEECH, AND SIGNAL PROCESSING, VOL. ASSP-30, NO, 3 , JUNE 1982 353

Enhancement of Sinusoids in Colored Noise and the White,ning Performance of Exact Least

Squares Predictors

Abstract-The extraction of sinusoids in white noise using least squares predictors has attracted a lot of attention in the past, mainiy in the con- text o€ the adaptive line enhancer (ALE). However, very few results exist for the practicd colored noise case or for the whitening perfor- mance of the predictors, We use a matrix formulation to derive the optimal least squares coefficients and frequency response of the D-step predictor or ALE for sinusoids (red or complex) in additive colared noise, Several cases are considered, and in particular, new formulas for the amplitude gain are obtained. In low-pass background noise, the amplitude gain of the sinusoids becomes essentially a monotonically increasing function of their frequency, and a decreasing function for high-pass noise. For the whitening application, signal-to-noise ratio {SNR) bounds of the output are derived when the input is a white signal plus a sinusoidal interference. We also give a state-space model and stochastic interpretations of our analysis of the D-step predictor, providing connections to other related areas. To enable filtering of nonstationary complex inputs, as well as multichannel and multi- experiment data, a complex vector version of the ladder algorithm is presented that can be used to implement the ALE, noise cancdling, and noise inversion for narrow-band interference rejection.

R I. INTRODUCTION

ECENTLY, there has been great interest in the problem of separating narrow-band and periodic signals from wide-

band signals or noise using least squares predictors. A consid- erable amount of work was based on gradient-type techniques, referred to as adaptive line enhancer (ALE) by.Widrow et al. [ 1 ] . The predictors can also be 'implemented by using exact least squares algorithms, such as the ladder algurithm of h e and Morf [ 2 ] , [I61 , [ 171 or other system identification algo- rithms [3]. This paper analyzes the steady-state behavior of the exact least squares predictors, which coincides with the optimal response of the ALE to sinusoids in additive colored noise.' The analysis is then complemented by a presentation of a complex vector version of the exact least squares ladder algorithm for prediction of nonstationary multichannel corn- plex signals +

The D-step predictor or the (prefiltered) ALE shown in Fig. I can be viewed as a special case of the joint process estimator [2] [3] or instrumental variable method .[4], where the reference input is a delayed (or suitably pxefdtered) version of

Manuscript received January 1& 1981; revised Nov'ember 3, 1981. This work was supported by the Defense Advanced Research Projects Agency under Contract MDA9U3-80-C-0331, and the Joint Services Etectronics Program under Contract DAAG29-79-C-004 7.

The authors .are with the Xnformation Systems Laboratory, Depart- ment of Electrical Engineering, Stanford University, Stanford, CA 94305.

the primary input dk. By properly choosing the delay D (or the prefilter), the wide-band components at the two inputs .tend t o become decorrelated, while the narrow-band compo- nents remain correlated. Since the coefficients of the trans- virsal filter are updated to minimize the error power, the predictor output yk attempts to track the narrow-band com- ponents of the input, while the residual error e& tracks, the wide-band component. Thus, the two outputs can be used for line enhancement and signal whitening, respectively. Almost all previous analyses for least .squares prediction of sinusoids in additive noise were limited to the white noise case only [ 11 , [SI -[lo] The main techniques used to analyze the corre- sponding Wiener-Hopf equation were the undetermined co- efficients method [8] and the use of the matrix inversion lemma; see,,e.g., [ 9 ] . The last methad was also used by Lacoss [ 1 l ] , to examine different spectral estimation methods for sinusoids in additive white noise. However, in most practical applications, such as a communication system where the signals are prefiltered, the output noise cannot be assumed to be white in general, and different results are expected to occur. Satorius and Zeidler [ 121 have considered a special case of this; where the noise can be modeled as a one-pole autoregres- sive (AR) noise. Using the undetermined coe€ficients method, they have evaluated the corresponding maximum entropy estimator. The one-pole AR background noise has also been independently, analyzed in [ 131 by using the matrix inversion lemma, providing new expressions for the amplitude gain of the predicted output sinusoids. This paper extends the results of [I33 h Section I1 we use the matrix formulation of [ 141 [ 15 ] to

obtain new expressions for the optimal least squares coefficients and frequency response of the D-step predictor with multiple complex sinusoids in general additive colored noise. Real sinu- soids or white noise'-'becbme special cases. The predictor behavior depends on one function of the observability Gramian investigated in the Appendix.

Section III applies the general results of Section X1 to special cases, such as the single complex sinusoid, real sinusoids, the one-pole AR noise, as well as general low-pass and high-pass noise. In particular, we derive new formulas for the amplitude gain of the output sinusoids as a function of their frequency and the colored noise covariance. With low-pass noise, the amplitude gain becomes essentially a monotonically increasing function of the frequency and a decreasing function for high- pass noise. This agrees with the results of the maximum en-

0096-35~8/~2~0600-0353$00.75 0 1982 ZEEE

354 XEEE TRANSACTIONS ON ACOUSTICS, SPEECH, AND SIGNAL PROCESSING, VOL. ASSP-SO, NO. 3 , J U N E 1982

d,= s p nk e k where T denotes transpose, H is the Hermitian transpose, and

(-) is the complex conjugate. o is the frequency and A , in ( l f ) will denote the amplitude of the mth complex sinusoid at the input and Cm for real sinusoids. Notice that while A , and Cm are scalars, all other capital letters are used for matrices. In this notation, the predictor output is

j r BROAD-BAND

OUTPUT PRIMARY INPUT

- t DELAY L-l REFERENCE

INPUT

i=o

Fig. 1 . Block diagram- of the D-step predictor or (preflltered) ALE. and i t s error ek = dk - y k .

OprimaE Coefficients Devivation To determine the optimal coefficients, assume that the input

consists of N complex sinusoids with additive zero-mean col- ored noise, Le.,

tropy estimator for the special case of one-pole AR noise found

Sectian IV derives an upper bound for the signal-to-noise ratio (SNR) of the residual output when used to filter an input consisting of a white signal that had been correlated by multiple stationary sinusoids.

In Section V we present the complex vector version of the exact least squares ladder algorithm that can be used to imple- ment the ALE: and noise canceller or inversion for narrowband interference rejection. This algorithm enables filtering of non- stationary complex inputs, as well as multichannel and multi- experiment data. The basic material on ladder forms can be found in 123 , [ 161 [ 171 . Fur the multiexperiment case, see also [ZS] +

The Appendix discusses the interpretations of the functions which determine the predictor response. A relationship to orthogonal polynomials provides an efficient method for the evaluation of these functions. The state-space model reveals the meaning of the predictor operation from the system theory point of view, and is also useful for further connections to this theory.

ill 1121.

m=l

where {p,} are independent and uniformly distributed over [O, Z ~ T ] and ylk is not necessarily white. The autocorrelation sequence of the input (3) is

m = l

where r,,(q) denotes the noise correlation.

Wiener-Hopf equation, The vector w , minimizes E { I ekI2}-hence, by the matrix

In OUT case, where the input is described by (3) and (4), the matrix R,, can be written as

11. MATRIX FORMULATION FOR LINE ENHANCEMENT I N CULORED NOISE

In this section, we derive the optimal coefficients, frequency response, and output signal of the D-step predictor or ALE for input of sinusoids with additive colored noise. The results here are based on [ 141 , [ 151 .

where R,, is the covariance matrix of n k . Applying the well- known matrix inversion lemma for (6 ) , then

where IL is the L X I, identity matrix and

To find th.e vector p , we assume that the delay D has been chosen correctly, Le., large enough to sufficiently decorrelate ylk, the wide-band component of the input. In this case, p includes only the sinusoidalpart given by

The optimal (complex) weight vector (5) can now be rewritten as

NEHORAI AND MORF: ENHANCEMENT OF SINUSOIDS 355

W , = RiL [IL - rR- 1 I' H p nn] r'uD to be solved in order to evaluate the frequency response (14) is the inversion of the N X N Hermitian matrix R. This, in gen- eral, . can be easiiy done (relative t o the original problem of

or using the defmition (8) of R , we finally get [ 141 inversion of R x x ) since N , the number of the complex sinu- soids, is much smaller than the fdter length L.

nitidn (13) or by using the alternative expression (A13). The Frequency Response second method is found from the relationship of $ to the

T h e matrix formulation can also be used to express the fie- reproducing kernels of orthogonal polynomials, and by the quency response of the predictor Christoffel-Darboux formula that appears in the Appendix.

While the definition (18) requires both matrix calculation and

W , =R-,l,rR-luD. (I2) The function $(ai, q ) - c a n be computed either by its defi-

L -1 H(w) = wiz = w;Eiy(o)e - (D+ i) -1 W D (1 3) the knowledge of Rik , the formula (A13) needs only scalar

eo z= e iw computations and the knowledge of the corresponding whiten-

and substituting (1 2) gives [ 141

The sinusoidal component at the predictor output is

m = l

where sk denotes the sinusoidal component of the data vector xk given by

m=l

where

ing filter of nk. This advantage will be demonstrated for the one-pole AR noise in the next section.

We remark that a function of a form similar to $(urn, a,) was used by Capon [28] for wavenumber analysis (see also [ 111, and note 'that these two functions would coincide if the signals were deterministic, which is different from our as- sumption). In particular, this implies that- our results on the relationship of to the reproducing kernels of orthogonal polynomials can be applied to efficiently compute that func- tion as well. It is also worth noting that the (complex) ladder algorithm can be very useful for this application since i t simul- taneously supplies all the whitening fdtters required for the computation of the corresponding formulas (A1 0) or (A14).

Finally, we note that another possible way to invert the matrix R,, is to use N rank one matrix updates. By repeated application of the matrix inversion lemma t o the summation in (6), we obtain the recursion

(1 7)

Remarks The complex signal notation which was used in this section

is a general framework that can be directly applied to real in- puts, as we shall show in the next section. We nate also that the same results can be obtained using different definitions of orthogonality between the complex sinusoids This indicates that the least squares solution is not very sensitive with respect to the type of orthogonality.

The results of this section show that the matrix FHRiiF, an obsexvability Gramian, plays an important role in the prediction behavior, as well as t h e following log-likelihood type variable (see the Appendix) :

I t is interesting to note now that the frequency response of the predictor is completely determined by this variable. To see this, observe that the entries of the vector y(w) which appear In (14) are $(q, a). The entries of R are obviously also functions of $ ( a i , c3l) by definition. Thus, it is clear that if the function $ were known, the only problem that remains

for Qo = R i i , and after N steps this gives the desired re-sult, Le., QN = R & , These and related techniques can be used to recursively derive bounds on the behavior of the predictor fur more complicated models as well.

111. SPECIAL CASES The results of Section I1 can now be applied to special cases.

Fur a single complex sinusoid, where N = 1. in (3), the com- plex predictor behaves differently from the real. Here R becomes a scalar and (12) yields

and the frequency response (14) becomes

Thus, if nk is a white noise, Le., J?;; = (l/oi)l, then

356 IEEE TRANSACTIONS ON ACOUSTICS, SPEECH, AND SIGNAL PROCESSING, VOL. ASSP-30, NO. 3 , JUNE 1982

pendent of D, in contrast to the real case [18] + We also note but we note that this result becomes exact for a single corn- here that a single complex sinusoid in white noise is a special plex sinusoid. For the particular input of real sinusoids, sub- case, since R,, has only one active eigenvector in the comer- stituting the relations (25)-(26) yields gence of adaptive algorithms; hence, gradient-type algorithms with a suitable step size are expected to converge as rapidly as I &? PmIl(om )LP the Newton type. This is not surprising, since in the white H2=l s y k - 1 f. pmT?(u&/2

cm cos (%& + cpm) (27)

Real Input sinusoidal component are defined by

Consider the input of the form Pm = n . 2 cm / C w

m=l Thus, each real sinusoid has amplitude gain given by

and therefore we can write for a white noise in [l] and [8] merely by introducing the likelihood weighting factor q(am>.' I t shows that amplitude

where is colored through the dependency of q(w,) on the sinusoid frequencies and the noise spectrum. (Note that q(wrn) = 1 for

The vector p is given by As already was mentioned in the Introduction, this case has

(24) also been considered by Satorius and Zeidler [ 121 who evalu- where ated the corresponding maximum entropy estimator. Our

analysis is different, as it is based on the matrix inversion lemma and the results of Sectiun 11. The matrix inversion

CMe j w P 1 T (24a) lemma was first used to analyze this case in [ 131 and then in [ 141 , [ 151 . We also recall that the analysis here assumes that D is chosen correctly, Le., large enough to sufficiently decorre- late F I ~ . This assumption was not made in [ 121. The results here will gve new expressions for tine amplitude gain.

Comparing (21)-(24) to the general notation of Section 11, we fmd not surprisingly that the real input can be solved as a special case of the complex by substituting

N = 2 M

The result that the complex predictor is specialized to the real could be anticipated by viewing the real sinusoids in (21) as a sum of 2M (orthogonal) complex sinusoids. From this point of view, the relations (25) become obvious. Note that the two complex components of each real sinusoid are ortho- gonal, although their initial phases are clearly related.

In the case of a single real sinusoid, observe that R is a 2 X 2 matrix that can be easily inverted.

Sinusoidal Transfer Characteristics When the input consists of more than one complex sinusoid,

the resulting output (1 7) includes coupling terns that compli- cate the analysis. However, if L is large and the interactions between different sinusoids can be neglected, (17) simplifies to

The one-pole AR noise can be described as the output of a system with white noise input and whose z transform is

1 H(z) = O < a < l .

1 - a2-l

If we assume that the variance of the white noise input is given bY

0; = ai(1 - a2), (3 1 4

then the correlation sequence of nk is

In urder to be able to apply the results of Section 11, we first have to compute the function $(ai, q). We shall do this by the two alternative methods mentioned in Section 11.

A similar generalization via a likelihood variable appears in exact least squares ladder forms in [ 21, [ 16 1 , [ 1'11 compared to approximate - gradient techniques. See also Section V.

NEHORAI AND MORF: ENHANCEMENT OF SINU-SOIDS 3 5 7

For the first method of computation of $, we use ,the where the RHS of the inequality refers to the amplitude gain in expression

1 -a s

Z+a2 -a

c 0 -a 1

, . the white noise case, Thus, low frequency sinusoids with fre- quencies a, such that w, < cos-’ a will have lower gain than if the noise were white. Fur -urn > COS-’ a, the inequality is reversed, and the higher frequency sinusoids will have higher gain than in the white noise case and relative to the low. fre- quency sinusoids. This qualitatively agrees with the previous results of [ 121 found for the corresponding maximum entropy estimator. Observe now that the boundary cos% is a decrea- sing function of a, and since the normalized cutoff angular

(32)

Now ,inserting (32) into (18) gives, for wi = c 3 1 = LS,, L

L $(%I 1 wrn) = o;(l - 2 ) , .

’ 11 - a2 + 2a(a- cos 0,) (1. - l / L > ] , ( 3 3 4

and for ai # c s l we get [ 141

where

The result (33b) can. also be obtained by the alternative ex- pression (A13), however, by a much simpler calculation. To see this, first note that the corresponding normalized whitening fdter of n k , Le+, the filter whose input is YE^ and its output is a unit variance white noise, is here

Substituting (34) into (A13) now yields

(34)

frequency of t h i s low-pass noise can be written as c3, = I - a, this implies that the boundary cos-’ a is an increasing function of &&*.

Fig. 2 compares the approximate amplitude gain (30) and (35) for each of the M sinusoids with the exact gain for a single real sinusoid in the same colored noise. We note that the ripple in the exact gain is due to the interactions between the two complex components df the real sinusoid. The figures verify that for larger L , the interaction becomes smaller, and that the amplitude gain in (30) approaches unity.

Finally, we note that the result (33b) for $(ai, q ) can be generalized to other colored noise cases again by applying the Christoffel-Darboux formula (A13). Note also that with this fact, a single (real ox complex) sinusoid in general AR noise is an example that can easily be solved by the formulas of Section 11.

.The resulting behavior of the amplitude gain found above for sinusoids in one-pole AR noise can be generalized to other low-pass noise and high-pass cases.

For general low-pass (high-pass) noise, it is shown in the Appendix that the likelihood variable q(c3,) is a monotonic increasing (decreasing) function of om. Combining this with the fact that a, is a monotonic increasing function of q(um), we find that in a low-pass (high-pass) background noise, the amplitude gain of the sinusoids is essentially a monotonic increasing (decreasing) function of their frequencies, as long

which is equivalent t o (33b). These results together with the ones in Section 11 can be used to derive- the complete behavior of the predictor for the one-pole AR mise [ 141 . However, here we will examine only the effect on the amplitude gain. The factor q(um) was found to be [ 1.31

This expression can be found by insertion of (33a) into (29). In addition to this, by noting from (35) that here q(wm) < 1 for om < cos-’ a, it w3s indicated in5[13] that

as the interaction terms are negligible. We note that usually this is the case since, in practice, L is very large compared to the inverse of the frequency distance between the input shu- soids, which. makes the coupling very small [cf. (A1 2)] . The same results hold also for complex inputs since ?(urn) would appear there as well, In p,articular, for the case of a single com- plex sinusoid .where the result (26) becomes exact.

The essential nonuniformity in the amplitude gains is ex- plained as follows. In order to minimize the error power, the predictor “prefers” the spectral components of t h e input signal in. special regions where the noise is weak, a fact that is well known for the normalizable Wiener filters and was shown here to be true also for the realizable predictors. The result (30) shows that this effect becomes more significant when the power of the separate spectral lines is too small compared to

358 IEEE TRANSACTIONS ON ACOUSTICS, SPEECH, AND SIGNAL PROCESSING, VOL. ASSP-30, NO. 3 , JUNE 1982

1 .OO

0.80 n

E n W

0.20 0.20 t

0.00 0

/-. 4J u

0.20

Fig. 2. The amplitude gain of the sinusoids as a function of their frequency, for one-pole AR low-pass noise. (a) Approximate gain for each of the real sinusoids by (30), (35). L = 15, p m = 0.2. (b) Exact

=: 15, p1 = 0.2. (cj, (a> Same as (a), gain for a single real sinusoid. E (b), but L = 40.

The output ek has three components: the desired wide-band component vk, its filtered version from the predictor output which is a distortion component, and the attenuated sinusoids. The last component, namely, the sinusoids, can’ be expressed either by applying our previous result (30) to this special case or from the derivations in [8] . Assuming that L is sufficiently large (L >> 2 7 ~ / ( q - al), see, e.g., [8]), we find that the sinu- soids at ek are given by

the total noise power, e.g., for a large number of sinusoids corrupted by colored noise and a low SNR.

A possible way to reduce the amplitude distortions that are introduced by the colored noise is to c.hoose a large value of L , which causes’ the amplitude gain in (30) to approach unity, or choose prefiiltering that enhances estimated signal bands.

IV. OPTIMAL WHITENING PERFORMANCE OF THE ALE AND EXACT LEAST SQUARES PREDICTORS

In this section, we find the approximated optimal SNR per- formance of the ALE and the exact least squares predictors in whitening signals. We assume here that the input includes a zero-mean white signal that has been correlated by N additive stationary sinusoids, Le .,

Iv

m=l

where urn was defined in (30), and here ~(cc),) = I since uk is white. Consider now the average power of ek . Noting that all the components of ek are uncorrelated, and using part of the results in [ 103 , we find that

N (37)

with autocorrelation sequence given by

C37a) Referring now to the second term (which is related to the dis- tortion of vk) as a part of the output noise, the overall SNR at ek (Le ,, the signal-to-noise-distortion-plus-interfere~c~ ratio) is found from (39) to be

m-5.

where 0; is the average power of uk, We wish to find the steady-state signal-to-noise ratio at the error output.

NEHORAI AND MORF: ENHANCEMENT OF SINUSOIDS 359

and if L >> 1 SO that a, = I for all m, (40) reduces to

This result i s similar to the gain of the line enhancement case [ 101 . It can be interpreted as the ratio uf the filter to the model order.

V. THE EXACT LEAST SQUARES LADDER ALGORITHM FOR COMPLEX D ~ T E P PREDICTOR

Extension of the exact least squares ladder algorithm for the D-step predictor or the ALE can be done either by viewing them as special cases of a joint process estimator 121 and then using the results which were derived, e.g,, in [17] or by direct extension of the one-step ladder predictor t o a D-step predic- tor* The extension to multichannel complex inputs i s simple, as it is easy to show that all the projection theorems in [ 171 hold in the complex case as well. I t i s also possible to extend the algorithm to the multiexperiment case (see [25]), where for each time point the data of several experiments are available, The data may not only be complex, but they may even have a signature in the same sense that the covariance matrix of some of the experiments is negative definite; t h i s arises, for instance, in the sliding covariance method presented in [20] I The algorithm below In its simplest form is the complex prewin- dowed case; however, using the results i n - [25] [17 J , it also specializes to the growing covariance, sliding covariance case, and via embedding techniques, to pole-zero or autoregressive moving average (ARMA) models, etc. We present the recur- sions which are obtained in this way for nonstationary complex PZ X rn matrix inputs where PZ is the number of channels and m is the number of experiments. In the case of a signature, we would have to postmultiply each n X pn matrix input by the square root of the signature, or alternatively, the complex calculus can be extended itself in order toinclude the signature.

t = O

Time update;

The recursions in (42), (44), and (45) are adapted from the recursions (45)-(50) in [23] . The additional recursions (43) are required fop implementation of the D-step predictor. The algorithm above minimizes, at each time k , the sum

(47) t=o

for p = O , 1, * 9 L - 1 .. The scalar h is an exponential weight- ing factor which enables tracking of nonstationary inputs (0 < X < 1). g is a likelihood variable that yields the optimized step size for fast convergence. (This vasiable was defined as? in [2] and [I61 [17] .)

The implementation of the algorithm is demonstrated in the lower part of Fig. 3. in the scalar case, the outputs $, (p =

ALE with lengths p + 1 Thus, k in Fig. 3 is equivalent to ek in Fig. 1 However, note that different lengths (up to pmax) of predictors become available in the same ladder implementa- tion. Note also that the ladder algorithm gives exact minimiza- tion of the error function in (47), based on all previous data.

The upper part of Fig. 3 i s for the D-step smoothed estimates of . {dk-2D-p}p=D Prn ax based' on the data [dk+ ' - 3 dk-D-J . The suitable ladder algorithm can be derived in a similar way to the predictor algorithm above, We note that ladder recur- sions fur the combined predictor and smoother can also be derived by using the embedding approach in 127. This kind of fitter was suggested in [24] for enhancement of speech signals from noise.

For optimal choice of the €orgetting €actor A, note that the recursion ( 4 b ) suggests that the expectation of the estimates & k and ROY of the covariance of the input dk are one-pole low-pass versions of its, true value. The cutoff frequency of this filter is ~3,~ = 1 - X. Hence, 1 - X has to be chose-n greater than the rate of changes in the second-order statistics of the input. Note that this description ' is true also for other feast squares algorithms with exponential windowing.

We also present the square-root normalized version of the

0, l ; 3 t - I ) are equivalent to the error outputs of the 9

1 f

360 IEEE TRANSACTIONS O N ACOUSTICS, SPEECH, AND SIGNAL PROCESSING, VOE. ASSP-30, NO. 3, JUNE 1982

s S

r 1,-1 , k-n

Fig. 3. Ladder farm for multichannel D-step predictor and smoother.

previous algurithrn, which is numerically superior to the un- We note that the implementation of these equations can be normalized one. done efficiently using orthogonal transformations or rotations Squure-Rout Normalized Algurithrn (see [ 173, [26] ) or in VLSI implementation [27] ,

(4W

(48b)

VI. CONCLU SIUNS

The matrix formulation was used to derive the uptirnal co- efficients and frequency response of the exact least squares

K D - p O p = l ; * * Y Pmax* (484 predictors and the ALE for inputs of real or complex sinusoids '- 7 ~

o is a prior covariance estimate of the input dk. If no informa- tion is available on dk , o has to be chosen small, but greater than zero.

2) Main Loop: At time k we have available

and the new data d k

a) Set

In the next recursions, we define for an arbitrary matrix X the complements

xc = ( I - xx N ) 1/2

x-c = ( I - xx N ) -112

x-cH = ( I - xx H ) --HI2

This notation is simply extended to the matrix X H , Le., X HC , xHCH 3 etc.

in general additive colored noise. The derivation of the ampli- tude gains of the output sinusoids generalizes known results for the white noise case. The obtained expressions describe the dependency of the sinusoids amplitude gain on their fre- quencies and the noise covariance. In low-pass and high-pass background noise, the amplitude gains become esientially monotonically increasing and decreasing functions, respectively, of the sinusoid frequencies. It should be noted that expression (A14), utilized to prove these properties, indicates also that a similar behavior occurs in different colored noise cases (e.g., band-limited type). Specifically, the amplitude becomes smaller-in frequency regions where the noise is stronger. The amplitude distortions that are introduced by the correlated noise can be reduced by choosing a larger number of coeffi- cients L , the size of the averaging window, or by using suitable prefiltering. In evaluation of the performance of the predictors in whitening applications, an approximated upper bound on the output SNR was found when filtering a white signal that correlated by multiple stationary sinusoids,

The state-space model and other interpretations of the pre- dictor behavior that appear in the Appendix provide a few connections between our analysis and other areas. These con- nections particularly give an alternative furmula useful for efficient computation of the underlying variable $, as was demonstrated for the one-pole AIR background noise.

The complex vector version of the exact least squares ladder algorithm in Section V can be used t o implement the complex ALE, as well as the noise canceller or noise inversion [Z] for narrowband interference rejection. A major advantage of tlvs algorithm is its rapid rate of convergence since it computes the exact least squares solution; hence, it can actually achieve the upper bounds of the SNR performance that were found in Section IV and [ 101 for stationary inputs.

NEHORAI AND MORF: .ENHANCEMENT OF SINUSOIDS 361

APPENDIX STATE-SPACE AND STOCHASTIC INTERPRETATIONS

The purpose of the Appendix is to explore the related func- tions PHR&r, I), and 7 which determine the predictor .be- havior as shown in Section TI, We give different interpretations of these functions in the context of our analysis. The state- space model interpretation will reveal the meaning of I' as well as the predictor operation from the system theory point of view, hence- leading to further insight to this analysis. It can also be used for additional connections to system theory. The likelihood variable interpretation of ' + and the relation to reproducing kernels (see also [ 151) will enable us to evaluate the frequency dependence, of this function as well a's the func- tion 77, The connection to reproducing kernels will also partic- ularly provide the efficient alternative formula for the compu- tation of $L Finally, we point out that the matrix I? has the role of block eigenspace of R,, when nk is white.

State-Space Model Consider the general state-space model

where u k and rzk are assumed to be zero-rnean orthogonal processes with covariance matrices R,, and Q, respectively. For this model, it is well known that

where no = E{&&} and the observability matrix 8 is given 'OY

= [HT, FTHT, - - . 3 ( F T ) L - i H T ] Tm (A3)

T is a lower triangular ToepZitz matrix, with [TI i , j = HF~+' G if i > j , else 0.

Now we turn t o the D-step predictor. Here, the input signal dk (3) can also be described by the general state-space model (AI) with #k = 0 and

H = [ l , 1, 9 ; 11 a 1 X N row vector,

With (A3), we find that the normalized observability matrix of the above sinusoidal model with additive colored noise is given by

This means that r [defined in (Ig)] is the observability matrix for this signal, with each column multiplied by the amplitude Ai. - Note also that the observability matrix here is written explicitly as

The above statmpace model can be used to explore further connections to system theory. We do not pursue this matter here, but just note the similarity between (6) and (A2) where the second term in (A2) vanishes since #k = 0, Thus, the opera- tion of the D-step predictor ox ALE can be interpreted as an estimation of the sum of the states, Le., of all sinusoids.

Thus, the log-likelihood function is proportional to

ZI = In IRnnI + j ( U ] I R - l IA8) nn

where u = [uo , m . 0 , u T ] . The variables $(w, ? urn) and v(wm) as defined in (18) and

(29), respectively, can be seen to be proportional to the second term of the log likelihood (AS) for u = r(urn). This implies that they give a measure of ,the mismatch between a finite length data sample of a sinewave of frequency w, and the process nk. Hence, 'if nk i s a low-pass (high-pass) noise, these functions are expected to be increasing (decreasing) with the frequency a,. This fact will also be proved more rigorously by the next interpretation. The more general function $(wi, wd) can be interpreted as a cross-likelihood function between two short samples of sinusoids of different frequencies wi and c 3 1 with respect to nk.

n

Relution to Reproducing Kernels

[ 2 2 ] ) , the reproducing kernels can be defined by In the theory of orthogonal polynomials (see, e.g., [21] ,

These functions are also related to the Cholesky factorization and Christoffel-Darboux formulas [22] . The kernel can be also written as

p=o

where a&) has the interpretation of the z transform of the pth-order ' least squares (normalized) whitening filter of the process nk

Comparing (A9) to our definition (H), we find that

or using (AlO),

p=Q

where e P ( e jW') is the frequency response of the pth-order whitening filter of f lk at the frequency ai, and similarly for

QP(eiwI). Thus, $(mi, 01) is a linear combination of cross spectra of the whitening filters of nk .

From the Christoffel-Darboux formulas [22], we can also write

where we used the stationarity assumption for nk. For mi = q E a,, we find from (29) and ( A H ) that

p =Q

hence, q(wy,) is a normalized sum of the whitening spectra of fik in the frequency a,.

Consider now the case where nk is a low-pass (high-pass) sig- nal and its spectrum is a monotonically decreasing (increasing) of the frequency. In this case, each of the spectral terms 6‘,(eiW“)$(e-iwm) that appeal: in (A14) becomes an increas- ing (decreasing) function of c3, by definition of the whitening filters. Hence, ~ ( w , ) and $(urn, u,) are also increasing (decreasing) with w, .

Eigenspace Interpretation For completeness of the discussion on the possible interpre-

tations of the functions which determine the predictor response, we point out that the matrix I‘ has the function of block eigenspace of R,, when nk is white. This can be seen by noting that in this case the column space of the Hermitian matrix R,, [cf. (6)] is spanned by the columns of f. This, in particular, implies that for a single complex sinewave in white noise, the vector y(a I ) becomes an eigenvector of R,, . This result could also be obtained by observing that here R,, is a circulant matrix or by direct substitution [ 131 ,

REFERENCES [ 3 1 €3. Widrow et ai., “Adaptive noise cancelling: Principles and

app2icatians,”Pruc. E E E , vol, 63, pp. 1692-1716, Dee. 1975. [ Z J M. Morf and D.T.L. Lee, ‘Xecursive least-squares ladder forms

for fast parameter tracking,” in Proc. I7 th IEEE Conf. Decision Confr., San Diego, CA, Jan, 1979, pp. 1326-136 7.

[ 31 B. Friedlander, “System identification approach for adaptive noise cancelling,” IEEE Trans. A c o t l s ~ , Speech, SignalPmcessing, to be published.

[ 41 K. Y. Wong and E. Polak, “Identification of linear discrete time system using the instrumental variable method,” IEEE Trans. Automat. Contr., v d AC-12, no. 6 , pp. 707-708, 1967.

[ 5 ] L. J. Griffiths, “Rapid measurement of digital instantaneous frequency,” IEEE Trans. Acoust., Speech, Signal Processing, vol. ASSf-23, pp. 207-222, Apr. 1975.

[ 6 1 J . R. Treichler, “The spectral line enhancer-the concept, an implementation, and an application,” Ph.D. dissertation, Stan- ford Univ., Stanford, CA, June 1977.

enhancer,” IEEE Trans. Acousf., Speech, Signal Processing, vol. A S P - 2 7 , pp. 53-62, Feb. 1979.

181 J . R. Zeidler, E. H. Satorious, x). M. Chabries, and €3. T. Wexlew, “Adaptive enhancement of multiple sinusoids in uncorrelated noise,” IEEE Trms. Acoust., Speech, Signal Processing, vol. A S P - 2 6 , pp. 240-254, June 3978.

[71 - 1 “Transient and convergent behavior of the adaptive line

[ 9 ] F. W. Symons, “Narrow-band interference rejection using the complex linear prediction filter,” IEEE Trans. Acoust. , Speech, Signal Processing, vol. ASP-26 , pp. 94-98, Feb. 1978.

[ 101 A. Nehorai and I>. Malah, “On the stability and performance of the adaptive line enhancer,” in Roc . IEEE In[ . Conf. Acoust., Speech, Signal Processitzg, Denver, CU, Apr. 1980, pp. 478-481.

11 1 1 R. T. Lacoss, “Data adaptive spectral analysis methods,” Geo-

[ 121 E. €3. Satarius and .I. R. Zeidler, “Maximum entropy spectral analysis of multiple sinusoids in noise,” Geophysics, vol. 43, pp. 1111-111.8, Oct. 3978.

[ 13 1 A. Nehorai, “Adaptive filtering of speech signals from noise” (in Hebrew), M.S. thesis, Dep. Elec. Eng., Technion-IXT, Aug. 1979.

order colored noise,” Stanford EE 365 term paper, Spring 1980. [ 151 A. Nehorai and M. Morf, “Enhancement of sinusoids in colored

noise and the whitening performance of exact least-squares pre- dictors,” in Pvoc. IEEE lnt. ConJ Acoust., Speech, Signal Pro- cessing, Atlanta, GA, Mar. 3 98 1.

[ 161 M. Morf, A. Vieira, and De T. L* Lee, ‘‘Canonical ladder forms for identification and speech processing,’’ in Proc. IEEE Con$ Deci- sion ConfP., New Orleans, LA, Dec. 1977, pp. 1074-1078.

[ 171 D.T.L. Lee, M. Morf, and B. Friedlander, “Recursive least squares ladder estimation algorithms,” IEEE Trans. Circuits Syst. V O ~ .

CAS-28, pp. 627-64 1, June 1981. See also D. Lee, “Ladder form realizations af fast algorithms in estimation,” PhB. disser- tation, Dep. Elec. Eng., Stanford Univ., Stanford, CA, Aug. 1980.

[ 181 V. U. Reddy, B. Egardt, and T. Kailath, “‘Optimized lattice-form adaptive line enhancer for a sinusoidal signal in broad band noise,” IEEE Trans. Circuits Syst.? vol. CAS-28, pp. 702-709, June 1981.

[ 191 L. B. Jackson, D. W. Tufts, F- K. Soong, and R. M. Rao, “Fre- quency estimation by linear prediction,” in Proc. IEEE In t. Cmfi Acousf., Speech, Signal Processing, Tulsa, OK, Apr. 1978.

I201 €3. Porat, B. Friedlander, and M. Morf, “Square root covariance ladder algorithm,”IEEEE Tmzs. Automat. Contr. ? to be published.

[ 21 ] L. Y. Geronimus, Orthogonal Polynomials. New Yoxk: Consul- tant’s Bureau, 196 1 *

[ 221 T. Kailath, A. Vieira, and M. Morf, ‘‘Inverses of Toeplitz opera- tors, innovations and orthogonal polynomials,” SIAM Rev., no.

[ 23 1 B. Friedlander, ‘‘Recursive algorithms far all-pole ladder forms,’’ Systems Control, Inc., Palo Alto, CA, Tech. Rep. TM 5334-02, Jan. 1980,

[ 24 ] M. R. Sambur, “Adaptive noise canceling for speech signals,” IEEE Trans. Acoust., Speech, Signal Processing, vol. A S P - 2 6 ,

[ 251 B. Porat, D. Morgan, and M. Morf, “On the relationships among several square-root ladder algorithms,” presented at the Johns Wopkins Conf., 198 1 i

[26] M. Morf, C. Muravchik, and D* T. Lee, “Hilbert space array methods for finite rank process estimation and ladder realizations far speech and adaptive signal processing,” in Pmc. I 9 8 1 IEEE C m f . Acoust., Speech, Signal Processing, Atlanta, GA, Apr. 1981.

[ 27 ] €3. M. Ahmed, M. Moxf, D.T.L. Lee, and P.-H. Ang, “A VLSI speech analysis chip set based an square-root normalized ladder forms,” in Pmc. 1981 IEEE h t . Cmf : Acuusb., Speech, Signa! Processing, Atlanta, GA, Apr. 1 981.

[ 28 J J . Capon, “High-resolution frequency-wa\:e-number spectrum analysis,” Proc. IEEEE, vola 57, pp. 1408-1418, Aug. 1969.

physics, V O ~ . 36, pp. 661-675, Aug. 1971.

1141 - 1 “The behavior of the ALE with N complex sinusoids in first

20, pp. 106-107, 1978.

pp. 419-423, Oct. 1978.

Research Assistant in the Information System Laboratory at Stanford. His current research interests are in the areas of adaptive signal process- ing and system identification.

IEEE 'TRANSACTIONS ON ACOUSTICS, SPEECH, AND SIGNAL PROCESSING, VOL. ASSP-30, NO. 3, JUNE 1982 363

time data and signal processing, with emphasis on the structure of multivariable and multidimensional systems, estimation, identification, control, and efficient computation algorithms and architectures for VLSI. He is interested in the application of computers to speech processing, large-scale systems, image ' processing and reconstruction, and biomedical applications such as ultrasonic image processing and biological systems modeling His current research work focuses on algorithms and architectures for the design of VLSI: for real-time processing, as part of his involvement in the new Center for Integrated Systems at Stanford University.

Dr. Moxf is Associate Editor of the journal Circuits, Systems, and Signal Processirag, published by Birkhauser Verlag, and is a member of Sigma Xi, ACM, and S A M .

Recursive Processing by Noncausal Digital Filters ROLAND CZARNACH

Abstract-Digital filters having exactly linear phase are usually realized by nonrecursive systems. However, the fiiter degree, and thus the ex- pense, becomes rather large in the case of steep fiiters with high at- tenuation in the stopband.

This paper deals with an alternative possibility using noncausal recur- sive systems After known methods are generalized, suitable algorithms are proposed and systematic errors are investigated. Finally, the method is compared with nonrecuxsive filtering.

A I. INTRODUCTION

S is well known, digital Filtering with a system having ex- actly linear phase cannot only be done by using an ap-

propriately designed finite-duration impulse-response (FIR) filter, but by a recursive filter as well, if the sequence to be processed is of finite length. For this purpose, the sequence has to be processed twice, either in a parallel or cascade con- nection of identical systems (see Fig. 1). The key to the solu- tion is that sequences of finite length can be processed in rer versed order. With this possibility, it is clear from Fig. 1 that the noncausal overall transfer function will ideally be F(z) = G(z) + G( l/z) if the parallel connection is used, and F(z) = H(z)H( l/z) in the case of the cascade connection. For z - dn, the frequency responses will be real.

In practice, the zero phase of the system .will change to a linear phase due to the necessary processing time. But since this time delay is inconsequential to' the presented results, it is not considered in the sequel.

The described method is known from image processing. The one-dimensional case was considered in [ I ] , for example. But since the type af transfer functions which can be imple- mented is restricted, and since there are some open questions

-

Manuscript received May 11, 198 1 ; revised November 16, 198 1. The author is with the Lehrstuhl fur Nachrichtentechnik, Universitat

Eriangen-Niixnberg, Erlangen, West Germany.

concerning the approximative nature of the solution, the problem is reconsidered here.

From the realization in Fig. l(b), it is obvious that in this case, impulse responses must have the properties of an auto- correlation function. In order 'to overcome this restriction, we generalize the problem by replacing the identical blocks having the transfer functions G(z) or H(z) by different filters, which now have the transfer functions G I (z), Gz (z) or H I (z), Hz (z) respectively (see Fig, 2). Thus, any noncatlsal overall transfer functions F(z) = GI (2) + G2 (I /z) or F(z) = HI (2) H2 (1 /z) can be implemented, including the special cases HI ( 2 ) = Hz (2) = H(z) and GI (z) = G2 (2) = G(z). In Section 11, it is shown how G I , (2) and H I t 2(z) can be determined from requirements in the time or frequency domain.

The following sections deal with the realization of the method. In Section 111, it is shown that the output sequence y(k) can be calculated exactly if the input sequence u(k) is of finite length and if only a finite number of the output samples y(k) is of interest. While this is obvious in the case of the parallel connection, the cascaded subsystems require a modi- fied procedure, which will be explained. The above methods can also be applied to infinite length input sequences if u(k) is processed in segments. In this case, an unavoidable inherent error results from the noncausal filter part. In Section IV, an efficient overlap-save algorithm is presented, and the xe- rnaining error is investigated. The method is similar t o that proposed in [ I ] but differ,s in detail in order to achieve higher computational efficiency and a reduction of the errors. Fi- nally, the proposed method is compared to the usual non- recursive fdtering.

r1. DESIGN OF THE SUBSYSTEMS

While the design of causal recursive filters is well understood ( e g , [3]), there are some open questions concerning the de-

0096-3518/82~0600-03~3$00.75 0 1982 IEEE