Regularized algorithm for Raman lidar data processing

Regularized algorithm for Raman lidar data processing

Valery Shcherbakov1,2,*1OPGC, Laboratoire de Météorologie Physique UMR 6016 CNRS, Université Blaise Pascal, 24

Avenue des Landais F-63177, Aubière, France2Institute of Physics, 66 Independence Avenue, Minsk BY-220072, Belarus

*Corresponding author: [email protected]

Received 4 October 2006; revised 19 April 2007; accepted 5 May 2007;posted 7 May 2007 (Doc. ID 75791); published 3 July 2007

A regularized algorithm that has the potential to improve the quality of Raman lidar data processing ispresented. Compared to the conventional scheme, the proposed algorithm has the advantage, whichresults from the fact that it is based on a well-posed procedure. That is, the profile of the aerosolbackscatter coefficient is computed directly, using the explicit relationships, without numerical differ-entiation. Thereafter, the profile of the lidar ratio is retrieved as a regularized solution of a first-kindVolterra integral equation. Once these two steps have been completed, the profile of the aerosol extinctioncoefficient is computed by a straightforward multiplication. The numerical simulations demonstratedthat the proposed algorithm provides good accuracy and resolution of aerosol profile retrievals. The erroranalysis showed that the retrieved profiles are continuous functions of the measurement errors and of thea priori information uncertainties. © 2007 Optical Society of America

OCIS codes: 010.3640, 280.1100, 280.3640, 290.5860, 290.1350, 290.2200.

1. Introduction

Raman lidars became well-established tools that areused in the study of numerous areas of importance inthe atmospheric sciences (see [1] and referencestherein). In particular, the use of combined Ramanelastic-backscatter lidar permits the independentmeasurement of aerosol extinction and backscatterprofiles [2–4]. Numerous Raman lidars performroutine measurements within networks, e.g., theNetwork for the Detection of Stratospheric Change(NDSC) [5], the European Aerosol Research LidarNetwork (EARLINET) [6], and the Asian dust net-work (AD-Net) [7].

In regard to the aerosol profiles, Raman lidardata processing, as a rule, follows the scheme ofAnsmann et al. [2–4], hereafter called the conven-tional scheme–algorithm. According to that approach,a range-resolved extinction coefficient is retrieved firstfrom a nitrogen or oxygen Raman signal. The key andmost critical element is the need to compute the de-rivative of the logarithm of the ratio between the at-mospheric number density and the range-corrected

lidar-received power [8]. Thereafter profiles of theaerosol backscatter coefficient and the aerosolextinction-to-backscatter ratio, hereafter called the li-dar ratio, are evaluated using both the elastic andthe Raman backscattered signals together with theretrieved profile of the extinction coefficient. Thedistinction between algorithms mainly concernsdata-smoothing techniques and the evaluation of anumerical derivative [8,9]. The efficiency of the con-ventional scheme is evidenced by the practice. Inaddition, the blind intercomparison of Raman lidaralgorithms showed satisfactory results [8].

On the other hand, the problem of numerical dif-ferentiation is known to be ill-posed in the sense thatsmall perturbations in the function to be differenti-ated may lead to large errors in the computed deriv-ative [10]. Correspondingly, much care is needed indata-averaging and -handling operations when onedeals with Raman lidar data [8]. In particular, thisplaces more stringent requirements upon the datasmoothing, which in turn decreases the spatial reso-lution and affects the retrieved profiles of the aerosolbackscatter coefficient and the lidar ratio.

The aim of this paper is to present a flexible algo-rithm, which is believed to improve the quality of theRaman lidar data processing. The basic equations

0003-6935/07/224879-11$15.00/0© 2007 Optical Society of America

1 August 2007 � Vol. 46, No. 22 � APPLIED OPTICS 4879

and the a priori assumptions are recalled first. Thetwo milestones of the proposed algorithm are out-lined in Subsections 2.B and 2.C. Thereafter the re-sults of the simulations are described. Section 4 isdevoted to error analysis.

2. Algorithm

A. Basic Equations and Assumptions

The single-scattering elastic and Raman lidar equa-tions for the background-subtracted power can beexpressed as follows [1,3]:

P0�z� � K0

Q0�z�z2 ��M0�z� � �a0�z��exp��2�

0

z

��M0�z��

� �a0�z��dz��, (1)

PX�z� � KX

QX�z�z2 NX�z��d�X0��d��exp��

0

z

��M0�z��

� �a0�z�� MX�z�� aX�z��dz��, (2)

where the subscripts 0 and X relate the parametersto the laser wavelength 0 and the Raman wave-length X, respectively. The wavelength X has beenshifted from the laser wavelength 0 owing to inelas-tic Raman scattering by molecular species X. P0�z�and PX�z� are the powers (after any background con-tribution is subtracted) received from distance z. K0and KX contain all depth-independent system param-eters. Q0�z� and QX�z� are the overlap functions thatare unity for distances greater than zmin where thefield of view of the receiver completely overlaps thelaser beam. NX�z� is the number density of molecularspecies X that is being excited, and d�X0��d� isthe pertinent range-independent differential Ramancross section for the backward direction. �M0�z� and�a0�z� are the backscatter coefficients that are dueto Rayleigh and particle scattering. The coefficients�M0,MX�z� and �a0,aX�z� describe the extinction that isdue to absorption and Rayleigh scattering by atmo-spheric gases and aerosol extinction, respectively.

It is assumed that the two signals P0�z� and PX�z�are used to retrieve two aerosol profiles, i.e., �a0�z�and �a0�z�. Accordingly, other parameters of Eqs. (1)and (2) must be supplemented with a priori as-sumptions and�or data. The a priori assumptions,proposed in [3], are proved by the practice to beconvenient and fruitful. They are applied in thiswork and, for completeness’s sake, outlined in thefollowing.

Air density as a function of range z is determinedfrom actual radiosonde data of temperature and pres-

sure, if available, or from a standard atmospheremodel fitted to measured ground-level temperatureand pressure values. Consequently, there can be ob-tained the Rayleigh scattering profiles �M0�z�, �M0�z�,and �MX�z�, as well as the number density NX�z� ofmolecular species X with known percentage in air(e.g., nitrogen or oxygen).

The particle extinction is assumed to be propor-tional to �k, that is,

�aX�z� � �a0�z�0

Xk

� �a0�z�, (3)

where � �0�X�k. The Ångström coefficient k is aparameter, which is considered to be constant over arange interval. Its value has to be guessed or one canuse external data such as from a Sun photometer [11]to get a good estimate. It is common to choose kconsidering the origin of particulate matter. For in-stance, k � 0 corresponds to large dust or cirrusparticles, k � 2 corresponds to small smoke particles[11]. The overlap functions Q0�z� and QX�z� have to bemeasured beforehand [12] or the retrieval has to beperformed only within the range where these func-tions are unity.

It is often convenient to analyze lidar aerosol datain terms of the backscatter ratio R0�z� [13], which isdefined as the ratio of the total backscatter to themolecular backscatter R0�z� � ��a0�z� � �M0�z��M0�z�. It is equal to unity in a region of the atmo-sphere that is free of aerosols and exceeds unitywhere aerosols are present. An a priori value R0�zref�at a reference distance zref is most commonly used inlidar data processing algorithms to normalize the sig-nals and eliminate the depth-independent system pa-rameters [3,13] (in the considered case K0 and KX). Asa rule, zref is assigned within the distance intervalwhere the backscatter ratio is as close to unity aspossible, i.e., such that at zref the particle backscattercoefficient is negligible compared to the known mo-lecular backscatter value [3]. Such clear-air condi-tions are normally given in the upper troposphere forlaser wavelengths �700 nm [3].

Equations (1) and (2), along with the listed as-sumptions, are a closed system of equations with re-spect to the profiles �a0�z� and �a0�z�. An algorithm ofnumerical solution to the system is described in Sub-section 2.B. Errors that are due to the adopted as-sumptions and to measurement errors are addressedin Section 4.

B. Aerosol Backscatter Profile

The conventional scheme starts from the retrieval of�a0�z� on the base of Eq. (2) and the listed aboveassumptions. As was mentioned, this is associatedwith numerical differentiation of, broadly speaking, ameasured function. That is, one needs to solve anill-posed problem [10]. Consequently, the retrievedprofile is very sensitive to measurement errors. More-over, that high sensitivity in turn decreases the spa-tial resolution. Because of such features, if it is

4880 APPLIED OPTICS � Vol. 46, No. 22 � 1 August 2007

possible, a solution to an ill-posed problem should notbe used as the base of the subsequent data process-ing.

Accordingly, the proposed approach starts with theretrieval of the aerosol backscatter profile �a0�z�. Thisis the first milestone of the algorithm.

Hereafter, the following functions will be applied:

S0�z, zref� �P0�z�z2

P0�zref�zref2

Q0�zref�Q0�z�

R0�zref��M0�zref�

� exp��2�z

zref

�M0�z��dz� , (4)

SX�z, zref� �PX�z�z2

PX�zref�zref2

QX�zref�QX�z�

NX�zref�NX�z�

� exp��z

zref

��M0�z�� MX�z��dz��. (5)

These functions are the range- and overlap-correctedsignals normalized to the value at the reference dis-tance zref, and corrected for the functions of the airdensity. The a priori value of the backscatter ratioR0�zref� is used as well. There are no unknown param-eters or functions in the right-hand side of Eqs. (4)and (5), i.e., S0�z, zref� and SX�z, zref� can be computeddirectly.

It immediately follows from Eqs. (4) and (1) that

S0�z, zref� � ��M0�z� � �a0�z��exp�2�z

zref

�a0�z��dz� , (6)

and from Eqs. (2), (3), and (5) that

SX�z, zref� � exp��1 � ��z

zref

�a0�z��dz� . (7)

Taken together, Eqs. (6) and (7) lead to

�a0�z� �S0�z, zref�

�SX�z, zref��2��1�� M0�z�. (8)

The features of the explicit relationships [Eqs. (4),(5), and (8)] merit an extended discussion. It is seenthat the aerosol backscatter profile can be calculateddirectly from the Raman lidar signals, using the sameassumption as in the conventional algorithm, withoutthe retrieval of the aerosol extinction or the lidarratio. Another important point is that Eqs. (4), (5),and (8) do not require a solution to an ill-posed prob-lem. Consequently, compared to the numerical differ-

entiation, the sensitivity to measurement errorsshould be lower and the spatial resolution should behigher, all other factors being the same.

At any distance z, the computed value of �a0�z�depends only on the lidar signals at z and zref, and theair density within the interval �z, zref�. It means thatthe aerosol backscatter profile can be evaluated atany chosen interval or several intervals without ref-erence to exterior z. Moreover, if there are somegrounds, e.g., aerosol layers having distinct features,different values of the Ångström coefficient can beassigned to different intervals. The reference dis-tance can be assigned anywhere within the range ofinterest considering only errors in the lidar signals atzref and in the a priori value R0�zref�. There are nostrong nonlinear effects such as in algorithms for asingle-wavelength lidar [14].

The last part of this subsection is devoted to somerecommendations for applying Eqs. (4), (5), and (8) torecorded Raman lidar signals, i.e., real experimentaldata. In computing terms, the important feature oflidar signals is the steep decrease of the signal-to-noise ratio (SNR) with distance [15]. It means thefollowing. (i) Even if the SNR is satisfying at the nearend of the measurement range, as a rule, smoothingshould be evaluated somewhere at the far end. (ii) Itshould be range dependent, i.e., the higher z is, thestronger the smoothing.

When necessary, the smoothing should be appliedto the range-corrected signals. If a raw signal issmoothed, the computed curve mostly follows the fac-tor 1�z2, which overlaps the variations that are due tothe aerosol profile. This feature is most pronounced atthe far end. Accordingly, it is common practice tosmooth range-corrected signals and not raw signals(see, e.g., [9]). There exists a large variety of data-smoothing methods. Each of them has some advan-tages and disadvantages. The comparative analysisof the methods is beyond the scope of the presentwork. Weighted smoothing splines [16,17] (see Ap-pendix A) are well-adapted to the particularities ofreal lidar signals (range corrected). With weightedsplines, it is easy to obtain a range-dependentsmoothing and to adjust its degree. Moreover, therange dependence can rely directly on the standarddeviation of a lidar signal by means of a table or ananalytical function. Of course, the proposed algo-rithm does not impose restrictions and other data-smoothing methods can be used.

C. Profiles of Lidar Ratio and Extinction Coefficient

In the following, the aerosol optical thickness

a0�z, zref� � �z

zref

�a0�z��dz� (9)

between z (variable) and zref at the wavelength 0 isconsidered to be known. It can be calculated directlystarting from Eq. (7),


a0�z, zref� �1

1 � ln�SX�z, zref��, (10)

or Eq. (6) [having known �a0�z�],

a0�z, zref� �12 ln� S0�z, zref�

�M0�z� � �a0�z��. (11)

Equations (10) and (11) are mathematically equiva-lent, and they give the same results.

Consequently, the aerosol extinction profile can beretrieved. This can be done by the numerical deriva-tive means (as in the conventional scheme) or apply-ing regularization techniques (as it is recommendedby the inverse problems theory [10]). One shouldmeet the above-discussed difficulties when �a0�z� isretrieved.

The computing of the lidar ratio qa0�z� prior to �a0�z�is found to be more advantageous. This is the secondmilestone of the proposed algorithm. The calculationof qa0�z� is based on the equation

a0�z, zref� � �z

zref

qa0�z��a0�z��dz�, (12)

which follows from Eq. (9) along with the relationship�a0�z� � qa0�z��a0�z�.

With given zref, Eqs. (12) and (9) are a first-kindVolterra integral equation with a kernel that does notdepend on z. Compared to the numerical differentia-tion, a regularized solution of the integral equationoffers several advantages. Among other things, themost important for the atmospheric optics feature isthat the aerosol optical thickness is less affected. Inother words, the integral of retrieved extinction pro-files is closest to a0�z, zref� throughout the interval ofinterest.

The method of regularization employs an a priorismoothness constraint on retrieved characteristics(profiles) [10]. Generally, the better the constraintcorresponds to the features of the retrieved profile,the lower the retrieval errors. The aerosol extinctioncoefficient can vary by orders of magnitude through-out a distance interval, whereas the lidar ratio varieswithin a limited range. Accordingly, a priori smooth-ness constraints (such as a constraint on derivatives)are much more suitable to qa0�z� than to �a0�z�. Thatis the reason why not Eq. (9) but Eq. (12) is used inthe proposed algorithm. The flexibility of the algo-rithm lies in the fact that it provides a way to applywell-developed capabilities of the inverse problemtheory to retrieve a profile of the lidar ratio, i.e., tosolve Eq. (12).

The next part of this subsection is devoted to theapproach used to compute qa0�z�. The trapezoidal in-tegration rule approximates the integral equation(12) by the linear system, i.e., by the matrix equation(see, e.g., [18])

Bq � �, (13)

where the M � M matrix B is upper triangular, withzero entries below the diagonal. The range depen-dence of the measurement errors should also be con-sidered. Therefore, the regularized solution to Eq.(13) is of the form [19]

q � �BTW�1B � �HTH��1BTW�1�, (14)

where T denotes matrix transposition, � is the regu-larization parameter (the Lagrange multiplier), H isthe nth difference matrix [18,19], and W is the co-variance matrix of the vector �. It is seen from Eq.(10) that W corresponds to the function SX�z, zref� andits diagonal elements correlate with measurementerrors of the signal at the Raman wavelength. In thisstudy, we used the a priori smoothness constraint onprofiles of the lidar ratio through the matrix

H ��1 �2 1 0 0 · · · 00 1 �2 1 0 · · · 00 0 1 �2 1 · · · 0É É Ì Ì Ì

0 0 1 �2 1�, (15)

i.e., H is the �M � 1� � M second-difference matrix. Inthe inverse problem theory, the Lagrange multiplier� is defined as a nonnegative parameter that controlsthe strength of a priori constraints relative to thecontribution of the measurements. That is, the higher�, the smoother the retrieved profile qa0�z�. When� → �, the solution to Eq. (14) is qa0�z� � const. Thereexists a variety of methods to evaluate the value of �(see, e.g., recipes and discussions in [19]). In thisstudy, the regularization parameter was estimatedby means of the L-curve method [18–20]. Havingretrieved the profiles of the lidar ratio and the aerosolbackscatter, one can evaluate the range-dependentextinction coefficient just by the multiplication�a0�z� � qa0�z��a0�z�.

3. Numerical Simulations

In 2004, Pappalardo et al. [8] reported the results ofa blind intercomparison of the algorithms used forRaman lidar data processing within the networkEARLINET. The algorithms were tested against syn-thetic lidar data, simulated at 355, 532, 386, and607 nm, with realistic experimental and atmosphericconditions as well as with typical aerosol propertiesand load taken into account.

The same data were used as a training case for thenext generation intercomparison. It means that therewere provided the profiles of the extinction, the back-scatter, and the lidar ratio along with the lidar sig-nals, the atmospheric conditions, and the Ångströmcoefficient values. The proposed algorithm was testedagainst the EARLINET training case data as an ex-ternal party after the internal comparison was fin-ished. The results of our tests are outlined below.


The training case data consist of a series of fifteenlidar signals (realizations) at each wavelength. Eachsignal was simulated with a spatial resolution of15 m and with 2400 laser shots, i.e., it corresponds toa 2 min profile for a 20 Hz laser system. These con-ditions and the fixed maximum counting rate of20 MHz of an acquisition system resulted in syn-thetic Raman lidar signals with a typical SNR of 5 at2 km of altitude and of less than 2 at altitudes above4 km [8]. The proposed algorithm was tested in thefollowing way. Each pair of the Raman-elastic lidarsignals was processed and a set of solutions was ob-tained. The retrieved aerosol profiles were found byaveraging over the corresponding set of solutions.The standard deviation was calculated as a functionof height from the set of solutions. That is, the errorbars in Figs. 1 and 2, for all three parameters, wereobtained by Monte Carlo technique. In addition, theerrors were calculated by analytical techniques, i.e.,by application of the error propagation rules; andthese errors are equivalent to those reported in thefigures.

The results, obtained with the proposed algorithm,are displayed in Figs. 1 and 2. Figure 1 shows theretrieved aerosol profiles, averaged over the set ofsolutions, at the wavelength of 355 nm. The backscat-ter coefficient, the lidar ratio, and the extinction co-efficient are drawn with gray curves along with thecorresponding input profiles (black curves), i.e., pro-files that were used to compute the synthetic lidarsignals. Figure 2 shows the same data but at the

wavelength of 532 nm. The inversions were per-formed using the atmospheric model (temperatureand pressure profiles) of [8]. The reference distancezref was chosen at 5.98 km. The a priori values ofthe backscatter ratio R0�zref� were taken according tothe input backscatter coefficients at zref and wave-lengths of 355 and 532 nm. The preassigned value ofthe Ångström coefficient k � 1.8 was used. Conse-quently, the error bars in Figs. 1 and 2 correspond toonly statistical errors that are due to signal detection.

It is seen from Figs. 1 and 2 that the proposedalgorithm provides the possibility to retrieve profilesof the backscatter and extinction coefficients withgood accuracy and resolution. For example, the peakof width 180 m at the distance of 1.55 km is well-resolved in both cases. The resolution decreases withdistance due to the steep decrease of the SNR. Com-pared to Fig. 2, the values of the standard deviationare higher in Fig. 1 because the SNR is lower atwavelengths of 355 and 386 nm.

In this study, the regularization parameter � ofEq. (14) was estimated by means of the version ofthe L-curve method when the L-curve was a plotof �Bq � �2 versus �. Figure 3 shows an example ofa case at the wavelength of 532 nm. This plot has anL-shaped corner showing a point, labeled by the blacktriangle, with the value of � that is considered as anoptimum balance between measurements and thea priori constraint. The retrieval quality of the lidarratio [Figs. 1(b) and 2(b)] may be considered as sat-isfying in the both cases. The retrieved functions arewithin the range of the corresponding input profiles.There is tendency to follow the variation of the inputprofiles. On the other hand, the retrieved lidar ratiosseem to be oversmoothed. This is due, in part, to thefact that the profiles are computed by averaging overthe sets of solutions. At the same time, this suggeststhat the performance of the proposed algorithm canbe improved by optimization of inversion of Eq. (12).Unified principles addressing inversion optimizationare outlined in [19]. Of course, the optimization sug-

Fig. 1. Retrieved aerosol profiles (gray curves) and input data(black curves) at the wavelength of 355 nm: (a) the backscattercoefficients, (b) the lidar ratios, and (c) the extinction coefficients.

Fig. 2. Retrieved aerosol profiles (gray curves) and input data(black curves) at the wavelength of 532 nm: (a) the backscattercoefficients, (b) the lidar ratios, and (c) the extinction coefficients.

Fig. 3. �Bq � �2 versus the regularization parameter �; the blacktriangle marks the value of � estimated by means of the L-curvemethod.


gests a great quantity of numerical tests and sensi-tivity studies.

As mentioned above, a regularized solution of theintegral equation (12) is more advantageous com-pared to the numerical differentiation because theaerosol optical thickness is less affected. Figure 4illustrates that point in more detail. The examples,which were taken from the sets of solutions, areused. The gray curves display the profiles of theoptical thickness a0�z, zref� between z and the refer-ence distance zref � 5.98 km, which were computedaccording to Eq. (10). Figures 4(a) and 4(b) corre-spond to wavelengths 355 and 532 nm, respectively.There are fluctuations in a0�z, zref�, that is, the ran-dom errors were not totally eliminated by thesmoothing of the range-corrected signals. The fluc-tuations at the wavelength of 532 nm are higher justbecause the range-corrected signal at the wavelength607 nm was less smoothed. Accordingly, an a priorismoothness constraint on retrieved profile of the lidarratio qa0�z�, i.e., Eq. (14), should be employed. Theblack curves show the profiles of the optical thickness a0*�z, zref�, which were computed by direct integra-tion of �a0*�z� � qa0*�z��a0*�z�, where qa0*�z�, �a0*�z�,and �a0*�z� are the retrieved profiles of the lidar ratio,the aerosol backscatter and extinction coefficients,respectively. It is seen that the fluctuations wereeliminated. At the same time, the retrieved opticalthickness a0*�z, zref� follows quite well the main fea-tures of a0�z, zref�.

An insight into the potential of the proposed algo-rithm can be gained from comparing its results withthose of the conventional scheme. First of all, itshould be pointed out that we compared the proposedand the conventional algorithms for the case whenthe signals were averaged over fifteen realizations.This case represents a temporal average of 30 min,which increases the SNR. The temporal average of30 min produces an overall lidar signal with a typicalSNR of 20 at 2 km of altitude and of less than 10 ataltitudes above 4 km [8]. That is, the signals alsocorrespond to typical experimental conditions. Theinversions were performed in the case when the at-

mospheric model (temperature and pressure pro-files), the backscatter ratio R0�zref� at zref, and theÅngström coefficient were known. Figure 5 shows theprofiles of the backscatter coefficient, the lidar ratio,and the extinction coefficient at the wavelength of532 nm. The black curves are the input profiles thatwere used to compute the synthetic lidar signals. Theresults, obtained with the proposed algorithm, aredisplayed with the gray curves. The dotted curvesand the solid circles represent the profiles computedby means of the conventional algorithm.

More specifically, the conventional algorithm wasused as follows. A sliding linear least-squares fit (see,e.g., [9]) was applied to derive the extinction coeffi-cient profile with the spatial resolution 105 m up to700 m, 315 m up to 1430 m, 195 m up to 1580 m,765 m up to 4270 m, and 1665 m above 4270 m. Theprofile of the backscatter coefficient was computedusing sliding average of the range-corrected signalswith the same spatial resolution at the wavelength of607 nm and the spatial resolution 45 m up to 4940 m,and 765 m above 4940 m at the wavelength of 532nm. The profile of the lidar ratio was derived usingthe additional sliding average of 105 m up to 1900 m.Fluctuations of the extinction-to-backscatter ratioabove 2000 m were large. Hence, at the high alti-tudes, it was averaged over 900 m or more and onlyseveral values of the lidar ratio are shown by thepoints in Fig. 5(b).

It follows from the comparison between Figs. 2 and5 that the increase in the SNR resulted in the moreaccurate profiles retrieved with the proposed algo-rithm. The root-mean-square error (RMSE) and theroot-mean-square relative error (RMSRE), i.e., thedifference between the input and the retrieved pro-files, decreased for all three parameters. In addition,the profile of the lidar ratio is less smoothed andbetter follows the variation of the input profile.

As seen from Fig. 5(a), the both algorithms providethe backscatter profiles that are close to the inputone. Moreover, the values of the RMSRE are similar.The chief drawbacks of the profiles, obtained with theconventional scheme, are (i) the quite large noise inthe extinction [Fig. 5(c)] and lidar ratio profiles, and

Fig. 4. Profiles of the “measured” (gray curves) and retrieved(black curves) optical thicknesses a0�z, zref� at the wavelengths of(a) 355 and (b) 532 nm.

Fig. 5. Aerosol profiles retrieved with the proposed algorithm(PA, gray curves), with the conventional algorithm (CA, dottedcurves and solid circles) and input data (black curves) at the wave-length of 532 nm: (a) the backscatter coefficients, (b) the lidarratios, and (c) the extinction coefficients.


(ii) the extremely large noise in the lidar ratio above2 km [Fig. 5(b)]. It follows from Figs. 5(b) and 5(c)that the proposed algorithm provided more accurateprofiles of the extinction coefficient and the lidar ra-tio. For the extinction profile up to 6 km of height, theRMSE is 3.1 times lower and the RMSRE is 6.5 timeslower compared to the conventional scheme retrieval.For the lidar ratio profile up to 1.9 km of height, theRMSE is 2.4 times lower and the RMSRE is 2.6 timeslower. The same conclusion follows from the compar-ison at the wavelength of 355 nm (not shown). Theseresults suggest that the proposed algorithm has thepotential to improve the quality of Raman lidar dataprocessing.

The effect of the a priori smoothness constraint andof values of the regularization parameter � can beseen in Fig. 6. The simulations were performed forthe case of the temporal average of 30 min and at thewavelength of 532 nm. Figure 6(a) shows the lidarratio profiles computed according to Eq. (14) with thematrices H that are the first, second, and third dif-ference matrices [18] (see the legend). Figure 6(b)displays the lidar ratio profiles that were derivedusing the second difference matrix H and the valuesof the regularization parameter ��3, �, and 3�, where� is the value estimated according to the L-curvemethod. Although the retrieved profiles follow thevariations of the input profile, the effect of the a priorismoothness constraint is quite high. Thus, the opti-mization of inversion should be performed on thebase of those techniques of the inverse problem the-ory that show a good performance in the case when aretrieved function can be stepwise and the SNR israther low.

4. Error Analysis

The objective of this section is to provide insight intothe primary error sources that affect the retrievedprofiles.

In Figs. 1 and 2, the error bars represent the stan-dard deviation computed from the sets of solutions,i.e., illustrate errors estimated using a numerical pro-cedure based on a Monte Carlo technique [8]. Thevariations are due to random noise in the syntheticlidar signals. Accordingly, the error bars show thepropagation of measurement errors: all the system-atic effects being neglected. In addition, errors in�a0�z� were estimated by application of the error prop-agation rules in Eqs. (4), (5), and (8). The computedvalues of the standard deviation correspond to theerror bars shown in Figs. 1(a) and 2(a).

It is seen from Figs. 1 and 2 that the absolutevalues of the standard deviation depend on theheight, i.e., on the distance between z and zref. Con-sequently, it is of importance to examine the effect ofmeasurement errors in more detail.

For that purpose, the principal component analysis(PCA) technique (see [21] and references therein) wasused. The PCA is known to be designed to generate anew set of uncorrelated parameters, called principalcomponents, representative of the original data set.One of the most important advantages consists in thefact that the PCA provides the possibility to repro-duce almost all variability of a data set with reason-ably good accuracy and considering only a fewprincipal components. The first principal componentsof the sets of solutions, i.e., of the retrieved profiles ofthe aerosol backscatter coefficients, are drawn withgray curves in Fig. 7. The corresponding eigenvaluesof the first components, normalized as a percentage ofthe total variance, are approximately 28% and 57%for wavelengths 355 and 532 nm, respectively. Thedistinction between the eigenvalues is caused by thedifference in the SNRs.

The averaged profiles, i.e., the retrieved profilesfrom Figs. 1 and 2, are displayed in Fig. 7 as well.They are scaled in order to compare with the princi-pal components, and shown with black curves. It isseen from Fig. 7(b) that there is much in commonbetween the first component and the averaged pro-file. The same is true but less pronounced in Fig. 7(a)for the height range between approximately 0.2 and

Fig. 7. First principal components (gray curves) and scaled re-trieved profiles (black curves) at the wavelengths of (a) 355 and(b) 532 nm.

Fig. 6. Profiles of the lidar ratio derived for (a) different matricesof a priori smoothness constraint, and (b) different values of theregularization parameter.


2 km. Consequently, a quite large part (57% and28%) of the variations results from a coincident in-crease or decrease of values of the retrieved backscat-ter coefficients despite the fact that there is no spatialcorrelation in the noise of the signals.

Recall that a reference value R0�zref� of the back-scatter ratio at the reference distance zref is used inthe proposed algorithm in order to eliminate thedepth-independent system parameters. At the givenwavelength, R0�zref� is the same within the set of so-lutions. That is why there is no variation of the re-trieved profiles at the reference distance. With regardto that peculiarity, the above-discussed feature canbe explained by the effect of data errors at the refer-ence distance zref, i.e., errors in the smoothed range-corrected signals. This suggests that the accuracy ofthe proposed algorithm can be enhanced by optimi-zation of the smoothing in the vicinity of zref. Forexample, it could be used the constraint that the slopeof a smoothed range-corrected signal has to be close tothe slope of the known air density profile.

The correlation matrix, supplied by the PCA, pro-vides a convenient way to estimate the spatial res-olution of the backscatter profile retrievals byevaluating the range dependence of the correlationlength. Generally, the spatial resolution nonlinearlydecreases with the distance, i.e., it is in agreementwith the standard deviation of the range-correctedsignals. At the wavelength of 532 nm, it is 15 m up to900 m, 40 m around 2000 m, 135 m around 3000 m,360 m around 4000 m, and 640 m above 5000 m. Atthe wavelength of 355 nm, the values are about 10%higher.

As for the effect of uncertainty in the Ångströmcoefficient k, it can be expressed analytically (disre-garding all other errors). The equation looks muchsimpler in terms of the relative error of the backscat-ter ratio, i.e.,

�R0�z� � <R0*�z� � R0�z�=�R0�z�, (16)

where R0*�z� � <�a0*�z� � �M0�z�=��M0�z� and �a0*�z�is computed using Eq. (8), but with the Ångströmcoefficient value of k � �k.

It follows from Eqs. (7) and (8) that

�R0�z� � exp<2�1 a0�z, zref�= � 1, (17)

where

�1 � �0�X�k�0�X��k � 1

�0�X�k��k � 1. (18)

For the Raman lidar wavelengths, if �k � 0, �1 � 0,and �R0�z� � 0, that is, the aerosol backscattercoefficient is overestimated. It is underestimated if�k � 0. When the aerosol optical thickness a0�z, zref�is of 0.5, k � 1, and �k � �1, the relative error �R0�z�is within �7% and �4% at the wavelengths 532�607and 355�386 nm, respectively. �R0�z� is within �14%and �12% when a0�z, zref� � 1. Consequently, the

errors caused by the uncertainty in the Ångströmcoefficient are satisfactory, even though the relation-ship [Eq. (17)] is nonlinear.

Errors in the total backscatter caused by the un-certainty in the air density, i.e., in the molecularextinction, backscatter, and number density NX�z�,are satisfactory as well. That conclusion follows fromEqs. (4), (5), and (8) when all parameters, which de-pend on the a priori profile of the air density, aretaken together. The retrieved profile �0�z� � �a0�z�� M0�z� is proportional to the function

M�z� � �M0�zref��NX�z��NX�zref��2��1��

� exp��2�2 M0�z, zref��, (19)

where

�2 ��0�X�k � �0�X�4

�0�X�k � 1 , (20)

and M0�z, zref� is the molecular optical thickness. It isseen that the errors in the total backscatter are of theorder of magnitude of the uncertainty in the air den-sity.

It is seen from Eqs. (14) and (10) that the quality ofthe lidar ratio retrieval depends on random and sys-tematic errors in the optical thickness a0�z, zref� and,consequently, in the range-corrected signal SX�z, zref�at the Raman wavelength. At the same time, itshould be remembered that the matrix B depends onrandom errors in the range-corrected signal S0�z, zref�at the elastic wavelength and on the systematic er-rors. An insight into the primary error sources thataffect retrieved profiles of the lidar ratio can be ob-tained assuming that B remains constant.

According to the inverse problem theory (see Sec-tion 3.2 of [22]), the covariance matrix of the solutionto Eq. (14), i.e., of the retrieved profile of the lidarratio, is given by the expression

Cq � �BTW�1B � �HTH��1. (21)

The matrix Cq depends on the a priori constraint�HTH and the covariance matrix W of the vector �,i.e., errors of measurement. The square roots ofthe diagonal elements (variances) of Cq can beinterpreted as “uncertainty bars” [22]. If � → 0, thesolution to Eq. (14) is unstable due to the factthat BTW�1B is a quasi-degenerated matrix, i.e.,det�BTW�1B� is close to zero. �Cq�ii � ��BTW�1B� �HTH��1�ii → � for � → 0 and det�BTW�1B� → 0.This is the equivalent of the case when a numericalderivative is computed without data smoothing. Thea priori constraint is an essential tool for achievinga unique and stable solution (see, e.g., Section 4 of[19]). Recall that in this study the regularizationparameter � was estimated by means of the L-curvemethod. On the whole, the values of ��Cq�ii, com-


puted according to Eq. (21), correspond to the errorbars shown in Figs. 1(b) and 2(b).

It follows from the analysis of the matrix Cq thatthe uncertainties are highly correlated. This meansthat the parameters have not been independentlyresolved [22]. The spatial resolution of the lidar ratioprofiles was estimated from the range dependence ofthe correlation length. As in the case of the backscat-ter profiles, the spatial resolution nonlinearly de-creases with the distance. At the wavelength of532 nm, it is 165 m up to 700 m, 820 m around2000 m, 1000 m around 3000 m, and 2100 m above5000 m. At the wavelength of 355 nm, the values arethe same. This confirms the conclusion of Section 3that the L-curve method has led to the oversmoothedprofiles of the lidar ratio.

It immediately follows from Eqs. (10) and (14) thatthe uncertainty �k in the Ångström coefficient kleads to the relative error ��qa0�z�� in the computedvalue of the lidar ratio, which does not depend on thedistance and can be expressed by the relationship

��qa0�z�� 1, (22)

where �1 is given by Eq. (18). When k � 1 and �k� �1, the relative error ��qa0�z�� is of �6% and �4%at the wavelengths 532�607 and 355�386 nm, respec-tively. Thus, the errors caused by the uncertainty inthe Ångström coefficient are satisfactory.

Errors in the lidar ratio caused by the uncertaintyin the air density, i.e., in the molecular extinction andnumber density NX�z�, are satisfactory as well. Thatconclusion follows from Eqs. (5) and (10) when allparameters, which depend on the a priori profile ofthe air density, are taken together. The correspond-ing errors � a0�z, zref� in the optical thickness a0�z, zref� are given by the following equation:

� a0�z, zref� �1

1 � �ln�1 � �NX�zref�� ln�1 � �NX�z��

� �z

zref

��M0�z�� MX�z��dz��, (23)

where �NX�z� is the relative error in the number den-sity, and ��M0,MX�z� are the errors in the molecularextinctions. It is seen that the errors � a0�z, zref� are ofthe order of magnitude of the uncertainty in the airdensity. The errors �q in the lidar ratio can be writ-ten, in view of Eq. (14), as

�q � �BTW�1B � �HTH��1BTW�1��, (24)

where the vector �� approximates � a0�z, zref� of Eq.(23). As a rough approximation, we may say that �qcompensates for � a0�z, zref�.

With the range-dependent extinction coefficient �a0�z�computed by the multiplication �a0�z� � qa0�z��a0�z�,errors in �a0�z� can be estimated applying the law oferror propagation. Generally, the computed values ofthe standard deviation correspond to the error barsshown in Figs. 1(c) and 2(c). The spatial resolution ofthe extinction profile, estimated from the range de-pendence of the correlation length, is the same as forthe backscatter profile. This is because the variationsof the backscatter coefficient dominate. As for thesystematic errors caused by the uncertainty in the airdensity and in the Ångström coefficient, it is reason-able that they are satisfactory as well.

Overall, the main conclusion of this section is thatthe retrieved profiles are continuous functions of themeasurement errors and the a priori information un-certainties. There are no strong nonlinear effects suchas in algorithms for a single-wavelength lidar [14].

5. Discussion

First, it should be pointed out that the proposed algo-rithm cannot be used to generate real-time quick-viewplots. The algorithm is relatively time consuming. Ona PC running at 3 GHz, it takes approximately 2 minto retrieve profiles within the range of 6 km with thestep size of 15 m. The proposed algorithm is recom-mended for cases when the accuracy of estimates is ofimportance.

The errors caused by the uncertainties in the Ång-ström coefficient and in the air density are satisfac-tory. The quality of retrievals is mostly determinedby random errors of measurements. In common withother lidar methods, the spatial resolution and theaccuracy of estimates depend directly on the SNR ofthe lidar signals. It is a common practice to increasethe SNR by signal averaging of 30–120 min (see, e.g.,[23]). The necessary signal averaging can be per-formed if there is no significant evolution. As it wasproved by Ansmann et al. [3], the signal averagingcan introduce an additional and significant error ifthe particle optical properties vary strongly duringthe measurement. It is seen from the simulationsabove that the algorithm provides the possibility toperform retrievals with good accuracy and resolutionin the conditions of the typical SNR.

It is the usual practice to assign the reference dis-tance zref according to the clear-air conditions thatnormally prevail in the upper troposphere [3]. Forground-based lidars, one deals with a range wherethe SNR is relatively low. In fact, the errors in thea priori value R0�zref� and in the estimated values ofthe parameters P0,X�zref�, i.e., in the smoothed range-corrected signals, are of the same importance [seeEqs. (4), (5), and (8)]. Consequently, the referencedistance zref should be chosen at the range where theerror in the a priori value R0�zref� is low and the SNRvalues are as large as possible. In addition, the accu-racy of the proposed algorithm can be enhanced bythe constraint that the slope of a smoothed range-corrected signal has to be close to the slope of theknown air-density profile near zref.


It was shown above that the proposed algorithmhas the potential to reduce the root-mean-square er-rors of retrieved profiles of the lidar ratio. At the sametime, the retrieved lidar ratios seem to be over-smoothed. In reality, there can be large gradients ofthe lidar ratio, for example, at the base and top ofcirrus. The oversmoothed profile of the lidar ratio canlead to large errors in the extinction coefficients.When there are aerosol layers having distinct fea-tures, the proposed algorithm should be applied in-dividually to each interval of height. Alternatively,the performance of the proposed algorithm can beimproved by optimization of inversion of Eq. (12).

6. Conclusions

In this paper, a regularized algorithm has been pre-sented, which has the potential to improve the qual-ity of the Raman lidar data processing. Compared tothe conventional scheme, the proposed algorithm hasthe advantage, which results from the fact that it isbased on a well-posed procedure. That is, the profileof the aerosol backscatter coefficient is computed di-rectly, using the explicit relationships, without nu-merical differentiation. Thereafter, the profile of thelidar ratio is retrieved as a regularized solution of afirst-kind Volterra integral equation. The advantageof this originates from the fact that the lidar ratiovaries in a limited range, and therefore correspondsbetter to an a priori smoothness constraint. Oncethese two steps have been completed, the profile ofthe aerosol extinction coefficient can be computed bya straightforward multiplication. The flexibility ofthe proposed algorithm lies in the fact that it providesa way to apply well-developed capabilities of the in-verse problem theory to retrieve the profile of thelidar ratio and, consequently, the extinction coeffi-cient profile.

The numerical simulations demonstrated that theproposed algorithm provides good accuracy and res-olution of aerosol profile retrievals. The error analysisshowed that the retrieved profiles are continuousfunctions of the measurement errors and the a prioriinformation uncertainties.

Appendix A: Application of Weighted SmoothingSplines

As mentioned in Subsection 2.B, the important fea-ture of lidar signals is the steep decrease of the SNRwith distance. Consequently, smoothing, as a rule,should be applied to the range-corrected signals and itshould be range-dependent. Weighted cubic smooth-ing splines [16,17] are well-suited for this purpose.

Without going into the details of the mathematics,we briefly follow Chapter 3 of [16] with little adapta-tion to a lidar data case. Let us consider a grida � z0 � z1 � · · · � zn � b, where [a, b] is somesegment of distance z, and the corresponding setfk �k � 0, 1, . . . , n� of values of a range-corrected sig-nal. A function s(z) defined on the grid is called asmoothing cubic spline if it (i) is a cubic polynomial oneach partial segment �zk�1, zk�, (ii) has the continuoussecond derivatives s��z� on segment [a, b], (iii) mini-

mizes the functional

J�s� ��a

b

�s��2dz � �k�1

n

pk�s�zk� � fk�2, (A1)

where pk � 0 are given numbers, and (iv) satisfies theboundary conditions s��a� � s��b� � 0.

In a matrix representation, the values �k � s�zk� ofthe smoothed range-corrected signal at the knots zk

are given by the equation

� � f � P�1DTm, (A2)

where the vector m � �m1, m2, . . . , mn�1�T is the so-lution to the equation

�A � DP�1DT�m � Df, (A3)

� � ��0, �1, . . . , �n�T, f � �f0, f1, . . . , fn�T, where Tdenotes matrix transposition; the �n � 1��n � 1�matrix A and the �n � 1��n � 1� matrix D depend onlyon the grid. For an equidistant grid of step size�z � zk � zk�1, the matrices are as follows:

A ��2�z�3 �z�6 0 · · · 0 0�z�6 2�z�3 �z�6 · · · 0 0

0 �z�6 2�z�3 · · · 0 0· · · · · · · · · · · · · · · · · ·0 0 0 · · · �z�6 2�z�3

�, (A4)

D ��1��z �2��z 1��z 0 · · · 0 0

0 1��z �2��z 1��z · · · 0 00 0 1��z �2��z · · · 0 0

· · · · · · · · · · · · · · · · · · · · ·0 0 0 0 · · · �2��z 1��z

�.

(A5)

The diagonal �n � 1��n � 1� matrix P �Pkk � pk�serves to implement the range-dependent smoothing.It is seen from Eq. (A1) that the higher pk is, the closer�k is to fk; and the lower pk is, the smoother s(z) is inthe vicinity of zk. Consequently, the range-dependentsmoothing can be performed in agreement with themeasurement errors by setting pk inversely propor-tional to �2�zk�, where ��zk� is the standard deviation(or its estimate) of the range-corrected signal at thedistance zk. The proportionality coefficient is neededbecause lidar data are usually specified in relativeunits.

The matrix �A � DP�1DT� is a symmetric andpositive-definite matrix [16]. Thus, a solution to Eq.(A3) always exists.

Figure 8 shows two examples of application of theweighted smoothing splines to range-corrected signals.The original signals (drawn with the gray curves) aretaken from the training case data. Figures 8(a) and


8(b) represent the 532 and 607 nm cases, respectively.The black curves show the smoothed profiles. At thenear end, the smoothed profiles closely match theoriginal signals. The degree of the smoothing in-creases with the distance. It is seen that the weightedsplines demonstrate good performance and are well-adapted to the particularities of lidar signals.

The author is grateful to G. Pappalardo for pro-viding the training case data of the EARLINETintercomparison. The author thanks the anonymousreviewers for their fruitful comments.

References1. D. N. Whiteman, “Examination of the traditional Raman lidar

technique. I. Evaluating the temperature-dependent lidarequations,” Appl. Opt. 42, 2571–2592 (2003).

2. A. Ansmann, M. Riebesell, and C. Weitkamp, “Measurement ofatmospheric aerosol extinction profiles with a Raman lidar,”Opt. Lett. 15, 746–748 (1990).

3. A. Ansmann, U. Wandinger, M. Riebesell, C. Weitkamp, andW. Michaelis, “Independent measurement of the extinctionand backscatter profiles in cirrus clouds by using a combinedRaman elastic-backscatter lidar,” Appl. Opt. 31, 7113–7131(1992).

4. A. Ansmann, M. Riebesell, U. Wandinger, C. Weitkamp, E.Voss, W. Lahmann, and W. Michaelis, “Combined Ramanelastic-backscatter LIDAR for vertical profiling of moisture,aerosol extinction, backscatter, and LIDAR ratio,” Appl. Phys.B 55, 18–28 (1992).

5. M. R. Gross, T. J. McGee, R. A. Ferrare, U. N. Singh, and P.Kimvilakani, “Temperature measurements with a combinedRayleigh–Mie and Raman lidar,” Appl. Opt. 36, 5987–5995(1997).

6. J. Bösenberg, A. Ansmann, J. M. Baldasano, D. Balis, C. Böck-mann, B. Calpini, A. Chaikovsky, P. Flamant, A. Hågård, V.Mitev, A. Papayannis, J. Pelon, D. Resendes, J. Schneider, N.Spinelli, T. Trickl, G. Vaughan, G. Visconti, and M. Wiegner,“EARLINET: a European aerosol research lidar network,” in

Advances in Laser Remote Sensing, A. Dabas, C. Loth, and J.Pelon, eds. (Editions de L’Ecole Polytechnique, 2001), pp. 155–158.

7. T. Murayama, N. Sugimoto, I. Uno, K. Kinoshita, K. Aoki, N.Hagiwara, Z. Liu, I. Matsui, T. Sakai, T. Shibata, K. Arao, B.-J.Sohn, J.-G. Won, S.-C. Yoon, T. Li, J. Zhou, H. Hu, M. Abo,K. Iokibe, R. Koga, and Y. Iwasaka, “Ground-based networkobservation of Asian dust events of April 1998 in east Asia,”J. Geophys. Res. 106, 18345–18359 (2001).

8. G. Pappalardo, A. Amodeo, M. Pandolfi, U. Wandinger, A.Ansmann, J. Bösenberg, V. Matthias, V. Amiridis, F. De To-masi, M. Frioud, M. Iarlori, L. Komguem, A. Papayannis, F.Rocadenbosch, and X. Wang, “Aerosol lidar intercomparison inthe framework of the EARLINET project: 3. Raman lidar al-gorithm for aerosol extinction, backscatter, and lidar ratio,”Appl. Opt. 43, 5370–5385 (2004).

9. D. Whiteman, “Application of statistical methods to the deter-mination of slope in lidar data,” Appl. Opt. 38, 3360–3369(1999).

10. A. E. Tikhonov and V. Y. Arsenin, Solutions of Ill-posed Prob-lems (Wiley, 1977).

11. O. Dubovik, B. N. Holben, T. F. Eck, A. Smirnov, Y. J. Kauf-man, M. D. King, D. Tanré, and I. Slutsker, “Variability ofabsorption and optical properties of key aerosol types observedin worldwide locations,” J. Atmos. Sci. 59, 590–608 (2002).

12. U. Wandinger and A. Ansmann, “Experimental determinationof the lidar overlap profile with Raman lidar,” Appl. Opt. 41,511–514 (2002).

13. P. B. Russell, T. J. Swissler, and M. P. McCormick, “Method-ology for error analysis and simulation of lidar aerosol mea-surements,” Appl. Opt. 18, 3783–3797 (1979).

14. J. D. Klett, “Stable analytical inversion solution for processinglidar returns,” Appl. Opt. 20, 211–220 (1981).

15. R. M. Measures, Laser Remote Sensing. Fundamentals andApplications (Krieger, 1992).

16. G. I. Marchuk, Methods of Numerical Mathematics, 2nd ed.(Springer-Verlag, 1982).

17. E. V. Shikin and A. I. Plis, Handbook on Splines for the User(CRC, 1995).

18. W. H. Press, B. P. Flannery, S. A. Teukolsky, and W. T. Vet-terling, Numerical Recipes in C: The Art of Scientific Comput-ing (Cambridge U. Press, 1992).

19. O. Dubovik, “Optimization of numerical inversion in photopo-larimetric remote sensing,” in Photopolarimetry in RemoteSensing, G. Videen, Y. Yatskiv, and M. Mishchenko, eds. (Klu-wer, 2004), pp. 65–106.

20. P. C. Hansen, “Analysis of discrete ill-posed problems bymeans of the L-curve,” SIAM Rev. 34, 561–580 (1992).

21. V. Shcherbakov, J.-F. Gayet, O. Jourdan, A. Minikin, J. Ström,and A. Petzold, “Assessment of cirrus cloud optical and micro-physical data reliability by applying statistical procedures,” J.Atmos. Ocean. Technol. 22, 409–420 (2005).

22. A. Tarantola, Inverse Problem Theory and Methods for ModelParameter Estimation (SIAM, 2005).

23. K. Franke, A. Ansmann, D. Müller, D. Althausen, C. Venkat-araman, M. S. Reddy, F. Wagner, and R. Scheele, “Opticalproperties of the Indo-Asian haze layer over the tropical IndianOcean,” J. Geophys. Res. 108, doi:10.1029/2002JD002473(2003).

Fig. 8. Original signals (gray curves) and smoothed profiles(black curves) at the wavelengths of (a) 532 and (b) 607 nm.


Regularized algorithm for Raman lidar data processing

Documents

Transcript of Regularized algorithm for Raman lidar data processing