ZENGYAN ZT-IANG, SATYA N.V. KALLURI, JOSEPH JAJA, SHUNLIN LIANG, AND JOHN R.G. TOWNSI-IEND Univevsity of Maryland, College Park

The authors describe three models for retrieving information related to the scntter- ing of light on the earth’s su$ace. Using these models, they’ve developed

algorithms for the IBM SP2 that eficiently retrieve this information.

ost land-cover types are anisotropic; that is, they do not reflect solar radiation uni- formly in all directions. Characterizing the bidirectional veJectnnce distvibution finc-

tion of the earth’s surface is critical in understanding sur- face anisotropy. Although several methods exist for re- trieving the BRDF of various land-cover types, most of them have been applied over small data sets collected ei- ther on the ground or from aircraft a t limited spatial and temporal scales.

Satellite-based sensors can provide the data necessary for large-scale studies of BRDF. However, most sensor systems (for example, A VHRR-advanced very-high- resolution radiometer) take directional spectral mea- surements. Land-surface anisotropy causes variations in surface reflectances when measured under different il- lumination and view angles. So, these measurements (for example, reflectance) are valid only for a particular sen- sor-illumination geometry.

In this article, we’ll describe three models for deriving global BRDF that compensate for the directional limita- tions of satellite sensors. We’ve devised algorithms that im-

plement these models on a high-performance computing system, using an efficient method to handle the large data set involved. Our implementations optimize I/O access time and efficiently balance computations across the nodes.

BRDF A surface’s BRDF specifies the surface-scattering behav- ior as a function of illumination and viewing angles at particular wavelengths.

The BRDF can be written as shown in Equation 1 (see the sidebar for this and the other numbered equations), where 8, $, and A are the zenith angle, azimuth angle, and wavelength.’ The subscripts D and s denote the view and solar angles. L, is the reflected radiance in the view direction measured by the sensor, and E, is the parallel beam irradiance from the illumination direction in the wave band.f;(8,, e,, @,, q$, 1) has the unit ~v (sv means steradian). To make BRDF directly comparable to bi- directional surface reflectance and hemispherical re- flectance (which we’ll describe in the next section), we refer to BRDF as p =f;. z.’

16 1070-9924/98/$10.00 D 1998 IEEE IEEE COMPUTATIONAL SCIENCE & ENGINEERING

Equations

+ a4 cos( $1 + a5 sin(') + a6 cos( s) + a, s i n ( F )

ATAx = ATy , where

a11 a21 P: P: a12 a22

. . . .

OCTOBER-DECEMBER 1998 17

Why is understanding BRDF important? For some radiative-transfer and energy-

balance studies of the land surface, we need the surface reflectance that is integrated over all viewing angles in the upward hemisphere (the hemispherical reflectance) and over the visible and near-infrared wavelength (the broadband albedo). A surface’s albedo describes the ratio of radiant energy scattered upward and away from the sur- face in all directions to the down-welling irra- diance incident upon the surface.

Equation 2 gives the directional-hemispheri- cal spectral reflectance (ph(8,, A)), where p(@, e,, 4, A) is the surface BRDF, and 4 is the relative azimuth angle (4 = 4, -4J. When the equation is integrated over all possible solar-zenith angles (e,), it is known as the bihemisphevical albedo.

Another reason for understanding the surface BRDF is that it affects vegetation indices (for example, the NDVI-Normalized Difference Vegetation Index), which are derived from a combination of spectral bands. Many remote- sensing scientists use vegetation indices to char- acterize and monitor vegetation dynamics from satellite data.3

So, understanding the surface BRDF lets usZ

* correct multidate images taken at different view angles for BRDF effects (for example, in the creation of temporal composites), retrieve surface structural attributes (for example, leaf-area index and biomass) and land-cover information from the surface’s scattering behavior, and accurately retrieve the broadband albedo.

*

The land-surface BRDF measured at the top of the atmosphere is different from that measured at the surface because of atmospheric effects. However, the net shortwave energy balance re- trieved at the top of the atmosphere is linear and directly proportional to the net shortwave fluxes measured on the g r ~ u n d . ~ So, the albedo and BRDF retrieved at the top of the atmosphere can still he used in energybalance studies.

How to compute BRDF BRDF models fall into three broad classes:

physically based, empirical, or semiempirical.? Physically based models include geometric-

optical models, turbid medium models, hybrid models, and computer simulation models. These models are complex and computationally de- manding; so far, they have only been developed for specific land-cover types, with no known uni-

versal models for different cover types. They de- pend on the structural and state attributes of the land surface such as the leaf-angle distribution on plant canopies, photosynthetic activity, and the shape and size of plants (for example, cylindrical, spherical, or conical). Their development and ap- plication have been limited to BRDF modeling activities and have not been widely applied to large global data sets.

Empirical models are simple to use and have been applied fairly widely, although the model coefficients might not have a physical meaning. One of the most widely used empirical models is the Modified Walthall model5 (see Equation 3), where ao, a l , a2, and a3 are the model’s parameters.

Semiempirical models try to provide a balance between physically based models and empirical models by providing empirical coefficients that have a physical meaning. One successfully tested model is the Coupled Surface-Atmosphere Re- flectance (CSAR) model,6 which describes the surface BRDF as shown in Equation 4, where

cos g = cos 8, cos 8, + sin 8, sin 8, cos Cp 1- Po 1 + R( G ) = 1 + ~

l + G 1

G = [ tan2 0, + tanZ 0, - 2 tan 0, tan cos $17

The above set of equations has three unknown parameters, po, k, and 0, which must be deter- mined by model inversion and numerical itera- tion. The coefficient po represents the intensity of surface reflectance, although it is not a nor- malized albedo. Typical values of k range from 0 to 1, where lower values indicate higher surface anisotropy. The parameter 8 controls the rela- tive amount of forward and backward scattering.6

To model the BRDF, we need a set of re- flectance measurements of the target made at dif- ferent view and illumination angles so that we get a good sampling of the BRDF. However, current satellite instruments cannot make simultaneous multiangular measurements. Therefore, to get multiangular measurements, we use data that has been collected on different days to model the BRDF, because data collected from the AVHRR on different days has different observation ge- ometries. The long-term record of A W R R ob- servations provides an excellent opportunity to

18 IEEE COMPUTATIONAL SCIENCE & ENGINEERING

explore the angular signatures based on multi- temporal data assembly. The long time series are sufficient to link bidirectional reflectance with surface characteristics through different model- ing approaches.

When multiangular measurements are made, the surface conditions must be static. Otherwise, it is difficult to interpret whether the variations in reflectances at different angles are due to the BRDF or due to changes in surface conditions.

When we fit angular models to multitemporal observations, we hope to get long records to get enough angular sampling. We assume that the target does not change significantly over the pe- riod of measurements. However, many cover types have seasonal or annual changes. For ex- ample, a pixel of an agricultural-land image could correspond to a dense canopy in the grow- ing season and bare soil in the winter. Canopies have quite different angular behavior from soils. Almost all empirical models assume that only changes in viewing and illumination geometry cause variations in surface reflectance; they do not take temporal variations into account.

We've implemented a temporal angular model where a Fourier series approximates the tempo- ral component and the Modified Walthall model expresses the angular component. It describes surface BRDF as shown in Equation 5, where N is the number of data points for each year, and t varies from 0 to N-

Global BRDF retrieval computation To derive the global BRDF, we used the Patb- finder AVHRR Land data set and the Modified Walthall model, the CSAR model, and the tem- poral model. The input consists of four years (1983-1986) of the PAL data set with 8-km spa- tial resolution.8 Although the PAL data set spans from 1981 to 1994, we chose only these four years of data because the data from the other years has processing errors.

We used only the 10-day maximum-value NDVI composite data because this compositing procedure reduces the effects of atmosphere and cloud^.^ However, for visible and near-IR ob- servations, the compositing procedure elimi- nates larger view-zenith angles and favors data selection in the forward-scattering directi0n.j Also, it preferentially selects pixels with lower solar-zenith angles. In spite of these limitations in the sampling strategy, the AVHRR composite data is a valuable input for BRDF modeling be- cause it is the only global data available with

'I DOY (2 bytedpixel)

Relative azimuth angle (2 bytedpixel)

T 2,168 1 NDVI (1 bvte/pixel) .~ + 5,004 --

A VHRR Advanced very-high-resolution radiometer CLA VR Clouds from A VHRR DO)' Day of year NDVI Normalized difference vegetation index QC Quality control

Figure 1. Layout of the Pathfinder AVHRR (advanced very-high- resolution radiometer) Land IO-day composite data set that was usecl as input to derive the BRDF parameters (1 1 million pixels per image, 9 layers).

multiangular measurements. Such limitations are expected when only a single measurement of the target is made on any given day.

The visible and near-IR reflectances in the PAL, data have been corrected for Rayleigh scat- tering and ozone absorption. However, there is no correction for aerosol or water vapor effects. Each year comprises 3 6 1 0-day composite data sets, and we use nine out of 12 data layers from each composite, as Figure 1 shows.

For each pixel of the composite images, we use four types of information:

Pixel-selection information: QC (quality control) flags give information about data validity; CLAVR (clouds from AVHRR)9 flags provide cloud-cover information. Temporal and geometric information: day of year (DOY), sensor-scan angle, solar- zenith angle, and relative-azimuth angle. Reflectance values in channels 1 (visible) and 2 (near-IR). Information on surface phenology (sea- sonal variations in land cover such as peri- ods of peak greenness and leaf fall): The PAL data set also contains the NDVI, which is derived from channel l(pchl) and 2(pch2) reflectance measurements of AVHRR, as shown in Equation 6. Spatial and temporal variations in the NDVI cor- relate with the surface phenology."

To minimize the influence of surface phenol-

OCTOBER-DECEMBER 1998 19

ogy on BRDF retrieval by the Modified Walt- hall model and the CSAR model, we divide the input data into four quarters: January-March, April-June, July-September, and October-De- cember. For these two models, BRDF retrieval is done at quarterly intervals. Each quarter contains data grouped from all four years. However, the temporal model does not require this division of data into discrete time intervals because it uses Fourier series to describe the surface phenology.

For each pixel, we derive the coefficients re- quired to describe the surface BRDF. To do this, we use linear least-squares fit for the linear Modified Walthall and temporal models, and model inversion and iteration for the nonlinear CSAR model. For every pixel, we generate the coefficients a, (i = 1, 2, . . . , NJ for channels 1 and 2 for each model.

In addition, to allow a more in-depth anal- ysis, we compute statistical information on the model fit. In particular, for each pixel we compute

* the standard error between the model de- rived and the measured channel l and 2 reflectances, which provides a quantitative measure of the model fit’s accuracy. for the linear models, the regression- analysis coefficient R2, which is the ratio of the predicted data’s and the given data’s variances. R2 shows how well the model fits the given data and indicates the pro- portion of the variation “explained” by the regression line. the standard error between the NDVI in the PAL data set and the NDVI estimated by each BRDF model using the pixel geometry. the mean and standard deviations of the NDVI in the PAL data set, to study the land surface changes over the time period.

e

*

*

Now we’ll describe in detail the computation that takes place at each pixel. In the following analysis, Nd is the number of data points uscd for the model fitting, and N, is the number of model coefficients. k is the number of floating-point op- erations needed to evaluate a trigonometric func- tion such as sine, cosine, or arcsine. In practice, k = 25; we assume that the time required to per- form floating-point multiplication and to per- form addition and subtraction are the same.

Input data conditioning When the PAL data set was generated, the

physical values were scaled to either an eight-bit or a 16-bit unsigned integer value (see Figure l).’ For our application, we need to rescale the input PAL data from digital numbers to float- ing-point physical values, and convert the solar- zenith angle from degrees to radians and the sensor-scan angles to view-zenith angles. To condition the input, we need Nd (27 + 3k) float- ing-point operations per pixel.

Linear models The Modified Walthall model and the tem-

poral model are linear; we can express them as shown in Equation 7.

Taking channel 1 as an example, we can ex- press the BRDF model-fitting problem as

Given a set of data values (’~,’~, 4 ’ 7 pi’) , j = 1, 2, . . ., Nd, and the BRDF model equation (Equa- tion 7), choose the linear model coefficients that best describe the function relationship between p1 and the independent variables e,, e , , and q5.

The least-squares method chooses the “best” a, to minimize the cumulative error between the given value p{ and the model-predicted value F ; . Equation 8 shows the cumulative error.

Equation 7’s function has continuous partial derivatives in terms of a,, so we can get a neces- sary condition for the best a, (see Equation 9).

Simplifying, we obtain normal equations for channels 1 and 2 (see Equations 10 and 11). We can express Equations 10 and 11 as the linear system in Equation 12. ( O j , OJ, qY, p j , p i ) are the data points. p1 and p2 are the reflectances of channels 1 and 2. all and azl are the coefficients for channels 1 and 2. We can solve this linear system by Gaussian elimination and then back- ward substitution. Although the Cholesky de- composition method is the most efficient way to solve the normal equations, for many applica- tions Gaussian elimination with back substitu- tion is quite adequate.”

To form the normal equations, we need to evaluate the functionsfl(B,, e,, 4, A) for the given points. The number of floating-point operations needed to determine these functions depends on the model used. Suppose the evaluation off; at a single point requires Ffflops. So, to form the normal equations, we need Nd(F, + N,‘ + SN,) operations using the symmetric properties of the left-hand side ATA of Equation 12.

For the linear system in Equation 12, Gauss- ian elimination followed by backward substi- tution will require approximately

20 IEEE COMPUTATIONAL SCIENCE & ENGINEERING

2N: 5N2 IlN, -+l+-

3 2 6

flops. Hence, we need

flops per pixel.

The nonlinear model For the CSAR model, we need a nonlinear

least-squares fit. We can write the nonlinear model as p =f(a, e,, e,, 4, A), where a is the model coefficient vector. In particular, a = [Po, k, 01'' for the CSAR model. As with the linear least-squares fit, we aim a t minimizing the cu- mulative squared error for each channel. For channel 1, for example, the cumulative error is as shown in Equation 1 3 .

Several numerical methods to minimize this error are available.'* We used Powell's algo- rithm," because it has been successfully used in the inversion of BRDF and numerical radiative- transfer algorithms.13 The basic idea is to change the multidimensional minimization problem to a sequence of line minimizations, which mini- mizes the function ?(a) along some vector di- rection n using 1 D methods.

We can describe Powell's algorithm as

Let a. be an initial guess of the coefficients, and let ui (i = 1,2 , . . . , N,) be the N,-dimensional basis vector-that is, U, = e,. Then repeat the follow- ing steps until the functionJ(a) stops decreasing:

(I) Save the starting point as ao. (2) For i = 1, 2, ... N,, compute vi, which mini-

(3) For i = 1 , 2 , .. ., Nc-l, replace U, by U, +

(4) Replace uhTc by aNc - ao. (5) Compute 77, which minimizes J(ao + vuNc ,

and call this point ao.

mizesJ(a /-, + uj), and call this point a/.

1 We can express the line-minimization prob-

lem as

Given the input vector a, the direction vector n, and the functionJ(a), find the scalar 77 that min- imizes J(a + vn). Replace a by a + qn, and n by vn."

We can change the multidimensional function J(a) along the line going through a in the direc- tion n to a ID functionJ1(q)." That is, we first evaluate (a +qn) and then compute the function

value of the original function. So, we can think ofJl as a function of 7.

To solve such a ID line-minimization prob- lem, we first bracket the minimum along this line. We know that a minimum is bracketed only when there is a triplet of points, a < b < c, such thal-J,(b) is less than bothJ,(a) andJ,(c). In this case, we know that the functionJI(q) has a min- imum in the interval (a,c). We can start with some initial guess as the bracketing triplet's left point and then step downhill to find the brack- eting triplet's middle point. To find the middle point, we can increase the step size either by a constant factor or by the result of a parabolic ex- trapolation to the preceding points that will take us to the extrapolated turning point." Then we just need to take a big enough step to stop the downhill trend and get a high third point.

P/e can then solve the line-minimization prob- lem using Brent's method. The basic idea is to it- eratively shrink the bracketing interval that con- tains the minimum of the functionJl(q). Given the initial bracketing triplet (a, b, c), we can choose a new point x either between a and b or between b and c. For example, if x is between a and b, the next bracketing triplet is (x, b, c) when Jl(b) <Jl(x); otherwise, the next bracketing triplet is (a, x, b) whenJl(b) >Jl(x). In any case, the new triplet's middle point is the abscissa whose func- tion value is the best minimum so far. We find the minimum in the interval (a, c) by continuing the bracketing process until the distance between the triplet's two outer points is tolerably small. Brent's method first attempts to use parabolic interpolation to find x; if that fails, it switches to golden-section seart-h. For parabolic interpretation, the formula for the abscissa x that is the minimum of a paratbola through those three points is as shown in Elquation 14. In golden-section search, x is 0.3 8 197 (the golden ratio) of the larger of the two intervals measuring from the central point b of the triplet." XI determine the number of operations for the

CSlLR model, let 1, be the number of iterations to bracket the minimum in the line-minimiza- tion part. I, is the number of iterations required in Brent's method, and I3 is the number of itera- tions needed by Powell's method. For each iter- ation in Powell's method, we need to do (N, + 1) line minimizations and evaluate the error func- tion J(a) once, plus some other memory and boo keeping operatlons. In our case, Nd = 2 5, N, = 3,11 = 3,12 = 18, and13 = 22 onaverage. Hence, we need 3.9 Mflops per pixel to do the nonlinear leas-squares fit for the CSAR model.

OCTOBER-DECEMBER 1998 21

1983 1984 1985 1986

January February December January November December

1 0-day 1 0-day 1 0-day comoosite comoosite comoosite

1 0-day 1 0-day 1 0-day comoosite comoosite comoosite

NDVl QC DOY NDVl QC DOY layer layer layer layer layer layer

NDVl QC DOY NDVl QC DOY layer layer layer layer layer layer

Figure 2. The structure of the input file system. Each year has 12 months, each month has three 10-day composites, each composite has nine layers, and each layer is stored in a file.

Generating model-verification information As mentioned before, using the generated

model coefficients, we compute channels 1 and 2 reflectances and NDVI for each pixel in every composite using the pixel’s geometry. Then we also compute the standard error and R2 between the predicted and measured reflectances for all three models. We also compute the mean and standard deviation of NDVI in the PAL data. Overall, we need Nd(2N, + 22) flops per pixel to perform these computations.

High- performance retrieval algorithms Global BRDF retrieval involves large computa- tional requirements per pixel and requires ex- tensive handling of large amounts of data in external storage. So, we developed an imple- mentation on a multiprocessor system where each node has a sufficiently large storage sub- system. Our main goal is a general strategy that optimally uses the available resources.

Initial data layout We want to minimize the total U 0 time and

to balance the computational loads among the nodes. Therefore, we seek an efficient layout of the input imagery on the available disks and an efficient mapping of the computation across the available processors.

Our input (the PAL 10-day composite data) can be viewed as an array, Datapear] [Month] [Period] [Layer] [Row] [Column]. Figure 2 illus- trates the initial collection of input files. The in- put comprises four years, 12 months per year,

three periods per month, nine layers per period, and the global pixels given by 2,168 rows and 5,004 columns. The nine layers account for 17 bytes for each pixel. For example, on our multi- processor platform, a node’s disk array will con- tain nine 1 0-day composites comprising 8 1 im- age files. The image files can be accessed by giving the index wear] [Month] [Period] [Layer].

If we use a Time index to refer to (Year, Month, Period), we can view the input as a 4D array, Data[Time] [Layer] [Row] [Column]. The time will vary from 1 to 144 (periods) in our case. There are 1,296 input image files and 27 Gbytes of data (144 x 17 x 2,168 x 5,004). We distrib- ute this 4D array over the nodes according to the Time index. That is, if we have p nodes and the Time index ranges from 1 to T (assuming Tis a multiple ofp without loss of generality), Data[ 1 : T/p] [Layer] [Row] [Column] lies on the disks of node 1, Data[T/p + 1 : 2T/p] [Layer] [Row] [Col- umn] lies on the disks of node 2, ..., and Data[((% - l)r)/p + 1 : nT/p][Layer][Row][Column] lies on the disks of nodep.

In addition to the data, we use a land-cover map14 to classify a pixel as land or water. We also use the map to give the surface’s class type, which provides the initial guess values of p, k , and 0 in the CSAR model. The land-cover map consists of a global image with one byte per pixel and is approximately 11 Mbytes. This LandCov- erMap[Row][Column] is stored on one node.

Output data layout We will generate all the model coefficients,

the standard error, and the R2 coefficient for each channel, as well as the mean NDVI, the

22 IEEE COMPUTATIONAL SCIENCE & ENGINEERING

I Hiah-oedormance rswitch + Node 1

. . .

Figure 3. In this example, land data from three months (nine IO-day composite images) is stored on the local disk of each node. After redistribution, all the four years‘ data from the top c land pixels will reside on node 1, the second c land pixels will reside on node 2, and so on.

NDVI standard deviation, and the NDVI stan- dard error. Each generated output will be a 32- bit floating-point number for each pixel. Then, we will produce NumCoef= 2(Nc + 2) + 3 floating images (44 Mbytes each) and one unsigned short image output for the number of points used in each model fitting. This amounts to approxi- mately 670 Mbytes for the Modified Walthall model, 1 .O Gbytes for the temporal model, and 590 Mbytes for the CSAR nonlinear model. We can view these outputs as a 3D array O u t v u m - Coefl [Row]Column] and one Num[Row] [Col- umn] unsigned short image.

Data redistribution and computation mappings

Given a p-processor platform, we implement each BRDF algorithm as follows. Using the land-cover map, we generate a land-sea map that describes the relative land position in the original image. We then broadcast this land-sea map to all the nodes. We then extract the land- pixel data a t each node, using an intermediate data structure, DateIntervear] [Month] [Land- Pixel] [Period] payer]. That is, we first eliminate the nonland pixels in the data, then save the re- maining pixels. This is the preprocessing phase.

Next, we redistribute the land data evenly among the nodes so that all the four years’ in- formation for a pixel is at the same node. This results in approximately 380 Mbytes of data per node for p = 16. Given the data’s size, we per- form the redistribution iteratively as follows. We read c x p consecutive land pixels during each it- eration (cis the maximum number of pixels that

can fit into each node’s main memory). The time required for this step is Tlnput. The algorithm ships the first c land pixels to the first node, the second c land pixels to the second node, and so on (see Figure 3). So, each node gets all the Iyear] [Month] [Period] [Layer] information for its c land pixels. This involves data movement in each processor and data communication among processors. We use the Message Passing Inter- fact. collective communication primitive MPI- AI 1. t oal1 to do the interprocessor communi- cation in the land-pixel redistribution. The time required for this step is T,,,.

We are now ready to perform the BRDF model fitting, for all the pixels on each node concurrently. We first select the clear-day data to form the curve-fitting data set according to the CLAVR layer. We then compute the model coefficients, generate the model-verification in- formation, and save them locally as temporary files. The time required for this step is Tcomp.

Figure 4 gives the histogram of condition num- bers for matrix A in our applications. A condition number is a measure of the sensitivity of the so- lution’s relative error to the changes in the right- hand side b of Equation 12. It gives us the solu- tion’s accuracy using Gaussian elimination and backward substitution to solve the system. (Con- dition numbers can be measured only for linear curve fitting.) The curve fitting is ill-conditioned if the condition number is too large; that is, if the reciprocal of the condition number approaches the machine’s floating-point precision. Figure 4 shows that most of our curve fitting is well- conditioned.

OCTOBER-DECEMBER 1998 23

h & 10 ul a, - .% 8

p 6

4

2

- m

0 20 40 60 80 100 120 140 160 180 200 20 40 60 81

Condition number la) Condition number (b) L D 100 120 140 160 180 200

Figure 4. Condition numbers for the least-square fits of the (a) Modified Walthall model and (b) temporal model.

8

60 17~ lnput Temporal model

Data movement - 50 Computation ~

output

z 4 0 - Modified Walthall model

E 30

a, 3 - v

E I= 20

n; --

5 E 10

E 8

6

4

2

n 0

v

a,

I-

1G

output " 1 2 4 8 1 6 1 2 4 8 1 6 Input Data movement Computation (4 Number of nodes (b)

Figure 5. Timing of global BRDF retrieval from the four-year PAL data set on an SP2: (a) running time and (b) scalability (for the Modified Walthall model). The figure does not show the time to run the CSAR (Coupled Surface-Atmosphere Re- flectance) model on 16 nodes (150 minutes).

After we've processed all the land pixels, we read back the coefficients from the temporary file one by one and collect them into one node, using the MPI-Gather communication primi- tive. Using the land-sea map, we reinsert the nonland information to transform the output map back to the original global image. The time required for this step is Toutput.

Given that the loads are distributed equally among the processors-that is, each node will have l/p of the total land pixels-TComp and Tnput should scale linearly as the number of processors increases. Tu,,,, should scale almost linearly as the number of processors increases because we are using balanced collective-communication primitives. Toutput should stay constant as the

number of nodes increases, because one node collects all the output data.

Implementation and performance results Our testbed consists of a 16-node IBM SP2.

E,ach node consists of one Power2 processor, a 40-Mbyte-per-second bidirectional link for each processor to a multistage high-performance switch, and two fast-wide SCSI buses. Each SCSI bus has three 2.2-Gbyte SCSI disks.

Figure 5 summarizes the performance of our algorithms on this testbed. The preprocessing phase achieves linear speedup as the number of processors increases. As we expected, Tcomp and T,nput scale linearly, Tmove scales almost linearly, and Toutput remains fairly constant.

24 IEEE COMPUTATIONAL SCIENCE & ENGINEERING

0 0 I -4 0'0

0 0 20.0 40.0 60.0 0.0 20.0 40.0 60.0 (a) Reflectance (%) (b) Reflectance (%)

Figure 6. The coefficient a3 from the Modified Walthall model for channels (a) 1 and (b) 2 of AVHRR for the third quarter (July-September) of 1983-86. a3 represents nadir reflectance values for this model.

Regarding raw performance, on 16 nodes, the least-squares fit computation achieves approxi- mately 0.89 Gflops for the Modified Walthall model and 0.91 Gflops for the temporal model. The nonlinear least-squares fit achieves approx- imately 1.18 Gflops for the CSAR model. These numbers represent approximately 2 5 % of the peak performance on our machine. The input bandwidth is approximately 6.7 Mbytes per sec- ond, and the output bandwith is approximately 6.0 Mbytes per second. The peak of either is 8 Mbytes per second for each disk. The inter- processor-communication bandwidth is about 2 5 Mbytes per second with a theoretical peak of 40 Mbytes per second. These numbers clearly indicate that our code efficiently uses the avail- able resources.

Comparing the BRDF rnodels The coefficients for each model corresponding to channels 1 and 2 of AVHRR could serve to discriminate between different land-cover types.6> Consider, for example, the coefficient a3 from the Modified Walthall model for the third quarter Uuly-September) (see Figure 6). This coefficient is the nadir reflectance value for an overhead sun position in the individual AVHRR bands for this model. To our knowl- edge, this is the first time the coefficients of any BRDF model have been determined at a global scale.

The spatial patterns of visible and near-IR re- flectances shown here are consistent with dif- ferent land-cover ~ l a s s e s . ' ~ ~ ' ~ Densely vegetated areas (for example, central Africa, Brazil, and temperate and boreal forests in Asia and Eu- rope) and agricultural regions (for example, cen- tral USA) show high reflectances in channel 2

and very low reflectances in channel 1. How- ever, deserts (for example, the Sahara, central Australia, and the Kalahari) show high re- flectances in both bands. Insufficient data across the snow-covered Himalayas, over the polar re- gions, and along the west coast of South Amer- ica resulted in null values in these areas, which are caused by CLAW'S rejection of these pix- els. On the other hand, the spatial structure of coefficients a. to a2 of the Modified Walthall model was poor when compared with the spa- tial stratification of the land-cover map.

Figure 7 shows the parameter k for channels 1 and 2 of AVHRR for the CSAR model for the third quarter. This parameter describes the vari- ations in reflectance from the view and illumi- nation angles. Thus, it indicates the level of sur- fact. anisotropy.6 As Figure 7 clearly shows, this parameter's spatial variations are closely related to the variations in land-cover types. Lower val- ues of k mean higher anisotropy. Vegetated areas show higher anisotropy in channel 1 compared to channel 2. Deserts, on the other hand, are less anisotropic in both channels compared to vege- tated areas. Among the vegetated areas, high- latitude deciduous forests have higher values of k compared to tropical areas. The other two co- efficients of the CSAR model @o and 0) exhibit similar spatial stratification corresponding to variations in different land-cover types.

The standard errors in channel 1 and 2 re- flectances were very similar from all three mod- els when we divided the data into quarters. Fig- ure 8 shows the histograms of the standard errors in channels 1 and 2 for the third quarter. However, if we retrieve the BRDF without di- viding the input data into quarters, the temporal model gives better results than the other two, because it accounts for phenological changes.

OCTOBER-DECEMBER 1998 25

0 io ole (a) -1 .o 0 0 1 .o (b) -1 0 0 0 1 .o

Figure 7. The coefficient k from the CSAR model for channels (a) 1 and (b) 2 of AVHRR for the third quarter of 1983-86. Lower values of k represent regions of high surface anisotropy.

Figure 9 shows the standard error between the model-predicted and the measured re- flectances from all three algorithms over Eu- rope, when the data is not divided into quarters. For seasonally invariant land-cover types such as the Sahara, all three models perform equally well. In higher latitudes with deciduous vegeta- tion that has a strong seasonal variation, the temporal model performs better.

So, the temporal model has a distinct advan- tage over the other two because it does not re- quire dividing the data into discrete time inter- vals. Also, we can apply a single set of coefficients from this model to estimate the BRDF at any given time. Computationally, the CSAR model is more expensive (150 minutes) than either the Modified Walthall model (15 minutes) or the temporal model (1 8 minutes).

he results from this study are unique and should provide valuable inputs into BRDF retrieval from future Earth Observation System (EOS)

instruments such as the Moderate Resolution Imaging Spectroradiometer (MODIS) and the Multiangle Imaging Spectroradiometer (MISR), because both the CSAR and the Modified Walthall models are among the suite of BRDF algorithms chosen for these missions. Based on the data set we used, our results show that com- putationally intensive BRDF models such as the CSAR model do not necessarily provide better results than simple empirical models. However, further research needs to be conducted to assess the relative merits of inverting simple and com- plex BRDF models in characterizing land-cover information. Clearly, our results show that such data and computationally intensive satellite-data- processing problems require high-performance

computing methods, and carrying out such applications operationally at a global scale in a single-processor environment might not be feasible.

Acknowledgments We thank Anurag Acharya for the helpful discussion on the SP2’s //0 configuration. We also thank the UMlACS parallel-system staff, Mitchell Murphy, Matt Beal, and Galen Wilkerson, for their retrieving of the data and system support. Useful comments by three anonymous reviewers are greatly appreciated. NSF Grand Challenge Grant No. BIR-93 181 83 partlysupported this work.

References 1. F.E. Nicodemus e t al., Geometrical Considerations

and Nomenclature for Reflectance, Report NBS MN- 160, Nat’l Bureau of Standards, Washington, D.C., 1977.

2. A.H. Strahler and I . Muller, MODIS BRDF/Albedo Product: Algorithm Theoretical Basis Document Ver- sion 4.0; ATBD-MOD-09, NASA Goddard Space Flight Center, Greenbelt, Md., 1996; http:// modarch.gsfc.nasa.gov/MODIS/ATBD/atbd-mod 09.pdf.

3. B.N. Holben, “Characteristics of Maximum-Value Composite Images from Temporal AVHRR Data,” Int?). Remote Sensing, Vol. 7, No. 11, Nov. 1986, pp. 141 7-1 434.

4. V. Ramanathan, Scientific Use of Surface Radiation Budget Data for Climate Studies, NASA Reference Publication 11 69, NASA, Washington, D.C., 1986.

5 . T. Nilson and A. Kuusk, “A Reflectance Model for the Homogeneous Plant Canopy and Its Inver- sion,‘‘ Remote Sensing of Environment, Vol. 27, No. 2, Feb. 1989, pp. 157-1 67.

6. H. Rahman, B. Pinty, and M.M. Verstraete, “ C o u - pled Surface-Atmosphere Reflectance Model (CSAR), 2. Semi-Empirical Surface Model Usable with N O M AVHRR Data,“ I . Geophysical Research,

26 IEEE COMPUTATIONAL SCIENCE & ENGINEERING

"I

2 3 4 5 6 Standard error (Yo)

0

7 -

6 -

8 1

n

7

6 h

E5

2 3 4 5 6 Standard error (%)

2 3 4 5 Standard error (%)

U 0 1 2 3 4 5 6 " 0 1 (4 Standard error (%) (6 Standard error (%)

Figure 8. Standard error from the third quarter of 1983-86: Modified Walthall model, channels (a) 1 and (b) 2; CSAR model, channels (c) 1 and (d) 2; temporal model, channels (e) 1 and (9 2. For the temporal model, we use all four years' data t o derive the model coefficients and apply the coefficients t o compute the reflectances and standard errors for the third quarter.

OCTOBER-DECEMBER 1998 27

9.0 12.0 15.0 0.0 3.0 6.0 Standard errors in retrieved reflectances (%)

Figure 9. Spatial distribution of standard error, for the combined input data from all four years over Eu- rope (that is, the data is not divided into quarters for retrieving the BRDF coefficients): Modified Walt- hall model, channels (a) 1 and (b) 2; CSAR model, channels (c) 1 and (d) 2; and temporal model, chan- nels (e) 1 and (f) 2.

28 IEEE COMPUTATIONAL SCIENCE & ENGINEERING

Vol. 98, No. D1 1, Nov. 1993, pp. 20791-20801. 7. S. Liang and J.R.G. Townshend, “Angular Signa-

tures of NASA/NOAA Pathfinder AVHRR Land Data and Applications to Land Cover Identification,” Proc. lGARSS ’97: lnt‘l Geoscience and Remote Sens- ing Symp., IEEE Press, Pisacataway, N.J., 1997, pp. 1781-1 783.

8. M.E. James and S.N.V. Kalluri, “The Pathfinder AVHRR Land Data Set: An Improved Coarse Reso- lution Data Set for Terrestrial Monitoring,” lnt’ l ]. Remote Sensing, Vol. 15, No. 17, Nov. 1994, pp.

9. L.L. Stowe e t al., ”Global Distribution of Cloud Cover Derived from NOAA/AVHRR Operational Satellite Data,” Advances in Space Research, Vol. 11, No. 3, Pergamon Press, Oxford, UK, 1991, pp. 51-54.

10. R.S. Defries and J.R.G. Townshend, “NDVI-Derived Land Cover Classifications a t a Global Scale,” lnt? 1. Remote Sensing, Vol. 15, No. 1 7, Nov. 1994, pp.

11. W.H. Press et al., Numerical Recipes in C, 2nd ed., Cambridge Univ. Press, New York, 1992.

12. R.B. Myneni et al., ”Optical Remote Sensing of Vegetation: Modeling, Caveats, and Algorithms,” Remote Sensing of Environment, Vol. 51, No. 1, Jan.

13. S. Liang and A.H. Strahler, “Retrieval of Surface BRDF from Multiangle Remotely Sensed Data,” Remote Sensing of Environment, Vol. 50, No. 1,

14. R.S. Defries et al., ”Global Land Cover Classifica- tions a t 8km Spatial Resolution: The Use of Train- ing Data Derived from Landsat Imagery in Deci- sion Tree Classifiers,” to be published in the lnt? 1. Remote Sensing.

3347-3363.

3567-3586.

1995, pp. 169-1 88.

Oct. 1994, pp. 18-30.

Zengyan Zhang is a research assistant at the Univer- sity of Maryland’s Institute for Advanced Computer Studies and is a PhD candidate in electrical engineer- ing a t the university. His research interests are high- performance computing, data-intensive computing, distributed and parallel systems, remote sensing, and image processing. He received his BS in electrical en- gineering from Fudan University, Shanghai, China, and his MS in computer engineering from the University of Maryland, College Park. Contact him a t UMIACS, AWV Bldg., Univ. of Maryland, College Park, MD 20742; zengyan@umiacs.umd.edu.

Satya N.V. Kalluri is an assistant research scientist in the University of Maryland’s Department of Geogra- phy. His research interests include the development of algorithms for satellite data processing and land-

surface characterization for energy-balance studies. He previously was a validation scientist for the National Oceanic and Atmospheric Administration-NASA Path- finder AVHRR Land project a t NASA’s Goddard Space Flight Center. He received his MSc in geographyfrom Osmania University, Hyderabad, India, and his PhD in geography from the University of Maryland, College Park. Contact him a t the Dept. of Geography, Univ. of Maryland, College Park, MD 20742; sk71 @urnail. umd.edu.

Joseph JaJa is the director of the University of Mary- land’s Institute for Advanced Computer Studies and a professor of electrical engineering at the university. His research interests are parallel algorithms and high- performance computing, and particularly data-inten- sive applications such as those in earth systems sci- ence. He wrote An lntroduction to Parallel Algorithms (Addison-Wesley, 1992), is the subject-area editor on parallel algorithms for the journal of Parallel and Dis- tributed Computing, and is an associate editor of / €E€ Transactions on Parallel and Distributed Systems. He received his PhD in applied mathematics from Har- vard University. Contact him a t UMIACS, AWV Bldg., Univ. of Maryland, College Park, MD 20742; joseph@ umiacs.umd.edu.

Shunlin Liang is an assistant professor in the University of Maryland’s Department of Geography. His research interests are computer cartography, land surface- atmosphere radiative transfer modeling, quantitative information extraction from satellite data, and large- volume spatial-data handling. He previously was a val- idation scientist for the National Oceanic and Atmos- pheric Administration-NASA Pathfinder AVHRR Land project. He received his PhD in remote sensing and geographic information systems from Boston Univer- sity. Contact him a t the Dept. of Geography, Univ. of Maryland, College Park, MD 20742; sliang@geog. umd.edu.

John R.G. Townshend is a professor of geography a t the University of Maryland, College Park, is a member of the University’s Institute for Advanced Computer Studies, and is the acting director of the University’s Earth Systems Science Interdisciplinary Center. He is also a member of the EOS-AM MODIS (Earth Observ- ing System-AM Moderate Resolution Imaging Spec- troradiometer) Science Team, and he chairs the Joint Scientific and Technical Committee of the Global Cli- mate Observing System. He received his BSc and PhD in geography from University College, London. Con- tact him a t the Dept. of Geography, Lefrak Hall, Rrn. 113, Univ. of Maryland, College Park, MD 20742; jt59@umail.umd.edu.

OCTOBER-DECEMBER 1998 29