Ratliff et al. Vol. 19, No. 9 /September 2002 /J. Opt. Soc. Am. A 1737

An algebraic algorithm for nonuniformitycorrection in focal-plane arrays

Bradley M. Ratliff and Majeed M. Hayat

Department of Electrical and Computer Engineering, University of New Mexico, Albuquerque,New Mexico 87131-1356

Russell C. Hardie

Department of Electrical and Computer Engineering, University of Dayton, Dayton, Ohio 45469-0226

Received December 17, 2001; revised manuscript received April 10, 2002; accepted April 16, 2002

A scene-based algorithm is developed to compensate for bias nonuniformity in focal-plane arrays. Nonunifor-mity can be extremely problematic, especially for mid- to far-infrared imaging systems. The technique isbased on use of estimates of interframe subpixel shifts in an image sequence, in conjunction with a linear-interpolation model for the motion, to extract information on the bias nonuniformity algebraically. The per-formance of the proposed algorithm is analyzed by using real infrared and simulated data. One advantage ofthis technique is its simplicity; it requires relatively few frames to generate an effective correction matrix,thereby permitting the execution of frequent on-the-fly nonuniformity correction as drift occurs. Additionally,the performance is shown to exhibit considerable robustness with respect to lack of the common types of tem-poral and spatial irradiance diversity that are typically required by statistical scene-based nonuniformity cor-rection techniques. © 2002 Optical Society of America

OCIS codes: 100.2000, 100.2550, 110.4280, 110.3080, 100.3020.

1. INTRODUCTIONFocal-plane array (FPA) sensors are used in many mod-ern imaging and spectral sensing systems. An FPA sen-sor consists of a mosaic of photodetectors placed at the fo-cal plane of an imaging system. Despite the widespreaduse and popularity of FPA sensors, their performance isknown to be affected by the presence of fixed-patternnoise, which is also known as spatial nonuniformity noise.Spatial nonuniformity occurs primarily because each de-tector in the FPA has a photoresponse slightly differentfrom that of its neighbors.1 Despite significant advancesin FPA and detector technology, spatial nonuniformitycontinues to be a serious problem, degrading radiometricaccuracy, image quality, and temperature resolution. Inaddition, spatial nonuniformity tends to drift temporallyas a result of variations in the sensor’s surroundings.Such drift requires that the nonuniformity be compen-sated for repeatedly during the course of camera opera-tion.

There are two main types of nonuniformity correction(NUC) techniques, namely, calibration-based and scene-based techniques. The most common calibration-basedcorrection is the two-point calibration.2 This method re-quires that normal imaging system operation be inter-rupted while the camera images a flat-field calibrationtarget at two distinct known temperatures. The nonuni-formity parameters are then solved for in a linear fashion.For IR systems, a blackbody radiation source is typicallyemployed as the calibration target. Calibration must beused when accurate temperature or radiometric measure-ment is required. However, radiometric accuracy is notneeded in some applications. Therefore, a considerable

1084-7529/2002/091737-11$15.00 ©

amount of research has recently focused on developingscene-based correction algorithms, which can provide sig-nificant improvement in image quality at the expense ofreduced radiometric accuracy. These techniques typi-cally use a digital image sequence and rely on motion toprovide diversity in the irradiance observed by each de-tector. Some scene-based algorithms in the literature in-clude those by Narendra and Foss,3,4 Harris andChiang,5,6 and Chiang and Harris,7 whose algorithms re-peatedly compensate for gain and bias nonuniformity.These methods rely on the concept of constant statistics,which assumes that, over time, the mean and variance ofthe irradiance of the scene become spatially invariant.O’Neil8,9 and Hardie et al.10 developed motion-based algo-rithms that use the principle that detectors should havean identical response when observing the same scenepoint at different times. Hayat et al.11 developed a sta-tistical algorithm that relies on the assumption that alldetectors in the array are exposed to the same range ofcollected irradiance within a sequence of frames. Thistechnique therefore relaxes the constant-statistics as-sumption to a constant-range assumption. Torres andcolleagues12,13 recently adopted the constant-range as-sumption and developed a Kalman-filtering techniquethat also captures and estimates stochastic drift in thegain and bias nonuniformity.

A new scene-based algorithm that is algebraic in na-ture is proposed in this paper. We assume that each FPAdetector output obeys an approximate linear irradiance–voltage model in which only detector bias nonuniformityis considered. Global motion between adjacent imageframes is estimated with a reliable gradient-based shift-

2002 Optical Society of America

1738 J. Opt. Soc. Am. A/Vol. 19, No. 9 /September 2002 Ratliff et al.

estimation algorithm. Through use of these motion esti-mates, a number of gradient-type matrices are computedusing pairs of consecutive frames demonstrating pure ver-tical and pure horizontal subpixel shifts. These matricesare then combined to form an overall correction matrixthat is used to compensate for bias nonuniformity acrossthe entire image sequence. The algorithm provides aneffective bias correction with relatively few frames. Thiswill be demonstrated with both simulated and real infra-red data. Because of its highly localized nature in timeand space, the algorithm is easily implemented and com-putationally efficient, permitting quick computation ofthe correction matrix that is needed to compensate con-tinually for nonuniformity drift. Another advantage ofthis algorithm is that it requires only local temporal irra-diance variation. Many algorithms (including those thatare based on constant-statistics or constant-range as-sumptions) rely on the assumption that each detector isexposed to a wide range of irradiance levels. Such algo-rithms typically suffer in performance when a portion ofan image sequence lacks sufficient irradiance diversityover time. As demonstrated in this paper, the algebraictechnique exhibits considerable robustness to such limita-tions.

This paper is organized as follows. The sensor andmotion models are given in Section 2. The algorithm isderived in Section 3, and its performance is studied inSection 4. Finally, the conclusions and future extensionsof the proposed technique are given in Section 5.

2. SENSOR AND MOTION MODELSConsider an M 3 N image sequence yn generated by aFPA sensor, where n 5 1, 2, 3,... represents the imageframe number. A commonly used approximate linearmodel14 for a FPA-sensor output is given by

yn~i, j ! 5 a~i, j !zn~i, j ! 1 b~i, j !, (1)

where zn(i, j) is the irradiance, integrated over the detec-tor’s active area within the frame time, and a(i, j) andb(i, j) are the detector’s gain and bias, respectively. Inmany sensors, the bias nonuniformity dominates the gainnonuniformity and the latter can be neglected. In thispaper we will restrict our attention to such cases andtherefore assume that the gain is uniform across all de-tectors with a common value—without loss ofgenerality—of unity. Thus the observation model be-comes

yn~i, j ! 5 zn~i, j ! 1 b~i, j !. (2)

It is convenient to associate with each M 3 N sensor abias nonuniformity matrix B defined by

B 5 F b~1, 1 ! b~1, 2 ! ¯ b~1, N !

b~2, 1 ! b~2, 2 ! ¯ b~2, N !

] ] ] ]

b~M, 1! b~M, 2! ¯ b~M, N !

G . (3)

Moreover, we define I(B) to be the collection of all imagesthat can be generated by a sensor whose bias matrix is B,

according to the model given by Eq. (2). A sensor havinga bias matrix B will be referred to as a B-sensor.

We also assume that the temperature of the observedobjects does not change during the interframe time inter-val. Thus if two consecutive frames are seen to exhibitstrictly horizontal or vertical subpixel global motion, wemay then approximate the irradiance at a given pixel inthe second frame as a linear interpolation of the irradi-ance at the pixel and its neighbor from the first frame.This linear interpolation model is selected for its simplic-ity as an approximation. More precisely, for a pair of con-secutive frames exhibiting a purely vertical subpixel shiftof a pixels, denoted as an a-pair, the kth and the (k1 1)th frames are related by

yk11~i 1 1, j ! 5 azk~i, j ! 1 ~1 2 a!zk~i 1 1, j !

1 b~i 1 1, j !, 0 , a < 1. (4)

Similarly, for a pair of adjacent frames with a purely hori-zontal subpixel shift of b pixels (denoted as a b-pair), wehave

ym11~i, j 1 1 ! 5 bzm~i, j ! 1 ~1 2 b!zm~i, j 1 1 !

1 b~i, j 1 1 !, 0 , b < 1. (5)

By convention, a positive a represents downward mo-tion of the scene and a positive b represents rightwardmotion. In the next section we exploit the relationshipsgiven by Eqs. (4) and (5) and use pairs of observed framesto compute algebraically a bias correction map.

3. ALGORITHM DESCRIPTIONWe begin by discussing the key principle behind the algo-rithm. The algorithm is based on the ability to exploitshift information between two consecutive image framesexhibiting a purely vertical shift, say, to convert the biasvalue in a detector element to the bias value of its verticalneighbor. This mechanism will, in turn, allow us to con-vert the biases of detectors in an entire column to a com-mon bias value. The procedure can then be repeated forevery column in the image pair, resulting in an imagethat suffers from nonuniformity across rows only (i.e.,each column has a different, yet uniform, bias value).Now, with an analogous procedure and by using a pair ofhorizontally shifted images, we can unify the bias valuesacross all rows, which ultimately allows for the unifica-tion of all biases in the array to a common value. Thedetails of the algorithm are given next.

A. Bias Unification in Adjacent DetectorsSuppose that we have two consecutive image frames(frames n and n 1 1, say) for which there is a purely ver-tical shift a between them. Without loss of generality,consider the output at detectors (1, 1) and (2, 1) of theFPA. According to Eq. (2),

yn~1, 1 ! 5 zn~1, 1 ! 1 b~1, 1 !, (6)

yn~2, 1 ! 5 zn~2, 1 ! 1 b~2, 1 !. (7)

Moreover, we know from the interpolation model of Eq. (4)that

Ratliff et al. Vol. 19, No. 9 /September 2002 /J. Opt. Soc. Am. A 1739

yn11~2, 1 ! 5 azn~1, 1 ! 1 ~1 2 a!zn~2, 1 ! 1 b~2, 1 !.(8)

Now the key step is to form a linear combination ofyn(1, 1), yn(2, 1), and yn11(2, 1) so that the irradiancevalues are canceled and only the biases remain. Moreprecisely, we form

VB~2, 1 ! 51

a@ayn~1, 1 ! 1 ~1 2 a!yn~2, 1 !

2 yn11~2, 1 !#, (9)

(we will formally define the matrix VB later) which, upon

use of Eqs. (6), (7), and (8), reduces to

VB~2, 1 ! 5 b~1, 1 ! 2 b~2, 1 !. (10)

Now if VB(2, 1) is added to yn(2, 1), we obtain

yn~2, 1 ! 1 VB~2, 1 ! 5 zn~2, 1 ! 1 b~1, 1 !. (11)

Hence the bias of the (2, 1)th detector element b(2, 1) isconverted to b(1, 1).

This same procedure may now be applied to the bias-adjusted (2, 1)th pixel [that is, yn(2, 1) 1 VB(2, 1)] andthe original (3, 1)th pixel effectively to convert the bias ofthe (3, 1)th detector from b(3, 1) to b(1, 1). This proce-dure may be applied down the entire column, effectivelyunifying all biases of the column to b(1, 1).

Clearly we can repeat the same procedure for other col-umns with the same a-pair. Moreover, an analogous pro-cedure can be performed across all rows with a b-pair.The general algorithm will be discussed next.

B. Vertical and Horizontal Correction MatricesWe begin by defining an intermediate vertical correctionmatrix VB with an a-pair for a B-sensor as follows: Forj 5 1, 2,...,N, put VB(1, j) 5 0, and for i 5 2, 3,..., M, de-fine

VB~i, j ! 51

a@ayn~i 2 1, j ! 1 ~1 2 a!yn~i, j !

2 yn11~i, j !#

5 b~i 2 1, j ! 2 b~i, j !. (12)

Hence

Now the vertical correction matrix VB is calculated byperforming a partial cumulative sum down each columnof VB . More precisely, for i 5 2, 3,..., M and j5 1, 2,...,N, we define

VB~i, j ! 5 (r52

i

VB~r, j ! 5 b~1, j ! 2 b~i, j !, (14)

so that

Indeed, in the vertically corrected frame of Eq. (16), thebias values down each column have effectively been uni-fied to the bias value of the topmost pixel (the column-bias ‘‘leader’’).

Now if we define the column-corrected sensor bias ma-trix

B8 5 F b~1, 1 ! b~1, 2 ! ¯ b~1, N !

] ] ] ]

b~1, 1 ! b~1, 2 ! ¯ b~1, N !G , (17)

VB 5 F 0 0 ¯ 0

b~1, 1 ! 2 b~2, 1 ! b~1, 2 ! 2 b~2, 2 ! ¯ b~1, N ! 2 b~2, N !

b~2, 1 ! 2 b~3, 1 ! b~2, 2 ! 2 b~3, 2 ! ¯ b~2, N ! 2 b~3, N !

] ] ] ]

b~M, 1, 1 ! 2 b~M, 1! b~M 2 1, 2 ! 2 b~M, 2! ¯ b~M 2 1, 2 2 b~M, N !

G . (13)

VB 5 F 0 0 ¯ 0

b~1, 1 ! 2 b~2, 1 ! b~1, 2 ! 2 b~2, 2 ! ¯ b~1, N ! 2 b~2, N !

b~1, 1 ! 2 b~3, 1 ! b~1, 2 ! 2 b~3, 2 ! ¯ b~1, N ! 2 b~3, N !

] ] ] ]

b~1, 1 ! 2 b~M, 1! b~1, 2 ! 2 b~M, 2! ¯ b~1, N ! 2 b~M, N !

G . (15)

Observe now that if VB is added to an arbitrary raw frame yk from I(B),then

yk 1 VB 5 F zk~1, 1 ! 1 b~1, 1 ! zk~1, 2 ! 1 b~1, 2 ! ¯ zk~1, N ! 1 b~1, N !

zk~2, 1 ! 1 b~1, 1 ! zk~2, 2 ! 1 b~1, 2 ! ¯ zk~2, N ! 1 b~1, N !

] ] ] ]

zk~M, 1! 1 b~1, 1 ! zk~M, 2! 1 b~1, 2 ! ¯ zk~M, N ! 1 b~1, N !

G . (16)

1740 J. Opt. Soc. Am. A/Vol. 19, No. 9 /September 2002 Ratliff et al.

then we can identify yk 1 VB as a member of I(B8).More compactly, we maintain that VB 1 I(B) 5 I(B8).

Clearly a similar procedure can be followed with ab-pair to obtain a horizontal correction matrix HB , which,when added to any raw image, will unify the bias valuesacross each row. More precisely, for i 5 1, 2,...,M and j5 2, 3,...,N, put H(i,1) 5 0, and

H~i, j ! 51

b@bym~i, j 2 1 ! 1 ~1 2 b!ym~i, j !

2 ym1i~i, j !#. (18)

HB is then computed by performing a partial cumulativesum across each row of H(i, j). Thus the resulting hori-zontal correction matrix HB becomes

Next we use the ability to unify the bias values acrosscolumns or rows to unify the biases of the entire array toa common value.

C. Total Correction MatrixThe principle of total correction is now clear. First, forany raw a-pair obtained from a B-sensor, compute a ver-tical correction matrix VB . Next, add VB to a b-pair fromthe B-sensor. [Recall that adding VB to any raw frameresults in an image in I(B8) as if it were obtained fromthe vertically corrected B8-sensor, where B8 is defined inEq. (17).] Next, use the vertically corrected b-pair, de-noted as a b8-pair, to compute the horizontal correctionmatrix HB8 . The total correction matrix C is thenformed by adding HB8 to VB . A straightforward calcula-tion shows that

which is the desired total correction matrix. Notice thatwhen C is added to any raw image frame in I(B), all biasvalues become b(1, 1), as desired.

In practice, it is observed that there is error associatedwith the linear interpolation approximation of Eqs. (4)and (5) and error in estimating the shifts a and b, which

collectively introduce error (or noise) in the bias correc-tion matrix. To reduce the effect of this noise, we first ob-tain two collections Ca and Cb which consist of many dis-tinct a- and b-pairs, respectively. With these collections,we can compute many vertical and horizontal correctionmatrices and form averaged vertical and horizontal cor-rection matrices, denoted VB and HB8 , respectively.Moreover, we observe from Eq. (20) that all rows of HB8are ideally identical; therefore a second average can beperformed down each column of HB8 , resulting in a hori-zontal row vector H¢ B8 . This vector can then be repli-cated M times to form the M 3 N averaged horizontalcorrection matrix H% B8 . Thus in practice, it is these VB

and H% B8 that are summed to generate the final correction

matrix C. As will be shown in Section 4, relatively fewframe pairs from each collection are needed to compute aneffective correction matrix. A block diagram of the pre-sented NUC algorithm is shown in Fig. 1.

D. Shift EstimationThe collections of image pairs Ca and Cb are generatedthrough the use of the gradient-based shift-estimation al-gorithm described by Irani and Peleg15 and Hardie et al.16

The gradient-based shift estimator first estimates thegradient at each point in one image by use of a Prewittoperator. With a first-order Taylor series approximation,it is possible to predict the values of a second frame (as-sumed to be a shifted version of the first) based on thefirst frame, the gradient from the first frame, and theshift between the frames. In our application, we have

knowledge of the two frames and we are able to estimatethe gradient. What we do not know is the shift betweenthe frames. The shifts are estimated through a least-squares technique that minimizes the error between theobserved second frame and the predicted second frame.

While the gradient-based technique has been shown to

HB 5 F 0 b~1, 1 ! 2 b~1, 2 ! b~1, 1 ! 2 b~1, 3 ! ¯ b~1, 1 ! 2 b~1, N !

0 b~2, 1 ! 2 b~2, 2 ! b~2, 1 ! 2 b~2, 3 ! ¯ b~2, 1 ! 2 b~2, N !

] ] ] ] ]

0 b~M, 1! 2 b~M, 2! b~M, 1! 2 b~M, 3! ¯ b~M, 1! 2 b~M, N !

G . (19)

HB8 5 F 0 b~1, 1 ! 2 b~1, 2 ! b~1, 1 ! 2 b~1, 3 ! ¯ b~1, 1 ! 2 b~1, N !

0 b~1, 1 ! 2 b~1, 2 ! b~1, 1 ! 2 b~1, 3 ! ¯ b~1, 1 ! 2 b~1, N !

] ] ] � ]

0 b~1, 1 ! 2 b~1, 2 ! b~1, 1 ! 2 b~1, 3 ! ¯ b~1, 1 ! 2 b~1, N !

G , (20)

and indeed

C 5 VB 1 HB8 5 F 0 b~1, 1 ! 2 b~1, 2 ! b~1, 1 ! 2 b~1, 3 ! ¯ b~1, 1 ! 2 b~1, N !

b~1, 1 ! 2 b~2, 1 ! b~1, 1 ! 2 b~2, 2 ! b~1, 1 ! 2 b~2, 3 ! ¯ b~1, 1 ! 2 b~2, N !

] ] ] � ]

b~1, 1 ! 2 b~M, 1! b~1, 1 ! 2 b~M, 2! b~1, 1 ! 2 b~M, 3! ¯ b~1, 1 ! 2 b~M, N !

G , (21)

Ratliff et al. Vol. 19, No. 9 /September 2002 /J. Opt. Soc. Am. A 1741

Fig. 1. Block diagram of the proposed NUC algorithm.

yield reliable estimates, Armstrong et al.17 found that theaccuracy of the shift estimates is compromised by thepresence of spatial nonuniformity. This estimation errorcan be minimized by first smoothing the original imagesequence with an R 3 R mask, which effectively reducesthe nonuniformity. Through this prefiltering of the im-age sequence, shift estimate accuracy is improved signifi-cantly. The resulting shift estimates are then analyzedfor acceptable shifts. Acceptable subpixel shifts are de-fined as those in the interval [21, 1] for both pure hori-zontal and pure vertical motion. Moreover, we define atolerance parameter e, which is used to define the maxi-mum allowable deviation from the ideal zero shift in a di-rection orthogonal to the dominant motion direction.

Other computationally efficient registration techniquesare available that are shown to perform well in the pres-ence of fixed-pattern noise18 and may be used in place ofthe gradient-based algorithm. It is also possible that wecan substitute controlled motion (i.e., induced motionsuch that the shifts are known), thereby obviating theneed for a motion-estimation algorithm.

E. Further RemarksAlgorithm robustness is improved by considering threeimportant issues: (1) selection of the starting collection(i.e., Ca or Cb), (2) treatment of frame pairs with positiveand negative shifts, and (3) sensitivity to the toleranceparameter e.

Once shift estimation is completed, the best startingcollection must be determined. This decision is madesimply by selecting the collection with the highest num-ber of pairs of frames. Such a selection is made becausemost of the spatial nonuniformity is encountered whenthe first averaged correction matrix is computed. Withuse of the larger collection, error in the bias estimates isreduced as more correction matrices are averaged.

Each pair of frames in collections Ca and Cb may exhibita respective shift that is either positive or negative.When each correction matrix is computed, the shift polar-ity determines the starting row or column. As an ex-ample, if an M 3 N frame pair having a positive horizon-tal shift is used—with our shift sign convention—thealgorithm starts in column 1 and computes the bias esti-mates in a rightward direction. If the frame pair exhib-its a negative horizontal shift, the algorithm instead be-gins with column N and computes the bias estimates in aleftward direction. The vertical motion cases are analo-gous. By considering each of these cases, the algorithmcan find and use more frame pairs in accordance with theshift direction.

Finally, for all data tested, we learned that sufficientframe pairs were found when parameter e was given avalue of 0.05 pixels. As higher values of e are allowed, nomajor decrease in the visual quality of the corrected im-ages is observed until this parameter approaches 0.2pixel, beyond which striping artifacts across rows or col-umns will begin to appear.

4. PERFORMANCE ANALYSISA. Accuracy of Shift EstimationA study was performed on the accuracy of the gradient-based shift-estimation technique used in this paper. Ingeneral, the shift-estimation algorithm, in the presence oflittle or no spatial nonuniformity, demonstrates a highlevel of accuracy. However, the accuracy of shift estima-tion suffers as the amount of nonuniformity increases.To observe this effect, the average absolute error in theshift estimates was calculated as a function of bias non-uniformity standard deviation for a typical 8-bit, 1283 128, 200-frame image sequence. The result is shownin Fig. 2. A zero-mean Gaussian model was used to in-dependently generate the simulated bias nonuniformities.It is seen that the error in shift estimation increasessharply as the level of nonuniformity increases beyond acertain cutoff in signal-to-noise ratio, which depends onthe spatial-frequency content of the true scene sequence.In the example considered, this cutoff occurred at a biasnonuniformity standard deviation of 5 (based on an 8-bitgray scale).

Fig. 2. Average absolute error in the shift estimates as a func-tion of the standard deviation of the bias nonuniformity for a200-frame sequence. A true shift of 0.5 pixel was used in thesimulations.

1742 J. Opt. Soc. Am. A/Vol. 19, No. 9 /September 2002 Ratliff et al.

Application of a smoothing filter to the raw images be-fore shift estimation greatly reduced the shift error, sincethe spatial nonuniformity was considerably reduced as aresult of smoothing. In practice the best mask size willdepend on the strength of the nonuniformity and thespatial-frequency content of the scene. A surface plot isdisplayed in Fig. 3 that shows the average absolute errorin the shift estimates as a function of the standard devia-tion of bias nonuniformity and the mask size. As seenfrom the plot, the shift error can be reduced significantlyif a sufficiently large smoothing filter is employed, evenfor severe nonuniformity cases, as long as the smoothedimages maintain sufficient spatial diversity to allow shiftestimation.

B. Nonuniformity Correction with Simulated DataA sequence of clean images (in the visible-light range) wasused to generate two types of simulated image sequenceswith bias nonuniformity. The first type consisted of se-quences for which subpixel motion was simulated by us-ing linear interpolation, which is fully compatible withour algorithm. As expected when this type of simulateddata was used in the algorithm along with the true shiftvalues, perfect NUC was achieved with only four imageframes (one a-pair and one b-pair).

The second type of image sequence consists of a morerealistic nonlinear shift mechanism. Global motion isgenerated by first shifting a high-resolution image (e.g., a1280 3 1280 image) by multiples of a whole pixel andthen down-sampling the shifted image back to the 1283 128 grid by means of pixel averaging. This shiftmechanism simulates the practical situation when thetrue object (i.e., the high-resolution image) is moving andeach detector in the FPA generates a voltage proportionalto the total irradiance collected by its active area.Through this down-sampling technique we can generatesubpixel shifts between image frames that are as small asthe down-sampling ratio (i.e., 0.1 pixel in this case). Allimage sequences created were down-sampled by factorsranging from 2 to 10. As a result of down-sampling, theproduced sequences were aliased. Clearly as the amountof aliasing increases, the accuracy of our linear-interpolation approximation [given in Eqs. (4) and (5)] de-creases, in which case we would expect to see degradationin the NUC capability.

A 300-frame, 128 3 128, 8-bit aliased image sequence(down-sampled by a factor of 4) was generated with 0.5-pixel shifts. The sequence originally contained 150a-pairs and 149 b-pairs. Zero-mean Gaussian noise withstandard deviation of 20 was added to each frame of theimage sequence to bring about bias nonuniformity. Thesequence was then blurred with a 10 3 10 smoothingmask before shift estimation. We obtained and used 144acceptable a-pairs and 138 acceptable b-pairs. Figures4(a) and 4(b) show frame 1 from the image sequence be-fore and after addition of the simulated bias nonunifor-mity, respectively. Figure 4(c) shows frame 1 from theimage sequence after correction with our algorithm. De-spite the presence of aliasing, very good NUC is achieved.

1. Effect of Lack of Temporal Scene DiversityThe simulated sequence was also designed to show an-other advantage of the algebraic technique. The se-

quence had the property that detectors in the top part ofthe image were exposed to high values (i.e., the sky in thevisible-light range) while the detectors in the bottom por-tion of the image were exposed to low irradiance (i.e.,buildings and trees). The simulated motion allowed onlythe detectors in the middle portion of the sequence tohave the benefit of temporal scene diversity as a result ofbeing exposed to both low and high scene values. To de-termine the effect of this lack of scene diversity on statis-tical scene-based algorithms, we subjected the simulatedframes to the (bias-only version) constant-statistics algo-rithm of Harris and Chiang.6 The corrected image frame1 is shown in Fig. 4(d). As can be seen, this techniquehas difficulty with regions where irradiance diversity isnot present (i.e., in the top and bottom portions of the im-age). In contrast, the algebraic algorithm does not sufferfrom such conditions. This is so provided that scene-irradiance does not significantly change (i.e., the tempera-ture of objects within the scene does not vary greatly) be-tween adjacent image frames, as the algorithm would besensitive to such conditions. However, it is important tonote that because we are using the shift-estimation algo-rithm, two adjacent image frames must present somescene diversity in order for the algorithm to detect and ac-curately estimate the motion. In cases where motion isknown a priori (through precisely controlled motion),such scene diversity is not required.

2. Dependence of Performance on Image SequenceLengthA study was performed to determine the number of framepairs required to generate accurate estimates of the biasnonuniformity. The simulated sequence consisted of 500frames of 128 3 128, 8-bit images, and the bias-nonuniformity standard deviation was selected as 20.The dependence of the average absolute error (overall pix-els) on the number of frame pairs used for correction isdepicted in Fig. 5. We observe that in the example con-sidered, the average absolute error in the bias estimationreduces to a very low constant level (approximately1.25%) after approximately 100 frames. We suspect thatthis limiting value is primarily attributed to shift-estimation error, which does not vanish, as expected, withan increase in the number of frames used by the NUC al-gorithm.

Fig. 3. Average absolute error in the shift estimates as a func-tion of bias nonuniformity standard deviation and the size R ofthe smoothing mask.

Ratliff et al. Vol. 19, No. 9 /September 2002 /J. Opt. Soc. Am. A 1743

Fig. 4. (a) Frame 1 from the down-sampled image sequence before addition of bias nonuniformity, (b) frame 1 from the down-sampledimage sequence after the addition of bias nonuniformity, (c) frame 1 from the image sequence corrected by use of the algebraic NUCalgorithm, (d) frame 1 from the image sequence corrected by use of Harris’s (bias-only version) constant-statistics NUC technique.

3. Sensitivity to Error in Shift EstimationWe now examine the sensitivity of algorithm performanceto error in the shift estimates. We used a simulated8-bit, 200-frame, aliased image sequence, similar to thesequence of Fig. 4, where the true shift values in the purehorizontal and pure vertical motion were 0.5 pixel. Withthe true shifts known, the algorithm was deliberatelysupplied with inaccurate shifts and the resulting averageabsolute bias-estimate error was computed for differentlevels of bias nonuniformity, as shown in Fig. 6. We mayfirst observe that when ideal shifts of 0.5 are used, thebias estimate error is approximately zero, regardless ofthe nonuniformity level. However, for incorrectly as-sumed shifts beyond 0.5, we notice an almost linear in-crease in the bias-estimate error. The more interestingcase occurs when the incorrectly assumed shift values arebelow 0.5. In this case, the shift error increases progres-sively with the incorrect shift. This behavior is ex-

Fig. 5. Average absolute error in the estimation of bias nonuni-formity as a function of the number of frame pairs used in gen-erating the correction matrix. In this example, the bias nonuni-formity has a zero mean and a standard deviation of 20.

1744 J. Opt. Soc. Am. A/Vol. 19, No. 9 /September 2002 Ratliff et al.

plained by examining the role of the a and b shift param-eters in Eqs. (12) and (18). Computing the entries of thecorrection matrix involves a division by the appropriatepresumably correct shift value, forcing the calculation ofthe correction matrix to be particularly sensitive to smallvalues (and the corresponding fluctuations) of the as-sumed shift. This adverse effect will be re-examined andreduced when we consider real infrared imagery in sub-section 4.C.

We also observe that bias-estimation error is insensi-tive to the level of bias nonuniformity regardless of the er-ror in the assumed shift estimates. However, we need tobe cautious in interpreting this result, bias nonuniformitydoes affect the accuracy of shift estimation (as was shownin Fig. 3). In generating the graph in Fig. 6, however, weexercised full control over the error in the assumed shift.In general, our results show that the combined effect ofnonuniformity and shift-estimation error begins to se-verely degrade the visual quality of the corrected imageswhen the relative error in the shift estimates is in excessof approximately 20%. Thus we conclude that accurateknowledge of the shift is critical for the operation of ourNUC algorithm.

4. Effect of Gain NonuniformityWe now examine the algorithm’s performance in the pres-ence of gain nonuniformity in addition to bias nonunifor-mity. Gain nonuniformity was simulated by usingGaussian random noise with a mean of 1 and varyingstandard deviation. An 8-bit, 100-frame, aliased (downsampled by a factor of 10:1) image sequence, similar tothe sequences previously considered, was processed withour algorithm. Figure 7 shows the error in the calculatedbias values as a function of the standard deviations of thebias and gain nonuniformities under the assumption thattrue shifts are used. We also repeated the calculationsusing estimated shift values, as shown in Fig. 8. A10 3 10 smoothing mask was employed before the shiftestimation process. The two plots show similar behavior,namely, that as the severity of the gain nonuniformity in-creases, so does the error in the bias estimates. Whentrue shifts are used, the role of bias nonuniformity on theperformance, unlike the level of gain nonuniformity, be-comes irrelevant. On the other hand, when the shifts areestimated, both types of nonuniformity contribute to theerror. This is expected since we have observed earlierthat the shift estimates are degraded as a result of thebias nonuniformity.

Overall, we observe that gain nonuniformity can have aharmful effect on the accuracy of the bias estimates.Though the accuracy of the bias estimates is affected, it isimportant to note that the visual quality of the correctedimage sequences is still good. This is so because the al-gorithm effectively incorporates the gain nonuniformityinto the bias correction map and therefore removes muchof it. It is important to note that in all cases where idealshift values were used, the nonuniformity pattern was al-ways removed after correction. In the case where shiftestimates were used, striping artifacts appeared in thecorrected sequence whenever the nonuniformity was se-vere. These artifacts arise mainly as a result of signifi-cant error in the shift estimates.

C. Nonuniformity Correction with Real Infrared DataThe proposed algorithm was tested with real infrared im-age sequences. The data sets were collected using an8-bit, 128 3 128-array, InSb FPA camera (Amber ModelAE-4128) operating in the 3–5-mm range (the data weregenerated at the Air Force Research Laboratory, Dayton,Ohio). For all IR data, a 3 3 3 mask was used to smooththe images before shift estimation. Using two sets of realinfrared data sequences, we found that 50–75 frame pairs

Fig. 6. Average absolute error of the bias estimates as a func-tion of bias nonuniformity standard deviation and shift value.

Fig. 7. Average absolute error in the computed bias values as afunction of the standard deviations of gain and bias. The cor-rection algorithm incorporated the true shift values.

Fig. 8. Average absolute error in the computed bias values as afunction of the standard deviations of gain and bias. The cor-rection algorithm incorporated the estimated shift values.

Ratliff et al. Vol. 19, No. 9 /September 2002 /J. Opt. Soc. Am. A 1745

in the first direction and 10–20 frame pairs in the seconddirection were sufficient to produce an effective correctionmatrix.

In the first data set, which consisted of 500 frames,there were 139 a-pairs and 47 b-pairs when a value of0.05 was employed for the shift tolerance parameter e.Figure 9(a) displays frame 1 from the 500-frame infraredimage sequence. Frame 1 of the infrared image sequenceafter correction by Harris’s algorithm is shown in Fig. 9(b)and is compared with correction by the algebraic tech-nique, which is shown in Fig. 9(c). The corrected imageobtained from the algebraic algorithm contains undesir-able striping artifacts, which are due mainly to inaccurateshift estimates, and can be understood in the context ofFig. 6. When inaccurate shifts with values close to zeroare used, error in the bias estimates is known to increasedramatically, as discussed earlier in Subsection 4.B.

The striping problem can be largely overcome by limit-ing the acceptable shift range to shifts that are in the in-terval [0.5, 1.0]. The result for this limited shift range is

shown in Fig. 9(d); the striping has effectively disap-peared. It is interesting to note that the use of the re-stricted shifts resulted in only 49 a-pairs and 18 b-pairs;this excluded many of the problematic shift estimates.Again, these problematic shifts are those that containedsignificant error and had a value close to zero.

For comparison, the matrices (scaled to 256 dynamiclevels) for the correction of Figs. 9(c) and 9(d) are dis-played in Figs. 10(a) and 10(b), respectively. In the firstcorrection map, the undesirable striping effects areclearly visible; however, in the second one the artifactsare not noticeable.

Examples from a second 8-bit, 400-frame real infraredimage sequence, collected 4 h later by the same camera,are shown in Figs. 11(a) and 11(b). Figure 11(c) displaysthe correction made with Harris’s algorithm. In this ex-ample, which also used the same shift-tolerance param-eter (e 5 0.05) and restricted shifts (shifts in the interval[0.5, 1]), 43 a-pairs and 11 b-pairs were found and used bythe algorithm.

Fig. 9. (a) Frame 1 from infrared data set 1, (b) correction by Harris’s method, (c) correction by the algebraic algorithm (unrestrictedshifts), (d) correction by the algebraic technique with the shifts restricted to the interval [0.5, 1.0].

1746 J. Opt. Soc. Am. A/Vol. 19, No. 9 /September 2002 Ratliff et al.

Fig. 10. (a) Nonuniformity correction map associated with the image of Fig. 9(c), (b) nonuniformity correction map associated with theimage of Fig. 9(d).

Fig. 11. (a) Frame 100 from infrared data set 2, (b) correction bythe algebraic technique (restricted shifts), (c) correction by Har-ris’s constant-statistics algorithm.

Ratliff et al. Vol. 19, No. 9 /September 2002 /J. Opt. Soc. Am. A 1747

5. CONCLUSIONSWe have developed an algebraic scene-based technique forcorrection of bias nonuniformity in focal-plane arrays.The performance of the algorithm was thoroughly studiedthrough the use of both simulated and real infrared data.The strength of the proposed algorithm is in its simple al-gebraic nature. Because of this, the algorithm is compu-tationally efficient and easily implemented. It is shownthat an effective correction map can be obtained quicklywith use of relatively few image frames. Also, the algo-rithm’s performance has shown robustness to lack ofscene-irradiance diversity, thus demonstrating its advan-tage over some statistical scene-based techniques.

We showed that accurate motion estimation is crucialfor acceptable nonuniformity correction, and that thegradient-based, shift-estimation algorithm employed pro-duced reliable shift estimates. We also showed that ouralgorithm is most sensitive to error in small shift values.This adverse effect was largely overcome by restrictingthe acceptable subpixel shift range to shifts in excess of0.5 pixel, which drastically improved bias estimationwhile reducing the number of frame pairs used for correc-tion. We also showed that significant gain nonuniformitycan have a severe effect on the accuracy of the bias esti-mates, though good visual correction is still possible inthe presence of gain nonuniformity. In principle, gener-alization of the algorithm to incorporate two-directionalshifts is possible at the expense of substantial added com-plexity.

A promising course that we are currently undertakingis to use our algorithm in conjunction with absolute cali-bration to achieve radiometrically accurate nonuniformitycorrection. Our expectation is that if absolute blackbodycalibration is performed on only a fraction of the array el-ements without halting the function of remaining arrayelements, then the algorithm can be used to transfer thecalibration from the calibrated detectors to the remaininguncalibrated detectors. In this way, we achieve radio-metrically accurate correction without halting the opera-tion of the camera during the course of the calibrationprocess. Clearly, the algebraic (nonstatistical) nature ofthe algorithm is key to linking scene-based andcalibration-based techniques without compromising ra-diometric accuracy. The performance analysis carriedout in this paper can be useful in setting a guideline forthe expected degree of radiometric accuracy.

ACKNOWLEDGMENTSThis research was supported by the National ScienceFoundation Faculty Early Career Development (CA-REER) Program MIP-9733308. We are grateful to J.Scott Tyo for many valuable discussions regarding thepossibility of integrating calibration-based techniqueswith the reported algorithm. We are also grateful toErnest Armstrong and the Air Force Research Laboratory,Dayton, Ohio, for providing us with IR imagery.

M. Hayat’s e-mail address is hayat@eece.unm.edu.

REFERENCES1. A. F. Milton, F. R. Barone, and M. R. Kruer, ‘‘Influence of

nonuniformity on infrared focal plane array performance,’’Opt. Eng. 24, 855–862 (1985).

2. D. L. Perry and E. L. Dereniak, ‘‘Linear theory of nonuni-formity correction in infrared staring sensors,’’ Opt. Eng.32, 1853–1859 (1993).

3. P. M. Narendra and N. A. Foss, ‘‘Shutterless fixed patternnoise correction for infrared imaging arrays,’’ in TechnicalIssues in Focal Plane Development, W. S. Chan and E.Krikorian, eds., Proc. SPIE 282, 44–51 (1981).

4. P. M. Narendra, ‘‘Reference-free nonuniformity compensa-tion for IR imaging arrays,’’ in Smart Sensors II, D. F.Barbe, ed., Proc. SPIE 252, 10–17 (1980).

5. J. G. Harris, ‘‘Continuous-time calibration of VLSI sensorsfor gain and offset variations,’’ in Smart Focal Plane Arraysand Focal Plane Array Testing, M. Wigdor and M. A.Massie, eds., Proc. SPIE 2474, 23–33 (1995).

6. J. G. Harris and Y. M. Chiang, ‘‘Nonuniformity correctionusing constant average statistics constraint: Analog anddigital implementations,’’ in Infrared Technology and Appli-cations XXIII, B. F. Andersen and M. Strojnik, eds., Proc.SPIE 3061, 895–905 (1997).

7. Y. M. Chiang and J. G. Harris, ‘‘An analog integrated circuitfor continuous-time gain and offset calibration of sensor ar-rays,’’ J. Analog Integr. Circuits Signal Process. 12, 231–238 (1997).

8. W. F. O’Neil, ‘‘Dithered scan detector compensation,’’ pre-sented at the 1993 International Meeting of the Infrared In-formation Symposium Specialty Group on Passive Sensors,Ann Arbor, Mich., 1993).

9. W. F. O’Neil, ‘‘Experimental verification of dithered scannonuniformity correction,’’ in Proceedings of the 1996 Inter-national Meeting of the Infrared Information SymposiumSpecialty Group on Passive Sensors (Infrared InformationAnalysis Center, Ann Arbor, Michigan, 1997) Vol. 1, pp.329–339.

10. R. C. Hardie, M. M. Hayat, E. E. Armstrong, and B. Yasuda,‘‘Scene-based nonuniformity correction using videosequences and registration,’’ Appl. Opt. 39, 1241–1250(2000).

11. M. M. Hayat, S. N. Torres, E. E. Armstrong, and B. Yasuda,‘‘Statistical algorithm for nonuniformity correction in focal-plane arrays,’’ Appl. Opt. 38, 772–780 (1999).

12. S. N. Torres, M. M. Hayat, E. E. Armstrong, and B. Yasuda,‘‘A Kalman-filtering approach for nonuniformity correctionin focal-plane array sensors,’’ in Infrared Imaging Systems:Design, Analysis, Modeling, and Testing XI, G. C. Holst andJCD Publishing, eds., Proc. SPIE 4030, 196–203 (2000).

13. S. N. Torres and M. M. Hayat, ‘‘Compensation for gain andbias nonuniformity and drift in array detectors: AKalman-filtering approach,’’ manuscript available from M.M. Hayat; hayat@eece.unm.edu.

14. G. C. Holst, CCD Arrays, Cameras and Displays (SPIE Op-tical Engineering Press, Bellingham, Wash., 1996).

15. M. Irani and S. Peleg, ‘‘Improving resolution by image reg-istration,’’ CVGIP: Graph. Models Image Process. 53, 231–239 (1991).

16. R. C. Hardie, K. J. Barnard, J. G. Bognar, and E. A. Watson,‘‘High-resolution image reconstruction from a sequence ofrotated and translated frames and its application to an in-frared imaging system,’’ Opt. Eng. 37, 247–260 (1998).

17. E. E. Armstrong, M. M. Hayat, R. C. Hardie, S. N. Torres,and B. Yasuda, ‘‘Nonuniformity correction for improved reg-istration and high-resolution image reconstruction in IRimagery,’’ in Application of Digital Image Processing XXII,A. G. Tescher and Lockheed Martin Missions Systems, eds.,Proc. SPIE 3808, 150–161 (1999).

18. S. C. Cain, M. M. Hayat, and E. E. Armstrong, ‘‘Projection-based image registration in the presence of fixed-patternnoise,’’ IEEE Trans. Image Process. 10, 1860–1872 (2001).