Blind adaptive sampling of images.bak

1478 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 21, NO. 4, APRIL 2012

Blind Adaptive Sampling of ImagesZvi Devir and Michael Lindenbaum

Abstract—Adaptive sampling schemes choose different samplingmasks for different images. Blind adaptive sampling schemes usethe measurements that they obtain (without any additional or di-rect knowledge about the image) to wisely choose the next samplemask. In this paper, we present and discuss two blind adaptivesampling schemes. The first is a general scheme not restricted toa specific class of sampling functions. It is based on an underlyingstatistical model for the image, which is updated according to theavailable measurements. A second less general but more practicalmethod uses the wavelet decomposition of an image. It estimatesthe magnitude of the unsampled wavelet coefficients and samplesthose with larger estimated magnitude first. Experimental resultsshow the benefits of the proposed blind sampling schemes.

Index Terms—Adaptive sampling, blind sampling, image repre-sentation, statistical pursuit, wavelet decomposition.

I. INTRODUCTION

I MAGE sampling is a method for extracting partial infor-mation from an image. In its simplest form, i.e., point sam-

pling, every sample provides value of image at somelocation . In generalized sampling methods, every samplemeasures the inner product of image with some samplingmask . Linear transformsmay be considered as sampling pro-cesses, which use a rich set of masks, defined by their basis func-tions; examples include the discrete cosine transform (DCT) andthe discrete wavelet transform (DWT).A progressive sampling scheme is a sequential sampling

scheme that can be stopped after any number of samples, whileproviding a “good” sampling pattern. Progressive sampling ispreferred when the number of samples is not predefined. Thisis important, e.g., when we want to minimize the number ofsamples and terminate the sampling process when sufficientknowledge about the image is obtained.Adaptive schemes generate different sets of sampling masks

for different images. Common sampling schemes (e.g., DCTand DWT) are nonadaptive, in the sense of using a fixed setof masks, regardless of the image. Adaptive schemes are poten-tially more efficient as they are allowed to use direct informa-tion from the image in order to better construct or choose theset of sampling masks. For example, using a directional DCTbasis [21] that is adapted to the local gradient of an image patchis more efficient than using a regular 2-D DCT basis. Other

Manuscript received July 08, 2010; revised October 10, 2011; acceptedNovember 15, 2011. Date of publication December 23, 2011; date of currentversion March 21, 2012. The associate editor coordinating the review of thismanuscript and approving it for publication was Prof. Charles Creusere.The authors are with the Computer Science Department, Technion–Israel In-

stitute of Technology, Haifa 32000, Israel (e-mail: [email protected];[email protected]).Color versions of one or more of the figures in this paper are available online

at http://ieeexplore.ieee.org.Digital Object Identifier 10.1109/TIP.2011.2181523

examples for progressive adaptive sampling methods are thematching pursuit and the orthogonal matching pursuit (OMP)[11], [12], [14], which sample the image with a predefined setof masks and choose the best mask according to various criteria.In this paper, we consider a class of schemes that we refer to

as blind adaptive sampling schemes. Those schemes do not havedirect access to the image and make exclusive use of indirect in-formation gathered from previous measurements for generatingtheir adaptive masks. A geological survey for underground oilreserves may be considered as an example of such a scheme.Drilling can be regarded as point sampling of the undergroundgeological structure, and the complete survey can be regardedas a progressive sampling scheme, which ends when either oilis found or funds are exhausted. This sampling scheme is blindin the sense that the only data directly available to the geologistsare the measurements. The sampling scheme is adaptive in thesense that the previous measurements are taken into an accountwhen choosing the best spot for the next drill.While blindness to the image necessarily makes the samples

somewhat less efficient, it implies that the masks are depen-dent only on the previous measurements. Therefore, storing thevarious sampling masks is unnecessary. The sampling schemefunctions as a kind of decoder, i.e., given the same set of mea-surements, it will produce the corresponding set of samplingmasks. This property of blind sampling schemes stems from thedeterministic nature of the sampling process. Similar ideas standat the root of various coding schemes, such as adaptive Huffmancodes (e.g., [20]) or binary compression schemes, which con-struct the same online dictionary during encoding and decoding(e.g., the Lempel–Ziv–Welch algorithm).In this paper, we present two blind adaptive schemes, which

can be considered as generalizations of the adaptive farthestpoint strategy (AFPS), a progressive blind adaptive samplingscheme proposed in [6].The first method, called statistical pursuit, generates an op-

timal (in a statistical sense) set of masks and can be restrictedto dictionaries of masks. The scheme maintains an underlyingstatistical model of the image, derived from the available infor-mation about the image (i.e., the measurements). The model isupdated online as additional measurements are obtained. Thescheme uses this statistical model to generate an optimal maskthat provides the most additional information possible from theimage.The second method, called blind wavelet sampling, works

with the limited yet large set of wavelet masks. It relies on anempirical statistical model of wavelet coefficients. Using thismodel, it estimates the magnitude of unknown wavelet coeffi-cients from the sampled ones and samples the coefficients withlarger estimated magnitude first. Our experiments indicate thatthis method collects information about the image much moreefficiently than alternative nonadaptive methods.

1057-7149/$26.00 © 2011 IEEE

http://ieeexploreprojects.blogspot.com

DEVIR AND LINDENBAUM: BLIND ADAPTIVE SAMPLING OF IMAGES 1479

Fig. 1. One iteration of the nonadaptive FPS algorithm: (a) the sampled points(sites) and their corresponding Voronoi diagram; (b) the candidates for sampling(Voronoi vertices); (c) the farthest candidate chosen for sampling; and (d) theupdated Voronoi diagram.

The remainder of this paper is organized as follows: InSection II, the AFPS scheme is described as a particular caseof a progressive blind adaptive sampling scheme. Section IIIpresents the statistical pursuit scheme, followed by a presen-tation of the blind wavelet sampling scheme. Experimentalresults are shown in Section V. Section VI concludes this paperand proposes related research paths.

II. AFPS ALGORITHM

We begin by briefly describing an earlier blind adaptivesampling scheme [6], which inspired the work in this paper.The algorithm, denoted as AFPS, is based on the farthest pointstrategy (FPS), a simple, progressive, but nonadaptive pointsampling scheme.

A. FPS Algorithm

In the FPS, a point in the image domain is progressivelychosen, such that it is the farthest from all previously sampledpoints. This intuitive rule leads to a truly progressive samplingscheme, providing after every single sample a cumulative set ofsamples, which is uniform in a deterministic sense and becomescontinuously denser [6].To efficiently find its samples, the FPS scheme maintains a

Voronoi diagram of the sampled points. A Voronoi diagram [2]is a geometric structure that divides the image domain into cellscorresponding to the sampled points (sites). Each cell containsexactly one site and all points in the image domain, which arecloser to the site than to all other sites. An edge in the Voronoidiagram contains points equidistant to two sites. A vertex in theVoronoi diagram is equidistant to three sites (in the general case)and is thus a local maximum of the distance function. There-fore, in order to find the next sample, it is sufficient to consideronly the Voronoi vertices (with some special considerations forpoints on the image boundary).After each sampling iteration, the new sampled point be-

comes a site, and the Voronoi diagram is accordingly updated.Fig. 1 describes one iteration of the FPS algorithm. Note that,because the FPS algorithm is nonadaptive, it produces a uni-form sampling pattern regardless of the given image content.

B. AFPS Algorithm

An adaptive more efficient sampling scheme is derived fromthe FPS algorithm. Instead of using the Euclidean distance asa priority function, the geometrical distance is used, along with

Fig. 2. First 1024, 4096, and 8192 point samples of the cameraman image,taken according to the AFPS scheme.

either the estimated local variance of the image intensities or theequivalent local bandwidth. The resulting algorithm, denotedAFPS, samples the image more densely in places where it ismore detailed and more sparsely where it is relatively smooth.Fig. 2 shows the first 1024, 4096, and 8192 sampling points pro-duced by the AFPS algorithm for the cameraman image, usingthe priority function , where is the dis-tance of the candidate to its closest neighbors and is a localvariance estimate.A variant of the AFPS scheme, designed for range sampling

using a regularized grid pattern, was presented in [5].

III. STATISTICAL PURSUIT

In this section, we propose a sampling scheme based on a di-rect statistical model of the image. In contrast to point samplingschemes such as the AFPS, this scheme may choose samplingmasks from an overcomplete family of basis functions or cal-culate optimal masks. The scheme updates an underlying statis-tical model for the image as more information is gathered duringthe sampling process.

A. Simple Statistical Model for Images

Images are often regarded as 2-D arrays of scalar values (graylevels or intensities). Yet, it is clear that arbitrary arrays of valuesdo not resemble natural images. Natural images contain struc-tures that are difficult to explicitly define. Several attempts weremade to formulate advanced statistical models, which can ap-proximate such global structures and provide good predictionfor missing parts [9]. Still, the local structure is easier to pre-dict, and there exist some low-level statistical models, whichmodel local behavior fairly well.



Here, we consider a common and particularly simple second-order local statistical model for images. We regard image asa Gaussian random vector, i.e.,

(1)

where is the mean vector and is the covariance matrix. Forsimplicity and without loss of generality, we assume .Two neighboring pixels in an image often have similar gray

level values (colors). Statistically speaking, their colors havea strong positive correlation, which weakens as their distancegrows. The exponential correlation model [10], [13] is a second-order stationary model based on this observation. According tothis model, the covariance between the intensities of two arbi-trary pixels and exponentially depends on their distance, i.e.,

(2)

where is the variance of the intensities and determines howquickly the correlation drops.

B. Statistical Reconstruction

We consider linear sampling where the th sample is gen-erated as an inner product of image with some sampling mask. That is, , where both the image and the mask

are regarded as column vectors. A sampling process provides uswith a set of sampling masks and their corresponding measure-ments . We wish to reconstruct an image fromthis partial information.Let be a matrix containing the masks as

its columns, and let be a column vector of themeasurements. Using those matrix notations, .The underlying statistical model of the image may be used to

obtain an image estimate based on measurements .For the second-order statistical model (1), the optimal estimatoris linear [15]. The linear estimator is optimal in the sense,i.e., it minimizes the mean square error (MSE) between the trueand reconstructed images MSE .It is not hard to show that the image estimate , its covariance

, and its MSE can be written in matrix form as

(3)

(4)

MSE trace (5)

assuming is linearly independent.It should be noted that the statistical reconstruction is analo-

gous to the algebraic one. The algebraic (consistent) reconstruc-tion of an image from its measurements is(assuming the linear independence of ). That is, the alge-braic reconstruction is a statistical reconstruction, assuming thepixels are independent identically distributed, i.e., .Searching for a new mask that minimizes the algebraic re-

construction error , leads to the OMP [14]. Analo-gously, searching for a new mask that minimizes the expectederror , leads to the statistical pursuit, which is dis-cussed next.

C. Reconstruction Error Minimization

A greedy sampling strategy is to find a sampling mask thatminimizes the MSE of .If is a linear combination of the previous masks, it is trivial

to show that theMSE does not change (as no additional informa-tion is gained) and MSE . Therefore, we can assumethat the new mask is linearly independent of .Proposition: The reduction of the MSE, given a new mask(linearly independent of ), is

MSE (6)

where is the covariance of , defined in (4).Proof: See Appendix A.

The aforementioned proposition justifies selection criteria forthe next best mask(s) in several scenarios.For the sake of brevity, we define and as the estimated

image and its covariance, after sampling the image with masks. MSE is the MSE after sampling masks, and

MSE is the expected reduction of MSE given an arbitrarymask as the next selected mask. We further denote and

. Without subscript, and shall refer to the estimatedimage and its covariance, given all known masks.

is a positive-semidefinite symmetric matrix, with the pre-vious masks as its eigenvectors with corresponding zero eigen-values. may be regarded as the “portion” of the covari-ance matrix , which is statistically independent of the previousmasks.

D. Progressive Sampling Schemes

1) Predetermined Family of Masks: The masks are often se-lected from a predefined set of masks (a dictionary) , such asthe DCT or DWT basis. In such cases, the next best mask is de-termined by calculating MSE for each mask in the dictionaryand choosing MSE .2) Parametrized Masks: Suppose depends on several

parameters . We can differentiate (6) by the parame-ters of the mask, solve the resulting system of equations

MSE , and check all the extrema masks for theone that maximizes MSE . However, solving the resultingsystem of equations is not a trivial task.3) Optimal Mask: If the next mask is not restricted, the op-

timal mask is an eigenvector corresponding to the largest eigen-value. See Appendix B for details. We shall denote the eigen-vector corresponding to the largest eigenvalue of the largesteigenvector of and mark it as .4) Optimal Set of Masks: The largest eigenvector of is theoptimal mask. If that mask is chosen, it becomes an eigen-

vector of with a corresponding eigenvalue of 0. Therefore,the optimal mask is the largest eigenvector of , whichis the second largest eigenvector of . Collecting optimalmasks together is equivalent to finding the largest eigenvec-tors of .If we begin with no initial masks, the optimal first masks

are simply the largest eigenvectors of . Those are, notsurprisingly, the first components obtained from the principal



component analysis (PCA) [7], [8] of , i.e., the covariance ofthe image .

E. Adaptive Progressive Sampling

Using the MSE minimization criteria, it is easy to constructa nonadaptive progressive sampling scheme that selects the op-timal mask, i.e., the one that minimizes the estimated error ateach step. Such a nonadaptive sampling scheme makes use ofa fixed underlying statistical model for image .However, as we gain information about the image, we can up-date the underlying model accordingly.In the exponential model of (2), the covariance between two

pixels depends only on their spatial distance. However, pairsassociated with a large intensity dissimilarity are more likelyto belong to different segments in the image and to thus be lesscorrelated. If we have some image estimate, a pair of pixels maybe characterized by their spatial distance and their estimatedintensity dissimilarity.Using the partial knowledge about image obtained during

the iterative sampling process, we now redefine the exponentialcorrelation model (2). The new model is based on both the spa-tial distance and the estimated intensity distance, i.e.,

(7)

Such correlation models are implicitly used in nonlinear fil-ters such as the bilateral filter [19]. Naturally, other color-awaremodels can be used. For example, instead of taking Euclideandistances, we can take geodesic distances on the image colormanifold [17].Introducing the model of (7) into a progressive sampling

scheme, we can construct an adaptive sampling scheme pre-sented as Algorithm 1.

F. Image Reconstruction From Adaptive Sampling

The statistical reconstruction of (3) requires the measure-ments, along with their corresponding masks. For nonadaptivesampling schemes, the masks are fixed regardless of the image,and there is no need to store them. For blind adaptive samplingschemes, where the set of masks differs for different images,there is no need to store them either.At each iteration of the sampling process, a new sampling

mask is constructed, and the image is sampled. Because of thedeterministic nature of the sampling process, those masks canbe recalculated during reconstruction. The reconstruction algo-rithm is almost identical to Algorithm 1, except for step 3(c),which now reads, “Pick from a list of stored measure-ments,” and the stopping criteria at step 4 is accordingly up-dated.

IV. BLIND WAVELET SAMPLING

The statistical pursuit algorithm is quite general, but up-dating the direct underlying space-varying statistical model ofthe image is computationally costly. We now present an alter-native blind sampling approach, which is limited to a familyof wavelet masks and relies on an indirect statistical modelof the image. The scheme chooses first the coefficient that isestimated to carry most of the energy, using the measurementsthat it obtains.A trivial adaptive scheme stores the largest wavelet coef-

ficients and their corresponding masks. Such a scheme samples(i.e., decomposes) the complete image and sorts the coefficientsaccording to their energy. However, it clearly uses all the imageinformation and is therefore not blind.The proposed sampling scheme is based on the statistical

properties of a wavelet family; we use the statistical correlationsbetween magnitudes (or energies) of the wavelet coefficients.Those statistical relationships are used to construct a numberof linear predictors, which predict the magnitude of the unsam-pled coefficients from the magnitude of the known coefficients.This way, the presented scheme chooses the larger coefficientswithout direct knowledge about the image.The proposed sampling scheme is divided into three stages.1) Learning Stage: The statistical properties of the waveletfamily are collected and studied. This stage is done offlineusing a large set of images and is considered a constantmodel.

2) Sampling Stage: At each iteration of this blind samplingscheme, the magnitudes of all unsampled coefficients areestimated, and the coefficient with the largest magnitude isgreedily chosen.

3) Reconstruction Stage: The image is reconstructed usingthe measurements obtained from the sampling stage.

As with the other blind schemes, it is sufficient to store thevalues of the sampled coefficients since their correspondingmasks are recalculated during reconstruction.

A. Correlation Between Wavelet CoefficientsWavelet coefficients are weakly correlated among them-

selves [3], [9]. However, their absolute values or their energiesare highly correlated. For example, the correlation betweenthe magnitude of wavelet coefficients at different scales but



similar spatial locations is relatively high. Nevertheless, thesign of the coefficient is hard to predict, and therefore, thecorrelations between the coefficients remain low. This propertyis the foundation of zerotree encoding [16], which is used toefficiently encode quantized wavelet coefficients.Each wavelet coefficient corresponds to a discrete wavelet

basis function, which can also be interpreted as a samplingmask. The mask (and its associated coefficient) is specified byorientation ( , , , or ), level of decomposition (orscale), spatial location, and support.Three relationships are defined.1) Each coefficient has four (spatial) direct neighbors in thesame block.

2) Each coefficient from the , , and blocks (for) has four children. The children are of the same

orientation one level below and occupy approximately thesame spatial location of the parent coefficient. Each coef-ficient in the , , and blocks (expected at thehighest level) has a parent coefficient.

3) Each coefficient from the , , and blocks hastwo cousins that occupy the same location but with dif-ferent orientations. At the highest level , a third cousin inthe block exists. Similarly, each coefficient in theblock has three cousins in the , , and blocks.

We can further define second-order family relationships, suchas grandparents and grandchildren, cousin’s children, diagonalneighbors, and so on. Those relationships carry lower correla-tion and can be indirectly approximated from the first-order rel-atives.

B. Learning Stage

The learning stage is done offline, and the statistical relation-ships are stored for the later sampling and reconstruction stages.The measured correlations are considered as a fixed model. Ourexperiments showed the statistical characteristics to be almostindifferent to the image classes. That is, the proposed method isrobust for varying image classes.At the learning stage, the wavelet coefficients are considered

as instances of random variables. We assume that the statisticsof the wavelet coefficients are independent of their spatial lo-cation, and we study complete blocks of wavelet coefficients asinstances of a single random variable. In addition, we assumetransposition invariance of the image and wavelet coefficients.Therefore, we expect the same behavior from wavelet coeffi-cients of opposite orientations (e.g., the and blocks).It is common to assume scaling invariance as well, but our ex-periments showed this assumption is not completely valid fordiscrete images.See Section V-B for experimental results of the statistical

model for different types of wavelet families.It is straightforward to build a linear predictor of the magni-

tude of a certain coefficient , assuming it has knownrelatives , i.e., [15]. However,the actual predictors differ according to the available observa-tions.

C. Sampling Stage

The sampling stage is divided into an initial phase and a pro-gressive phase.1) Initial Sampling Phase: Before any coefficient is sampled,

the predictors can rely only on the expected mean. Therefore,the coefficients with the highest expected mean should be sam-pled first. For wavelet decompositions, the expected mean of thecoefficients from the block is the highest, and we start bysampling them all.The coefficients of the block carry low correlation with

their cousins (at the , , and blocks). Therefore,after sampling the block, we further decompose it “on thefly” into , , , and , in order to makeuse of the higher parent–children correlations. This way, we getbetter predictors for , , and .2) Progressive Sampling Phase: The blind sampling algo-

rithm has three types of coefficients, i.e., the coefficients it hasalready sampled, which we refer to as known coefficients; rela-tives of the known coefficients, which are the candidates for thesampling; and the remaining coefficients. The algorithm keepsthe candidates in a heap data structure, sorted according to theirestimated magnitude.The output of the algorithm is an array of the coefficient

values, as sampled at steps 1(a) and 2(a).

D. Reconstruction Stage

Again, we mark the coefficients as known, candidates, andthe remaining coefficients. The input for the reconstruction al-gorithm is the array of values generated by the sampling algo-rithm (Algorithm 2).Note that, while the value of the coefficient is stored in the

array, its corresponding wavelet mask is not part of the storedinformation and is obtained during the reconstruction stage (step2), using the predictors. Consequently, the algorithm does not



Fig. 3. First 30 nonadaptive sampling masks.

Fig. 4. First 30 adaptive sampling masks.

Fig. 5. (Left) Image patch and (right) the reconstruction error using adaptiveand nonadaptive unrestricted masks.

need to keep the masks associated with the coefficients or theirindexes.

Fig. 6. Ratio between the average reconstruction errors of adaptive and non-adaptive schemes, for unrestricted masks (PCA) and three overcomplete dictio-naries (DB3, DB2, and Haar). The reference line (100%) represents the non-adaptive schemes.

V. EXPERIMENTAL RESULTS

A. Masks Generated by Statistical Pursuit

We start by illustrating the difference between the nonadap-tive and adaptive masks. Figs. 3 and 4 present the first 30 masksused to sample a 32 32 image patch, shown in Fig. 5. Themasks are produced by both nonadaptive and adaptive schemes,where, in both cases, the masks are unrestricted and the same ex-ponential correlation model is used.The nonadaptive sampling masks, shown in Fig. 3, closely

resemble the DCT basis and not by coincidence (the DCT is anapproximation of the PCA for periodic stationary random sig-nals [1]). The adaptive sampling masks, shown in Fig. 4, presentmore complicated patterns. As the sampling process advances,it attempts to “study” the image at interesting regions, e.g., thevertical edge at the center of the patch (see Fig. 5).Fig. 5 presents the true reconstruction error for a varying

number of samples. For both adaptive and nonadaptive sam-pling schemes, the reconstruction is done using the linear es-timator (3).Fig. 6 presents the ratio between the reconstruction errors

of the adaptive and nonadaptive schemes. We took 256 smallpatches (16 16 pixels each) and compared the average re-construction errors. This experiment was repeated for severalclasses of masks, i.e., unrestricted masks (PCA) and an over-complete family of Daubechies wavelets of order three (DB3),of order two (DB2), and of order one (Haar wavelets) [4].Using the adaptive schemes reduces the reconstruction error

by between 5% and 10% compared with the nonadaptiveschemes (which use the same family of masks). Using optimalrepresentation (i.e., the PCS basis), the error is reduced by 5%,whereas using a less optimal basis (the Haar basis), the error isreduced even further, as the corresponding nonadaptive schemeis known to be suboptimal. For the first few coefficients, theadaptive scheme does not have much information to work with.Therefore, until more samples are gathered, the relative gainover the nonadaptive schemes is erratic. After sampling somemore coefficients, the benefits of the adaptivity become moreapparent.

B. Correlation Models for Blind Wavelet Sampling

In our experiments, we used a data set of 20 images. All theimages were converted to grayscale and rescaled to 256 256pixels. The images are shown in Fig. 7. The first two subsets are



Fig. 7. Image sets used in our experiments.

TABLE ICORRELATIONS BETWEEN DB2

“natural” images, the third set is taken from a computed-tomog-raphy brain scan, and the fourth set is a collection of animations.Two examples for experimental correlation models are

presented in Tables I and II. Those models were studiedfor Daubechies wavelets [4] of second and third orders. Afour-level wavelet decomposition was carried out over theimage data set, and the correlation coefficients between dif-ferent kinds of wavelet coefficients were estimated accordingto the maximum-likelihood principle.Observing Tables I and II, we see that Daubechies wavelets of

second and third orders exhibit similar behavior. This behaviorwas also experimentally found for other wavelet families. Asthe order of the wavelet family increases, most of the correla-tion coefficients decrease. We also see that the images are notscale invariant, and as the decomposition level decreases, thecorrelation coefficients between related coefficients increase.Our experiments show the statistical characteristics to be al-

most indifferent to the image classes. We tested the benefits ofusing a specific (rather than the generic) model for each class ofimages and found that it reduces the error by a negligible 1%.It appears that the correlation model characterizes the statistical

TABLE IICORRELATIONS BETWEEN DB3

behavior of the wavelet family and not of the different imageclasses.

C. Blind Wavelet Sampling Results

Having obtained the correlation model for the wavelet family,we now present some results of the blind sampling scheme.Taking an image, we decompose it using third-level waveletdecomposition with DB2. We compare the adaptive order, ob-tained by the blind adaptive sampling scheme, to a nonadaptiveraster order, and to a optimal order (which is not blind).The nonadaptive raster-order scheme samples the coefficients

according to their block order, from the highest level to lowerones. The optimal-order scheme assumes full knowledge of thecoefficients and samples them according to their energy.Figs. 8–10 present partial reconstructions of the cameraman

image, where only some of the wavelet coefficients are used forthe reconstruction. There are three columns in the figures, i.e.,the left, where the selected coefficients are marked; the middle,where the reconstructed images, using the selected coefficients,are presented; and the right, where the error images are shown.Reconstruction error is also shown.All three schemes start with the block and continue to

sample the remaining wavelet coefficients in different orders.Figs. 8–10 correspond to raster, adaptive, and optimal orders,respectively.In Fig. 11, we can see the reconstruction errors for the raster,

adaptive and optimal orders, averaged over all 20 images. Notethat the same reconstruction error may be achieved by the adap-tive scheme using about half of the samples required by the non-adaptive scheme.The main advantage of blind sampling schemes is that the

sampling results (the actual coefficients) need to be storedbut not their locations. Nonblind adaptive schemes, such aschoosing the largest coefficients of the decomposition (theoptimal-order scheme), require an additional piece of informa-tion, i.e., the exact location of each coefficient, to be storedalongside its value.From a compression point of view, we have to estimate the

number of bits required for storing the additional informa-tion needed for the reconstruction. As a rough comparison,



Fig. 8. Reconstruction of the cameraman image using (a) the first 1024 coeffi-cients of the block; (b) 4096 coefficients of the , , , andblocks; and (c) 16 384 coefficients of the , , , , , ,and the blocks.

Fig. 9. Reconstruction of the cameraman image using (a) 4096 coefficientstaken according to the blind sampling order and (b) 16 384 coefficients takenaccording to the blind sampling order.

Fig. 10. Reconstruction of the cameraman image using (a) 4096 coefficientstaken according to the optimal order and (b) 16 384 coefficients taken accordingto the optimal order.

we assume that the location (index) of a coefficient takesbits. Optimistic assumptions on entropy

encoders reduce the size required for storing the coefficientindexes to about 8 bits, disregarding quantization. Using thisrough estimation on storage requirements, each coefficient ofthe optimal order is equivalent, in a bit-storage sense, to abouttwo coefficients of the blind adaptive scheme. In Fig. 11, the

Fig. 11. Reconstruction error averaged over 20 images, using up to 16 384wavelet coefficients, taken according to the raster, adaptive, and optimal orders.(Dashed line) Compression-aware comparison of the optimal order, taking intoan account the space required to store the coefficient indexes of the optimalorder.

dashed line marks the reconstruction error of the optimal ordertaking into account the storage considerations.

VI. CONCLUSION

In this paper, we have presented two novel blind adaptiveschemes. Our statistical pursuit scheme, presented in Section III,maintains a second-order statistical model of the image, which isupdated as information is gathered during the sampling process.Experimental results have shown that the reconstruction erroris smaller by between 5% and 10%, as compared with regularnonadaptive sampling schemes, depending on the class of basisfunctions. Due to its complexity, however, this scheme is mostsuitable for small patches and a small number of coefficients.Our blind wavelet sampling scheme, presented in Section IV,

is more suitable for complete images. It uses the statistical cor-relation between the magnitudes of wavelet coefficients.Naturally, the optimal selection of the coefficients with thehighest magnitude shall produce superior results, but suchunblind methods require storage of the coefficient indexes,whereas the blind scheme only stores the coefficients. Takinginto an account the additional bit space, the blind waveletsampling scheme produces results almost as good as optimalselection of the masks.Some open problems are left for future research, such as the

application of the statistical-pursuit scheme to image compres-sion. Including quantization in the scheme and introducing anappropriate entropy encoder can turn the sampling scheme intoa compression scheme. Replacing DCT or DWT sampling withtheir statistical-pursuit counterparts reduces the error foreach patch by 5%–10%. However, some of the gain is expectedto be lost by the entropy encoder.The blind wavelet sampling scheme makes use of linear pre-

dictors. However, it is known that the distribution of wavelet co-efficients is not Gaussian [9], [18]. Therefore, higher order pre-dictors for modeling the relationships between the coefficients



may yield better predictors and a better blind adaptive samplingscheme.

APPENDIX A

Let be a linearly independent set ofmasks, where is the new mask. According to (5), MSE ,i.e., the MSE of the whole image estimate after sampling theth mask, is

MSE

where , is a unit column vector with 1 at and0 elsewhere, and is the image domain.We now rewrite MSE while separating the elements influ-

enced by , i.e., the new mask, from the elements that areindependent of

MSE

We denote as the matrix of the pre-vious masks, excluding the th new mask . Using ma-trix notation, , , ,

, and .A matrix blockwise inversion

implies that

MSE

MSE

Now, we have a more precise expression for the expectedreduction of the MSE at the th iteration, i.e.,

MSE

The numerator of MSE is

and the denominator is

Both the numerator and the denominator have similarquadratic forms. Therefore, let us define

Plugging back into the numerator and the denominator ofMSE yields

MSE

Surprisingly, is the covariance of , i.e., the estimatedimage based on the first measurements.

APPENDIX B

Proposition: , which is an eigenvector corresponding tothe largest eigenvalue of , is an optimal mask.Proof: Let be the eigendecomposition of ,

where are the orthonormal eigenvectorsof , sorted in descending order of their corresponding eigen-values .Let be an arbitrary mask. If , MSE. Otherwise, let be represented by the eigenvector basis as

. According to (6)

MSE

As is the largest eigenvalue of , the following holds:

Since MSE , we see that,indeed

MSE MSE

Hence, maximizes MSE and is an optimal mask.

REFERENCES

[1] N. Ahmed, T. Natarajan, and K. R. Rao, “Discrete cosine transform,”IEEE Trans. Comput., vol. C-23, no. 1, pp. 90–93, Jan. 1974.

[2] M. de Berg, M. van Kreveld, M. Overmars, and O. Schwarzkopf, Com-putational Geometry, Algorithms and Applications, 2nd ed. NewYork: Springer-Verlag, 2000.

[3] R. W. Buccigrossi and E. P. Simoncelli, “Image compression via jointstatistical characterization in the wavelet domain,” IEEE Trans. ImageProcess., vol. 8, no. 12, pp. 1688–1701, Dec. 1999.

[4] I. Daubechies, Ten Lectures on Wavelets. Philadelphia, PA: SIAM,1992.



[5] Z. Devir and M. Lindenbaum, “Adaptive range sampling using a sto-chastic model,” J. Comput. Inf. Sci. Eng., vol. 7, no. 1, pp. 20–25, Mar.2007.

[6] Y. Eldar, M. Lindenbaum, M. Porat, and Y. Zeevi, “The farthest pointstrategy for progressive image sampling,” IEEE Trans. Image Process.,vol. 6, no. 9, pp. 1305–1315, Sep. 1997.

[7] H. Hotelling, “Analysis of a complex of statistical variables into prin-cipal components,” J. Educ. Psychol., vol. 24, no. 6, pp. 417–441, Sep.1933.

[8] H. Hotelling, “Analysis of a complex of statistical variables into prin-cipal components,” J. Edu. Psychol., vol. 24, no. 7, pp. 498–520, Oct.1933.

[9] J. Huang and D. Mumford, “Statistics of natural images and models,”in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Fort Collins, CO,1999, pp. 541–547.

[10] A. K. Jain, Fundamentals of Digital Image Processing. Upper SaddleRiver, NJ: Prentice-Hall, 1989.

[11] S. Mallat, A Wavelet Tour of Signal Processing. San Diego, CA: Aca-demic, 1999.

[12] S. Mallat and Z. Zhifeng, “Matching pursuits with time-frequency dic-tionaries,” IEEE Trans. Signal Process., vol. 41, no. 12, pp. 3397–3415,Dec. 1993.

[13] A. N. Netravali and B. G. Haskell, Digital Pictures. New York:Plenum Press, 1995.

[14] Y. C. Pati, R. Rezaiifar, and P. S. Krishnaprasad, “Orthogonalmatchingpursuit: recursive function approximation with applications to waveletdecomposition,” in Proc. 27th Annu. Asilomar Conf. Signals, Syst.,Comput., 1993, pp. 40–44.

[15] S. Papoulis, Probability, Random Variables and Stochastic Pro-cesses. New York: McGraw-Hill, 2002.

[16] J. M. Shapiro, “Embedded image coding using zerotrees of waveletcoefficients,” IEEE Trans. Signal Process., vol. 41, no. 12, pp.3445–3462, Dec. 1993.

[17] N. Sochen, R. Kimmel, and R. Malladi, “A general framework for lowlevel vision,” IEEE Trans. Image Process., vol. 7, no. 3, pp. 310–318,Mar. 1998.

[18] A. Srivastava, A. B. Lee, E. P. Simoncelli, and S. C. Zhu, “On advancesin statistical modeling of natural images,” J. Math. Imag. Vis., vol. 18,no. 1, pp. 17–33, Jan. 2003.

[19] C. Tomasi and R. Manduchi, “Bilateral filtering for gray and color im-ages,” in Proc. IEEE Int. Conf. Comput. Vis., 1998, pp. 839–846.

[20] J. S. Vitter, “Design and analysis of dynamic Huffman codes,” J. ACM,vol. 34, no. 4, pp. 825–845, Oct. 1987.

[21] B. Zeng and J. Fu, “Directional discrete cosine transforms for imagecoding,” inProc. IEEE Int. Conf. Multimedia Expo, 2006, pp. 721–724.

Zvi Devir received the B.A. degrees in mathematicsand in computer science and theM.Sc. degree in com-puter science from the Technion–Israel Institute ofTechnology, Haifa, Israel, in 2000, 2000, and 2007,respectively.From 2006 to 2010, he was with Medic Vision

Imaging Solutions, Haifa, Israel, a company he co-founded, where he was the Chief Scientific Officer.Previously, he was with Intel, Haifa, Israel, mainlyworking on computer graphic and mathematicaloptimizations. He is currently with IARD Sensing

Solutions, Yagur, Israel, focusing on advanced video processing and spectralimaging. His research interests include video and image processing, mainlyalgebraic representations and differential methods for images.

Michael Lindenbaum received the B.Sc., M.Sc.,and D.Sc. degrees from the Department of Elec-trical Engineering, Technion–Israel Institute ofTechnology, Haifa, Israel, in 1978, 1987, and 1990,respectively.From 1978 to 1985, he served in the IDF in

Research and Development positions. He did hisPostdoc with the Nippon Telegraph and TelephoneCorporation Basic Research Laboratories, Tokyo,Japan. Since 1991, he has been with the Departmentof Computer Science, Technion. He was also a Con-

sultant with Hewlett-Packard Laboratories Israel and spent sabbaticals in NECResearch Institute, Princeton, NJ, in 2001 and in Telecom ParisTech, in 2011.He also spent shorter research periods in the Advanced TelecommunicationsResearch, Kyoto, Japan, and the National Institute of Informatics, Tokyo.He worked in digital geometry, computational robotics, learning, and variousaspects of computer vision and image processing. Currently, his main researchinterest is computer vision, particularly statistical analysis of object recognitionand grouping processes.Prof. Lindenbaum served on several committees of computer vision con-

ferences and is currently an Associate Editor of the IEEE TRANSACTIONS OFPATTERN ANALYSIS AND MACHINE.


Blind adaptive sampling of images.bak

Documents

Transcript of Blind adaptive sampling of images.bak