igny,

clatheriolow

n betedniforicaim

recent advances in sound and natural image processing. Edgesx noning andometrof loca. Thesthe mas.

tions are introduced in [8] and can be processed using multiscalemethods. A global manifold model is used by Baraniuk and Wakin[9] to reconstruct an image fromcompressive sensingmeasurements.

1.1.2. Local edge manifold and cartoon imagesThe study of sets of patches of 3 3 pixels extracted from nat-

ural images has been carried over by Lee et al. [10]. They report sta-tistical evidences showing that the set of high contrast patches is

ned by Carlsson et al. [11] who perform a simplicial approxima-

Some natural textures are composed of nearly parallel stripesthat are modeled as local oscillations. This model of locally paralleltextures is the extension to images of themodel of locally stationarysounds. This model is studied Ben-Shahar and Zucker [20] whoemphasis the role of the regularity of the underlying ow in imageperception. Spatially varying orientations has been used in psycho-physics as the simplest model for geometric textures, see for in-stance themodel of secondorder edges of Landy and co-worker [21].

Demanet and Ying [22] propose a waveatom basis to captureefciently the anisotropic regularity of such textures. AdaptiveE-mail address: gabriel.peyre@ceremade.dauphine.fr

Computer Vision and Image Understanding 113 (2009) 249260

Contents lists availab

I

.eURL: http://www.ceremade.dauphine.fr/Dimensionality reductionmethods such as Isomap [1], eigenmaps[2], LLE [3] or diffusion geometries [4] have been used to study themanifold structure of a library of images. The manifold regularity ofcertain images ensembles is emphasized by Donoho and Grimes [5]and Wakin et al. [6]. These global non-linenar models have applica-tions to image synthesis incomputer vision [7].Manifoldvalued func-

themselves regular. Several tools from harmonic analysis give opti-mal representations for such cartoon functions, including wedg-elets [16], curvelets [17] and bandlets [18,19].

Cartoon images and the edge manifold is studied in Section 3.2.

1.1.3. Locally parallel textures1.1.1. Global manifold models for image librariesmetric images is the cartoon model introduced by Donoho [16]. Acartoon function is regular outside a set of edge curves which areand texture patterns create complemust be captured to improve denoislution. This paper studies these geimages and textures models. The setis modeled using smooth manifoldscontinuous curve (resp. surface) onthat is used to solve inverse problem

1. Introduction

1.1. Previous works1077-3142/$ - see front matter 2008 Elsevier Inc. Adoi:10.1016/j.cviu.2008.09.003-local interactions thatinverse problem reso-

ies for several sounds,l patches in the datasete local features trace anifold, which is a prior

tion of the manifold.Manifold parameterizations of local structures such as edges and

corners is used in computer vision to detect salient features inimages. Bakeret al. [12]propose fast algorithmsto search ina featuremanifold. Huggins and Zucker [13] propose a local principal compo-nents analysis over a feature manifold to speed up computations.

Images with contours contain sharp variations along regularcurves that make wavelets sub-optimal because of their squaresupport [14]. Total variation methods [15] cannot make use ofthe regularity of the edge curves since they only constraint theoverall length of the edges. A simple image model to describe geo-Modeling the geometry of signals and images is at the core of located around the manifold of edges. These results have been re-Manifold models for signals and images

Gabriel PeyrCNRS and CEREMADE, Universit Paris-Dauphine, Place du Marchal De Lattre, De Tass

a r t i c l e i n f o

Article history:Received 19 September 2007Accepted 30 September 2008Available online 17 October 2008

Keywords:Signal processingImage modelingTextureManifold

a b s t r a c t

This article proposes a newof patches extracted fromstructure is detailed for vaThese manifolds provide aThese manifold models carestored signal is represenward measurements. A maularization problem. Numemanifolds models bring an

Computer Vision and

journal homepage: wwwll rights reserved.75775 Paris Cedex 16, France

ss of models for natural signals and images. These models constrain the setdata to analyze to be close to a low-dimensional manifold. This manifoldus ensembles suitable for natural signals, images and textures modeling.-dimensional parameterization of the local geometry of these datasets.used to regularize inverse problems in signal and image processing. Theas a smooth curve or surface traced on the manifold that matches the for-ld pursuit algorithm computes iteratively a solution of the manifold reg-l simulations on inpainting and compressive sensing inversion show thatprovement for the recovery of data with geometrical features.

2008 Elsevier Inc. All rights reserved.

le at ScienceDirect

mage Understanding

l sevier .com/locate /cviu

decompositions such as the grouplets bases of Mallat [23] captureefciently turbulent textures. The idea of characterizing textures ashighly oscillating functions has been introduced by Meyer [24].Efcient algorithms to separate such textures from a cartoonsketch have been proposed by Aujol et al. [25] among others.

Section3.3studies locallyparallel texturesandshowshowapatchmanifold can be optimized to capture specic texture patterns.

1.1.4. Non-local and sparse patch processingThe most successful texture synthesis algorithms perform a

consistent recopy of pixels and patches [26,27]. Local manifolds

8t 2 s=2; s=2d; pxf t f x t: 1The point x is the center of the patch pxf and the width s controlsthe size of typical features to analyze.

A signal ensemble H L20;1d gathers typical data one isinterested in. The patch manifold associated to this ensemble is

M fpxg n x 2 0;1d; g 2 Hg L2s=2; s=2d: 2Section 3 details the structure of M for various signal and imageensembles H. Section 4 uses a xed manifold M to regularize in-

250 G. Peyr / Computer Vision and Image Understanding 113 (2009) 249260of patches have been used for texture synthesis and modication[28,29]. Non-local ltering based on patch comparison has be pro-posed by Buades et al. [30] to perform denoising, and adaptedorthogonal basis can be constructed from non-local graphs [31].Non-local regularization is able to solve inverse problems such assuper-resolution [32]or compressive sensing [33].

Wavelets and more recent tools from harmonic analysis [14]leads to efcient image compression, but fail to capture the geom-etry of natural images and textures [34]. The weakness of xedrepresentations can be alleviated by learning a dictionary fromexamples. Olshausen and Field [35] obtained an optimized set oforiented lters trained on patches extracted from natural images.Other algorithms have been proposed that minimizes various spar-sity enforcing criterions, see for instance [3639]. These dictionar-ies on small patches can be used to perform image denoising [40],facial image compression [41] and to solve inverse problems suchas inpainting [42] or image separation [43].

Section 3.5 details a sparse model for patches originally intro-duced by Peyr [44] for texture synthesis. It ts into our patchmanifold framework, and can thus be used to regularize inverseproblems, as shown in Section 4.

1.2. Contributions

Section 3 details several patch manifolds for signals and images.This extends previous studies that consider mainly local edge mod-els and makes the connection with non-local ltering and sparsepatch decompositions.

Section 4 proposes a manifold regularization to solve inverseproblems, which relies on the local manifold structure of images.Numerical results are shown on inpainting and compressive sens-ing reconstruction. This extends global manifold regularizationthat works for library of images and regularization with sparsityprior in patch dictionaries.

2. Manifolds of signals and images patches

2.1. Local manifold model for patches

A patch pxf of width s > 0 extracted from a signal (d 1) orimage (d = 2) f 2 L20;1d around x 2 0;1d isFig. 1. Examples of patches pxf 2M extracted from images f 2 H in the cartoon imageSection 3.4).verse problems.Fig. 1 shows two examples of image and texture ensembles H.

The patches extracted from these images are parameterized by asmall number of variables (for instance the direction of the edge,the orientation and frequency of the texture) so that the set Mhas the structure of a smooth manifold.

2.2. Manifold parameterization

The setM is assumed to have a smooth manifold structure andis thus locally parameterized around a feature p 2M using asmooth mapping

u : X Rm ! uX M;

where puX. The dimension m gives the number of parametersthat describe the local geometry of images in H.

All the manifold studied in the sequel are globally describedusing a small numberm of parameters. A global parameter domainX Rmk S1k is used to parameterize the whole manifold. Thecircle S1 fh n jhj 1g 0;2p is used to account for directionalfeatures and can be replaced occasionally by the set of orientations~S1 (half circle).

Thanks to this global parameterization, any signal f 2 H is rep-resented as a curve (for signals when d 1) or a surface (forimages when d 2) traced on the manifold M and equivalentlyover the parameter domain X:

cf : x#pxf 2M;Cf : x#u1pxf X:

3

For signals (resp. images) with periodic boundary conditions, thecorresponding curve (resp. surface) is closed (resp. is topologicallyequivalent to a torus).

The mapping cf is a manifold valued function and in the follow-ing VM is the set of such functions from 0;1d toM. These func-tions have been studied in [8].

2.3. Global manifold model for signals and images

The local manifoldM introduced in (2) denes a patch manifoldmodel that assigns to any patch p 2 L2s=2; s=2d its distance toM:s ensemble (left, see Section 3.2) and in the oscillating textures ensemble (right, see

dp;M kp ProjMpk where ProjMp argmin

q2Mkp qk: 4

The goodness of t dp;M to the manifold model is extended to asignal or image f by averaging the distances of all the patches

EMf Z0;1d

dpxf ;M2dx

Z0;1dkpxf ProjMpxf k2dx: 5

A signal or an image f with a low energy EMf traces a curve(resp. a surfaces) cf fpxf gx20;1d close to the low-dimensionalmanifold M. This curve is projected on the manifold M and onthe parameter domain X as follow

~cf x ProjMpxf 2M and ~Cf x u1~cf x 2 X: 6

This leads us to consider the patch manifold of afne functions

~M fpa;b n jaj 6 amax and jbij 6 bmaxg; where pa;b: t#a hb; ti; 8

which is close to the true manifoldM generated by patches of H asdened in 2. In the following, for simplicity, we do not make the dis-tinction between ~M and M.

This afne manifold is parameterized globally as follows:

u : amax; amax bmax; bmaxd ! M;

a; b # t#a hb; ti:

(This shows thatM is a at Euclidean manifold of dimension d 1.Geodesic distances along M are equal, up to a constant, to theEuclidean distance over the parameter domain. This analysis canbe carried over for functions of higher smoothness or for bandlim-ited functions. These smooth signals ensembles are processed opti-

G. Peyr / Computer Vision and Image Understanding 113 (2009) 249260 251The projection ProjMf of a signal f on the patch manifold modelgenerated by M is achieved by computing the set of patchescf x pxf , projecting these patches on the manifold~cf x ProjMcf x and then reconstructing a signalProjMf Aver~cf from ~cf using the averaging operator

Avercx 1sd

Zjxzj6s=2

pzx zdz with pz cz: 7

Section 3 studies for several signal and image ensembles H thegeometry of ~cf . Section 4 uses the manifold energy EM to regularizeinverse problems.

3. Examples of patch manifolds

This section studies the manifold model introduced in the pre-vious section for various image and signal ensembles H.

3.1. Manifold of smooth variations

Smooth signals and images f belongs to the set

H f 2 C10;1d n kfk1 6 amax and@f@xi

16 bmax

:

Patches from functions f 2 H are well approximated by afnefunctions

8jtj 6 s=2; pxf t ax hbx; ti whereax f x;bx rxf :

Fig. 2. Manifold of smally using linear Fourier decomposition and linear ltering.The projection ProjMp of p 2 L2s=2; s=2d is computed by a

linear regression.

3.1.1. Uniformly regular 1D signalsSmooth periodic signals in 1D are locally described using the 2D

manifoldM of afne 1D functions. Each signal f denes a 1D sub-manifold ~cf M, see Eq. (6). This submanifold corresponds to acurve over the parametric domain

8x 2 0;1; ~Cf x u1ProjMpxf f x; f 0x 2 R2:Fig. 2 shows an example of a smooth function f together with thecorresponding embedding ~cf in (b). This curve ~cf exihibits self inter-sections, which correspond to points xy where f x f y andf 0x f 0y.

3.1.2. Uniformly regular 2D imagesSmooth images f with periodic boundary conditions are de-

scribed using the 3D manifold M of afne images. The manifoldM thus denes a surface

8x 2 0;12; ~Cf x1; x2 u1ProjMpxf f x; @f@x1 ;@f@x2

2 R3:

Fig. 3, left, shows an example of a regular image, that is generated asa realization of a Gaussian random eld X whose power spectrumsatises jbXxj 1 jxjr for a regularity exponent r > 1 (forthe gure we have set r 3).mooth signals.

of s

252 G. Peyr / Computer Vision and Image Understanding 113 (2009) 249260Fig. 3, right, shows the corresponding 2D surface ~cf embeddedin the 3D volume X u1M. This surface is topologically equiv-alent to a torus but exhibits self intersections along 1D curves.

3.2. Manifold of cartoon images

Uniformly smooth signals and images are of little practicalinterest to model natural datasets. Signals with step discontinuitiesare efciently processed with wavelets that avoid ringing artifactsof a linear Fourier approximation. Images with contours containsharp variations along regular curves that make wavelets sub-opti-mal because of their square support [14].

A simple model of binary images is dened as

H f1B hx n B 0;12 with @B regularg;where h is a regular kernel. The set B represents the object of inter-est in the scene. It is supposed to be connected with @B of boundedcurvature. This model can be extended to multiple objects as long astheir boundaries are separated by a distance larger than s.

Locally a patch of f is well approximated by a single straightedge

pxf t Phx;dxt where Ph;dt PRht d; 0;where Rh is the planar rotation of angle h. The step is P h ~Pwhere ~Pt 0 if t1 < 0 and ~Pt 1 otherwise. Fig. 4 shows someexamples of typical edge patches.

This leads to the following 2D parameterization of the manifold

Fig. 3. Manifoldof binary edge patches:

u :S1 R !M;h; d#Ph;d:

(9

Fig. 4. (left) An example of cartoon image. (right) ParameterizaThe manifoldM is thus topologically equivalent to a cylinder, how-ever, due to the lack of translation invariance when the edge ap-proaches the boundary of s=2; s=22, the manifold is not at.Indeed the two constant functions equal to 0 and 1 play a specialrole of poles. Fig. 5 shows a 3D display of the corresponding embed-ding. We note that for this edge manifold, one generally hasM ~Cfsince the boundary @B covers all the orientations h 2 S1. Howeverthe surface Cf traced by f on the parameter domain S

1 R is com-plex if @B exhibits concavities that lead to self intersections of Cf .Fig. 5 shows curves that correspond to a 1D sections in Cf . Thesecurves trace closed loops on the manifold M.

3.3. Manifold of locally stationary sounds

Natural sounds are usually modeled as highly oscillating signalswith a phase that is slowly varying. Such a signal can be written as

f x Ax cosWx;where AxP 0 is the local amplitude andW0xP 0 the local phaseof the oscillations. Such a decomposition is, however, non-uniquelydened and one usually assumes that A andW0x are slowly varyingwith respect to the signal sampling so that they can be reliably esti-mated. This leads to the following signals ensemble:

H fx#f x Ax cosWx n kA0k1 6 Amax and kW00k16 Wmaxg:

Patches extracted from locally stationary signals are close to the

mooth images.manifold

M fPA;q;d n AP 0 and qP 0 and d 2 S1g wherePA;q;dx A cosqx d:

tion of the manifold of edge patches and some examples.

The parameterization A;q; d#PA;q;d shows thatM is equivalent toX R R S1.

The projection of a patch p 2 L2s=2; s=2 on M can be car-ried over approximately using a windowed Fourier transform.One uses a smooth window function h supported on s=2; s=2and denes the windowed Fourier transform of p:

p^x Z

htpt expixtdt:

A 1D signal f denes a 1D curve ~cf M traced on the manifold anda 1D curve ~Cf in 3D parameter space

~Cf fAx;qx; dxgx20;1 where PAx;qx;dx ProjMpxf :Fig. 6 shows examples of a locally stationary oscillating signal to-gether with its spectrogram and the corresponding curve ~Cf overthe parametric space.

Fig. 5. (left) A cartoon image. (right) 3D representation of the edge manifold M (depicted in 3D as a cylinder). The two curves on the manifold corresponds to patchesextracted along the two lines in the image.

G. Peyr / Computer Vision and Image Understanding 113 (2009) 249260 253Following Delprat et al. [45] (see also [14]), the projection of p isthen given as

ProjMp PA;q;d whereq argmax

xP0jp^xj;

p^q A expid:

8

prole of the oscillations. This model of locally parallel textures isthe extension of locally stationary sounds to images, if one consid-ers the case hx cosx. Fig. 1, right, shows examples of locallyparallel textures.

A local texture patch is approximated as

pxf PhAx;Bx;qx;hx;dx where PhA;B;q;h;dx Ahqhx; hi d B:

The parameter h 2 ~S1 is the direction of the oscillations, q their fre-quency and d is a phase shift. The manifold M is parameterized as

u :R R R ~S1 S1 !M;A; B;q; h; d#PhA;B;q;h;d:

(11

An approximate estimation of qx; hx; dx from a texture f canbe carried over using local Fourier expansions over windows of sizes as dened in the 1D case in Eq. (10).

pxf Xi

sxiui Dsx

using a dictionary D fuigP1i0 of P atoms. Each ui 2 L2s=2; s=2dis an atomic template and sxi 2 R is the corresponding coefcient.The sparsity of the decomposition is ensured by constraining the 0

pseudo-norm of the coefcients to be smaller than S 2 N:ksxk0 #fi n sxi0g 6 S:This leads to the manifold of sparse patches

M MD Xi

siui n ksk0 6 S( )

:

This set M is not a smooth manifold but rather a non-linear unionof S-dimensional linear spaces. It is parameterized by the dictionaryD and the sparsity level S.

254 G. Peyr / Computer Vision and Image Understanding 113 (2009) 249260The feature manifold depends on the specic choice of the pro-le h:

M Mh fPhA;B;q;h;d n A;B;q; h; d 2 Xg:An adaptation of the manifold to an exemplar f 2 L20;1d to pro-cess is achieved by optimizing the prole h so that the patches of fare as close as possible from the manifoldMh. This amount to min-imizing the energy EMh dened in (5) to select an adapted prole h

:

h argminh2H

EMh f ; 12

where H L2R is a set of proles that might for instance containssome smoothness constraints on the texture prole.

We restrict the computation to a simple set of 1D prolesparameterized by a contrast c 2 0;1:hcx signcosxj cosxjc: 13The optimal c minimizing EMhc f is computed by testing a discreteset of values in 0;1. Fig. 7 shows examples of projection withc 0:35, which is the value minimizing EMhc f for the ngerprintimage f. More elaborated adaptation strategies could be used tot arbitrary texture proles.

3.5. Manifold of sparse patches

3.5.1. Sparse patch expansionThe sparse patch model assumes that each patch pxf extracted

from a signal or image f has a sparse expansionFig. 7. An image and its projection ProjMf on the manifold model of oscillating texturespatches pxf , to better see the importance of translation invariance.The corresponding signal ensemble H with sparse patches is

H HD ff n 8x;pxf Dsx with ksxk0 6 Sg: 14This model has been introduced by Peyr [44] for texture synthesisand is re-casted in our manifold patch model.

The projection on the sparse manifoldM requires the computa-tion of

ProjMp D argmins2Rm

kp Dsk2 subject to ksk0 6 S:

The exact computation of this projection is NP-hard for an arbitrarydictionary D, but it can be solved approximately using for instance Ssteps of orthogonal matching pursuit algorithm, see [14].

3.5.2. Dictionary learningThe manifold MD is optimized so that a given exemplar f is as

close as possible to HD. This corresponds to the learning of the dic-tionary D, that is optimized in order to minimize the goodness of tof f to the model generated by D:

D argminD

EMDf ; 15

where the energy EMD is dened in (5). This is similar to the adap-tation of the prole to optimize the oscillating texture model (12).In the optimization (15), the atoms ui of D are only constrainedto be of unit norm, kuik 1.

The optimization (15) is re-written by optimizing over both thedictionary D and the coefcients sx of the patches pxf :D; sx

x

argminD;fsxgx

Xx

kpxf Dsxk2 subject to ksxk0 6 S:

16. The center image displays a projection computed with a subset of non-overlapping

This optimization problem is highly non-convex and a stationarypoint of EMD with respect to D can be computed using several iter-ative algorithms, for instance the MOD algorithm [36] or K-SVD[39]. Table 1 details the MOD optimization process that alternatesbetween the computation of the coefcients fsxgx and the optimiza-tion of the dictionary atoms ui.

Fig. 8 shows an example of a dictionary learned from an homo-geneous texture.

4. Inverse problem regularization with manifold models

4.1. Inverse problems

The manifold prior model introduced in Section 2.3 is used toregularize the inversion of an operator U : L20;1d ! V , whereV is an Hilbert space of nite or innite dimension. The mappingU is typically ill-posed and difcult to invert since Uf gathers onlya limited amount of information from the original f to recover.

Forward measurements correspond to the computations of

y Uf 2 V ;where f is the data to recover and is an additive noise.

Regularization theory assumes that f belongs to some functionalspace H such as a Sobolev space (linear regularization) or the spaceof bounded variations (non-linear regularization). The recoveredsignal f is the solution of an optimization problem

f argming2H

kyUgk2 kEg; 17

where E should be small when g is close to the smoothness model.The weight k should be adapted to match the amplitude of the noise, which might be a non-trivial task in practical situations.

Classical variational priors include

Total variation: The bounded variation model imposes that f hasa nite bounded variation and uses.

Eg kgkTV Zjrxgjdx: 18

This prior has been introduced by Rudin, Osher and Fatemi [15] fordenoising purpose.

Sparsity priors: Given an orthogonal basis fwkgk of L20;1d, asparsity enforcing prior is dened as

where EMg; c kyUgk k0;1dkpxg cxk dx; 21

Table 1MOD dictionary learning algorithm.

(1) Initialization: set each ui as a realization of a white noise, normalized so thatkuik 1.

(2) Coefcient update: compute the projection Dsx of each patch pxf , on themanifold M

G. Peyr / Computer Vision and Image Understanding 113 (2009) 249260 255sx argminskpxf Dsk subject to ksk0 6 S

This is computed approximately with S steps of orthogonal matching pursuitapplied to pxf

(3) Dictionary update: the dictionary D is computed by minimizing

minD

Xx

kpxf Dsxk2;

whose solution is given as

D PR where R RTR1RT

where R fsxgx is the matrix whose columns are the coefcients sx andP fpxf gx is the matrix whose columns are the (discretized) patches pxf .

(4) Normalization: for all i, ui ui=kuik.(5) Stop: while not converged, go back to 2.Fig. 8. (left) Input texture f and (right) dictionawhose solution f also solves the original problem (17) for E EMand c ~cf .

The manifold valued mapping c should be close to the originalprojected curve (or surface when d 2) ~cf as dened in (6). Themanifold regularization of inverse problems (20) computes a curve(or surface) traced on the manifold M that also matches the for-ward measurements y.

A stationary point of (20) is computed by alternatively minimiz-ing over f and c:Eg Xk

jhg;wkij: 19

This prior has been introduced by Donoho and Johnstone [46] withthe wavelet basis for denoising purpose. It has then been used tosolve more general inverse problems, see for instance [47] andthe references therein. It can also be used in conjunction withredundant frames instead of orthogonal bases, see for instance[48,49].

4.2. Regularization with manifold model

This paper explores the global manifold energy E EM denedin Eq. (5) to perform the regularization. The optimization (17) withE EM is performed by introducing a manifold valued functionc 2 VM. At each location x, the patch cx 2M is tracking a lo-cal feature of f and should be close to pxf .

The optimization (17) is re-written using c as

f ; c argming;c2L20;1dVM

EMg; c; 20

2Z

2ry D learned to sparsity the patches pxf .

The image is xed. The manifold valued function c0 minimizingEMg; cwith respect to c 2 VM is computed with the projec-tion dened in Eq. (4)

8x 2 0;1d; c0x ProjMpxg: 22 The manifold-valued mapping is xed. The signal g0 minimizingEMg; c with respect to g 2 L20;1d is the solution of

UTU kIdg0 UTy kAverc;where the averaged signal Averc 2 L20;1d of c is dened in(7).The iteration of these two steps corresponds to the manifoldpursuit detailed in Table 2.

The algorithm of Table 2 shares some similarities with iterativethresholdings methods used to solve the non-linear regularizedinversion (17) with a sparsity prior (19), see for instance [47]. Tohandle the noiseless case 0, the value of the regularizationparameter k can be decreased toward 0 during the iterations.

The main computational burden in the manifold pursuit is thenumerical computation of the projection ProjM on the manifold.Depending on the image model considered, specic solvers canbe used. For complex manifolds without any analytical description,a dense sampling of the manifold is used together with a fast clos-est-point algorithm.

Since the energy to optimize (20) is non-convex, the manifoldpursuit might fail to converge to the global minimizer of the prob-lem. For a smooth manifold M, the iterates f k; ck of the algo-rithm 2 converge to a stationary point f ; c of the energy EM

Inpainting corresponds to the operation of removing pixels froman input data

Uf x 0 if x 2 X;f x if x R X;

where X 0;1d is the region where the input data have been dam-aged. Classical methods for inpainting use partial differential equa-tions that propagate the information from the boundary of X to itsinterior, see for instance [5154]. Sparsity promoting prior such as(19) in wavelets frames and local cosine bases have been used tosolve inpainting as a inverse problem [48,49].

256 G. Peyr / Computer Vision and Image Understanding 113 (2009) 249260since this energy satises the hypotheses of [50].

4.3. Numerical experiments

Although any linear operator can be treated within our regular-ization framework, this paper focusses on the following examples.

Table 2Manifold pursuit to minimize (20).

(1) Initialization: set f 0 UTy and k 0(2) Manifold closest point: update the manifold valued function as

8x 2 0;1d; ck1x ProjMpxf kwhere the manifold projection ProjM is dened in (4)

(3) Least square t: update the current estimate as

f k1 UTU kId1UTy kAverck1and where the averaging Averck1 of ck1 is dened in (7).

(4) Stopping criterion: while not converged, set k k 1 and go back to 2Fig. 9. Iterations of the inpainting algori Compressive sensing is a new sampling theory that uses a xedset of linear measurements together with a non-linear recon-struction [55,56], see also the review [57]. The sensing operatorcomputes the projection of the data on a nite set of k vectors

Uf fhf ; uiigk1i0 2 Rk: 23The signal are discretized nite-dimensional vectors f 2 Rn. Com-pressive sensing states hypotheses on both the input signal f andthe sensing vectors fuigi for this non-uniform sampling process tobe invertible with 1 minimization. The sensing vectors fuigi mustbe incoherent, which is the case with high probability if they aredrawn randomly from unit norms Gaussian white noise vectors. Un-der the additional condition that f is sparse in some orthogonal ba-sis fwkgk:#fk n hf ;wki0 6 Sg;the optimization of (17) with the sparsity prior (19) leads to a per-fect recovery f f if k OS logn=S, where n is the dimension ofthe sampled signal f. This result holds in the noiseless case 0; k! 0 and can be extended to an approximate recovery inthe noisy case 0; k > 0.

The following numerical experiments compare the efciency ofsparsity regularization with 1 prior (19) with the patch manifoldenergy EM (5). Sparsity regularization leads to a convex optimiza-tion, which is an advantage over the non-convex manifold regular-ization that might be trapped in a local minimum.

For numerical computation, discretized signals and images areobtained by an uniform sampling at n points. The correspondingmanifolds of patches M Rsdn are embedded in a nite-dimen-sional space. The following numerical experiments are performedwith patches of width w 10 pixels. Compressive sensing experi-ments are performed with sensing vectors ui that are random dis-crete Fourier vectors, so that the sensing operator U is computedwith the FFT algorithm.

4.3.1. Smooth imagesFig. 9 shows iterations of the algorithm 2 to solve the inpainting

problem on a smooth image using a manifold prior with 2D linearthm on an uniformly regular image.

patches, as dened in 8, with a low amplitude noise . This mani-fold regularization together with the overlapping of the patchesperforms a smooth interpolation of the missing pixels.

The iterations of the algorithm are similar to a linear diffusionthat propagates the available information inside the set X of re-moved pixels. The performances of the algorithm are similar to lin-ear methods such as inpainting with a Sobolev regularizerEf R jrf j2 that corresponds to a heat diffusion inside X.4.3.2. Cartoon images

Figs. 10 and 11 show iterations of the projection algorithm 2with a manifold model of binary edges, as dened in Eq. (9). Forthis numerical optimization, the manifold of edges is discretizedas already done for the display of Fig. 5 and the projection ProjMis computed with a fast nearest-neighbor search. For both inpaint-ing and compressive sampling, the manifold of edges allows to

reconstruct with good precision the boundary of a single smoothobject (here a disk).

Fig. 12 shows a more challenging compressive sensing problemwhere the image is composed of layers of occluding objects withsmooth boundaries and varying intensities. In order to cope witha non-binary image, the manifold of afne edges is used

M faPh;d b n h; d 2 S1 R and a; b 2 R Rg: 24This manifold is four dimensional, but the projection on this mani-fold is computed efciently by iteratively optimizing the projectionover the h; d parameters and then the a; b parameters. Patchesextracted from Fig. 12, left, are however not always close to thismanifold because of crossings that occur when two singularitycurves meet.

Fig. 12 compares the compressive sensing reconstruction with asparsity prior (19) in a translation invariant wavelet frame (center)

Fig. 10. Iterations of the inpainting algorithm on a geometrical image with the binary edge model.

Fig. 11. Iterations of the compressive sensing reconstruction algorithm on a geometrical image with the binary edge model. The number of sensed vectors is n0 n=10, wheren is the number of pixels.

G. Peyr / Computer Vision and Image Understanding 113 (2009) 249260 257Fig. 12. Compressive sensing reconstruction results on a geometrical image with sparssensed vectors is n0 n=8, where n is the number of pixels.ity prior in wavelets and with the manifold model of afne edges. The number of

and with a manifold prior in the afne edges manifold (24). Theoptimization of the sparsity energy (19) is performed with an iter-ative thresholding algorithm, see [47], whereas the optimization ofthe manifold energy (20) is performed with the algorithm 2. Thereconstruction with a manifold prior is of better quality than thesparsity prior in a wavelet frame. This is because, in 2D, waveletscannot take advantage of the regularity of the boundaries of theobjects.

4.3.3. Locally parallel texturesFig. 13 shows an example of inpainting of a ngerprint image

using the manifold of 2D oscillations (11). Fig. 14 shows the com-pressive sensing reconstruction from the same image, where themanifold prior is compared to a sparsity prior (19) in a local Gaborredundant frame, see [14]. Such local oscillating atoms have beenintroduced with success for texture decomposition and inpainting

in [48,49]. The manifold prior is better able to capture the geomet-ric regularity of the texture than the sparsity prior that diffuses theorientation information over several Gabor coefcients.

4.3.4. Sparse patchesFig. 15 shows a reconstruction from compressive sensing using

a manifold prior in the sparse texture ensemble (14). The dictio-nary D is learned by minimizing EMD fe for an exemplar texturefe that is close to the texture f to recover. In practice, both imageare extracted from different localization of the same image. Thecentral part of the texture is badly reconstructed, because this partis not similar to the exemplar fe.

Fig. 16 shows a comparison of inpainting with a sparse prior(19) in a translation invariant wavelet frame and a sparse patchmanifold model prior. The dictionary D is learned from a set ofpatches pxi y extracted from the observed data, where the point

Fig. 13. Iterations of the inpainting reconstruction algorithm on a locally parallel texture.

ith

258 G. Peyr / Computer Vision and Image Understanding 113 (2009) 249260Fig. 14. Compressive sensing reconstruction results on a locally parallel texture wnumber of sensed vectors is n0 n=4, where n is the number of pixels.Fig. 15. Iterations of compressive sensing reconstruction results using a manifold modenumber of pixels.sparsity prior in a redundant Gabor dictionary and with the manifold model. Thel in a learned dictionary. The number of sensed vectors is n0 n=6, where n is the

d th

ge Uxi are located at a distance larger than s from the missing region tobe inpainted. The resulting algorithm is similar to the inpaintingalgorithm of Mairal et al. [42], although they use a more compli-cated scheme to learn a dictionary with missing data.

5. Conclusion

This paper has reviewed several manifold models for sounds,images and textures. These models constrain the set of patches ex-tracted from the image and describe efciently the non-lineargeometry of some classes of natural signals and images. A newmanifold pursuit algorithm is used to regularize ill-posed inverseproblem while maintaining the manifold model constraints. Re-sults on various images and textures show how this manifold-dri-ven restoration enhances variational models based on sparseexpansions for inpainting and compressive sensing reconstruction.

Acknowledgement

I thank the anonymous reviewers for their comments thathelped to improve the quality of the manuscript.

References

[1] J.B. Tenenbaum, V. de Silva, J.C. Langford, A global geometric framework fornonlinear dimensionality reduction, Science 290 (5500) (2000) 23192323.

[2] M. Belkin, P. Niyogi, Laplacian eigenmaps for dimensionality reduction anddata representation, Neural Computation 15 (6) (2003) 13731396.

Fig. 16. Comparison of inpainting using a sparsity prior in a wavelet basis an

G. Peyr / Computer Vision and Ima[3] S. Roweis, L. Saul, Nonlinear dimensionality reduction by locally linearembedding, Science 290 (5500) (2000) 23232326.

[4] R.R. Coifman, S. Lafon, A.B. Lee, M. Maggioni, B. Nadler, F. Warner, S.W. Zucker,Geometric diffusions as a tool for harmonic analysis and structure denition ofdata: diffusion maps, Proceedings of the National Academy of Science 102(2005) 74267431.

[5] D. Donoho, C. Grimes, Image manifolds isometric to Euclidean space, Journal ofMathematical Imaging and Vision 23 (1) (2005) 524.

[6] M. Wakin, D. Donoho, H. Choi, R. Baraniuk, The multiscale structure of non-differentiable image manifolds, in: Optics and Photonics, San Diego, CA, 2005.

[7] M. Aharon, R. Kimmel, Representation analysis and synthesis of lip imagesusing dimensionality reduction, International Journal of Computer Vision 67(3) (2006) 297312.

[8] I. Ur-Raman, I. Drori, V. Stodden, P. Schroeder, Multiscale representations ofmanifold-valued data, SIAM Multiscale Modeling and Simulation 4 (4) (2005)12011232.

[9] R.G. Baraniuk, M.B. Wakin, Random projections of smooth manifolds,Foundations of Computational Mathematics, doi:10.1007/s10208-007-9011-z.Published online 18 December 2007.

[10] A. Lee, K. Pedersen, D. Mumford, The nonlinear statistics of high-contrastpatches in natural images, International Journal of Computer Vision 54 (13)(2003) 83103.

[11] G. Carlsson, T. Ishkhanov, V. de Silva, A. Zomorodian, On the local behavior ofspaces of natural images, International Journal of Computer Vision 76 (1)(2008) 112.

[12] S. Baker, S. Nayar, H. Murase, Parametric feature detection, InternationalJournal of Computer Vision 27 (1) (1998) 2750.[13] P.S. Huggins, S.W. Zucker, Representing edge models via local principalcomponent analysis, in: Proceedings of ECCV02, Springer LNCS 2350 (2002)384398.

[14] S. Mallat, A Wavelet Tour of Signal Processing, Academic Press, San Diego,1998.

[15] L.I. Rudin, S. Osher, E. Fatemi, Nonlinear total variation based noise removalalgorithms, Physica D 60 (14) (1992) 259268.

[16] D. Donoho, Wedgelets: Nearly-minimax estimation of edges, Annals ofStatistics 27 (1999) 353382.

[17] E. Cands, D. Donoho, New tight frames of curvelets and optimalrepresentations of objects with piecewise C2 singularities, Communicationson Pure and Applied Mathematics 57 (2) (2004) 219266.

[18] E. Le Pennec, S. Mallat, Bandelet image approximation and compression, SIAMMultiscale Modeling and Simulation 4 (3) (2005) 9921039.

[19] S. Mallat, G. Peyr, Orthogonal bandlet bases for geometric imagesapproximation, Communications on Pure and Applied Mathematics 61 (9)(2008) 11731212.

[20] O. Ben-Shahar, S. Zucker, The perceptual organization of texture ow: Acontextual inference approach, IEEE Transaction on Pattern Analysis andMachine Intelligence 25 (4) (2003) 401417.

[21] M.S. Landy, I. Oruc, Properties of 2nd-order spatial frequency channels, VisionResearch 42 (2002) 23112329.

[22] L. Demanet, L. Ying, Wave atoms and sparsity of oscillatory patterns, Appliedand Computational Harmonic Analysis 3 (23) (2007) 368387.

[23] S. Mallat, Geometrical grouplets, Applied and Computational HarmonicAnalysis, doi:10.1016/j.acha.2008.03.004. Published online 7 April 2008.

[24] Y. Meyer, Oscillating Patterns in Image Processing and Nonlinear EvolutionEquations, American Mathematical Society, Boston, MA, 2001.

[25] J.F. Aujol, G. Aubert, L. Blanc-Feraud, A. Chambolle, Image decomposition into abounded variation component and an oscillating component, Journal ofMathematical Imaging and Vision 22 (1) (2005) 7188.

[26] A.A. Efros, T.K. Leung, Texture synthesis by non-parametric sampling, in:ICCV99: Proceedings of the International Conference on Computer Vision vol. 2, IEEE Computer Society, Silver Spring, MD, 1999, p. 1033.

[27] L.-Y. Wei, M. Levoy, Fast texture synthesis using tree-structured vectorquantization, in: SIGGRAPH00: Proceedings of the 27th Annual Conference

e sparse manifold model with a dictionary learned from the observed data y.

nderstanding 113 (2009) 249260 259on Computer Graphics and Interactive Techniques, Addison-Wesley, Reading,MA, 2000, pp. 479488.

[28] J. Wang, X. Tong, S. Lin, M. Pan, C. Wang, H. Bao, B. Guo, H.-Y. Shum,Appearance manifolds for modeling time-variant appearance of materials, in:Proceedings of Siggraph06, ACM Press, New York, NY, 2006, pp. 754761.

[29] W. Matusik, M. Zwicker, F. Durand, Texture design using a simplicial complexof morphable textures, ACM Transaction on Graphics 24 (3) (2005) 787794.

[30] A. Buades, B. Coll, J.M. Morel, On image denoising methods, SIAM MultiscaleModeling and Simulation 4 (2) (2005) 490530.

[31] G. Peyr, Manifold spectral bases for the non-local processing of images, SIAMMultiscale Modeling and Simulation 7 (2) (2008) 703730.

[32] M. Protter, M. Elad, H. Takeda, P. Milanfar, Generalizing the non- local-meansto super-resolution reconstruction, IEEE Transactions on Image Processing,(2008), accepted for publication.

[33] G. Peyr, S. Bougleux, L. Cohen, Non-local regularization of inverse problems,in: Proceedings of ECCV 2008, 2008.

[34] E.P. Simoncelli, B.A. Olshausen, Natural image statistics and neuralrepresentation, Annual Review of Neuroscience 24 (2001) 11931215.

[35] B.A. Olshausen, D.J. Field, Emergence of simple-cell receptive-eld propertiesby learning a sparse code for natural images, Nature 381 (6583) (1996) 607609.

[36] K. Engan, S.O. Aase, J.H. Husoy, Method of optimal directions for frame design,in: Proceedings of ICASSP99, IEEE Computer Society, Washington, DC, 1999,pp. 24432446.

[37] K. Kreutz-Delgado, J.F. Murray, B.D. Rao, K. Engan, T.-W. Lee, T. Sejnowski,Dictionary learning algorithms for sparse representation, Neural Computation15 (2) (2003) 349396.

[38] M.S. Lewicki, T.J. Sejnowski, Learning overcomplete representations, NeuralComputation 12 (2) (2000) 337365.

[39] M. Aharon, M. Elad, A. Bruckstein, The K-SVD: an algorithm for designing ofovercomplete dictionaries for sparse representation, IEEE Transactions onSignal Processing 54 (11) (2006) 43114322.

[40] M. Elad, M. Aharon, Image denoising via sparse and redundant representationsover learned dictionaries, IEEE Transactions on Image Processing 15 (12)(2006) 37363745.

[41] O. Bryt, M. Elad, Compression of facial images using the K-SVD algorithm,Journal of Visual Communication and Image Representation 19 (4) (2008)270282.

[42] J. Mairal, M. Elad, G. Sapiro, Sparse representation for color image restoration,IEEE Transactions on Image Processing 17 (1) (2008) 5369.

[43] G. Peyr, J. Fadili, J. Starck, Learning dictionaries for geometry and textureseparation, in: Proceedings of SPIE Wavelet XII, vol. 6701, International Societyfor Optical Engineering, 2007, pp. 67011T.

[44] G. Peyr, Sparse modeling of textures, Journal of Mathematical Imaging andVision, doi:10.1007/s10851-008-0120-3. Published online 6 November 2008.

[45] N. Delprat, B. Escudi, P. Guillemain, R. Kronland-Martinet, P. Tchamitchian, B.Torrsani, Asymptotic wavelet and gabor analysis: extraction of instantaneousfrequencies, IEEE Transaction on Information Theory 38 (1992) 644664.

[46] D. Donoho, I. Johnstone, Ideal spatial adaptation via wavelet shrinkage,Biometrika 81 (1994) 425455.

[47] I. Daubechies, M. Defrise, C.D. Mol, An iterative thresholding algorithm forlinear inverse problems with a sparsity constraint, Communications on Pureand Applied Mathematics 57 (2004) 14131541.

[48] M. Elad, J.-L. Starck, D. Donoho, P. Querre, Simultaneous cartoon and textureimage inpainting using morphological component analysis (MCA), Journal onApplied and Computational Harmonic Analysis 19 (2005) 340358.

[49] M.J. Fadili, J.-L. Starck, F. Murtagh, Inpainting and zooming using sparserepresentations, The Computer Journal, doi:10.1093/comjnl/bxm055.Published online July 2007.

[50] P. Tseng, Convergence of a block coordinate descent method fornondifferentiable minimization, Journal of Optimization Theory andApplications 109 (3) (2001) 475494.

[51] S. Masnou, Disocclusion: a variational approach using level lines, IEEETransactions on Image Processing 11 (2) (2002) 6876.

[52] C. Ballester, M. Bertalmio, V. Caselles, G. Sapiro, J. Verdera, Filling-in by jointinterpolation of vector elds and gray levels, IEEE Transactions on ImageProcessing 10 (8) (2001) 12001211.

[53] M. Bertalm` o, G. Sapiro, V. Caselles, C. Ballester, Image inpainting, in: Siggraph2000, 2000, pp. 417424.

[54] D. Tschumperl, R. Deriche, Vector-valued image regularization with PDEs: Acommon framework for different applications, IEEE Transaction on PatternAnalysis and Machine Intelligence 27 (4) (2005) 506517.

[55] E. Cands, T. Tao, Near-optimal signal recovery from random projections:universal encoding strategies?, IEEE Transactions on Information Theory 52(12) (2006) 54065425

[56] D. Donoho, Compressed sensing, IEEE Transactions on Information Theory 52(4) (2006) 12891306.

[57] E. Cands, Compressive sampling, in: Proceedings of the InternationalCongress of Mathematicians, Madrid, Spain (3) (2006) 14331452.

260 G. Peyr / Computer Vision and Image Understanding 113 (2009) 249260

Manifold models for signals and imagesIntroductionPrevious worksGlobal manifold models for image libraries. librariesLocal edge manifold and cartoon images. imagesLocally parallel textures. texturesNon-local and sparse patch processing. processing

Contributions

Manifolds of signals and images patchesLocal manifold model for patchesManifold parameterizationGlobal manifold model for signals and images

Examples of patch manifoldsManifold of smooth variationsUniformly regular 1D signals. signalsUniformly regular 2D images. images

Manifold of cartoon imagesManifold of locally stationary soundsManifold of locally parallel texturesManifold of sparse patchesSparse patch expansion. expansionDictionary learning. learning

Inverse problem regularization with manifold modelsInverse problemsRegularization with manifold modelNumerical experimentsSmooth images. imagesCartoon images. imagesLocally parallel textures. texturesSparse patches. patches

ConclusionAcknowledgementReferences