arX

iv:1

504.

0829

1v5

[cs.

NE

] 14

Mar

201

61

Deep Neural Networks with Random GaussianWeights: A Universal Classification Strategy?

Raja Giryes* , Guillermo Sapiro** , and Alex M. Bronstein*

* School of Electrical Engineering, Faculty of Engineering,Tel-Aviv University, Ramat Aviv 69978, Israel.{raja@tauex, bron@eng}.tau.ac.il

** Department of Electrical and Computer Engineering, Duke University, Durham, North Carolina, 27708, USA.{guillermo.sapiro}@duke.edu.

Abstract—Three important properties of a classification ma-chinery are: (i) the system preserves the core information of theinput data; (ii) the training examples convey information aboutunseen data; and (iii) the system is able to treat differently pointsfrom different classes. In this work we show that these funda-mental properties are satisfied by the architecture of deep neuralnetworks. We formally prove that these networks with randomGaussian weights perform a distance-preserving embeddingofthe data, with a special treatment for in-class and out-of-classdata. Similar points at the input of the network are likely tohave a similar output. The theoretical analysis of deep networkshere presented exploits tools used in the compressed sensingand dictionary learning literature, thereby making a formalconnection between these important topics. The derived resultsallow drawing conclusions on the metric learning propertiesof the network and their relation to its structure, as well asproviding bounds on the required size of the training set such thatthe training examples would represent faithfully the unseen data.The results are validated with state-of-the-art trained networks.

I. I NTRODUCTION

Deep neural networks (DNN) have led to a revolution inthe areas of machine learning, audio analysis, and computervision, achieving state-of-the-art results in numerous applica-tions [1], [2], [3]. In this work we formally study the propertiesof deep network architectures with random weights appliedto data residing in a low dimensional manifold. Our resultsprovide insights into the outstanding empirically observedperformance of DNN, the role of training, and the size ofthe training data.

Our motivation for studying networks with random weightsis twofold. First, a series of works [4], [5], [6] empiricallyshowed successful DNN learning techniques based on ran-domization. Second, studying a system with random weightsrather than learned deterministic ones may lead to a betterunderstanding of the system even in the deterministic case.Forexample, in the field of compressed sensing, where the goal isto recover a signal from a small number of its measurements,the study of random sampling operators led to breakthroughsin the understanding of the number of measurements requiredfor achieving a stable reconstruction [7]. While the boundsprovided in this case are universally optimal, the introductionof a learning phase provides a better reconstruction perfor-mance as it adapts the system to the particular data at hand

[8], [9], [10]. In the field of information retrieval, randompro-jections have been used for locality-sensitive hashing (LSH)scheme capable of alleviating the curse of dimensionality forapproximate nearest neighbor search in very high dimensions[11]. While the original randomized scheme is seldom used inpractice due to the availability of data-specific metric learningalgorithms, it has provided many fruitful insights. Other fieldssuch as phase retrieval, gained significantly from a study basedon random Gaussian weights [12].

Notice that the technique of proving results for deep learn-ing with assumptions on some random distribution and thenshowing that the same holds in the more general case is notunique to our work. On the contrary, some of the strongerrecent theoretical results on DNN follow this path. For exam-ple, Arora et al. analyzed the learning of autoencoders withrandom weights in the range[−1, 1], showing that it is possibleto learn them in polynomial time under some restrictions onthe depth of the network [13]. Another example is the series ofworks [14], [15], [16] that study the optimization perspectiveof DNN.

In a similar fashion, in this work we study the properties ofdeep networks under the assumption of random weights. Be-fore we turn to describe our contribution, we survey previousstudies that formally analyzed the role of deep networks.

Hornik et al. [17] and Cybenko [18] proved that neural net-works serve as a universal approximation for any measurableBorel functions. However, finding the network weights for agiven function was shown to be NP-hard.

Bruna and Mallat proposed the wavelet scatteringtransform– a cascade of wavelet transform convolutions withnonlinear modulus and averaging operators [19]. They showedfor this deep architecture that with more layers the resultingfeatures can be made invariant to increasingly complex groupsof transformations. The study of the wavelet scattering trans-form demonstrates that deeper architectures are able to bettercapture invariant properties of objects and scenes in images

Anselmi et al. showed that image representations invari-ant to transformations such as translations and scaling canconsiderably reduce the sample complexity of learning andthat a deep architecture with filtering and pooling can learnsuch invariant representations [20]. This result is particularlyimportant in cases that training labels are scarce and in totallyunsupervised learning regimes.

2

Montufar and Morton showed that the depth of DNN allowsrepresenting restricted Boltzmann machines with a numberof parameters exponentially greater than the number of thenetwork parameters [21]. Montufaret al. suggest that eachlayer divides the space by a hyper-plane [22]. Therefore, adeep network divides the space into an exponential number ofsets, which is unachievable with a single layer with the samenumber of parameters.

Brunaet al. showed that the pooling stage in DNN resultsin shift invariance [23]. In [24], the same authors interpretthis step as the removal of phase from a complex signal andshow how the signal may be recovered after a pooling stageusing phase retrieval methods. This work also calculates theLipschitz constants of the pooling and the rectified linear unit(ReLU) stages, showing that they perform a stable embeddingof the data under the assumption that the filters applied in thenetwork are frames, e.g., for the ReLU stage there exist twoconstants0 < A ≤ B such that for anyx,y ∈ R

n,

A ‖x− y‖2 ≤ ‖ρ(Mx)− ρ(My)‖2 ≤ B ‖x− y‖2 , (1)

whereM ∈ Rm×n denotes the linear operator at a given layer

in the network withm andn denoting the input and outputdimensions, respectively, andρ(x) = max(0, x) is the ReLUoperator applied element-wise. However, the values of the Lip-schitz constantsA andB in real networks and their behavior asa function of the data dimension currently elude understanding.To see why such a bound may be very loose, consider theoutput of only the linear part of a fully connected layer withrandom i.i.d. Gaussian weights,mij ∼ N(0, 1/

√m), which

is a standard initialization in deep learning. In this case,A

and B scale like√

2nm

(

1±√

m2n

)

respectively [25]. Thisundesired behavior is not unique to a normally-distributedM,being characteristic of any distribution with a bounded fourthmoment. Note that the addition of the non-linear operators,theReLU and the pooling, makes these Lipschitz constants evenworse.

A. Contributions

As the former example teaches, the scaling of the data intro-duced byM may drastically deform the distances throughouteach layer, even in the case wherem is very close ton, whichmakes it unclear whether it is possible to recover the input ofthe network (or of each layer) from its output. In this work, themain question we focus on is: What happens to the metric ofthe input data throughout the network? We focus on the abovementioned setting, assuming that the network has random i.i.d.Gaussian weights. We prove that DNN preserve the metricstructure of the data as it propagates along the layers, allowingfor the stable recovery of the original data from the featurescalculated by the network.

This type of property is often encountered in the literature[24], [26]. Notice, however, that the recovery of the input ispossible if the size of the network output is proportional tothe intrinsic dimension of the data at the input (which is notthe case at the very last layer of the network, where we haveclass labels only), similarly to data reconstruction from asmall

number of random projections [27], [28], [29]. However, un-like random projections that preserve the Euclidean distancesup to a small distortion [30], each layer of DNN with randomweights distorts these distances proportionally to the anglesbetween its input points: the smaller the angle at the input,thestronger the shrinkage of the distances. Therefore, the deeperthe network, the stronger the shrinkage we get. Note that thisdoes not contradict the fact that we can recover the input fromthe output; even when properties such as lighting, pose andlocation are removed from an image (up to certain extent), theresemblance to the original image is still maintained.

As random projection is a universal sampling strategy forany low dimensional data [7], [29], [31], deep networks withrandom weights are a universal system that separates anydata (belonging to a low dimensional model) according tothe angles between its points, where the general assumptionis that there are large angles between different classes [32],[33], [34], [35]. As training of the projection matrix adapts itto better preserve specific distances over others, the trainingof a network prioritizes intra-class angles over inter-classones. This relation is alluded by our proof techniques andis empirically manifested by observing the angles and theEuclidean distances at the output of trained networks, asdemonstrated later in the paper in Section VI.

The rest of the paper is organized as follows: In Section II,we start by utilizing the recent theory of 1-bit compressedsensing to show that each DNN layer preserves the metric ofits input data in the Gromov-Hausdorff sense up to a smallconstantδ, under the assumption that these data reside ina low-dimensional manifold denoted byK. This allows usto draw conclusions on the tessellation of the space createdby each layer of the network and the relation between theoperation of these layers and local sensitive hashing (LSH)[11]. We also show that it is possible to retrieve the input ofalayer, up to certain accuracy, from its output. This impliesthatevery layer preserves the important information of the data.

In Section III, we proceed by analyzing the behavior ofthe Euclidean distances and angles in the data throughout thenetwork. This section reveals an important effect of the ReLU.Without the ReLU, we would just have random projectionsand Euclidean distance preservation. Our theory shows thatthe addition of ReLU makes the system sensitive to the anglesbetween points. We prove that networks tend to decrease theEuclidean distances between points with a small angle betweenthem (“same class”), more than the distances between pointswith large angles between them (“different classes”).

Then, in Section IV we prove that low-dimensional data atthe input remain such throughout the entire network, i.e., DNN(almost) do not increase the intrinsic dimension of the data.This property is used in Section V to deduce the size of dataneeded for training DNN.

We conclude by studying the role of training in Section VI.As random networks are blind to the data labels, trainingmay select discriminatively the angles that cause the distancedeformation. Therefore, it will cause distances between dif-ferent classes to increase more than the distances within thesame class. We demonstrate this in several simulations, someof which with networks that recently showed state-of-the-art

3

performance on challenging datasets, e.g., the network by [36]for the ImageNet dataset [37]. Section VII concludes the paperby summarizing the main results and outlining future researchdirections.

It is worthwhile emphasizing that the assumption thatclasses are separated by large angles is common in the liter-ature (see [32], [33], [34], [35]). This assumption can furtherrefer to some feature space rather than to the raw data space.Of course, some examples might be found that contradict thisassumption such as the one of two concentric spheres, whereeach sphere represents a different class. With respect to suchparticular examples two things should be said: First, thesecases are rare in real life signals, typically exhibiting someamount of scale invariance that is absent in this example.Second, we prove that the property of discrimination based onangles holds for DNN with random weights and conjecture inSection VI that a potential role (or consequence) of trainingin DNN is to favor certain angles over others and to selectthe origin of the coordinate system with respect to which theangles are calculated. We illustrate in Section VI the effect oftraining compared to random networks, using trained DNNthat have achieved state-of-the-art results in the literature.Our claim is that DNN are suitable for models with clearlydistinguishable angles between the classes if random weightsare used, and for classes with some distinguishable anglesbetween them if training is used.

For the sake of simplicity of the discussion and presentationclarity, we focus only on the role of the ReLU operator [38],assuming that our data are properly aligned, i.e., they arenot invariant to operations such as rotation or translation, andtherefore there is no need for the pooling operation to achieveinvariance. Combining recent results for the phase retrievalproblem with the proof techniques in this paper can lead toa theory also applicable to the pooling operation. In addition,with the strategy in [39], [40], [41], [42], it is possible togeneralize our guarantees to sub-Gaussian distributions and torandom convolutional filters. We defer these natural extensionsto future studies.

II. STABLE EMBEDDING OF A SINGLE LAYER

In this section we consider the distance between the metricsof the input and output of a single DNN layer of the formf(M·), mapping an input vectorx to the output vectorf(Mx), where M is an m × n random Gaussian matrixand f : R → R is a semi-truncated linear function appliedelement-wise.f is such if it is linear on some (possibly,semi-infinite) interval and constant outside of it,f(0) = 0,0 < f(x) ≤ x, ∀x > 0, and 0 ≥ f(x) ≥ x, ∀x < 0.The ReLU, henceforth denoted byρ, is an example of sucha function, while the sigmoid and the hyperbolic tangentfunctions satisfy this property approximately.

We assume that the input data belong to a manifoldK withthe Gaussian mean width

ω(K) := E[ supx,y∈K

〈g,x− y〉], (2)

where the expectation is taken over a random vectorg withnormal i.i.d. elements. To better understand this definition,

Fig. 1. Width of the setK in the direction ofg. This figure is a variant ofFig. 1 in [29]. Used with permission of the authors.

note thatsupx,y∈K〈g,x−y〉 is the width of the setK in the

direction ofg as illustrated in Fig. 1. The mean provides anaverage over the widths of the setK in different isotropicallydistributed directions, leading to the definition of the Gaussianmean widthω(K).

The Gaussian mean width is a useful measure for thedimensionality of a set. As an example we consider thefollowing two popular data models:

a) Gaussian mixture models:K ⊂ B2 (the ℓ2-ball)

consists ofL Gaussians of dimensionk. In this caseω(K) =O(

√k + logL).b) Sparsely representable signals:The data inK ⊂ B

2

can be approximated by a sparse linear combination of atomsof a dictionary, i.e.,K = {x = Dα : ‖α‖0 ≤ k, ‖x‖2 ≤ 1},where‖·‖0 is the pseudo-norm that counts the number of non-zeros in a vector andD ∈ R

n×L is the given dictionary. Forthis modelω(K) = O(

√

k log(L/k)).Similar results can be shown for other models such as

union of subspaces and low dimensional manifolds. For moreexamples and details onω(K), we refer the reader to [29],[43].

We now show that each standard DNN layer performs astable embedding of the data in the Gromov-Hausdorff sense[44], i.e., it is aδ-isometry between(K, dK) and (K ′, dK′),whereK andK ′ are the manifolds of the input and outputdata anddK anddK′ are metrics induced on them. A functionh : K → K ′ is a δ-isometry if

|dK′(h(x), h(y)) − dK(x,y)| ≤ δ, ∀x,y ∈ K, (3)

and for everyx′ ∈ K ′ there existsx ∈ K such thatdY (x

′, h(x)) ≤ δ (the latter property is sometimes calledδ-surjectivity). In the following theorem and throughout thepaperC denotes a given constant (not necessarily the sameone) andSn−1 the unit sphere inRn.

Theorem 1:Let M be the linear operator applied at aDNN layer,f a semi-truncated linear activation function, andK ⊂ S

n−1 the manifold of the input data for that layer.If

√mM ∈ R

m×n is a random matrix with i.i.d normallydistributed entries, then the mapg : (K ⊂ S

n−1, d Sn−1) →(g(K), dHm) defined byg(x) = sgn(f(Mx)) is with highprobability aδ-isometry, i.e.,

|d Sn−1(x,y) − dHm(g(x), g(y))| ≤ δ, ∀x,y ∈ K,

with δ ≤ Cm−1/6 ω1/3(K).

4

In the theoremdSn−1 is the geodesic distance onSn−1,dH is the Hamming distance and the sign function,sgn(·), isapplied elementwise, and is defined assgn(x) = 1 if x > 0and sgn(x) = −1 if x ≤ 0. The proof of the approximateinjectivity follows from the proof of Theorem 1.5 in [43].

Theorem 1 is important as it provides a better understandingof the tessellation of the space that each layer creates. Thisresult stands in line with [22] that suggested that each layerin the network creates a tessellation of the input data by thedifferent hyper-planes imposed by the rows inM. However,Theorem 1 implies more than that. It implies that each cellin the tessellation has a diameter of at mostδ (see alsoCorollary 1.9 in [43]), i.e., ifx andy fall to the same sideof all the hyperplanes, then‖x− y‖2 ≤ δ. In addition, thenumber of hyperplanes separating two pointsx andy in Kcontains enough information to deduce their distance up to asmall distortion. From this perspective, each layer followedby the sign function acts as locality-sensitive hashing [11],approximately embedding the input metric into the Hammingmetric.

Having a stable embedding of the data, it is natural toassume that it is possible to recover the input of a layer fromits output. Indeed, Mahendran and Vedaldi demonstrate thatitis achievable through the whole network [26]. The next resultprovides a theoretical justification for this, showing thatit ispossible to recover the input of each layer from its output:

Theorem 2:Under the assumptions of Theorem 1 thereexists a programA such that‖x−A(f(Mx))‖2 ≤ ǫ, where

ǫ = O(

ω(K)√m

)

.The proof follows from Theorem 1.3 in [29]. IfK is a cone

then one may use also Theorem 2.1 in [45] to get a similarresult.

Both theorems 1 and 2 are applying existing results from1-bit compressed sensing on DNN. Theorem 1 deals withembedding into the Hamming cube and Theorem 2 uses thisfact to show that we can recover the input from the output.Indeed, Theorem 1 only applies to an individual layer andcannot be applied consecutively throughout the network, sinceit deals with embedding into the Hamming cube. One wayto deal with this problem is to extend the proof in [43]to an embedding fromSn−1 to S

m−1. Instead, we turn tofocus on the ReLU and prove more specific results about theexact distortion of angles and Euclidean distances. These alsoinclude a proof about a stable embedding of the network ateach layer fromRn to R

m.

III. D ISTANCE AND ANGLE DISTORTION

So far we have focused on the metric preservation of thedeep networks in terms of the Gromov-Hausdorff distance. Inthis section we turn to look at how the Euclidean distancesand angles change within the layers. We focus on the case ofReLU as the activation function. A similar analysis can alsobe applied for pooling. For the simplicity of the discussionwedefer it to future study.

Note that so far we assumed thatK is normalized and lieson the sphereSn−1. Given that the data at the input of thenetwork lie on the sphere and we use the ReLUρ as the

cos( 6 (x,y))-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1

1 π

(

sin(6(x,y

))−cos(

6(x,y

))6(x,y

))

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Fig. 2. Behavior ofψ(x, y) (Eq. (5)) as a function ofcos∠(x,y). Thetwo extremitiescos∠(x,y) = ±1 correspond to the angles zero andπ,respectively, betweenx and y. Notice that the larger is the angle betweenthe two, the larger is the value ofψ(x,y). In addition, it vanishes for smallangles between the two vectors.

0 0.5 1 1.5 2 2.5 3 3.50

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

6 (x, y)

[π/4, 0.71]

[π/2, cos− 1(1/π )]

[π , π/2]

E [ 6 (ρ(Mx), ρ(My))]0.8 6 (x, y)0.95 6 (x, y)

Fig. 3. Behavior of the angle between two pointsx andy in the output of aDNN layer as a function of their angle in the input. In the ranges[0, π/4] and[π/4, π/2] the output behaves approximately like0.95∠(x,y) and 0.21 +23∠(x,y), respectively.

activation function, the transformationρ(M·) keeps the outputdata (approximately) on a sphere (with half the diameter, see(31) in the proof of Theorem 3 in the sequel). Therefore, inthis case the normalization requirement holds up to a smalldistortion throughout the layers. This adds a motivation forhaving a normalization stage at the output of each layer, whichwas shown to provide some gain in several DNN [1], [46].

Normalization, which is also useful in shallow represen-tations [47], can be interpreted as a transformation makingthe inner products between the data vectors coincide with thecosines of the corresponding angles. While the bounds weprovide in this section do not require normalization, they showthat the operations of each layer rely on the angles betweenthe data points.

The following two results relate the Euclidean and angulardistances in the input of a given layer to the ones in its output.We denote byBn

r ⊂ Rn the Euclidean ball of radiusr.

Theorem 3:Let M be the linear operator applied at a givenlayer,ρ (the ReLU) be the activation function, andK ⊂ B

n1 be

the manifold of the data in the input layer. If√mM ∈ R

m×n

is a random matrix with i.i.d. normally distributed entriesand

5

6 (x, y)0 0.2 0.4 0.6 0.8 1 1.2

6 (x,y) ∈ [0,π/4]6 (x,y) ∈ [π/4,π/2]6 (x,y) ∈ [π/2,π]

6 (x, y)/6 (x,y)0 0.2 0.4 0.6 0.8 1 1.2

6 (x, y)− 6 (x,y)-2 -1.5 -1 -0.5 0

Fig. 4. Left: Histogram of the angles in the output of the8-th (dashed-lines) and16-th (continuous-lines) layers of the ImageNet deep networkfor differentinput angle ranges. Middle: Histogram of the ratio between the angles in the output of the8-th (dashed-line) and16-th (continuous-line) layers of this networkand the angles in its input for different input angle ranges.Right: Histogram of the differences between the angles in the output of the8-th (dashed-line) and16-th (continuous-line) layers of this network and the anglesin its input for different input angles ranges.

m ≥ Cδ−4w(K)4, then with high probability∣

∣

∣

∣

∣

‖ρ(Mx)− ρ(My)‖22 − (4)

(

1

2‖x− y‖22 + ‖x‖2 ‖y‖2 ψ(x,y)

)

∣

∣

∣

∣

∣

≤ δ,

where0 ≤ ∠(x,y) , cos−1(

xTy

‖x‖2‖y‖2

)

≤ π and

ψ(x,y) =1

π

(

sin∠(x,y) − ∠(x,y) cos∠(x,y))

. (5)

Theorem 4:Under the same conditions of Theorem 3 andK ⊂ B

n1 \ Bn

β , whereδ ≪ β2 < 1, with high probability∣

∣

∣

∣

∣

cos∠ (ρ(Mx), ρ(My)) − (6)

(cos∠(x,y) + ψ(x,y))

∣

∣

∣

∣

∣

≤ 15δ

β2 − 2δ.

Remark 1 As we have seen in Section II, the assumptionm ≥ Cδ−4w(K)2 impliesm = O(k2) if K is a GMM andm = O(k2 logL) if K is generated byk-sparse represen-tations in a dictionaryD ∈ R

n×L. As we shall see laterin Theorem 6 in Section IV, it is enough to have the modelassumption only at the data in the input of the DNN. Finally,the quadratic relationship betweenm and w(K)2 might beimproved.

We leave the proof of these theorems to Appendices A andB, and dwell on their implications.

Note that ifψ(x,y) were equal to zero, then theorems 3 and4 would have stated that the distances and angles are preserved(in the former case, up to a factor of0.51) throughout the net-work. As can be seen in Fig. 2,ψ(x,y) behaves approximatelylike 0.5(1 − cos∠(x,y)). The larger is the angle∠(x,y),the larger isψ(x,y), and, consequently, also the Euclideandistance at the output. If the angle betweenx andy is closeto zero (i.e.,cos∠(x,y) close to 1),ψ(x,y) vanishes and

1More specifically, we would have‖ρ(Mx)− ρ(My)‖22∼= 1

2‖x− y‖22.

Notice that this is the expectation of the distance‖ρ(Mx−My)‖22.

therefore the Euclidean distance shrinks by half throughoutthe layers of the network. We emphasize that this is not incontradiction to Theorem 1 which guarantees approximatelyisometric embedding into the Hamming space. While thebinarized output with the Hamming metric approximatelypreserves the input metric, the Euclidean metric on the rawoutput is distorted.

Considering this effect on the Euclidean distance, thesmaller the angle betweenx andy, the larger the distortionto this distance at the output of the layer, and the smallerthe distance turns to be. On the other hand, the shrinkage ofthe distances is bounded as can be seen from the followingcorollary of Theorem 3.

Corollary 5: Under the same conditions of Theorem 3, withhigh probability

1

2‖x− y‖22 − δ ≤ ‖ρ(Mx)− ρ(My)‖22 ≤ ‖x− y‖22 + δ. (7)

The proof follows from the inequality of arithmetic andgeometric means and the behavior ofψ(x,y) (see Fig. 2). Weconclude that DNN with random Gaussian weights preservelocal structures in the manifoldK and enable decreasingdistances between points away from it, a property muchdesired for classification.

The influence of the entire network on the angles is slightlydifferent. Note that starting from the input of the second layer,all the vectors reside in the non-negative orthant. The cosineof the angles is translated from the range[−1, 1] in the firstlayer to the range[0, 1] in the subsequent second layers. Inparticular, the range[−1, 0] is translated to the range[0, 1/π],and in terms of the angle∠(x,y) from the range[π/2, π] to[cos−1(1/π), π/2]. These angles shrink approximately by half,while the ones that are initially small remain approximatelyunchanged.

The action of the network preserves the order between theangles. Generally speaking, the network affects the anglesin the range[0, π/2] in the same way. In particular, in therange [0, π/4] the angles in the output of the layer behavelike 0.95∠(x,y) and in the wider range[0, π/2] they arebounded from below by0.8∠(x,y) (see Fig. 3). Therefore,

6

Fig. 5. Sketch of the distortion of two classes with distinguishable anglebetween them as obtained by one layer of DNN with random weights. Thesenetworks are suitable for separating this type of classes. Note that the distancebetween the blue and red points shrinks less than the distance between thered points as the angle between the latter is smaller.

we conclude that the DNN distort the angles in the inputmanifold K similarly and keep the general configuration ofangles between the points.

To see that our theory captures the behavior of DNNendowed with pooling, we test how the angles change throughthe state-of-the-art19-layers deep network trained in [36] forthe ImageNet dataset. We randomly select3 ·104 angles (pairsof data points) in the input of the network, partitioned to threeequally-sized groups, each group corresponding to a one of theranges[0, π/4], [π/4, π/2] and[π/2, π]. We test their behaviorafter applying eight and sixteen non-linear layers. The lattercase corresponds to the part of the network excluding the fullyconnected layers. We denote byx the vector in the output ofa layer corresponding to the input vectorx. Fig. 4 presents ahistogram of the values of the angles∠(x, y) at the output ofeach of the layers for each of the three groups. It shows alsothe ratio∠(x, y)/∠(x,y) and difference∠(x, y) − ∠(x,y),between the angles at the output of the layers and their originalvalue at the input of the network.

As Theorem 4 predicts, the ratio∠(x, y)/∠(x,y) corre-sponding to∠(x,y) ∈ [π/2, π] is half the ratio correspondingto input angles in the range[0, π/2]. Furthermore, the ratios inthe ranges[0, π/4] and[π/4, π/2] are approximately the same,where in the range[π/4, π/2] they are slightly larger. This isin line with Theorem 4 that claim that the angles in this rangedecrease approximately in the same rate, where for largerangles the shrink is slightly larger. Also note that accordingto our theory the ratio corresponding to input angles in therange[0, π/4] should behave on average like0.95q, whereqis the number of layers. Indeed, forq = 8, 0.958 = 0.66 andfor q = 16, 0.9516 = 0.44; the centers of the histograms forthe range[0, π/4] are very close to these values. Notice thatwe have a similar behavior also for the range[π, π/2]. Thisis not surprising, as by looking at Fig. 3 one may observethat these angles also turn to be in the range that has theratio 0.95. Remarkably, Fig. 4 demonstrates that the networkkeeps the order between the angles as Theorem 4 suggests.Notice that the shrinkage of the angles does not cause largeangles to become smaller than other angles that were originallysignificantly smaller than them. Moreover, small angles in theinput remain small in the output as can be seen in Fig. 4(right).

We sketch the distortion of two sets with distinguishableangle between them by one layer of DNN with random weightsin Fig. 5. It can be observed that the distance between points

with a smaller angle between them shrinks more than thedistance between points with a larger angle between them.Ideally, we would like this behavior, causing points belongingto the same class to stay closer to each other in the outputof the network, compared to points from different classes.However, random networks are not selective in this sense: ifa pointx forms the same angle with a pointz from its classand with a pointy from another class, then their distance willbe distorted approximately by an equal amount. Moreover, theseparation caused by the network is dependent on the settingofthe coordinate system origin with respect to which the anglesare calculated. The location of the origin is dependent on thebias termsb (in this case each layer is of the formρ(Mx+b)),which are set to zero in the random networks here studied.These are learned by the training of the network, affecting theangles that cause the distortions of the Euclidean (and angular)distances. We demonstrate the effect of training in SectionVI.

IV. EMBEDDING OF THE ENTIRE NETWORK

In order to show that the results in sections II and IIIalso apply to theentire network and not only to one layer,we need to show that the Gaussian mean width does notgrow significantly as the data propagate through the layers.Instead of bounding the variation of the Gaussian mean widththroughout the network, we bound the change in the coveringnumberNǫ(K), i.e., the smallest number ofℓ2-balls of radiusǫ that coverK. Having the bound on the covering number, weuse Dudley’s inequality [48],

ω(K) ≤ C

∫ ∞

0

√

logNǫ(K)dǫ, (8)

to bound the Gaussian mean width.Theorem 6:Under the assumptions of Theorem 1,

Nǫ(f(MK)) ≤ Nǫ/

(

1+ω(K)√

m

) (K) . (9)

Proof: We divide the proof into two parts. In the first one,we consider the effect of the activation functionf on the sizeof the covering, while in the second we examine the effectof the linear transformationM. Starting with the activationfunction, letx0 ∈ K be a center of a ball in the covering ofK and x ∈ K be a point that belongs to the ball ofx0 ofradiusǫ, i.e., ‖x− x0‖2 ≤ ǫ. It is not hard to see that since asemi-truncated linear activation function shrinks the data, then‖f(x)− f(x0)‖2 ≤ ‖x− x0‖2 ≤ ǫ and therefore the size ofthe covering does not increase (but might decrease).

For the linear part we have that [49, Theorem 1.4]

‖Mx−Mx0‖2 ≤(

1 +ω(K)√m

)

‖x− x0‖2 . (10)

Therefore, after the linear operation each covering ball withinitial radius ǫ is not bigger than(1 + ω(K)√

m)ǫ. Since the

activation function does not increase the size of the covering,we have that after a linear operation followed by an activationfunction, the size of the covering balls increases by a factorof (1 + ω(K)√

m). Therefore, the size of a covering with balls of

radiusǫ of the output dataf(MK) is bounded by the size ofa covering with balls of radiusǫ/(1 + ω(K)√

m). �

7

Theorem 6 generalizes the results in theorems 2, 3 and4 such that they can be used for the whole network: thereexists an algorithm that recovers the input of the DNN fromits output; the DNN as a whole distort the Euclidean distancesbased on the angels of the input of the network; and the angulardistances smaller thanπ are not altered significantly by thenetwork.

Note, however, that Theorem 6 does not apply to Theorem 1.In order to do that for the later, we need also a versionof Theorem 1 that guarantees a stable embedding using thesame metric at the input and the output of a given layer,e.g., embedding from the Hamming cube to the Hammingcube or fromS

n−1 to Sm−1. Indeed, we have exactly such a

guarantee in Corollary 5 that implies stable embedding of theEuclidean distances in each layer of the network. Though thiscorollary focuses on the particular case of the ReLU, unlikeTheorem 1 that covers more general activation functions, itimplies stability for the whole network in the Lipschitz sense,which is even stronger than stability in the Gromov-Hausdorffsense that we would get by having the generalization ofTheorem 1.

As an implication of Theorem 6, consider low-dimensionaldata admitting a Gaussian mixture model (GMM) withLGaussians of dimensionk or ak-sparse represention in a givendictionary withL atoms. For GMM, the covering number isNǫ(K) = L

(

1 + 2ǫ

)kfor ǫ < 1 and 1 otherwise (see [31]).

Therefore we have thatω(K) ≤ C√k + logL and that at each

layer the Gaussian mean width grows at most by a factor of1+O

(√k+logL√

m

)

. In the case of sparsely representable data,

Nǫ(K) =(

Lk

) (

1 + 2ǫ

)k. By Stirling’s approximation we have

(

Lk

)

≤(

eLk

)kand thereforeω(K) ≤ C

√

k log(L/k). Thus, ateach layer the Gaussian mean width grows at most by a factor

of 1 +O

(√k log(L/k)√

m

)

.

V. TRAINING SET SIZE

An important question in deep learning is what is theamount of labeled samples needed at training. Using Sudakovminoration [48], one can get an upper bound on the size ofa covering ofK, Nǫ(K), which is the number of balls ofradius ǫ that include all the points inK. We have demon-strated that networks with random Gaussian weights realizea stable embedding; consequently, if a network is trainedusing the screening technique by selecting the best amongmany networks generated with random weights as suggestedin [4], [5], [6], then the number of data points needed inorder to guarantee that the network represents all the data isO(exp(ω(K)2/ǫ2)). Sinceω(K)2 is a proxy for the intrinsicdata dimension as we have seen in the previous sections(see [29] for more details), this bound formally predicts thatthe number of training points grows exponentially with theintrinsic dimension of the data.

The exponential dependency is too pessimistic, as it is oftenpossible to achieve a better bound on the required trainingsample size. Indeed, the bound developed in [13] requiresmuch less samples. As the authors study the data recoverycapability of an autoencoder, they assume that there existsa

‘ground-truth autoencoder’ generating the data. A combinationof the data dimension bound here provided, with a prior onthe relation of the data to a deep network, should lead to abetter prediction of the number of needed training samples.Infact, we cannot refrain from drawing an analogy with the fieldof sparse representations of signals, where the combined useof the properties of the system with those of the input data ledto works that improved the bounds beyond the naıve manifoldcovering number (see [50] and references therein).

The following section presents such a combination, byshowing empirically that the purpose of training in DNN isto treat boundary points. This observation is likely to leadtoa significant reduction in the required size of the training data,and may also apply to active learning, where the training setis constructed adaptively.

VI. T HE ROLE OF TRAINING

The proof of Theorem 3 provides us with an insight on therole of training. One key property of the Gaussian distribution,which allows it to keep the ratio between the angles in the data,is its rotational invariance. The phase of a random Gaussianvector with i.i.d. entries is a random vector with a uniformdistribution. Therefore, it does not prioritize one direction inthe manifold over the other but treats all the same, leading tothe behavior of the angles and distances throughout the netthat we have described above.

In general, points within the same class would have smallangles within them and points from distinct classes wouldhave larger ones. If this holds for all the points, then randomGaussian weights would be an ideal choice for the networkparameters. However, as in practice this is rarely the case,animportant role of the training would be to select in a smart waythe separating hyper-planes induced byM in such a way thatthe angles between points from different classes are ‘penalizedmore’ than the angles between the points in the same class.

Theorem 3 and its proof provide some understanding ofhow this can be done by the learning process. Consider theexpectation of the Euclidean distance between two pointsx

andy at the output of a given layer. It reads as (the derivationappears in Appendix A)

E[‖ρ(Mx)− ρ(My)‖22] =1

2‖x‖22 +

1

2‖y‖22 (11)

+‖x‖2 ‖y‖2

π

∫ π−∠(x,y)

0

sin(θ) sin(θ + ∠(x,y))dθ.

Note that the integration in this formula is done uniformlyover the interval[0, π − ∠(x,y)], which contains the rangeof directions that have simultaneously positive inner productswith x andy. With learning, we have the ability to pick theangleθ that maximizes/minimizes the inner product based onwhetherx andy belong to the same class or to distinct classesand in this way increase/decrease their Euclidean distances atthe output of the layer.

Optimizing over all the angles between all the pairs ofthe points is a hard problem. This explains why randominitialization is a good choice for DNN. As it is hard tofind the optimal configuration that separates the classes on themanifold, it is desirable to start with a universal one that treats

8

0 0.5 1 1.5 2 2.5 3||x− z||2/||x− y||2

RandomNet1 25% ErrorNet2 21% ErrorNet3 18% Error

(a) Inter-class Euclidean distanceratio

0 0.5 1 1.5 2 2.5 3||x− z||2/||x− y||2

(b) Intra-class Euclidean distanceratio

0 1 2 3 4 56 (x, z)/ 6 (x , y)

(c) Inter-class angular distance ra-tio

0 1 2 3 4 56 (x, z)/ 6 (x , y)

(d) Intra-class angular distance ra-tio

Fig. 6. Ratios ofclosest inter- (left) and farthest intra-class (right) classEuclidean (top) and angular (bottom) distances for CIFAR-10. For each datapoint we calculate its Euclidean distance to the farthest point from its classand to the closest point not in its class, both at the input of the DNN and atthe output of the last convolutional layer. Then we compute the ratio betweenthe two, i.e., ifx is the point at input,y is its farthest point in class,x is thepoint at the output, andz is its farthest point from the same class (it shouldnot necessarily be the output ofy), then we calculate

‖x−z‖2‖x−y‖2

and ∠(x,z)∠(x,y)

.We do the same for the distances between different classes, comparing theshortest Euclidean and angular distances.

most of the angles and distances well, and then to correct itin the locations that result in classification errors.

To validate this hypothesized behavior, we trained two DNNon the MNIST and CIFAR-10 datasets, each containing10classes. The training of the networks was done using thematconvnettoolbox [51]. The MNIST and CIFAR-10 networkswere trained with four and five layers, respectively, followedby a softmax operation. We used the default settings providedby the toolbox for each dataset, where with CIFAR-10 wealso used horizontal mirroring and64 filters in the first twolayers instead of32 (which is the default in the exampleprovided with the package) to improve performance. Thetrained DNN achieve1% and 18% errors for MNIST andCIFAR-10 respectively.

For each data point we calculate its Euclidean and angulardistances to its farthest intra-class point and to its closestinter-class point. We compute the ratio between the distancesat the output of the last convolutional layer (the input ofthe fully connected layers) and the ones at the input. Letx be the point at the input,y be its farthest point fromthe same class,x be the point at the output, andz be itsfarthest point from the same class (it should not necessarilybe the output ofy), then we calculate‖x−z‖2

‖x−y‖2for Euclidean

distances and∠(x, z)/∠(x,y) for the angles. We do the samefor the distances between different classes, comparing the

-2 -1 0 1 2||x− z||2 − ||x− y||2

RandomNet1 25% ErrorNet2 21% ErrorNet3 18% Error

(a) Inter-class Euclidean distancedifference

-2 -1 0 1 2||x− z||2 − ||x− y||2

(b) Intra-class Euclidean distancedifference

−1.5 −1 −0.5 0 0.5 1 1.56 (x, z) − 6 (x , y)

(c) Inter-class angular distance dif-ference

−1.5 −1 −0.5 0 0.5 1 1.56 (x, z) − 6 (x , y)

(d) Intra-class angular distance dif-ference

Fig. 7. Differences ofclosest inter- (left) and farthest intra- (right) classEuclidean (top) and angular (bottom) distances for CIFAR-10 (a counterpartof Fig. 6 with distance ratios replaced with differences). For each data pointwe calculate its Euclidean distance to the farthest point from its class and tothe closest point not in its class, both at the input of the DNNand at the outputof the last convolutional layer. Then we compute the difference between thetwo, i.e., if x is the point at input,y is its farthest point in class,x is thepoint at the output, andz is its farthest point from the same class (it shouldnot necessarily be the output ofy), then we calculate‖x− z‖2 −‖x− y‖2and∠(x, z) − ∠(x,y). We do the same for the distances between differentclasses, comparing the shortest Euclidean and angular distances.

shortest ones. Fig. 6 presents histograms of these distanceratios for CIFAR-10. In Fig. 7 we present the histogramsof the differences of the Euclidean and angular distances,i.e., ‖x− z‖2 − ‖x− y‖2 and ∠(x, z) − ∠(x,y). We alsocompare the behavior of all the inter and intra-class distancesby computing the above ratios for all pairs of points(x,y) inthe input with respect to their corresponding points(x, y) atthe output. These ratios are presented in Fig. 8. We present alsothe differences‖x− y‖2 − ‖x− y‖2 and∠(x, y) − ∠(x,y)in Fig. 9. We present the results for three trained networks,inaddition to the random one, denoted by Net1, Net2 and Net3.Each of them corresponds to a different amount of trainingepochs, resulting with a different classification error.

Considering the random DNN, note that all the histogramsof the ratios are centered around1 and the ones of thedifferences around0, implying that the network preserves mostof the distances as our theorems predict for a network withrandom weights. For the trained networks, the histograms overall data point pairs (Figs. 8 and 9) change only slightly dueto training. Also observe that the trained networks behavelike their random counterparts in keeping the distance of arandomly picked pair of points. However, they distort thedistances between points on class boundaries “better” thanthe random network (Figs. 6 and 7), in the sense that thefarthest intra class distances are shrunk with a larger factor

9

0 0.5 1 1.5 2 2.5 3||x− y||2/||x− y||2

RandomNet1 25% ErrorNet2 21% ErrorNet3 18% Error

(a) Inter-class Euclidean distanceratio

0 0.5 1 1.5 2 2.5 3||x− y||2/||x− y||2

(b) Intra-class Euclidean distanceratio

0 1 2 3 4 56 (x, y)/ 6 (x , y)

(c) Inter-class angular distance ra-tio

0 1 2 3 4 56 (x, y)/ 6 (x , y)

(d) Intra-class angular distance ra-tio

Fig. 8. Ratios of inter- (left) and intra- (right) class Euclidean (top) andangular (bottom) distances betweenrandomly selected pointsfor CIFAR-10.We calculate the Euclidean distances between randomly selected pairs of datapoints from different classes (left) and from the same class(right), both atthe input of the DNN and at the output of the last convolutional layer. Thenwe compute the ratio between the two, i.e., for all pairs of points (x,y) inthe input and their corresponding points(x, y) at the output we calculate‖x−y‖2‖x−y‖2

and ∠(x,y)∠(x,y)

.

than the ones of the random network, and the closest interclass distances are set farther apart by the training. Notice thatthe shrinking of the distances within the class and enlargementof the distances between the classes improves as the trainingproceeds. This confirms our hypothesis that a goal of trainingis to treat the boundary points.

A similar behavior can be observed for the angles. The clos-est angles are enlarged more in the trained network comparedto the random one. However, enlarging the angles betweenclasses also causes the enlargement of the angles within theclasses. Notice though that these are enlarged less than theones which are outside the class. Finally, observe that theenlargement of the angles, as we have seen in our theorems,causes a larger distortion in the Euclidean distances. Therefore,we may explain the enlargement of the distances in within theclass as a means for shrinking the intra-class distances.

Similar behavior is observed for the MNIST dataset. How-ever, the gaps between the random network and the trainednetwork are smaller as the MNIST dataset contains data whichare initially well separated. As we have argued above, for suchmanifolds the random network is already a good choice.

We also compared the behavior of the validation data, ofthe ImageNet dataset, in the network provided by [36] and inthe same network but with random weights. The results arepresented in Figs. 10, 11, 12 and 13. Behavior similar to theone we observed in the case of CIFAR-10, is also manifestedby the ImageNet network.

-2 -1 0 1 2||x− y||2 − ||x− y||2

RandomNet1 25% ErrorNet2 21% ErrorNet3 18% Error

(a) Inter-class Euclidean distancedifference

-2 -1 0 1 2||x− y||2 − ||x− y||2

(b) Intra-class Euclidean distancedifference

−1.5 −1 −0.5 0 0.5 1 1.56 (x, y) − 6 (x , y)

(c) Inter-class angular distance dif-ference

−1.5 −1 −0.5 0 0.5 1 1.56 (x, y) − 6 (x , y)

(d) Intra-class angular distance dif-ference

Fig. 9. Differences of inter- (left) and intra- (right) class Euclidean (top)and angular (bottom) distances betweenrandomly selected pointsfor CIFAR-10 (a counterpart of Fig. 8 with distance ratios replaced with differences).We calculate the Euclidean distances between randomly selected pairs of datapoints from different classes (left) and from the same class(right), both atthe input of the DNN and at the output of the last convolutional layer. Thenwe compute the difference between the two, i.e., for all pairs of points(x,y)in the input and their corresponding points(x, y) at the output we calculate‖x− y‖2 − ‖x− y‖2 and∠(x, y)− ∠(x,y).

VII. D ISCUSSION ANDCONCLUSION

We have shown that DNN with random Gaussian weightsperform a stable embedding of the data, drawing a connec-tion between the dimension of the features produced by thenetwork that still keep the metric information of the originalmanifold, and the complexity of the data. The metric preser-vation property of the network provides a formal relationshipbetween the complexity of the input data and the size ofthe required training set. Interestingly, follow-up studies [52],[53] found that adding metric preservation constraints to thetraining of networks also leads to a theoretical relation betweenthe complexity of the data and the number of training samples.Moreover, this constraint is shown to improve in practice thegeneralization error, i.e., improves the classification resultswhen only a small number of training examples is available.

While preserving the structure of the initial metric isimportant, it is vital to have the ability to distort some ofthe distances in order to deform the data in a way that theEuclidean distances represent more faithfully the similaritywe would like to have between points from the same class.We proved that such an ability is inherent to the DNNarchitecture: the Euclidean distances of the input data aredistorted throughout the networks based on the angles betweenthe data points. Our results lead to the conclusion that DNNare universal classifiers for data based on the angles of theprincipal axis between the classes in the data. As these are

10

0 0.5 1 1.5 2 2.5 3||x− z||2/||x− y||2

RandomTrained

(a) Inter-class Euclidean distanceratio

0 0.5 1 1.5 2 2.5 3||x− z||2/||x− y||2

(b) Intra-class Euclidean distanceratio

6 (x, z)/ 6 (x,y)0 1 2 3 4 5

(c) Inter-class angular distance ra-tio

6 (x, z)/ 6 (x,y)0 1 2 3 4 5

(d) Intra-class angular distance ra-tio

Fig. 10. Ratios ofclosest inter- (left) and farthest intra- (right) classEuclidean (top) and angular (bottom) distances for ImageNet. For each datapoint we calculate its Euclidean distance to the farthest point from its classand to the closest point not in its class, both at the input of the DNN and atthe output of the last convolutional layer. Then we compute the ratio betweenthe two, i.e., ifx is the point at input,y is its farthest point in class,x is thepoint at the output, andz is its farthest point from the same class (it shouldnot necessarily be the output ofy), then we calculate

‖x−z‖2‖x−y‖2

and ∠(x,z)∠(x,y)

.We do the same for the distances between different classes, comparing theshortest Euclidean and angular distances.

not the angles we would like to work with in reality, thetraining of the DNN reveals the actual angles in the data. Infact, for some applications it is possible to use networks withrandom weights at the first layers for separating the pointswith distinguishable angles, followed by trained weights atthe deeper layers for separating the remaining points. Thisispracticed in the extreme learning machines (ELM) techniques[54] and our results provide a possible theoretical explanationfor the success of this hybrid strategy.

Our work implies that it is possible to view DNN as astagewise metric learning process, suggesting that it might bepossible to replace the currently used layers with other metriclearning algorithms, opening a new venue for semi-supervisedDNN. This also stands in line with the recent literature onconvolutional kernel methods (see [55], [56]).

In addition, we observed that a potential main goal of thetraining of the network is to treat the class boundary points,while keeping the other distances approximately the same.This may lead to a new active learning strategy for deeplearning [57].

Acknowledgments-Work partially supported by NSF, ONR,NGA, NSSEFF, and ARO. A.B. is supported by ERC StG335491 (RAPID). The authors thank the reviewers of themanuscript for their suggestions which greatly improved thepaper.

-2 -1 0 1 2||x− z||2 − ||x− y||2

RandomTrained

(a) Inter-class Euclidean distancedifference

-2 -1 0 1 2||x− z||2 − ||x− y||2

(b) Intra-class Euclidean distancedifference

−1.5 −1 −0.5 0 0.5 1 1.56 (x, z) − 6 (x , y)

(c) Inter-class angular distance dif-ference

−1.5 −1 −0.5 0 0.5 1 1.56 (x, z) − 6 (x , y)

(d) Intra-class angular distance dif-ference

Fig. 11. Differences ofclosestinter- (left) andfarthest intra- (right) classEuclidean (top) and angular (bottom) distances for ImageNet (a counterpart ofFig. 10 with distance ratios replaced with differences). For each data point wecalculate its Euclidean distance to the farthest point fromits class and to theclosest point not in its class, both at the input of the DNN andat the outputof the last convolutional layer. Then we compute the difference between thetwo, i.e., if x is the point at input,y is its farthest point in class,x is thepoint at the output, andz is its farthest point from the same class (it shouldnot necessarily be the output ofy), then we calculate‖x− z‖2 −‖x− y‖2and∠(x, z) − ∠(x,y). We do the same for the distances between differentclasses, comparing the shortest Euclidean and angular distances.

APPENDIX APROOF OFTHEOREM 3

Before we turn to prove Theorem 3, we present two propo-sitions that will aid us in its proof. The first is the Gaussianconcentration bound that appears in [48, Equation 1.6].

Proposition 7: Let g be an i.i.d. Gaussian random vectorwith zero mean and unit variance, andη be a Lipschitz-continuous function with a Lipschitz constantcη. Then foreveryα > 0, with probability exceeding1−2 exp(−α2/2cη),

|η(g)− E[η(g)]| < α. (12)

Proposition 8: Let m ∈ Rn be a random vector with zero-

mean i.i.d. Gaussian distributed entries with variance1m , and

K ⊂ Bn1 be a set with a Gaussian mean widthw(K). Then,

with probability exceeding1− 2 exp(−ω(K)2/4),

supx,y∈K

(

ρ(mTx)− ρ(mTy))2< 4

ω(K)2

m. (13)

Proof: First, notice that from the properties of the ReLUρ, itholds that∣

∣ρ(mTx)− ρ(mTy)∣

∣ ≤∣

∣mTx−mTy∣

∣ =∣

∣mT (x− y)∣

∣ . (14)

Let g =√m ·m be a scaled version ofm such that each entry

in g has a unit variance (asm has a variance1m ). From theGaussian mean width charachteristics (see Proposition 2.1in

11

||x− y||2/||x− y||2

0 0.5 1 1.5 2 2.5 3

RandomTrained

(a) Inter-class Euclidean distanceratio

||x− y||2/||x− y||2

0 0.5 1 1.5 2 2.5 3

(b) Intra-class Euclidean distanceratio

6 (x, y)/ 6 (x,y)0 1 2 3 4 5

(c) Inter-class angular distance ra-tio

6 (x, y)/ 6 (x,y)0 1 2 3 4 5

(d) Intra-class angular distance ra-tio

Fig. 12. Ratios of inter- (left) and intra- (right) class Euclidean (top) andangular (bottom) distances betweenrandomly selected pointsfor ImageNet.We calculate the Euclidean distances between randomly selected pairs of datapoints from different classes (left) and from the same class(right), both atthe input of the DNN and at the output of the last convolutional layer. Thenwe compute the ratio between the two, i.e., for all pairs of points (x,y) inthe input and their corresponding points(x, y) at the output we calculate‖x−y‖2‖x−y‖2

and ∠(x,y)∠(x,y)

.

[29]), we haveE supx,y∈K

∣

∣gT (x− y)∣

∣ = w(K). Therefore,combining the Gaussian concentration bound in Proposition7together with the fact thatsup

x,y∈K

∣

∣gT (x− y)∣

∣ is Lipschitz-continuous with a constantcη = 2 (sinceK ⊂ B

n1 ), we have

∣

∣

∣

∣

supx,y∈K

gT (x− y)− ω(K)

∣

∣

∣

∣

< α, (15)

with probability exceeding(1− 2 exp(−α2/4)). Clearly, (15)implies

supx,y∈K

gT (x− y) ≤ 2ω(K), (16)

where we setα = ω(K). Combining (16) and (14) with thefact that

supx,y∈K

(

ρ(gTx) − ρ(gTy))2

= ( supx,y∈K

(

ρ(gTx)− ρ(gTy))

)2

leads to

supx,y∈K

(

ρ(gTx) − ρ(gTy))2< 4ω(K)2. (17)

Dividing both sides bym completes the proof. �

Proof of Theorem 3:Our proof of Theorem 3 consists ofthree keys steps. In the first one, we show that the bound in(4) holds with high probability for any two pointsx,y ∈ K.In the second, we pick anǫ-cover forK and show that thesame holds for each pair in the cover. The last generalizes thebound for any point inK.

||x− y||2 − ||x− y||2

-2 -1 0 1 2

RandomTrained

(a) Inter-class Euclidean distancedifference

||x− y||2 − ||x− y||2

-2 -1 0 1 2

(b) Intra-class Euclidean distancedifference

6 (x, y)− 6 (x,y)-1.5 -1 -0.5 0 0.5 1 1.5

(c) Inter-class angular distance dif-ference

6 (x, y)− 6 (x,y)-1.5 -1 -0.5 0 0.5 1 1.5

(d) Intra-class angular distance dif-ference

Fig. 13. Differences of inter- (left) and intra- (right) class Euclidean (top) andangular (bottom) distances betweenrandomly selected pointsfor ImageNet (acounterpart of Fig. 12 with distance ratios replaced with differences). Wecalculate the Euclidean distances between randomly selected pairs of datapoints from different classes (left) and from the same class(right), both atthe input of the DNN and at the output of the last convolutional layer. Thenwe compute the difference between the two, i.e., for all pairs of points(x,y)in the input and their corresponding points(x, y) at the output we calculate‖x− y‖2 − ‖x− y‖2 and∠(x, y)− ∠(x,y).

Bound for a pair x,y ∈ K: Denoting bymi the i-thcolumn ofM, we rewrite

‖ρ(Mx)− ρ(My)‖22 =

m∑

i

(

ρ(mTi x)− ρ(mT

i y))2. (18)

Notice that since all themi have the same distribution, therandom variables

(

ρ(mTi x) − ρ(mT

i y))2

are also equally-distributed. Therefore, our strategy would be to calculatetheexpectation of these random variables and then to show, usingBernstein’s inequality, that the mean of these random variablesdoes not deviate much from their expectation.

We start by calculating their expectation

E

[

(

ρ(mTi x)− ρ(mT

i y))2]

(19)

= E(

ρ(mTi x)

)2+ E

(

ρ(mTi y)

)2 − 2Eρ(mTi x)ρ(m

Ti y).

For calculating the first term at the right hand side (rhs) notethatmT

i x is a random Gaussian vector with variance‖x‖22 /m.Therefore, from the symmetry of the Gaussian distribution wehave that

E(

ρ(mTi x)

)2=

1

2E(

mTi x)2

=1

2m‖x‖22 . (20)

In the same way,E(

ρ(mTi y)

)2= 1

2m ‖y‖22. For calculatingthe third term at the rhs of (19), notice thatρ(mT

i x)ρ(mTi y)

is non-zero if both the inner product betweenmi andx and

12

the inner product betweenmi and y are positive. There-fore, the smaller the angle betweenx and y, the higherthe probability that both of them will have a positive innerproduct with mi. Using the fact that a Gaussian vector isuniformly distributed on the sphere, we can calculate theexpectation ofρ(mT

i x)ρ(mTi y) by the following integral,

which is dependent on the angle betweenx andy:

E[ρ(mTi x)ρ(m

Ti y)] = (21)

‖x‖2 ‖y‖2mπ

∫ π−∠(x,y)

0

sin(θ) sin(θ + ∠(x,y))dθ

=‖x‖ ‖y‖mπ

(sin(∠(x,y)) − cos(∠(x,y))∠(x,y) − π) .

Having the expectation of all the terms in (19) calculated, wedefine the following random variable, which is the differencebetween

(

ρ(mTi x)− ρ(mT

i y))2

and its expectation,

zi ,(

ρ(mTi x) − ρ(mT

i y))2 − 1

m

(

1

2‖x− y‖22 (22)

+‖x‖2 ‖y‖2

π

(

sin(∠(x,y)) − cos(∠(x,y))∠(x,y))

.

Clearly, the random variablezi is zero-mean. To finish the firststep of the proof, it remains to show that the sum

∑mi=1 zi does

not deviate much from zero (its mean). First, note that

P

(∣

∣

∣

∣

∣

m∑

i=1

zi

∣

∣

∣

∣

∣

> t

)

≤ P

(

m∑

i=1

|zi| > t

)

, (23)

and therefore it is enough to bound the term on the rhs of(23). By Bernstein’s inequality, we have

P

(

m∑

i=1

|zi| > t

)

≤ exp

(

− t2/2∑m

i=1 Ez2i +Mt/3

)

, (24)

where M is an upper bound on|zi|. To calculateEz2i , one needs to calculate the fourth momentsE(ρ(mT

i x))4 and E(ρ(mT

i y))4, which is easy to compute

by using the symmetry of Gaussian vectors, and thecorrelations Eρ(mT

i x)3ρ(mT

i y), Eρ(mTi x)ρ(m

Ti y)

3 andEρ(mT

i x)2ρ(mT

i y)2. For calculating the later, we use as

before the fact that a Gaussian vector is uniformly distributedon the sphere and calculate an integral on an interval whichis dependent on the angle betweenx andy. For example,

E[

ρ(mTi x)

3ρ(mTi y)

]

=4 ‖x‖32 ‖y‖2

m2π· (25)

∫ π−∠(x,y)

0

sin3(θ) sin(θ + ∠(x,y))dθ,

whereθ is the angle betweenmi andx. We have a similarformula for the other terms. By simple arithmetics and usingthe fact thatK ∈ B

n1 , we have thatEz2i ≤ 2.1/m2.

The type of formula in (25), which is similar to the one in(21), provides an insight into the role of training. As randomlayers ‘integrate uniformly’ on the interval[0, π − ∠(x,y)],learning picks the angleθ that maximizes/minimizes the innerproduct based on whetherx andy belong to the same classor to distinct classes.

Using Proposition 8, the fact thatK ∈ Bn1 and the behavior

of ψ(x,y) (see Fig. 2) together with the triangle inequal-ity imply that M < 4ω(K)2+3

m with probability exceeding(1 − 2 exp(−ω(K)2/4)). Plugging this bound with the onewe computed forEz2i into (24) leads to

P

(

m∑

i=1

|zi| >δ

2

)

≤ exp

(

− mδ2/8

2.12 + 4δω(K)2/3 + δ

)

(26)

+2 exp(−ω(K)2/4),

where we included the probability of Proposition 8 in theabove bound. Since by the assumption of the theoremm =O(δ−4ω(K)4), we can write

P

(

m∑

i=1

|zi| >δ

2

)

≤ C exp

(

− mδ2

4w(K)2

)

. (27)

Bound for all x,y ∈ Nǫ(K): Let Nǫ(K) be anǫ coverfor K. By using a union bound we have that for every pair inNǫ(K),

P

(

m∑

i=1

|zi| >δ

2

)

≤ C |Nǫ(K)|2 exp(

− mδ2

4w(K)2

)

. (28)

By Sudakov’s inequality we havelog |Nǫ(K)| ≤ cǫ−2w(K)2.Plugging this inequality into (28) leads to

P

(

m∑

i=1

|zi| >δ

2

)

≤ C exp

(

− mδ2

4w(K)2+ cǫ−2w(K)2

)

. (29)

Settingǫ ≥ 150δ, we have by the assumptionm ≥ Cδ−4ω(K)2

that the term in the exponent at the rhs of (29) is negative andtherefore the probability decays exponentially asm increases.

Bound for all x,y ∈ K: Let us rewritex,y ∈ K asx0+xǫ

andy0+yǫ, with x0,y0 ∈ Nǫ(K) andxǫ,yǫ ∈ (K−K)∩Bnǫ ,

whereK − K = {x− y : x,y ∈ K}. We get to the desiredresult by settingǫ < 1

40δ and using the triangle inequalitycombined with Proposition 8 to controlρ(Mxǫ) andρ(Myǫ),the fact thatw(K −K) ≤ 2w(K) and the Taylor expansionsof thecos(·), sin(·) andcos−1(·) functions to control the termsrelated toψ (see (5)). �

APPENDIX BPROOF OFTHEOREM 4

Proof: Instead of proving Theorem 4 directly, we deduce itfrom Theorem 3. First we notice that (4) is equivalent to∣

∣

∣

∣

∣

1

2‖ρ(Mx)‖22 +

1

2‖ρ(My)‖22 −

1

4‖x‖22 −

1

4‖y‖22 (30)

−(

ρ(Mx)T ρ(My) − ‖x‖2 ‖y‖22

cos(∠(x,y)) −

‖x‖2 ‖y‖22π

(

sin(∠(x,y)) − cos(∠(x,y))∠(x,y))

)

∣

∣

∣

∣

∣

≤ δ

2.

As E ‖ρ(Mx)‖22 = 12 ‖x‖

22, we also have that with high

probability (like the one in Theorem 3),∣

∣

∣

∣

‖ρ(Mx)‖22 −1

2‖x‖22

∣

∣

∣

∣

≤ δ, ∀x ∈ K. (31)

13

(The proof is very similar to the one of Theorem 3). Applyingthe reverse triangle inequality to (30) and then using (31),followed by dividing both sides by‖x‖2‖y‖2

2 , lead to∣

∣

∣

∣

∣

(2ρ(Mx)T ρ(My)

‖x‖2 ‖y‖2− cos(∠(x,y)) − (32)

1

π

(

sin(∠(x,y)) − cos(∠(x,y))∠(x,y))

)

∣

∣

∣

∣

∣

≤ 3δ

‖x‖2 ‖y‖2.

Using the reverse triangle inequality with (32) leads to∣

∣

∣

∣

∣

( ρ(Mx)T ρ(My)

‖ρ(Mx)‖2 ‖ρ(My)‖2− cos(∠(x,y)) − (33)

1

π

(

sin(∠(x,y)) − cos(∠(x,y))∠(x,y))

)

∣

∣

∣

∣

∣

≤ 3δ

‖x‖2 ‖y‖2

+

∣

∣

∣

∣

2ρ(Mx)Tρ(My)

‖x‖2 ‖y‖2− ρ(Mx)T ρ(My)

‖ρ(Mx)‖2 ‖ρ(My)‖2

∣

∣

∣

∣

.

To complete the proof it remains to bound the rhs of (33). Forthe second term in it, we have

∣

∣

∣

∣

2ρ(Mx)Tρ(My)

‖x‖2 ‖y‖2− ρ(Mx)T ρ(My)

‖ρ(Mx)‖2 ‖ρ(My)‖2

∣

∣

∣

∣

= (34)

2ρ(Mx)T ρ(My)

‖x‖2 ‖y‖2

∣

∣

∣

∣

1− ‖x‖2 ‖y‖22 ‖ρ(Mx)‖2 ‖ρ(My)‖2

∣

∣

∣

∣

.

BecauseK ⊂ Bn1 \ βBn

2 , it follows from (31) that

‖ρ(Mx)‖22 ≥ 1

2β2 − δ. (35)

Dividing by(

‖ρ(Mx)‖2 + 1√2‖x‖2

)

‖ρ(Mx)‖2 both sides

of (31) and then using (35) and the fact that‖x‖2 ≥ β, provide∣

∣

∣

∣

∣

1− ‖x‖2√2 ‖ρ(Mx)‖2

∣

∣

∣

∣

∣

(36)

≤ δ(

‖ρ(Mx)‖2 + 1√2‖x‖2

)

‖ρ(Mx)‖2

≤ δ(√

12β

2 − δ + 1√2β)√

12β

2 − δ≤ δ

β2 − 2δ, ∀x ∈ K,

where the last inequality is due to simple arithmetics. Usingthe triangle inequality and then the fact that the inequality in(36) holds∀x ∈ K, we have∣

∣

∣

∣

1− ‖x‖2 ‖y‖22 ‖ρ(Mx)‖2 ‖ρ(My)‖2

∣

∣

∣

∣

≤∣

∣

∣

∣

∣

1− ‖y‖2√2 ‖ρ(My)‖2

∣

∣

∣

∣

∣

(37)

+

∣

∣

∣

∣

∣

1− ‖x‖2√2 ‖ρ(Mx)‖2

∣

∣

∣

∣

∣

·∣

∣

∣

∣

∣

‖y‖2√2 ‖ρ(My)‖2

∣

∣

∣

∣

∣

≤ δ

β2 − 2δ+

δ

β2 − 2δ(1 +

δ

β2 − 2δ) ≤ 3δ

β2 − 2δ.

Combining (32) with the facts thatcos(∠(x,y)) +1π

(

sin(∠(x,y)) − cos(∠(x,y))∠(x,y) is bounded by1 (seeFig. 2) andK ⊂ B

n1 \ βBn

2 lead to

2ρ(Mx)T ρ(My)

‖x‖2 ‖y‖2≤(

1 +3δ

β2

)

. (38)

Plugging (38) and (37) into (34) and then the outcome into(33) lead to the bound

(

1 + 3δβ2

)

3δβ2−2δ + 3δ

β2 ≤ 15δβ2−2δ �

REFERENCES

[1] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenetclassificationwith deep convolutional neural networks,” inAdvances in Neural Infor-mation Processing Systems (NIPS), 2012.

[2] Y. Bengio, A. Courville, and P. Vincent, “Representation learning: Areview and new perspectives,”IEEE Trans. Pattern Anal. Mach. Intell.,vol. 35, no. 8, pp. 1798–1828, Aug. 2013.

[3] J. Schmidhuber, “Deep learning in neural networks: An overview,”Neural Networks, vol. 61, pp. 85–117, 2015.

[4] N. Pinto, D. Doukhan, J. J. DiCarlo, and D. D. Cox, “A high-throughputscreening approach to discovering good forms of biologically inspiredvisual representation,”PLoS Comput Biol, vol. 5, no. 11, p. e1000579,11 2009.

[5] A. Saxe, P. W. Koh, Z. Chen, M. Bhand, B. Suresh, and A. Y. Ng,“On random weights and unsupervised feature learning,” inInt. Conf.on Machine Learning (ICML), 2011, pp. 1089–1096.

[6] D. Cox and N. Pinto, “Beyond simple features: A large-scale featuresearch approach to unconstrained face recognition,” inIEEE Interna-tional Conference on Automatic Face Gesture Recognition and Work-shops (FG), March 2011, pp. 8–15.

[7] E. J. Candes and T. Tao, “Near-optimal signal recovery from randomprojections: Universal encoding strategies?”IEEE Trans. Inf. Theory,vol. 52, no. 12, pp. 5406 –5425, Dec. 2006.

[8] M. Elad, “Optimized projections for compressed sensing,” IEEE Trans.Signal Process., vol. 55, no. 12, pp. 5695–5702, Dec 2007.

[9] J. M. Duarte-Carvajalino and G. Sapiro, “Learning to sense sparsesignals: Simultaneous sensing matrix and sparsifying dictionary opti-mization,” IEEE Trans. Imag. Proc., vol. 18, no. 7, pp. 1395–1408, July2009.

[10] C. Hegde, A. C. Sankaranarayanan, W. Yin, and R. G. Baraniuk, “Nu-max: A convex approach for learning near-isometric linear embeddings,”IEEE Trans. Signal Proc, vol. 63, no. 22, pp. 6109–6121, Nov 2015.

[11] A. Andoni and P. Indyk, “Near-optimal hashing algorithms for nearneighbor problem in high dimensions,” inProceedings of the Symposiumon the Foundations of Computer Science, 2006, pp. 459–468.

[12] E. J. Candes, T. Strohmer, and V. Voroninski, “Phaselift: Exact andstable signal recovery from magnitude measurements via convex pro-gramming,”Communications on Pure and Applied Mathematics, vol. 66,no. 8, pp. 1241–1274, 2013.

[13] S. Arora, A. Bhaskara, R. Ge, and T. Ma, “Provable boundsfor learningsome deep representations,” inInt. Conf. on Machine Learning (ICML),2014, pp. 584–592.

[14] A. Saxe, J. McClelland, and S. Ganguli, “Exact solutions to the nonlineardynamics of learning in deep linear neural network,” inInternationalConference on Learning Representations (ICLR), 2014.

[15] A. Choromanska, M. B. Henaff, M. Mathieu, G. B. Arous, and Y. LeCun,“The loss surfaces of multilayer networks,” inInternational Conferenceon Artificial Intelligence and Statistics (AISTATS), 2015.

[16] Y. Dauphin, R. Pascanu, C. Gulcehre, K. Cho, S. Ganguli,and Y. Bengio,“Identifying and attacking the saddle point problem in high-dimensionalnon-convex optimization,” inAdvances in Neural Information Process-ing Systems (NIPS), 2014.

[17] K. Hornik, M. Stinchcombe, and H. White, “Multilayer feedforwardnetworks are universal approximators,”Neural Netw., vol. 2, no. 5, pp.359–366, Jul. 1989.

[18] G. Cybenko, “Approximation by superpositions of a sigmoidal function,”Math. Control Signals Systems, vol. 2, pp. 303–314, 1989.

[19] J. Bruna and S. Mallat, “Invariant scattering convolution networks,”IEEE Trans. Pattern Analysis and Machine Intelligence (TPAMI),vol. 35, no. 8, pp. 1872–1886, Aug 2013.

[20] F. Anselmi, J. Z. Leibo, L. Rosasco, J. Mutch, A. Tacchetti, and T. Pog-gio, “Unsupervised learning of invariant representations,” TheoreticalComputer Science, 2015.

[21] G. F. Montfar and J. Morton, “When does a mixture of products contain aproduct of mixtures?”SIAM Journal on Discrete Mathematics (SIDMA),vol. 29, no. 1, pp. 321–347, 2015.

[22] G. F. Montufar, R. Pascanu, K. Cho, and Y. Bengio, “On the numberof linear regions of deep neural networks,” inAdvances in NeuralInformation Processing Systems (NIPS), 2014.

[23] J. Bruna, Y. LeCun, and A. Szlam, “Learning stable groupinvariantrepresentations with convolutional networks,” inICLR Workshop, Jan.2013.

14

[24] J. Bruna, A. Szlam, and Y. LeCun, “Signal recovery fromlp poolingrepresentations,” inInt. Conf. on Machine Learning (ICML), 2014.

[25] M. Rudelson and R. Vershynin, “Non-asymptotic theory of randommatrices: extreme singular values,” inInternational Congress of Math-ematicans, 2010.

[26] A. Mahendran and A. Vedaldi, “Understanding deep imagerepresenta-tions by inverting them,” inIEEE Conference on Computer Vision andPattern Recognition (CVPR), 2015.

[27] V. Chandrasekaran, B. Recht, P. A. Parrilo, and A. S. Willsky, “The con-vex geometry of linear inverse problems,”Foundations of ComputationalMathematics, vol. 12, no. 6, pp. 805–849, 2012.

[28] D. Amelunxen, M. Lotz, M. B. McCoy, and J. A. Tropp, “Living onthe edge: phase transitions in convex programs with random data,”Information and Inference, 2014.

[29] Y. Plan and R. Vershynin, “Robust 1-bit compressed sensing and sparselogistic regression: A convex programming approach,”IEEE Trans. Inf.Theory, vol. 59, no. 1, pp. 482–494, Jan. 2013.

[30] W. B. Johnson and J. Lindenstrauss, “Extensions of lipschitz mappingsinto a hilbert space,” inConf. in Modern Analysis and Probability, 1984,p. 189206.

[31] S. Mendelson, A. Pajor, and N. Tomczak-Jaegermann, “Uniform uncer-tainty principle for Bernoulli and sub-Gaussian ensembles,” ConstructiveApproximation, vol. 28, pp. 277–289, 2008.

[32] O. Yamaguchi, K. Fukui, and K. Maeda, “Face recognitionusingtemporal image sequence,” inIEEE Int. Conf. Automatic Face andGesture Recognition, 1998, pp. 318–323.

[33] L. Wolf and A. Shashua, “Learning over sets using kernelprincipalangles,” Journal of Machine Learning Research, vol. 4, pp. 913–931,Oct. 2003.

[34] E. Elhamifar and R. Vidal, “Sparse subspace clustering: Algorithm,theory, and applications,”IEEE Trans. Pattern Anal. Mach. Intell.,vol. 35, no. 11, pp. 2765–2781, 2013.

[35] Q. Qiu and G. Sapiro, “Learning transformations for clustering andclassification,” Journal of Machine Learning Research, vol. 16, p.187225, Feb. 2015.

[36] K. Simonyan and A. Zisserman, “Very deep convolutionalnetworks forlarge-scale image recognition,” inInternational Conference on LearningRepresentations (ICLR), 2015.

[37] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma,Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei, “ImageNet Large Scale Visual Recognition Challenge,”2014.

[38] V. Nair and G. E. Hinton, “Rectified linear units improverestrictedboltzmann machines,” inInt. Conf. on Machine Learning (ICML), 2010,pp. 807–814.

[39] J. Haupt, W. Bajwa, G. Raz, and R. Nowak, “Toeplitz compressedsensing matrices with applications to sparse channel estimation,” IEEETrans. Inf. Theory, vol. 56, no. 11, pp. 5862–5875, Nov. 2010.

[40] V. Saligrama, “Aperiodic sequences with uniformly decaying correla-tions with applications to compressed sensing and system identification,”IEEE Trans. Inf. Theory, vol. 58, no. 9, pp. 6023–6036, Sept. 2012.

[41] H. Rauhut, J. Romberg, and J. A. Tropp, “Restricted isometries for par-tial random circulant matrices,”Appl. Comput. Harmon. Anal., vol. 32,no. 2, pp. 242–254, Mar. 2012.

[42] A. Ai, A. Lapanowski, Y. Plan, and R. Vershynin, “One-bit compressedsensing with non-Gaussian measurements,”Linear Algebra and itsApplications, vol. 441, pp. 222 – 239, 2014, special Issue on SparseApproximate Solution of Linear Systems.

[43] Y. Plan and R. Vershynin, “Dimension reduction by random hyperplanetessellations,”Discrete and Computational Geometry, vol. 51, no. 2, pp.438–461, 2014.

[44] M. Gromov, Metric Structures for Riemannian and Non-RiemannianSpaces. Birkhauser, Boston, US, 1999.

[45] Y. Plan, R. Vershynin, and E. Yudovina, “High-dimensionalestimation with geometric constraints,”Arxiv, 2014. [Online]. Available:http://arxiv.org/abs/1404.3749

[46] K. Chatfield, K. Simonyan, A. Vedaldi, and A. Zisserman,“Return ofthe devil in the details: Delving deep into convolutional nets,” in BritishMachine Vision Conference, 2014.

[47] F. Perronnin, J. Sanchez, and T. Mensink, “Improving the fisher kernelfor large-scale image classification,” inECCV, 2010, vol. 6314, pp. 143–156.

[48] M. Ledoux and M. Talagrand,Probability in Banach Spaces. Springer-Verlag, 1991.

[49] B. Klartag and S. Mendelson, “Empirical processes and random pro-jections,” Journal of Functional Analysis, vol. 225, no. 1, pp. 229–245,Aug. 2005.

[50] R. Gribonval, R. Jenatton, and F. Bach, “Sparse and spurious:dictionary learning with noise and outliers,” Sep. 2014. [Online].Available: https://hal.archives-ouvertes.fr/hal-01025503

[51] A. Vedaldi and K. Lenc, “Matconvnet – convolutional neural networksfor matlab,” in Proceeding of the ACM Int. Conf. on Multimedia.

[52] J. Huang, Q. Qiu, G. Sapiro, and R. Calderbank, “Discriminative robusttransformation learning,” inAdvances in Neural Information ProcessingSystems (NIPS), 2015.

[53] ——, “Geometry-aware deep transform,” inIEEE International Confer-ence on Computer Vision (ICCV), 2015.

[54] G.-B. Huang, D. H. Wang, and Y. Lan, “Extreme learning machines:a survey,”International Journal of Machine Learning and Cybernetics,vol. 2, no. 2, pp. 107–122, 2011.

[55] J. Mairal, P. Koniusz, Z. Harchaoui, and C. Schmid, “Convolutionalkernel networks,” inAdvances in Neural Information Processing Systems(NIPS), 2014.

[56] Z. Lu, A. May, K. Liu, A. B. Garakani, D. Guo, A. Bellet, L.Fan,M. Collins, B. Kingsbury, M. Picheny, and F. Sha, “How to scaleup kernel methods to be as good as deep neural nets,”Arxiv, 2014.[Online]. Available: http://arxiv.org/abs/1411.4000

[57] E. Elhamifar, G. Sapiro, A. Yang, and S. Sasrty, “A convex optimizationframework for active learning,” inIEEE International Conference onComputer Vision (ICCV), Dec 2013, pp. 209–216.