Apprentissage,réseauxdeneuronesetmodèlesgraphiques (RCP209...

Apprentissage, réseaux de neurones et modèles graphiques(RCP209)

Neural Networks and Deep Learning

Nicolas [email protected]

http://cedric.cnam.fr/vertigo/Cours/ml2/

Département InformatiqueConservatoire Nationnal des Arts et Métiers (Cnam)

http://cedric.cnam.fr/vertigo/Cours/ml2/

DL Strengths DL Weaknesses History

Outline

1 Deep Learning Strengths

2 Deep Learning Weaknesses

3 Deep Learning History

[email protected] RCP209 / Deep Learning 2/ 54

mailto:[email protected]


MLP: Universal Function Approximators

• Neural network with one single hidden layer ⇒ universal approximatorCan represent any function on compact subsets of Rn [Cyb89]

Approximate any continuous function to any desired precision

Ex pour regression: any function can be interpolated





• Neural network with one single hidden layer ⇒ universal approximatorCan represent any function on compact subsets of Rn [Cyb89]Ex pour classification: any decision boundaries can be expressed

⇒ very rich modeling capacities





• 2 layers, i.e. one hidden layer, is enough

• Challenge is NOT fitting training dataSimple models already have very large (infinite) modeling power

• Challenge: optimization, overfitting





• 2 layers, i.e. one hidden layer, is enough ... theoretically:BUT: exponential number of hidden units [Bar93]




Deep Models: Universal Function Approximators

• Deeper Models: less units required to represent the desired functionFunctions representable compactly with k layers may require exponentially size with k − 1layers [Has89, Ben09]

• Same modeling power, fewer parameters⇒ better generalization!




Deep Models

Depth improves generalization: multi-digit recognition, from [GBC16]




Local vs Distributed Representations

• Local Representations: one neuron ↔ one concept• Deep Learning ⇒ Distributed Representations:

Each concept ↔ many neurons, each neuron ↔ many concepts

⇒ Exponentially more efficient than local representations

From [BD11]




Deep Learning & Distributed Representations

• DL architectures: distributed representations shared across classes

Credit: M.A Ranzato




Deep ConvNets

• Deep models: hierarchy of sequential layers

• Layers: fully connected,

convolution + non linearity³¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹·¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹µ

convolution layer , pooling• Supervised training for classification⇒ Representation Learning




Error Back-Propagation ConvNets

• Convolution: example for 1d scalar conv with mask w = [w1 w2 w3]T

• Shared weights: simple chain rule application

⇒ Sum gradients for every region yk !

∂L

∂w=

N

∑

k=1

∂L

∂yk

∂yk∂w

−∂yk∂w

= [xk−1 xk xk+1]T




Error Back-Propagation with ConvNets

• Pooling: example for pooling area of size 2L + 1:• yk = f (xk−L, ..., xk+L)

∂L∂xh=

∂L∂yk

∂yk∂xh= δk ∂yk

∂xh




Error Back-Propagation for Average pooling

• Average pooling: yk = 1N

k+L∑

h=k−Lxh

∂L∂xh= δk ∂yk

∂xh=

1N δ

k

⇒ Gradient propagated through each input node ( 1N)




Error Back-Propagation for Max pooling

• Max pooling: yk = maxh′

xh

∂L∂xh

=

⎧⎪⎪⎪⎨⎪⎪⎪⎩

δk if xh = maxh′∈{k−L,...,k+L}

xh′

0 otherwise

⇒ Gradient propagated through arg max input node




ConvNets & Prior Distribution

• Prior: imposing distribution on fully connected parameters• Weak prior: high entropy (uncertainty), strong prior: low entropy• Infinitely strong prior: zero probability on some parameters




ConvNet as Infinitely Strong Prior

• ConvNet ∼ Infinitely Strong Prior on Fully Connected net weights• Convolution: local interactions, shared weights ⇒ zero probability elsewhere




ConvNet for Learning Representation

• ConvNet ∼ Infinitely Strong Prior on Fully Connected net weights• N.B.: weights adjusted for classification with back-prop• Convolution ⇒ support learning translation-equivariant features• Pooling ⇒ support features invariant (stable) wrt local translations




ConvNet for Learning Representation

• ConvNet Infinitely Strong Prior: adapted to data with local interactions, e.g. image,speech, etc

• Very rich modeling capacities: local interactions ⇒ global with depth• Significantly reduce # parameters ⇒ reducing over-fitting

From [GBC16]




ConvNet for Learning Compositions

• Conv/Pool hierarchies: feature compositionDepth: gradual complexity, larger spatial extendIntuitive processing for hierarchical information modelingBiological foundations: simple cells, complex cells




ConvNet for Learning Compositions

Credit: M.A Ranzato

• Hierarchical CompositionsLow-level: edges, colorMid-level: corner, partsHigher levels: objects, scene concepts

• Distributed Representations: sharingLower-level: shared by many classesHigher-levels: more class specific




Representation Learning & Deep Learning

• X-class classification, K classes: last hidden layer size L→ K

• Classification layer: linear projection + soft-max activationIn RL space, linear separation between classesDeep Learning (backprop) supports learning representations thatgradually project data to RL spaces where linear separation possible




Deep Learning & Manifold Untangling

• DL: gradually projecting data to RL spaces where linear separation possible• This is the definition of manifold untangling!

• ConvNets: manifold untangling easier!Conv/Pool prior : stability




Manifold Untangling Visualization

• We want to visualize each layer activation for each class• high-dimensional visualization?⇒ Projection to lower (e.g. 2d) dimensions




t-distributed Stochastic Neighbor Embedding (t-SNE)

• t-SNE [vdMH08]:non linear projection

• Intuitively: close distances in initial space⇒ close distances in projected (2d) space

Distance preservationNeighborhood preservation i.e. small distancepreservation




t-SNE [vdMH08]

• Similarity between points (xi , xj) in initial space, e.g. Rd :

pij =e−

∣∣xi−xj ∣∣22σ2

∑

k≠le−

∣∣xk−xl ∣∣22σ2

P = {pij}(i,j)∈N×N

• Similarity between points (yi , yj) in projected space, e.g. R2 :

qij =(1 + ∣∣yi − yj ∣∣2)−1

∑

k≠l(1 + ∣∣yk − yl ∣∣2)−1

Q = {qij}(i,j)∈N×N

• Loss function: Kullback-Leiber divergence KL(P ∣∣Q)

C =∑

i

KL(P ∣∣Q) =∑

i

∑

j

pij logpijqij




t-SNE Visualization: MNIST example

• MNIST dataset: 28 × 28 grayscale images of digits• 10 classes ⇔ digit number ∈ {0;9}• Input space dimension: 282

= 784• Projection in 2d (3d) space for visualization

• t-SNE for computing projection: gradient descent∂C

∂yi= 4∑

i

(pij − qij)(yi − yj)(1 + ∣∣yi − yj ∣∣2)−1

• Optimization (projection) for a given closed dataset⇒ transductive learning





• Application of t-SNE in the test set of MNIST (10000) images• Color ⇔ class ID





• Classes visually appear in 2d space, BUT overlap• How to measure class separability?

Neighborhood Hit [PNML08]: NH = # pts in knn of the same class# pts in knn





• How to measure class separability?Fitting ellipses to each class pointsEllipses non-overlap ⇒ linear separability




Outline







Deep Neural Networks: Weaknesses & Drawbacks

Criticisms at two main levels

1 Modeling level: Neural Networks ⇔ Black Boxes2 Training level: ad hoc, expertise, efficiency, guaranty




Deep Neural Networks: Black Boxes

• Lack of explainability: why this decision?Hidden units not directly interpretable ≠ others, e.g. decision trees, expert systems

⇒ Challenges: Human machine interaction, failure analysis





• Lack of confidence estimate (uncertainty): how (un)certain about decision?• Uncertainty estimates often vital, e.g. medicine, autonomous driving





• Lack of theory for architecture design• How many layers, neurons?• Layer type: fully connected, convolution, pooling?• Trial/test: optimize architecture on validation set⇒ Ad hoc, no theory to guide you




Deep Neural Networks: Training Issues

• Optimization: non convex objectiveNo guaranty to reach global optimumSolution dependent on initialization

Importance of (random) initialization⇒ training reproducibilityExpertise: ad hoc hyper-parameter tuning:# epochs, decay, etcCostly Tuning





• Optimization: stochastic trainingStochastic training for big data:⇒ gradient approximation⇒ ↑ difficulty for tuning





• Big Data

• Deep models need huge annotated datasets⇒ Huge models, huge computational demand⇒ Long be impossible to train such models with existing resources





• Generalization: deep models need huge annotated datasets• Smaller datasets: inferior predictive performances

Small models: not enough expressive powerLarge models overfit⇒ Performances ↓ handcrafted features




Deep Neural Network Weaknesses: Conclusion

• Deep learning Weaknesses:Black box modelsTraining challenges at many levels

• Drawbacks vs NN strength over DL History




Outline







Deep Learning: Trends and methods in the last four decades

Slide credit: https://www.slideshare.net/deview/251-implementing-deep-learning-using-cu-dnn


https://www.slideshare.net/deview/251-implementing-deep-learning-using-cu-dnn



Case Study: LeNet 5 Model

80’s: 1st Convolutionnal Neural Networks

• LeNet 5 Model [LBD+89], trained using back-prop

• Input: 32x32 pixel image. Largest character is 20x20• 2 successive blocks [Convolution + Sigmoid + Pooling (+sigmoid)]Cx: Convolutional layer, Sx: Subsampling layer

• C5: convolution layer ∼ fully connected• 2 Fully connected layers Fx





C1 Layer

• Convolutional layer with 6 5x5 filters ⇒ 6 feature maps of size 28x28 (nopadding).

• # Parameters: 52 per filer + bias ⇒ (5 ∗ 5 + 1) ∗ 6 = 156If it was fully connected: (32*32+1)*(28*28)*6 parameters 5 ∼ 106 !





S2 Layer

• Subsampling layer = pooling layer• Pooling area : 2x2 in C1• Pooling stride: 2 ⇒ 6 features maps of size 14x14• Pooling type : sum, multiplied by a trainable param + bias⇒ 2 parameters per channel

• Total # Parameters: 2 ∗ 6 = 12





C3 Layer: Convolutional

• C3: 16 filters ⇒ 16 feature maps of size 10x10 (no padding)

• 5x5 filters connected to a subset of S2 maps⇒ 0-5 connected to 3, 6-14 to 4, 15 connected to 6

• # Parameters: 1516(5 ∗ 5 ∗ 3 + 1) ∗ 6 + (5 ∗ 5 ∗ 4 + 1) ∗ 9 + (5 ∗ 5 ∗ 6 + 1) = 456 + 909 + 151





S4 Layer

• Subsampling layer = pooling layer• Pooling area : 2x2 in C3• Pooling stride: 2 ⇒ 16 features maps of size 5x5• Pooling type : sum, multiplied by a trainable param + bias⇒ 2 parameters per channel

• Total # Parameters: 2 ∗ 16 = 32





C5 Layer: Convolutionnal layer

• 120 5x5x16 filters ⇒ whole depth of S4 (≠ C3)• Each maps in S4 is 5x5 ⇒ single value for each C5 maps• C5 120 features map of size 1x1 (vector of size 120)⇒ spatial information lost, ∼ to a fully connected layer

• Total # Parameters: (5 ∗ 5 ∗ 16 + 1) ∗ 120 = 48210





F6 Layer: Fully Connected layer

• 84 fully connected units.• # Parameters: 84*(120+1)=10164

F7 Layer (output): Fully Connected layer

• 10 (# classes) fully connected units.• # Parameters: 10*(84+1)=850





• Evaluation on MNIST• Total # parameters ∼ 60000

60,000 original datasets: test error: 0.95%540,000 artificial distortions + 60,000 original: Testerror: 0.8%

• Successful deployment for postal code reading in the US




LeNet 5 Model: Manifold Untangling

Input space Latent space




LeNet 5 Model: Manifold Untangling

Latent space MLP Latent space LeNet




80’s: Deep Learning Sucees

• ConvNet trainable with backprop, able to untangle class manifold• Very good performances for digit classification, industrial transfer• 2nd winter of Deep Learning⇒ following!

Source: Amazon




References I

A. R. Barron, Universal approximation bounds for superpositions of a sigmoidal function, Information

Theory, IEEE Transactions on 39 (1993), no. 3, 930–945.

Yoshua Bengio and Olivier Delalleau, On the expressive power of deep architectures, Proceedings of the

22Nd International Conference on Algorithmic Learning Theory (Berlin, Heidelberg), ALT’11,Springer-Verlag, 2011, pp. 18–36.

Yoshua Bengio, Learning deep architectures for ai, Found. Trends Mach. Learn. 2 (2009), no. 1, 1–127.

George Cybenko, Approximation by superpositions of a sigmoidal function, Mathematics of control, signals

and systems 2 (1989), no. 4, 303–314.

Ian Goodfellow, Yoshua Bengio, and Aaron Courville, Deep learning, MIT Press, 2016,http://www.deeplearningbook.org.

Johan Hastad, Almost optimal lower bounds for small depth circuits, RANDOMNESS ANDCOMPUTATION, JAI Press, 1989, pp. 6–20.

Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and

Lawrence D Jackel, Backpropagation applied to handwritten zip code recognition, Neural computation 1(1989), no. 4, 541–551.

Fernando Vieira Paulovich, Luis Gustavo Nonato, Rosane Minghim, and Haim Levkowitz, Least square

projection: A fast high-precision multidimensional projection technique and its application to documentmapping, IEEE Trans. Vis. Comput. Graph. 14 (2008), no. 3, 564–575.

Laurens van der Maaten and Geoffrey E. Hinton, Visualizing high-dimensional data using t-sne, Journal of

Machine Learning Research 9 (2008), 2579–2605.


http://www.deeplearningbook.org


Apprentissage,réseauxdeneuronesetmodèlesgraphiques (RCP209...

Documents

Transcript of Apprentissage,réseauxdeneuronesetmodèlesgraphiques (RCP209...