Apprentissage,réseauxdeneuronesetmodèlesgraphiques (RCP209...

Apprentissage, réseaux de neurones et modèles graphiques(RCP209)

Neural Networks and Deep Learning

Nicolas ThomePrenom.Nom@cnam.fr

http://cedric.cnam.fr/vertigo/Cours/ml2/

Département InformatiqueConservatoire Nationnal des Arts et Métiers (Cnam)

DL Strengths DL Weaknesses History

Outline

1 Deep Learning Strengths

2 Deep Learning Weaknesses

3 Deep Learning History

nicolas.thome@cnam.fr RCP209 / Deep Learning 2/ 54

MLP: Universal Function Approximators

• Neural network with one single hidden layer ⇒ universal approximatorCan represent any function on compact subsets of Rn [Cyb89]

Approximate any continuous function to any desired precision

Ex pour regression: any function can be interpolated

• Neural network with one single hidden layer ⇒ universal approximatorCan represent any function on compact subsets of Rn [Cyb89]Ex pour classification: any decision boundaries can be expressed

⇒ very rich modeling capacities

• 2 layers, i.e. one hidden layer, is enough

• Challenge is NOT fitting training dataSimple models already have very large (infinite) modeling power

• Challenge: optimization, overfitting

• 2 layers, i.e. one hidden layer, is enough ... theoretically:BUT: exponential number of hidden units [Bar93]

Deep Models: Universal Function Approximators

• Deeper Models: less units required to represent the desired functionFunctions representable compactly with k layers may require exponentially size with k − 1layers [Has89, Ben09]

• Same modeling power, fewer parameters⇒ better generalization!

Deep Models

Depth improves generalization: multi-digit recognition, from [GBC16]

Local vs Distributed Representations

• Local Representations: one neuron ↔ one concept• Deep Learning ⇒ Distributed Representations:

Each concept ↔ many neurons, each neuron ↔ many concepts

⇒ Exponentially more efficient than local representations

From [BD11]

Deep Learning & Distributed Representations

• DL architectures: distributed representations shared across classes

Credit: M.A Ranzato

Deep ConvNets

• Deep models: hierarchy of sequential layers

• Layers: fully connected,

convolution + non linearity³¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹·¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹µ

convolution layer , pooling• Supervised training for classification⇒ Representation Learning

Error Back-Propagation ConvNets

• Convolution: example for 1d scalar conv with mask w = [w1 w2 w3]T

• Shared weights: simple chain rule application

⇒ Sum gradients for every region yk !

∂yk∂w

−∂yk∂w

= [xk−1 xk xk+1]T

Error Back-Propagation with ConvNets

• Pooling: example for pooling area of size 2L + 1:• yk = f (xk−L, ..., xk+L)

∂L∂xh=

∂L∂yk

∂yk∂xh= δk ∂yk

Error Back-Propagation for Average pooling

• Average pooling: yk = 1N

k+L∑

h=k−Lxh

∂L∂xh= δk ∂yk

∂xh=

⇒ Gradient propagated through each input node ( 1N)

Error Back-Propagation for Max pooling

• Max pooling: yk = maxh′

∂L∂xh

⎧⎪⎪⎪⎨⎪⎪⎪⎩

δk if xh = maxh′∈{k−L,...,k+L}

0 otherwise

⇒ Gradient propagated through arg max input node

ConvNets & Prior Distribution

• Prior: imposing distribution on fully connected parameters• Weak prior: high entropy (uncertainty), strong prior: low entropy• Infinitely strong prior: zero probability on some parameters

ConvNet as Infinitely Strong Prior

• ConvNet ∼ Infinitely Strong Prior on Fully Connected net weights• Convolution: local interactions, shared weights ⇒ zero probability elsewhere

ConvNet for Learning Representation

• ConvNet ∼ Infinitely Strong Prior on Fully Connected net weights• N.B.: weights adjusted for classification with back-prop• Convolution ⇒ support learning translation-equivariant features• Pooling ⇒ support features invariant (stable) wrt local translations

ConvNet for Learning Representation

• ConvNet Infinitely Strong Prior: adapted to data with local interactions, e.g. image,speech, etc

• Very rich modeling capacities: local interactions ⇒ global with depth• Significantly reduce # parameters ⇒ reducing over-fitting

From [GBC16]

ConvNet for Learning Compositions

• Conv/Pool hierarchies: feature compositionDepth: gradual complexity, larger spatial extendIntuitive processing for hierarchical information modelingBiological foundations: simple cells, complex cells

ConvNet for Learning Compositions

Credit: M.A Ranzato

• Hierarchical CompositionsLow-level: edges, colorMid-level: corner, partsHigher levels: objects, scene concepts

• Distributed Representations: sharingLower-level: shared by many classesHigher-levels: more class specific

Representation Learning & Deep Learning

• X-class classification, K classes: last hidden layer size L→ K

• Classification layer: linear projection + soft-max activationIn RL space, linear separation between classesDeep Learning (backprop) supports learning representations thatgradually project data to RL spaces where linear separation possible

Deep Learning & Manifold Untangling

• DL: gradually projecting data to RL spaces where linear separation possible• This is the definition of manifold untangling!

• ConvNets: manifold untangling easier!Conv/Pool prior : stability

Manifold Untangling Visualization

• We want to visualize each layer activation for each class• high-dimensional visualization?⇒ Projection to lower (e.g. 2d) dimensions

t-distributed Stochastic Neighbor Embedding (t-SNE)

• t-SNE [vdMH08]:non linear projection

• Intuitively: close distances in initial space⇒ close distances in projected (2d) space

Distance preservationNeighborhood preservation i.e. small distancepreservation

t-SNE [vdMH08]

• Similarity between points (xi , xj) in initial space, e.g. Rd :

pij =e−

∣∣xi−xj ∣∣22σ2

k≠le−

∣∣xk−xl ∣∣22σ2

P = {pij}(i,j)∈N×N

• Similarity between points (yi , yj) in projected space, e.g. R2 :

qij =(1 + ∣∣yi − yj ∣∣2)−1

k≠l(1 + ∣∣yk − yl ∣∣2)−1

Q = {qij}(i,j)∈N×N

• Loss function: Kullback-Leiber divergence KL(P ∣∣Q)

C =∑

KL(P ∣∣Q) =∑

pij logpijqij

t-SNE Visualization: MNIST example

• MNIST dataset: 28 × 28 grayscale images of digits• 10 classes ⇔ digit number ∈ {0;9}• Input space dimension: 282

= 784• Projection in 2d (3d) space for visualization

• t-SNE for computing projection: gradient descent∂C

∂yi= 4∑

(pij − qij)(yi − yj)(1 + ∣∣yi − yj ∣∣2)−1

• Optimization (projection) for a given closed dataset⇒ transductive learning

• Application of t-SNE in the test set of MNIST (10000) images• Color ⇔ class ID

• Classes visually appear in 2d space, BUT overlap• How to measure class separability?

Neighborhood Hit [PNML08]: NH = # pts in knn of the same class# pts in knn

• How to measure class separability?Fitting ellipses to each class pointsEllipses non-overlap ⇒ linear separability

Outline

Deep Neural Networks: Weaknesses & Drawbacks

Criticisms at two main levels

1 Modeling level: Neural Networks ⇔ Black Boxes2 Training level: ad hoc, expertise, efficiency, guaranty

Deep Neural Networks: Black Boxes

• Lack of explainability: why this decision?Hidden units not directly interpretable ≠ others, e.g. decision trees, expert systems

⇒ Challenges: Human machine interaction, failure analysis

• Lack of confidence estimate (uncertainty): how (un)certain about decision?• Uncertainty estimates often vital, e.g. medicine, autonomous driving

• Lack of theory for architecture design• How many layers, neurons?• Layer type: fully connected, convolution, pooling?• Trial/test: optimize architecture on validation set⇒ Ad hoc, no theory to guide you

Deep Neural Networks: Training Issues

• Optimization: non convex objectiveNo guaranty to reach global optimumSolution dependent on initialization

Importance of (random) initialization⇒ training reproducibilityExpertise: ad hoc hyper-parameter tuning:# epochs, decay, etcCostly Tuning

• Optimization: stochastic trainingStochastic training for big data:⇒ gradient approximation⇒ ↑ difficulty for tuning

• Big Data

• Deep models need huge annotated datasets⇒ Huge models, huge computational demand⇒ Long be impossible to train such models with existing resources

• Generalization: deep models need huge annotated datasets• Smaller datasets: inferior predictive performances

Small models: not enough expressive powerLarge models overfit⇒ Performances ↓ handcrafted features

Deep Neural Network Weaknesses: Conclusion

• Deep learning Weaknesses:Black box modelsTraining challenges at many levels

• Drawbacks vs NN strength over DL History

Outline

Deep Learning: Trends and methods in the last four decades

Slide credit: https://www.slideshare.net/deview/251-implementing-deep-learning-using-cu-dnn

Case Study: LeNet 5 Model

80’s: 1st Convolutionnal Neural Networks

• LeNet 5 Model [LBD+89], trained using back-prop

• Input: 32x32 pixel image. Largest character is 20x20• 2 successive blocks [Convolution + Sigmoid + Pooling (+sigmoid)]Cx: Convolutional layer, Sx: Subsampling layer

• C5: convolution layer ∼ fully connected• 2 Fully connected layers Fx

C1 Layer

• Convolutional layer with 6 5x5 filters ⇒ 6 feature maps of size 28x28 (nopadding).

• # Parameters: 52 per filer + bias ⇒ (5 ∗ 5 + 1) ∗ 6 = 156If it was fully connected: (32*32+1)*(28*28)*6 parameters 5 ∼ 106 !

S2 Layer

• Subsampling layer = pooling layer• Pooling area : 2x2 in C1• Pooling stride: 2 ⇒ 6 features maps of size 14x14• Pooling type : sum, multiplied by a trainable param + bias⇒ 2 parameters per channel

• Total # Parameters: 2 ∗ 6 = 12

C3 Layer: Convolutional

• C3: 16 filters ⇒ 16 feature maps of size 10x10 (no padding)

• 5x5 filters connected to a subset of S2 maps⇒ 0-5 connected to 3, 6-14 to 4, 15 connected to 6

• # Parameters: 1516(5 ∗ 5 ∗ 3 + 1) ∗ 6 + (5 ∗ 5 ∗ 4 + 1) ∗ 9 + (5 ∗ 5 ∗ 6 + 1) = 456 + 909 + 151

S4 Layer

• Subsampling layer = pooling layer• Pooling area : 2x2 in C3• Pooling stride: 2 ⇒ 16 features maps of size 5x5• Pooling type : sum, multiplied by a trainable param + bias⇒ 2 parameters per channel

• Total # Parameters: 2 ∗ 16 = 32

C5 Layer: Convolutionnal layer

• 120 5x5x16 filters ⇒ whole depth of S4 (≠ C3)• Each maps in S4 is 5x5 ⇒ single value for each C5 maps• C5 120 features map of size 1x1 (vector of size 120)⇒ spatial information lost, ∼ to a fully connected layer

• Total # Parameters: (5 ∗ 5 ∗ 16 + 1) ∗ 120 = 48210

F6 Layer: Fully Connected layer

• 84 fully connected units.• # Parameters: 84*(120+1)=10164

F7 Layer (output): Fully Connected layer

• 10 (# classes) fully connected units.• # Parameters: 10*(84+1)=850

• Evaluation on MNIST• Total # parameters ∼ 60000

60,000 original datasets: test error: 0.95%540,000 artificial distortions + 60,000 original: Testerror: 0.8%

• Successful deployment for postal code reading in the US

LeNet 5 Model: Manifold Untangling

Input space Latent space

LeNet 5 Model: Manifold Untangling

Latent space MLP Latent space LeNet

80’s: Deep Learning Sucees

• ConvNet trainable with backprop, able to untangle class manifold• Very good performances for digit classification, industrial transfer• 2nd winter of Deep Learning⇒ following!

Source: Amazon

References I

A. R. Barron, Universal approximation bounds for superpositions of a sigmoidal function, Information

Theory, IEEE Transactions on 39 (1993), no. 3, 930–945.

Yoshua Bengio and Olivier Delalleau, On the expressive power of deep architectures, Proceedings of the

22Nd International Conference on Algorithmic Learning Theory (Berlin, Heidelberg), ALT’11,Springer-Verlag, 2011, pp. 18–36.

Yoshua Bengio, Learning deep architectures for ai, Found. Trends Mach. Learn. 2 (2009), no. 1, 1–127.

George Cybenko, Approximation by superpositions of a sigmoidal function, Mathematics of control, signals

and systems 2 (1989), no. 4, 303–314.

Ian Goodfellow, Yoshua Bengio, and Aaron Courville, Deep learning, MIT Press, 2016,http://www.deeplearningbook.org.

Johan Hastad, Almost optimal lower bounds for small depth circuits, RANDOMNESS ANDCOMPUTATION, JAI Press, 1989, pp. 6–20.

Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and

Lawrence D Jackel, Backpropagation applied to handwritten zip code recognition, Neural computation 1(1989), no. 4, 541–551.

Fernando Vieira Paulovich, Luis Gustavo Nonato, Rosane Minghim, and Haim Levkowitz, Least square

projection: A fast high-precision multidimensional projection technique and its application to documentmapping, IEEE Trans. Vis. Comput. Graph. 14 (2008), no. 3, 564–575.

Laurens van der Maaten and Geoffrey E. Hinton, Visualizing high-dimensional data using t-sne, Journal of

Machine Learning Research 9 (2008), 2579–2605.

Apprentissage,réseauxdeneuronesetmodèlesgraphiques (RCP209...

Documents

Transcript of Apprentissage,réseauxdeneuronesetmodèlesgraphiques (RCP209...

Connectionist Models: Backprop Jerome Feldman CS182/CogSci110/Ling109 Spring 2008.

Introduction a la programmation massivement …cedric.cnam.fr/~sailhanf/enseignements/p2p/gpu.pdfF. Sailhan (CNAM, d epartement informatique)Introduction a la programmation massivement

ECE 6504: Deep Learning for Perception Dhruv Batra Virginia Tech Topics: –Recurrent Neural Networks (RNNs) –BackProp Through Time (BPTT) –Vanishing / Exploding.

PLS generalised linear regression - cedric.cnam.fr · PLS generalised linear regression retains the rationale of ... P. Bastien et al./Computational Statistics & Data Analysis 48

Ghost Units Yield Biologically Plausible Backprop in Deep Neural … Ghost Units Yield... · Ghost Units Yield Biologically Plausible Backprop in Deep Neural Networks Thomas Mesnard

Using Backprop to Understand Apects of Cognitive Development

The Backpropagation Algorithm - Harvard Universityasaxe/papers/backpropagation.pdf– Backprop-as-algorithm • Implementation level – Haven’t touched on it – Few do! – Not

Functional Principal Component Analysis of Financial Time ...cedric.cnam.fr/~saporta/Fpca_Costanzo.pdf · Functional Principal Component Analysis of Financial Time Series G. Damiana

Java Card TechnologyJava Card Technology - …cedric.cnam.fr/~bouzefra/cours/JavaCard_Bouzefrane_dec11_Anglais.pdf · 1 samia.bouzefrane@cnam.fr - CEDRIC ( CNAM) - Java Card TechnologyJava

Using basis expansions for estimating functional PLS ...cedric.cnam.fr/fichiers/art_1978.pdf · Using basis expansions for estimating functional PLS regression Applications with chemometric

Heuristic Algorithms And Learning Techniques: Applications to …cedric.cnam.fr/~porumbed/papers/theseEn.pdf · 2014-11-06 · Heuristic Algorithms And Learning Techniques: Applications

Backprop, 25 years later…

Phorigami: Visualization of Digital Photo Collections by ...cedric.cnam.fr › fichiers › RC1895.pdf3.1 Meta-categorization for everyman photo collections ... (static and mobile).

Filtering Structures for Microblogging Content Ryadh ...cedric.cnam.fr/fichiers/art_3339.pdfFILTERING STRUCTURES FOR MICROBLOGGING CONTENT 4 flooding, users who follow active accounts

Semantic interactive image retrieval combining …cedric.cnam.fr/~crucianm/src/ferecatu08semanticPreprint.pdfSemantic interactive image retrieval combining visual and conceptual content

Using Backprop to Understand Apects of Cognitive Development PDP Class Feb 8, 2010.

CSC321 Lecture 6: Backpropagation - University of Torontorgrosse/courses/csc321_2017/slides/lec6.pdf · Roger Grosse CSC321 Lecture 6: Backpropagation 19 / 21. Backpropagation Backprop

Error Backpropagation - cedar.buffalo.edusrihari/CSE574/Chap5/Chap5.3-BackProp.pdf · Neural Network Training ... Overview of Backprop algorithm • Choose random weights for the

analyse Discriminante, Classification Supervisée, Scoring…cedric.cnam.fr/~saporta/discriminante.pdf · Tufféry: «Étude de cas en statistique décisionnelle »,Technip, 2009

CSC321 Lecture 15: Recurrent Neural Networks › ... › csc321_2018 › slides › lec15.pdf · Roger Grosse CSC321 Lecture 15: Recurrent Neural Networks 16 / 26. Backprop Through