Deep LearningandConvolutional Neural Networks

Ronald XIE 8th May, 2014

Deep Learning and Convolutional Neural Networks | Page 1

Tensor voting system

Robot operating system

Deep learning system

Robot localization and Scene Labeling

Robot operating system

Outline

Motivation

Deep Learning

Convolutional Neural Networks

Applications

Deep Learning and Convolutional Neural Networks | Page 2

Outline

Motivation

Deep Learning

Convolutional Neural Networks

Applications

Deep Learning and Convolutional Neural Networks | Page 3

Motivation feature representation

Deep Learning and Convolutional Neural Networks | Page 4

Motivation feature representation

Deep Learning and Convolutional Neural Networks | Page 5

Motivation feature representation

Deep Learning and Convolutional Neural Networks | Page 6

Motivation feature representation

Deep Learning and Convolutional Neural Networks | Page 7

Motivation feature representation

Deep Learning and Convolutional Neural Networks | Page 8

Given a dictionary of simple non-linear functions:

Proposal 1: linear combination

Motivation learning non-linear features

Proposal 2: composition

Deep Learning and Convolutional Neural Networks | Page 9

Given a dictionary of simple non-linear functions:

Proposal 1: linear combination

Motivation learning non-linear features

Proposal 2: composition

Deep Learning and Convolutional Neural Networks | Page 10

Motivation learning non-linear features Linear Combination

BAD: It may requirean exponential number oftemplates!!!

Deep Learning and Convolutional Neural Networks | Page 11

Motivation learning non-linear features Composition

GOOD: Re-use of

intermediate parts Distributed

representations is more efficient

Deep Learning and Convolutional Neural Networks | Page 12

Motivation learning non-linear features

Deep Learning and Convolutional Neural Networks | Page 13

Motivation deep learning in practice

Google Brain

- Big success on image & speech recognition

Microsoft

- Simultaneous interpretation system

Baidu

- Institute of Deep Learning

Deep Learning and Convolutional Neural Networks | Page 14

Motivation deep learning in practice

Deep Learning and Convolutional Neural Networks | Page 15

Outline

Motivation

Deep Learning

Convolutional Neural Networks

Applications

Deep Learning and Convolutional Neural Networks | Page 16

Deep Learning definition

What is deep learning?

It's a Convolutional Net.

It's a Contrastive Divergence.

It's a Feature Learning.

It's a Unsupervised Learning.

It's just old Neural Nets.

It's a Deep Belief Net.

Deep Learning and Convolutional Neural Networks | Page 17

Deep Learning definition

What is deep learning?

A Deep Learning method is: a method which makes

predictions by using a sequence of non-linear processing

stages. The resulting intermediate representations can be

interpreted as feature hierarchies and the whole system is

jointly learned from data.

Some deep learning methods are supervised, others are

unsupervised.

It's a large family!

Deep Learning and Convolutional Neural Networks | Page 18

Three types of deep architectures:

Feed-Forward: Multilayer Neural Nets, Convolutional Nets

Feed-Back: Sparse Coding, Deconvolutional Net

Bi-Directional: Deep Boltzmann Machines, Auto-Encoders

Deep Learning types

Deep Learning and Convolutional Neural Networks | Page 19

Three types of training protocols: Purely Supervised

- Initialize parameters randomly

- Train in supervised mode

- Used in most practical systems for speech and image recognition

Unsupervised layer-wise + supervised classifier on top- Train each layer unsupervised, one after the other

- Train a supervised classifier on top, keeping the other layers fixed

- Good when very few labeled samples are available

Unsupervised layer-wise + global supervised fine-tuning- Train each layer unsupervised, one after the other

- Add a classifier layer, and retrain the whole thing supervised

- Good when label set is poor

Deep Learning types

Deep Learning and Convolutional Neural Networks | Page 20

Outline

Motivation

Deep Learning

Convolutional Neural Networks

Applications

Deep Learning and Convolutional Neural Networks | Page 30

Convolution Basic concert

CNNs key ideas

Deep Learning and Convolutional Neural Networks | Page 32

CNNs key ideas

Deep Learning and Convolutional Neural Networks | Page 33

CNNs key ideas

Deep Learning and Convolutional Neural Networks | Page 34

CNNs key ideas

Deep Learning and Convolutional Neural Networks | Page 35

CNNs key ideas

Deep Learning and Convolutional Neural Networks | Page 36

CNNs key ideas

Deep Learning and Convolutional Neural Networks | Page 37

CNNs key ideas

Deep Learning and Convolutional Neural Networks | Page 38

CNNs key ideas

Deep Learning and Convolutional Neural Networks | Page 39

CNNs key ideas

Deep Learning and Convolutional Neural Networks | Page 40

CNNs key ideas

Deep Learning and Convolutional Neural Networks | Page 41

CNNs key ideas

Deep Learning and Convolutional Neural Networks | Page 42

CNNs key ideas

Deep Learning and Convolutional Neural Networks | Page 43

CNNs typical architecture

Deep Learning and Convolutional Neural Networks | Page 44

CNNs typical architecture

Deep Learning and Convolutional Neural Networks | Page 45

CNNs conclusion

Connect each hidden unit to a small

patch of the input.

Share the weight across hidden units.

Subsampling layers are useful to reduce

computational burden and increase

invariance.

Deep Learning and Convolutional Neural Networks | Page 46

Outline

Motivation

Deep Learning

Convolutional Neural Networks

Applications

Deep Learning and Convolutional Neural Networks | Page 47

ApplicationsScene recognition:

Deep Learning and Convolutional Neural Networks | Page 52

ApplicationsLocalization:

Deep Learning and Convolutional Neural Networks | Page 52