Convolutional Neural Networkshomes.soic.indiana.edu/classes/spring2018/info/i529-yye/... ·...

27
Convolutional Neural Networks Yuzhen Ye School of Informatics and Computing, Indiana University

Transcript of Convolutional Neural Networkshomes.soic.indiana.edu/classes/spring2018/info/i529-yye/... ·...

Page 1: Convolutional Neural Networkshomes.soic.indiana.edu/classes/spring2018/info/i529-yye/... · 2017-04-17 · Convolutional Neural Networks Yuzhen Ye School of Informatics and Computing,

ConvolutionalNeuralNetworks

YuzhenYeSchoolofInformaticsandComputing,IndianaUniversity

Page 2: Convolutional Neural Networkshomes.soic.indiana.edu/classes/spring2018/info/i529-yye/... · 2017-04-17 · Convolutional Neural Networks Yuzhen Ye School of Informatics and Computing,

Contents

• CNNbasics• CNNforvisualrecognition(toexplaintheconceptofconvolution)• CNNforbioinformatics

Page 3: Convolutional Neural Networkshomes.soic.indiana.edu/classes/spring2018/info/i529-yye/... · 2017-04-17 · Convolutional Neural Networks Yuzhen Ye School of Informatics and Computing,

ABeginner'sGuideToUnderstandingCNN• NeuralNetworkswithConvolutionlayers

Ref1:http://cs231n.stanford.eduRef2:https://adeshpande3.github.io/adeshpande3.github.io/A-Beginner%27s-Guide-To-Understanding-Convolutional-Neural-Networks/

Page 4: Convolutional Neural Networkshomes.soic.indiana.edu/classes/spring2018/info/i529-yye/... · 2017-04-17 · Convolutional Neural Networks Yuzhen Ye School of Informatics and Computing,
Page 5: Convolutional Neural Networkshomes.soic.indiana.edu/classes/spring2018/info/i529-yye/... · 2017-04-17 · Convolutional Neural Networks Yuzhen Ye School of Informatics and Computing,

MovingAverageIn2D

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 10 20 30 30 30 20 10

0 20 40 60 60 60 40 20

0 30 60 90 90 90 60 30

0 30 50 80 80 90 60 30

0 30 50 80 80 90 60 30

0 20 30 50 50 60 40 20

10 20 30 30 30 30 20 10

10 10 10 0 0 0 0 0

5Slidecredit:SteveSeitz

],[ yxf ],[ yxg

Page 6: Convolutional Neural Networkshomes.soic.indiana.edu/classes/spring2018/info/i529-yye/... · 2017-04-17 · Convolutional Neural Networks Yuzhen Ye School of Informatics and Computing,

CorrelationfilteringSaytheaveragingwindowsizeis2k+1x2k+1:

Loopoverallpixelsinneighborhoodaroundimagepixelf[i,j]

Attributeuniformweighttoeachpixel

Nowgeneralizetoallowdifferentweightsdependingonneighboringpixel’srelativeposition:

Non-uniformweights

6SlideadaptedfromKristenGrauman

Page 7: Convolutional Neural Networkshomes.soic.indiana.edu/classes/spring2018/info/i529-yye/... · 2017-04-17 · Convolutional Neural Networks Yuzhen Ye School of Informatics and Computing,

Correlationfiltering

Filteringanimage:replaceeachpixelwithalinearcombinationofitsneighbors.

Thefilter“kernel”or“mask”h[u,v]istheprescriptionfortheweightsinthelinearcombination.

Thisiscalledcross-correlation,denoted

7Slidecredit:MichaelS.Ryoo

Page 8: Convolutional Neural Networkshomes.soic.indiana.edu/classes/spring2018/info/i529-yye/... · 2017-04-17 · Convolutional Neural Networks Yuzhen Ye School of Informatics and Computing,

],[ yxf ],[ yxg

Averagingfilter• Whatvaluesbelonginthekernelh forthemovingaverageexample?

0 10 20 30 30

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

111111111

“boxfilter”

?

8Slidecredit:MichaelS.Ryoo

],[ vuh

Page 9: Convolutional Neural Networkshomes.soic.indiana.edu/classes/spring2018/info/i529-yye/... · 2017-04-17 · Convolutional Neural Networks Yuzhen Ye School of Informatics and Computing,

• Whatifwewantnearestneighboringpixelstohavethemostinfluenceontheoutput?

• Removeshigh-frequencycomponentsfromtheimage(“low-passfilter”).

Gaussianfilter

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

1 2 1

2 4 2

1 2 1

This kernel is an approximation of a 2d Gaussian function:

9Slidecredit:SteveSeitz

],[ yxf

],[ vuh

Page 10: Convolutional Neural Networkshomes.soic.indiana.edu/classes/spring2018/info/i529-yye/... · 2017-04-17 · Convolutional Neural Networks Yuzhen Ye School of Informatics and Computing,

f

h

Convolution• Convolutionisasimplemathematicaloperationwhichisfundamentaltomanycommonimageprocessingoperators.

• Convolutionisperformedbyslidingthekernelovertheimage,generallystartingatthetopleftcorner,soastomovethekernelthroughallthepositionswherethekernelfitsentirelywithintheboundariesoftheimage.

• Convolution:• Flipthefilterinbothdimensions(bottomtotop,righttoleft)• Thenapplycross-correlation

Notationforconvolutionoperator

10Slidecredit:MichaelS.Ryoo

Page 11: Convolutional Neural Networkshomes.soic.indiana.edu/classes/spring2018/info/i529-yye/... · 2017-04-17 · Convolutional Neural Networks Yuzhen Ye School of Informatics and Computing,

Convolutionvs.correlationConvolution

Cross-correlation

11SlideadaptedfromKristenGrauman

Page 12: Convolutional Neural Networkshomes.soic.indiana.edu/classes/spring2018/info/i529-yye/... · 2017-04-17 · Convolutional Neural Networks Yuzhen Ye School of Informatics and Computing,

Derivativesandedges

imageintensity function

(along horizontal scanline) first derivative

edges correspond toextrema of derivative

An edge is a place of rapid change in the image intensity function.

12Slidecredit:SvetlanaLazebnik

Page 13: Convolutional Neural Networkshomes.soic.indiana.edu/classes/spring2018/info/i529-yye/... · 2017-04-17 · Convolutional Neural Networks Yuzhen Ye School of Informatics and Computing,

DerivativeswithconvolutionFor2Dfunction,f(x,y),thepartialderivativeis:

Fordiscretedata,wecanapproximateusingfinitedifferences:

Toimplementaboveasconvolution,whatwouldbetheassociatedfilter?

ee

e

),(),(lim),(0

yxfyxfxyxf -+=

¶¶

®

1),(),1(),( yxfyxf

xyxf -+»

¶¶

13Slidecredit:KristenGrauman

Page 14: Convolutional Neural Networkshomes.soic.indiana.edu/classes/spring2018/info/i529-yye/... · 2017-04-17 · Convolutional Neural Networks Yuzhen Ye School of Informatics and Computing,

Partialderivativesofanimage

Which shows changes with respect to x?

-1 1

1 -1

or?

-1 1

xyxf

¶¶ ),(

yyxf

¶¶ ),(

(showing filters for correlation) 14Slidecredit:KristenGrauman

Page 15: Convolutional Neural Networkshomes.soic.indiana.edu/classes/spring2018/info/i529-yye/... · 2017-04-17 · Convolutional Neural Networks Yuzhen Ye School of Informatics and Computing,

Filtersasfeature(edge)detectors

sobel filter

15Slidecredit:KristenGrauman http://homepages.inf.ed.ac.uk/rbf/HIPR2/index.htm

Page 16: Convolutional Neural Networkshomes.soic.indiana.edu/classes/spring2018/info/i529-yye/... · 2017-04-17 · Convolutional Neural Networks Yuzhen Ye School of Informatics and Computing,

ImagegradientThegradientofanimage:

Thegradientpointsinthedirectionofmostrapidchangeinintensity

The gradient direction (orientation of edge normal) is given by:

The edge strength is given by the gradient magnitude

16Slidecredit:SteveSeitz

Page 17: Convolutional Neural Networkshomes.soic.indiana.edu/classes/spring2018/info/i529-yye/... · 2017-04-17 · Convolutional Neural Networks Yuzhen Ye School of Informatics and Computing,

EffectsofnoiseConsiderasingleroworcolumnoftheimage

• Plottingintensityasafunctionofpositiongivesasignal

Where is the edge?17

Slidecredit:SteveSeitz

Page 18: Convolutional Neural Networkshomes.soic.indiana.edu/classes/spring2018/info/i529-yye/... · 2017-04-17 · Convolutional Neural Networks Yuzhen Ye School of Informatics and Computing,

Effectsofnoise

•Differencefiltersrespondstronglytonoise• Imagenoiseresultsinpixelsthatlookverydifferentfromtheirneighbors• Generally,thelargerthenoisethestrongertheresponse

•Whatcanwedoaboutit?

Slidecredit:MichaelS.Ryoo

Page 19: Convolutional Neural Networkshomes.soic.indiana.edu/classes/spring2018/info/i529-yye/... · 2017-04-17 · Convolutional Neural Networks Yuzhen Ye School of Informatics and Computing,

Where is the edge?

Solution:smoothfirst

Look for peaks in 19Slidecredit:KristenGrauman

Page 20: Convolutional Neural Networkshomes.soic.indiana.edu/classes/spring2018/info/i529-yye/... · 2017-04-17 · Convolutional Neural Networks Yuzhen Ye School of Informatics and Computing,

Derivativetheoremofconvolution

Differentiationpropertyofconvolution.

20Slidecredit:SteveSeitz

Page 21: Convolutional Neural Networkshomes.soic.indiana.edu/classes/spring2018/info/i529-yye/... · 2017-04-17 · Convolutional Neural Networks Yuzhen Ye School of Informatics and Computing,

LaplacianofGaussianConsider

Laplacian of Gaussianoperator

Where is the edge? Zero-crossings of bottom graph 21Slidecredit:SteveSeitz

Page 22: Convolutional Neural Networkshomes.soic.indiana.edu/classes/spring2018/info/i529-yye/... · 2017-04-17 · Convolutional Neural Networks Yuzhen Ye School of Informatics and Computing,

2Dedgedetectionfilters

• is the Laplacian operator:

Laplacian of Gaussian

Gaussian derivative of Gaussian

22Slidecredit:SteveSeitz

Two commonly used discrete approximations to the Laplacian filter

Page 23: Convolutional Neural Networkshomes.soic.indiana.edu/classes/spring2018/info/i529-yye/... · 2017-04-17 · Convolutional Neural Networks Yuzhen Ye School of Informatics and Computing,
Page 24: Convolutional Neural Networkshomes.soic.indiana.edu/classes/spring2018/info/i529-yye/... · 2017-04-17 · Convolutional Neural Networks Yuzhen Ye School of Informatics and Computing,
Page 25: Convolutional Neural Networkshomes.soic.indiana.edu/classes/spring2018/info/i529-yye/... · 2017-04-17 · Convolutional Neural Networks Yuzhen Ye School of Informatics and Computing,
Page 26: Convolutional Neural Networkshomes.soic.indiana.edu/classes/spring2018/info/i529-yye/... · 2017-04-17 · Convolutional Neural Networks Yuzhen Ye School of Informatics and Computing,

http://onlinelibrary.wiley.com/doi/10.15252/msb.20156651/full#msb156651-fig-0002

CNNforbiologicalimageanalysis

Page 27: Convolutional Neural Networkshomes.soic.indiana.edu/classes/spring2018/info/i529-yye/... · 2017-04-17 · Convolutional Neural Networks Yuzhen Ye School of Informatics and Computing,

MolecularSystemsBiologyVolume12,Issue7,29JUL2016DOI:10.15252/msb.20156651http://onlinelibrary.wiley.com/doi/10.15252/msb.20156651/full#msb156651-fig-0002

CNNforpredictingmoleculartraits

Inputdata:one-dimensionalgenomicsequenceswithonechannelpernucleotide

[Visualrecognition:2D-imagewiththreecolorchannels]


Very Deep Convolutional Neural Networks for Noise …...convolutional layers with carefully tuned architecture. •ASR: Very Deep Convolutional Neural Networks uses up to 10 convolutional

Very Deep Convolutional Neural Networks for Noise …...convolutional layers with carefully tuned architecture. •ASR: Very Deep Convolutional Neural Networks uses up to 10 convolutional

CONVOLUTIONAL NEURAL NETWORKS: MOTIVATION, …

CONVOLUTIONAL NEURAL NETWORKS: MOTIVATION, …

Deep convolutional neural networks - Computer Science- …web.cs.ucdavis.edu/.../lee_lecture19_deeplearning_notes.pdf · Deep convolutional neural networks ... “Deep” architecture

Deep convolutional neural networks - Computer Science- …web.cs.ucdavis.edu/.../lee_lecture19_deeplearning_notes.pdf · Deep convolutional neural networks ... “Deep” architecture

Convolutional Neural Networks - Virginia Techjbhuang/teaching/ece5554-4554/fa17/... · Convolutional Neural Networks Computer Vision ... ImageNet Classification with Deep Convolutional

Convolutional Neural Networks - Virginia Techjbhuang/teaching/ece5554-4554/fa17/... · Convolutional Neural Networks Computer Vision ... ImageNet Classification with Deep Convolutional

Detecting Object Affordances with Convolutional Neural ...dkanou.github.io/publ/P9__Nguyen_Kanoulas_Caldwell_Tsagarakis__… · Detecting Object Affordances with Convolutional Neural

Detecting Object Affordances with Convolutional Neural ...dkanou.github.io/publ/P9__Nguyen_Kanoulas_Caldwell_Tsagarakis__… · Detecting Object Affordances with Convolutional Neural

Convolutional neural networks deepa

Convolutional neural networks deepa

Convolutional Neural Networks (Part II)lvelho.impa.br/ip17/proj/slides/1110GoingDeeperConv.pdf · Convolutional Neural Networks 1 Convolutional Neural Networks (Part II) 08, 10 &

Convolutional Neural Networks (Part II)lvelho.impa.br/ip17/proj/slides/1110GoingDeeperConv.pdf · Convolutional Neural Networks 1 Convolutional Neural Networks (Part II) 08, 10 &

Convolutional Neural Networks Arise From Ising Models …sunilpai/convolutional-neural-networks.pdf · Convolutional Neural Networks Arise From Ising Models and Restricted Boltzmann

Convolutional Neural Networks Arise From Ising Models …sunilpai/convolutional-neural-networks.pdf · Convolutional Neural Networks Arise From Ising Models and Restricted Boltzmann

O-CNN: Octree-based Convolutional Neural Networks for … · O-CNN: Octree-based Convolutional Neural Networks for 3D ... object classification, ... Octree-based Convolutional Neural

O-CNN: Octree-based Convolutional Neural Networks for … · O-CNN: Octree-based Convolutional Neural Networks for 3D ... object classification, ... Octree-based Convolutional Neural

Convolutional Neural Networks Arise From Ising Models and ...web.stanford.edu/~sunilpai/convolutional-neural-networks.pdf · Convolutional neural networks are an attractive option

Convolutional Neural Networks Arise From Ising Models and ...web.stanford.edu/~sunilpai/convolutional-neural-networks.pdf · Convolutional neural networks are an attractive option

Training Convolutional Neural Networksraul/RS/Training Convolutional... · • Data-dependent Initializations of Convolutional Neural Networks by Krähenbühl et al., ... In practice,

Training Convolutional Neural Networksraul/RS/Training Convolutional... · • Data-dependent Initializations of Convolutional Neural Networks by Krähenbühl et al., ... In practice,

Doubly Convolutional Neural Networks - papers.nips.ccpapers.nips.cc/paper/6340-doubly-convolutional-neural-networks.pdf · convolutional neural network (DCNN). In Figure 3 we have

Doubly Convolutional Neural Networks - papers.nips.ccpapers.nips.cc/paper/6340-doubly-convolutional-neural-networks.pdf · convolutional neural network (DCNN). In Figure 3 we have

Introduction to Convolutional Neural Networksweb.stanford.edu/class/cs231a/lectures/intro_cnn.pdf · 2018-02-25 · Convolutional Neural Networks To address this problem, bionic convolutional

Introduction to Convolutional Neural Networksweb.stanford.edu/class/cs231a/lectures/intro_cnn.pdf · 2018-02-25 · Convolutional Neural Networks To address this problem, bionic convolutional

ImageNet Classification with Deep Convolutional Neural ... · Convolutional neural networks Output Hidden Data Here's a one-dimensional convolutional neural network Each hidden neuron

ImageNet Classification with Deep Convolutional Neural ... · Convolutional neural networks Output Hidden Data Here's a one-dimensional convolutional neural network Each hidden neuron

ImageNet Classification with Deep Deep Convolutional …rita/visual_recog_course/talks/ImageNet... · ImageNet Classification with Deep Deep Convolutional Convolutional Neural Neural

ImageNet Classification with Deep Deep Convolutional …rita/visual_recog_course/talks/ImageNet... · ImageNet Classification with Deep Deep Convolutional Convolutional Neural Neural

Convolutional Neural Networks - arXiv

Convolutional Neural Networks - arXiv

AnApproachtoSemanticandStructuralFeaturesLearningfor ...downloads.hindawi.com/journals/mpe/2020/6038619.pdf3.2. Convolutional Neural Network. Convolutional neural network(CNN)isone

AnApproachtoSemanticandStructuralFeaturesLearningfor ...downloads.hindawi.com/journals/mpe/2020/6038619.pdf3.2. Convolutional Neural Network. Convolutional neural network(CNN)isone

Using Support Vector Machines, Convolutional … Support Vector Machines, Convolutional Neural Networks and ... Using Support Vector Machines, Convolutional Neural ... orcircuit ,andmuchANNresearch

Using Support Vector Machines, Convolutional … Support Vector Machines, Convolutional Neural Networks and ... Using Support Vector Machines, Convolutional Neural ... orcircuit ,andmuchANNresearch

Pixel-Adaptive Convolutional Neural Networksopenaccess.thecvf.com/content_CVPR_2019/papers/Su_Pixel-Adaptive_Convolutional_Neural...Pixel-Adaptive Convolutional Neural Networks Hang

Pixel-Adaptive Convolutional Neural Networksopenaccess.thecvf.com/content_CVPR_2019/papers/Su_Pixel-Adaptive_Convolutional_Neural...Pixel-Adaptive Convolutional Neural Networks Hang

Convolutional Neural Networks

Convolutional Neural Networks