Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning...

67

Transcript of Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning...

Page 1: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge
Page 2: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge

Francois Chollet - https://blog.keras.io/the-limitations-of-deep-learning.html

Page 3: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge

3rd November 2017

Page 4: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge

Yes, it’s abrain image: (

Page 5: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge

Su et al., One pixel attack for fooling deep neural networkshttps://arxiv.org/abs/1710.08864

Page 6: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge

https://arxiv.org/pdf/1707.07397.pdf

Page 7: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge
Page 8: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge
Page 9: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge

Connectomics:Neural nets for neural nets

[PatricHagmann]

Page 10: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge

Vision for understanding the brain

1mm cubed of brain

Image at 5-30 nanometers

How much data?

[Kaynig-Fittkau et al.]

Page 11: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge

Vision for understanding the brain

1mm cubed of brain

Image at 5-30 nanometers

How much data?

1 Petabyte –1,000,000,000,000,000

~ All photos uploaded to Facebook per day

[Kaynig-Fittkau et al.]

Page 12: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge

[Kaynig-Fittkau et al.]

Page 13: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge

[Kaynig-Fittkau et al.]

Page 14: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge

Vision for understanding the brain

Page 15: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge

[Haehn et al.]

Page 16: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge

[Haehn et al.]

Page 17: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge

Network Architecture

[Haehn et al.]

Page 18: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge

[Haehn et al.]

Page 19: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge
Page 20: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge
Page 21: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge

Any directed acyclic graph (DAG) of differentiable modules is allowed.

Page 22: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge
Page 23: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge

What about learning across ‘domains’?

Page 24: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge
Page 25: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge
Page 26: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge

Two-stream networks –action recognition

[Simonyan et al. 2014]

Page 27: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge

[Simonyan et al. 2014]

Page 28: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge

Learning Deep Representations For Ground-to-Aerial Geolocalization

Tsung-Yi Lin, Yin Cui, Serge Belongie, James Hays

CVPR 2015

Page 29: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge

Where was the photo taken?

View From Your Window ContestJune 9, 2010 – Feb. 4, 2015

Page 30: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge

Ans:Milano, Italy

Page 31: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge

To Geolocalize a Photo

• One can capture every corner on the earth

Page 32: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge

To Geolocalize a Photo

Page 33: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge
Page 34: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge
Page 35: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge

How To Match Ground-to-Aerial?

Shan et al., Accurate Geo-registration by Ground-to-Aerial Image Matching, 3DV’14Bansal et al., Ultra-wide baseline façade matching for geo-localization, ECCV workshop’12

Page 36: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge

Are these the same location?

Ground

Aerial

Page 37: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge

Are these the same location?

Ground

Aerial

Page 38: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge

Cross-view Pairs

Ground

Aerial

Page 39: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge

“Siamese” ConvNet for Ground-to-Aerial Matching

Page 40: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge

“Siamese” ConvNet for Ground-to-Aerial Matching

Page 41: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge

“Siamese” ConvNet for Ground-to-Aerial Matching

Page 42: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge

For ground-aerial image pairs, should A, B networks share thesame weights?

Page 43: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge

Quantitative Evaluation

Minor improvement with ‘domain-specific’ weights.

Page 44: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge

Big space of designs!But we still don’t even know how many layers we need.

Page 45: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge
Page 46: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge
Page 47: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge
Page 48: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge

VERY DEEP CONVOLUTIONAL NETWORKS FOR LARGE-SCALE IMAGE RECOGNITION

Karen Simonyan & Andrew Zisserman 2015

These are the pre-trained “VGG” networks

that you use in project 5

Beyond AlexNet

Page 49: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge
Page 50: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge
Page 51: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge

Google LeNet (2014)

22 layers

6.67% errorImageNet top 5

Page 52: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge

Inception!

Page 53: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge

Parallel layers

Page 54: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge

ResNet (He et al., 2015)

ResNet won ILSVRC 2015 with a top-5 error rate of 3.6%

Depending on their skill and expertise, humans generally hover around a 5-10% error.

But the task is arguablynot well defined.

Page 55: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge
Page 56: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge
Page 57: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge

CIFAR-10

• 60,000 32x32 color images, 10 classes

Page 58: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge
Page 59: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge
Page 60: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge
Page 61: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge

Regular net

Page 62: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge

Residual Unit

Page 63: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge

Residual Unit

The inputs of a lower layer is made available to a node in a higher layer.

Page 64: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge
Page 65: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge

Why so steep?

Page 66: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge

Flat regions in energy landscape

Page 67: Francois Chollet //cs.brown.edu/courses/cs143/2017_Fall/lectures... · 2017. 11. 3. · Learning Deep Representations For Ground-to-Aerial Geolocalization Tsung-Yi Lin, Yin Cui, Serge

Compute vs. parameters / multiply-adds

Hmm…efficient nets…might be useful for final project ???

James, do we have to go deeper?

https://www.infoq.com/news/2017/06/google-mobilenets-tensorflowhttps://arxiv.org/abs/1704.04861