Leveraging Deep Learning Applications with CNTK

30
Dr. Miguel González-Fierro @miguelgfierro https://miguelgfierro.com LEVERAGING DEEP LEARNING APPLICATIONS WITH CNTK

Transcript of Leveraging Deep Learning Applications with CNTK

Page 1: Leveraging Deep Learning Applications with CNTK

Dr. Miguel González-Fierro

@miguelgfierrohttps://miguelgfierro.com

LEVERAGING DEEP LEARNING APPLICATIONS WITH CNTK

Page 2: Leveraging Deep Learning Applications with CNTK

1. CNTK FRAMEWORK

2. TRANSFER LEARNING WITH DEEP NETWORKS

3. IMAGE RECOGNITION API

Page 3: Leveraging Deep Learning Applications with CNTK

1. CNTK FRAMEWORK

Page 4: Leveraging Deep Learning Applications with CNTK

MICROSOFT’S COGNITIVE TOOLKIT: CNTK

Dr. Miguel González-Fierro - @miguelgfierro

MPIBrainScript

https://github.com/Microsoft/CNTK

Page 5: Leveraging Deep Learning Applications with CNTK

CNTK STRENGHTS: RNN & MULTISERVER GPU

Dr. Miguel González-Fierro - @miguelgfierro

Source:

https://arxiv.org/abs/1608.07249

Multi-server algorithms:

1-bit SGD

Block momentum

Recurrent Neural Networks

http://vision03.csail.mit.edu/cnn_art/
Page 6: Leveraging Deep Learning Applications with CNTK

CNTK TUTORIALS & EXAMPLES

Dr. Miguel González-Fierro - @miguelgfierro

https://github.com/Microsoft/CNTK/tree/master/Tutorialshttps://github.com/Microsoft/CNTK/tree/master/Examples

https://github.com/Microsoft/CNTK/tree/master/Tutorials
https://github.com/Microsoft/CNTK/tree/master/Examples
Page 7: Leveraging Deep Learning Applications with CNTK

GPU VIRTUAL MACHINE

Dr. Miguel González-Fierro - @miguelgfierro

$5.3/h$2.65/h$1.32/h

NOTE:

as of May 2017

Page 8: Leveraging Deep Learning Applications with CNTK

DATA SCIENCE VIRTUAL MACHINE (DSVM)

Dr. Miguel González-Fierro - @miguelgfierro

Page 9: Leveraging Deep Learning Applications with CNTK

2. TRANSFER LEARNING WITH DEEP NETWORKS

Page 10: Leveraging Deep Learning Applications with CNTK

KAGGLE DATA SCIENCE BOWL 2017

Dr. Miguel González-Fierro - @miguelgfierro

$1 million in prizes !

CT scan: horizontal section of lung

Determine

whether a

patient has

cancer or notcompetition

Page 11: Leveraging Deep Learning Applications with CNTK

OBJECTIVE: BASELINE SOLUTION

Dr. Miguel González-Fierro - @miguelgfierro

END TO END PROCESS

DSVM & CNTK

less than 1h

Top 10% (Jan 19th)

Page 12: Leveraging Deep Learning Applications with CNTK

LUNG SCANS DATASET

Dr. Miguel González-Fierro - @miguelgfierro

1595 patients with a diagnostic

200-500 scans per patient

Images of 512x512

Page 13: Leveraging Deep Learning Applications with CNTK

AUTOMATIC FEATURE GENERATOR

Dr. Miguel González-Fierro - @miguelgfierro

Source:

http://vision03.csail.mit.edu/cnn_art/

https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf

http://vision03.csail.mit.edu/cnn_art/
https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf
Page 14: Leveraging Deep Learning Applications with CNTK

TRANSFER LEARNING

ImageNet dataset Lung cancer dataset

weight transference

Dr. Miguel González-Fierro - @miguelgfierro

Page 15: Leveraging Deep Learning Applications with CNTK

STANDARD TRAINING

Dr. Miguel González-Fierro - @miguelgfierro

input hidden hidden hidden hidden hidden output

Forward and backward propagation

Page 16: Leveraging Deep Learning Applications with CNTK

TRANSFERENCE 1: FREEZE N-1 LAYERS

Dr. Miguel González-Fierro - @miguelgfierro

input hidden hidden hidden hidden hidden output

Frozen layers

Page 17: Leveraging Deep Learning Applications with CNTK

TRANSFERENCE 2: FREEZE INITIAL LAYERS

Dr. Miguel González-Fierro - @miguelgfierro

input hidden hidden hidden hidden hidden output

Frozen layers

Forward and backward propagation

Page 18: Leveraging Deep Learning Applications with CNTK

TRANSFERENCE 3: MODEL INITIALIZATION

Dr. Miguel González-Fierro - @miguelgfierro

input hidden hidden hidden hidden hidden output

Forward and backward propagation

Weight initialization

Page 19: Leveraging Deep Learning Applications with CNTK

PRETRAINED RESNET MODEL

Dr. Miguel González-Fierro - @miguelgfierro

3

224

224

last

layer

ImageNet ResNet N layers

penultimate

layer

tabby

cat

Page 20: Leveraging Deep Learning Applications with CNTK

SOLUTION: CNN FEATURIZER

3

224

224

ResNet N-1 layers

penultimate

layer

CNTK

(53min)k batch

of images= 1 patient

features

Dr. Miguel González-Fierro - @miguelgfierro

Page 21: Leveraging Deep Learning Applications with CNTK

SOLUTION: BOOSTED TREE TRAINING

3

224

224

ResNet N-1 layers

penultimate

layer

no

cancer

CNTK

(53min)

LightGBM

(2min)

Boosted tree

k batch

of images= 1 patient

features

Dr. Miguel González-Fierro - @miguelgfierro

Page 22: Leveraging Deep Learning Applications with CNTK

KAGGLE SUBMISSION

Dr. Miguel González-Fierro - @miguelgfierro

top 10% as of Jan 19th

Page 23: Leveraging Deep Learning Applications with CNTK

RESOURCES

Kaggle script Cortana Gallery notebook

Blog in TechNet

https://aka.ms/dsb2017-cntk-notebookhttps://aka.ms/dsb2017-cntk-script https://aka.ms/dsb2017-cntk-blog

+6000 views +1500 views+300 shares on SN

+1500 views+300 downloads

TechNet

Dr. Miguel González-Fierro - @miguelgfierro

https://aka.ms/dsb2017-cntk-notebook
https://aka.ms/dsb2017-cntk-script
https://aka.ms/dsb2017-cntk-blog
Page 24: Leveraging Deep Learning Applications with CNTK

3. IMAGE RECOGNITION API

Page 25: Leveraging Deep Learning Applications with CNTK

SUCCESS IN IMAGE CLASSIFICATION

Dr. Miguel González-Fierro - @miguelgfierro

15.4%

7.3%

6.7%

3.6%3.1%

5.1% (human)

error (%)

ImageNet (image recognition competition) top-5 error

AlexNet(2012)

VGG(2014)

Inception(2015)

ResNet(2015)

Inception-ResNet(2016)

Page 26: Leveraging Deep Learning Applications with CNTK

ARCHITECTURE

Dr. Miguel González-Fierro - @miguelgfierro

Ilia Karmanov: https://github.com/ilkarman/CV_end_to_end

https://github.com/ilkarman/CV_end_to_end
Page 27: Leveraging Deep Learning Applications with CNTK

ARCHITECTURE

Dr. Miguel González-Fierro - @miguelgfierro

Ilia Karmanov: https://github.com/ilkarman/CV_end_to_end

https://github.com/ilkarman/CV_end_to_end
Page 28: Leveraging Deep Learning Applications with CNTK

SUCCESS IN IMAGE CLASSIFICATION

Dr. Miguel González-Fierro - @miguelgfierro

Ilia Karmanov: https://github.com/ilkarman/CV_end_to_end

https://github.com/ilkarman/CV_end_to_end
Page 29: Leveraging Deep Learning Applications with CNTK

SUCCESS IN IMAGE CLASSIFICATION

Dr. Miguel González-Fierro - @miguelgfierro

Ilia Karmanov: https://github.com/ilkarman/CV_end_to_end

https://github.com/ilkarman/CV_end_to_end
Page 30: Leveraging Deep Learning Applications with CNTK

Dr. Miguel González-Fierro

@miguelgfierrohttps://miguelgfierro.com

QUESTIONS?


PATTERN RECOGNITION AND MACHINE · PDF file · 2018-02-05PATTERN RECOGNITION AND MACHINE LEARNING Slide Set 6: Neural Networks and Deep Learning ... MS Cognitive TK (CNTK): Microsoft

PATTERN RECOGNITION AND MACHINE · PDF file · 2018-02-05PATTERN RECOGNITION AND MACHINE LEARNING Slide Set 6: Neural Networks and Deep Learning ... MS Cognitive TK (CNTK): Microsoft

Leveraging Frequency Analysis for Deep Fake Image Recognition

Leveraging Frequency Analysis for Deep Fake Image Recognition

Workshop: Redes neurais com keras, cntk e tensorflow no Azure … · 2019-05-16 · 1 Workshop: Redes neurais com keras, cntk e tensorflow no Azure machine learning

Workshop: Redes neurais com keras, cntk e tensorflow no Azure … · 2019-05-16 · 1 Workshop: Redes neurais com keras, cntk e tensorflow no Azure machine learning

The Market Opportunity for Productive Use Leveraging Solar … · 2019-10-03 · Kenya Market Deep Dive vi.Zimbabwe Market Deep Dive vii.Côte d’Ivoire ... Insights and Findings

The Market Opportunity for Productive Use Leveraging Solar … · 2019-10-03 · Kenya Market Deep Dive vi.Zimbabwe Market Deep Dive vii.Côte d’Ivoire ... Insights and Findings

Benchmarking and Analyzing Deep Neural Network Trainingpekhimenko/Papers/iiswc18-tbd.pdf · frameworks (TensorFlow [8], MXNet [22], CNTK [89]) across different hardware configurations

Benchmarking and Analyzing Deep Neural Network Trainingpekhimenko/Papers/iiswc18-tbd.pdf · frameworks (TensorFlow [8], MXNet [22], CNTK [89]) across different hardware configurations

Precipitation Nowcasting: Leveraging Deep Convolutional ... · Build a Precipitation Nowcasting deep learning model Gain insights into the full deep learning workflow Improve the

Precipitation Nowcasting: Leveraging Deep Convolutional ... · Build a Precipitation Nowcasting deep learning model Gain insights into the full deep learning workflow Improve the

Deep Virtual Stereo Odometry: Leveraging Deep Depth ...openaccess.thecvf.com/content_ECCV_2018/papers/Nan_Yang_Deep… · Deep Virtual Stereo Odometry: Leveraging Deep Depth Predictionfor

Deep Virtual Stereo Odometry: Leveraging Deep Depth ...openaccess.thecvf.com/content_ECCV_2018/papers/Nan_Yang_Deep… · Deep Virtual Stereo Odometry: Leveraging Deep Depth Predictionfor

DEEP LEARNING, MACHINE LEARNING, + ARTIFICIAL INTELLIGENCE, SYMPOSIUM · 2018-11-27 · Leveraging Deep Learning and Rapid Response Team Nurses to Improve Sepsis Management SPEAKER:

DEEP LEARNING, MACHINE LEARNING, + ARTIFICIAL INTELLIGENCE, SYMPOSIUM · 2018-11-27 · Leveraging Deep Learning and Rapid Response Team Nurses to Improve Sepsis Management SPEAKER:

Image Classification, Deep Learning and Convolutional ...1111144/FULLTEXT02.pdfTensorFlow and Microsoft CNTK, primarily based on their performance in the training time of neural networks.

Image Classification, Deep Learning and Convolutional ...1111144/FULLTEXT02.pdfTensorFlow and Microsoft CNTK, primarily based on their performance in the training time of neural networks.

CNTK: deep learning framework - Nvidiaimages.nvidia.com/events/sc15/pdfs/CNTK-Overview-SC150-Kamanev.pdfmeanFile=$WorkDir$/ImageNet1K_mean.xml ] labels=[ labelDim=1000 ] ] ] Next steps:

CNTK: deep learning framework - Nvidiaimages.nvidia.com/events/sc15/pdfs/CNTK-Overview-SC150-Kamanev.pdfmeanFile=$WorkDir$/ImageNet1K_mean.xml ] labels=[ labelDim=1000 ] ] ] Next steps:

Deep Learning Tools and Frameworks - microsoft.com · deep learning tools and frameworks hamid palangi deep learning group, microsoft research ai redmond, wa, usa ... (cntk), mxnet

Deep Learning Tools and Frameworks - microsoft.com · deep learning tools and frameworks hamid palangi deep learning group, microsoft research ai redmond, wa, usa ... (cntk), mxnet

High-Performance Training for Deep Learning and …mvapich.cse.ohio-state.edu/static/media/talks/slide/cvpr...Research Challenges to Exploit HPC Technologies CPU InfiniBand GPU CNTK

High-Performance Training for Deep Learning and …mvapich.cse.ohio-state.edu/static/media/talks/slide/cvpr...Research Challenges to Exploit HPC Technologies CPU InfiniBand GPU CNTK

Bright Computing - UGentkehoste/eum18/Bright_Computing...CafffeOnSpark static 0.1 Spark 2.x CNTK static 2.3.1 OpenMPI, MKL Keras N/A 2.1.2 Theano, CNTK and Tensorflow backends MXNet

Bright Computing - UGentkehoste/eum18/Bright_Computing...CafffeOnSpark static 0.1 Spark 2.x CNTK static 2.3.1 OpenMPI, MKL Keras N/A 2.1.2 Theano, CNTK and Tensorflow backends MXNet

Comparative Study of Deep Learning Framework in HPC ......2) CNTK The Microsoft Cognitive Toolkit (CNTK) is a unified deep-learning toolkit developed by Microsoft [2]. It provides

Comparative Study of Deep Learning Framework in HPC ......2) CNTK The Microsoft Cognitive Toolkit (CNTK) is a unified deep-learning toolkit developed by Microsoft [2]. It provides

A Survey on Leveraging Deep Neural Networks for Object ......Deep Learning for Object Tracking - End-to-End A Survey on Leveraging Deep Neural Networks for Object Tracking| Sebastian

A Survey on Leveraging Deep Neural Networks for Object ......Deep Learning for Object Tracking - End-to-End A Survey on Leveraging Deep Neural Networks for Object Tracking| Sebastian

Leveraging AI in Service Automation Modeling: from …Leveraging AI in Service Automation Modeling: from Classical AI Through Deep Learning to Combination Models Qing Wang 1[00001111

Leveraging AI in Service Automation Modeling: from …Leveraging AI in Service Automation Modeling: from Classical AI Through Deep Learning to Combination Models Qing Wang 1[00001111

DEEP LEARNING WITH MICROSOFT COGNITIVE TOOLKIT CNTK … · Microsoft Cognitive Toolkit library trains LSTM-models up to 4 times faster than TensorFlow, it parallelizes the processes

DEEP LEARNING WITH MICROSOFT COGNITIVE TOOLKIT CNTK … · Microsoft Cognitive Toolkit library trains LSTM-models up to 4 times faster than TensorFlow, it parallelizes the processes

DGXUPDATE - NVIDIA...NVIDIA Docker GPU DRIVER NVIDIA Driver SYSTEM Host OS Advantages: Instant productivity with NVIDIA optimized deep learning frameworks Caffe, CNTK, MXNet, PyTorch,

DGXUPDATE - NVIDIA...NVIDIA Docker GPU DRIVER NVIDIA Driver SYSTEM Host OS Advantages: Instant productivity with NVIDIA optimized deep learning frameworks Caffe, CNTK, MXNet, PyTorch,

Abstract · exible deep learning framework, The Microsoft Cognitive Toolkit (formerly CNTK). In this work we rst provide background on deep learning with the Cognitive Toolkit (section

Abstract · exible deep learning framework, The Microsoft Cognitive Toolkit (formerly CNTK). In this work we rst provide background on deep learning with the Cognitive Toolkit (section

Deep Dive into Leveraging New Features and Functions of SAP NetWeaver BW 7.4

Deep Dive into Leveraging New Features and Functions of SAP NetWeaver BW 7.4