With TensorFlow Google Brain team Large-Scale Deep Learning...
Transcript of With TensorFlow Google Brain team Large-Scale Deep Learning...
![Page 1: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/1.jpg)
Large-Scale Deep LearningWith TensorFlow
Jeff DeanGoogle Brain team
g.co/brainIn collaboration with many other people at Google
![Page 2: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/2.jpg)
What is the Google Brain Team?
● Research team focused on long term artificial intelligence research○ Mix of computer systems and machine learning
research expertise○ Pure ML research, and research in context of
emerging ML application areas:■ robotics, language understanding, healthcare, ...
g.co/brain
![Page 3: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/3.jpg)
We Disseminate Our Work in Many Ways
● By publishing our work○ See papers at research.google.com/pubs/BrainTeam.html
● By releasing TensorFlow, our core machine learning research system, as an open-source project
● By releasing implementations of our research models in TensorFlow
● By collaborating with product teams at Google to get our research into real products
![Page 4: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/4.jpg)
What Do We Really Want?
● Build artificial intelligence algorithms and systems that learn from experience
● Use those to solve difficult problems that benefit humanity
![Page 5: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/5.jpg)
What do I mean by understanding?
![Page 6: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/6.jpg)
What do I mean by understanding?
![Page 7: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/7.jpg)
What do I mean by understanding?
![Page 8: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/8.jpg)
What do I mean by understanding?
[ car parts for sale ]
Query
![Page 9: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/9.jpg)
What do I mean by understanding?
[ car parts for sale ]
Query
Document 1
… car parking available for a small fee.… parts of our floor model inventory for sale.
Document 2
Selling all kinds of automobile and pickup truck parts, engines, and transmissions.
![Page 10: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/10.jpg)
Example Needs of the Future● Which of these eye images shows symptoms of diabetic
retinopathy?
● Find me all rooftops in North America
● Describe this video in Spanish
● Find me all documents relevant to reinforcement learning for robotics and summarize them in German
● Find a free time for everyone in the Smart Calendar project to meet and set up a videoconference
● Robot, please fetch me a cup of tea from the snack kitchen
![Page 11: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/11.jpg)
Growing Use of Deep Learning at Google
AndroidAppsdrug discoveryGmailImage understandingMapsNatural language understandingPhotosRobotics researchSpeechTranslationYouTube… many others ...
Across many products/areas:
# of directories containing model description files
![Page 12: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/12.jpg)
Important Property of Neural Networks
Results get better with
more data +bigger models +
more computation
(Better algorithms, new insights and improved techniques always help, too!)
![Page 13: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/13.jpg)
AsideMany of the techniques that are successful now were developed 20-30 years ago
What changed? We now have:
sufficient computational resourceslarge enough interesting datasets
Use of large-scale parallelism lets us look ahead many generations of hardware improvements, as well
![Page 14: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/14.jpg)
What do you want in a machine learning system?● Ease of expression: for lots of crazy ML ideas/algorithms● Scalability: can run experiments quickly● Portability: can run on wide variety of platforms● Reproducibility: easy to share and reproduce research● Production readiness: go from research to real products
![Page 15: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/15.jpg)
Open, standard software for general machine learning
Great for Deep Learning in particular
First released Nov 2015
Apache 2.0 license
http://tensorflow.org/and
https://github.com/tensorflow/tensorflow
![Page 16: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/16.jpg)
http://tensorflow.org/whitepaper2015.pdf
![Page 17: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/17.jpg)
Preprint: arxiv.org/abs/1605.08695Updated version will appear in OSDI 2016
![Page 18: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/18.jpg)
Strong External Adoption
GitHub Launch Nov. 2015
GitHub Launch Sep. 2013
GitHub Launch Jan. 2012
GitHub Launch Jan. 2008
50,000+ binary installs in 72 hours, 500,000+ since November, 2015
![Page 19: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/19.jpg)
Strong External Adoption
GitHub Launch Nov. 2015
GitHub Launch Sep. 2013
GitHub Launch Jan. 2012
GitHub Launch Jan. 2008
50,000+ binary installs in 72 hours, 500,000+ since November, 2015Most forked new repo on GitHub in 2015 (despite only being available in Nov, ‘15)
![Page 21: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/21.jpg)
Motivations● DistBelief (our 1st system) was the first scalable deep
learning system, but not as flexible as we wanted for research purposes
● Better understanding of problem space allowed us to make some dramatic simplifications
● Define the industrial standard for machine learning● Short circuit the MapReduce/Hadoop inefficiency
![Page 22: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/22.jpg)
TensorFlow: Expressing High-Level ML Computations
● Core in C++○ Very low overhead
Core TensorFlow Execution System
CPU GPU Android iOS ...
![Page 23: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/23.jpg)
TensorFlow: Expressing High-Level ML Computations
● Core in C++○ Very low overhead
● Different front ends for specifying/driving the computation○ Python and C++ today, easy to add more
Core TensorFlow Execution System
CPU GPU Android iOS ...
![Page 24: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/24.jpg)
TensorFlow: Expressing High-Level ML Computations
● Core in C++○ Very low overhead
● Different front ends for specifying/driving the computation○ Python and C++ today, easy to add more
Core TensorFlow Execution System
CPU GPU Android iOS ...
C++ front end Python front end ...
![Page 25: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/25.jpg)
MatMul
Add Relu
biases
weights
examples
labels
Xent
Graph of Nodes, also called Operations or ops.
Computation is a dataflow graph
![Page 26: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/26.jpg)
with tensors
MatMul
Add Relu
biases
weights
examples
labels
Xent
Edges are N-dimensional arrays: Tensors
Computation is a dataflow graph
![Page 27: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/27.jpg)
● Build a graph computing a neural net inference.
import tensorflow as tf
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets('MNIST_data', one_hot=True)
x = tf.placeholder("float", shape=[None, 784])
W = tf.Variable(tf.zeros([784,10]))
b = tf.Variable(tf.zeros([10]))
y = tf.nn.softmax(tf.matmul(x, W) + b)
Example TensorFlow fragment
![Page 28: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/28.jpg)
with state
Add Mul
biases
...
learning rate
−=...
'Biases' is a variable −= updates biasesSome ops compute gradients
Computation is a dataflow graph
![Page 29: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/29.jpg)
● Automatically add ops to calculate symbolic gradients of variables w.r.t. loss function.
● Apply these gradients with an optimization algorithm
y_ = tf.placeholder(tf.float32, [None, 10])
cross_entropy = -tf.reduce_sum(y_ * tf.log(y))
opt = tf.train.GradientDescentOptimizer(0.01)
train_op = opt.minimize(cross_entropy)
Symbolic Differentiation
![Page 30: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/30.jpg)
● Launch the graph and run the training ops in a loop
init = tf.initialize_all_variables()
sess = tf.Session()
sess.run(init)
for i in range(1000):
batch_xs, batch_ys = mnist.train.next_batch(100)
sess.run(train_step, feed_dict={x: batch_xs, y_: batch_ys})
Define graph and then execute it repeatedly
![Page 31: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/31.jpg)
GPU 0 CPU
Add Mul
biases
learning rate
AssignSub
......
distributedComputation is a dataflow graph
![Page 32: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/32.jpg)
GPU 0 CPU
Add Mul
biases
learning rate
AssignSub
......
Assign Devices to Ops● TensorFlow inserts Send/Recv Ops to transport tensors across devices● Recv ops pull data from Send ops
Send Recv
![Page 33: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/33.jpg)
GPU 0 CPU
Add Mul
biases
learning rate
AssignSub
......
Assign Devices to Ops● TensorFlow inserts Send/Recv Ops to transport tensors across devices● Recv ops pull data from Send ops
Send Recv
Send Recv
Send
Recv SendRecv
![Page 34: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/34.jpg)
November 2015
![Page 35: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/35.jpg)
December 2015
![Page 36: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/36.jpg)
February 2016
![Page 37: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/37.jpg)
April 2016
![Page 38: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/38.jpg)
June 2016
![Page 39: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/39.jpg)
Activity
![Page 40: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/40.jpg)
Experiment Turnaround Time and Research Productivity
● Minutes, Hours:○ Interactive research! Instant gratification!
● 1-4 days○ Tolerable○ Interactivity replaced by running many experiments in parallel
● 1-4 weeks○ High value experiments only○ Progress stalls
● >1 month○ Don’t even try
![Page 41: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/41.jpg)
![Page 42: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/42.jpg)
Data Parallelism
Parameter Servers
...ModelReplicas
Data ...
![Page 43: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/43.jpg)
Data Parallelism
Parameter Servers
...ModelReplicas
Data ...
p
![Page 44: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/44.jpg)
Data Parallelism
Parameter Servers
...ModelReplicas
Data ...
p∆p
![Page 45: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/45.jpg)
Data Parallelism
Parameter Servers
...ModelReplicas
Data ...
p∆p
p’ = p + ∆p
![Page 46: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/46.jpg)
Data Parallelism
Parameter Servers
...ModelReplicas
Data ...
p’
p’ = p + ∆p
![Page 47: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/47.jpg)
Data Parallelism
Parameter Servers
...ModelReplicas
Data ...
p’∆p’
![Page 48: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/48.jpg)
Data Parallelism
Parameter Servers
...ModelReplicas
Data ...
p’∆p’
p’’ = p’ + ∆p
![Page 49: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/49.jpg)
Data Parallelism
Parameter Servers
...ModelReplicas
Data ...
p’∆p’
p’’ = p’ + ∆p
![Page 50: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/50.jpg)
Distributed training mechanisms
Graph structure and low-level graph primitives (queues) allow us to play with synchronous vs. asynchronous update algorithms.
![Page 51: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/51.jpg)
/job:worker/cpu:0 /job:ps/gpu:0
Add Mul
biases
learning rate
AssignSub
......
Cross process communication is the same!● Communication across machines over the network abstracted identically to
cross device communication.
Send Recv
Send Recv
Send
Recv SendRecv
No specialized parameter server subsystem!
![Page 52: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/52.jpg)
Image Model Training Time
Hours
10 GPUs50 GPUs
1 GPU
![Page 53: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/53.jpg)
Hours
2.6 hours vs. 79.3 hours (30.5X)
10 GPUs50 GPUs
1 GPU
Image Model Training Time
![Page 54: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/54.jpg)
Sync converges faster (time to accuracy)
Synchronous updates (with backup workers) trains to higher accuracy fasterBetter scaling to more workers (less loss of accuracy)
Revisiting Distributed Synchronous SGD, Jianmin Chen, Rajat Monga, Samy Bengio, Raal Jozefowicz, ICLR Workshop 2016, arxiv.org/abs/1604.00981
![Page 55: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/55.jpg)
Sync converges faster (time to accuracy)
Synchronous updates (with backup workers) trains to higher accuracy fasterBetter scaling to more workers (less loss of accuracy)
Revisiting Distributed Synchronous SGD, Jianmin Chen, Rajat Monga, Samy Bengio, Raal Jozefowicz, ICLR Workshop 2016, arxiv.org/abs/1604.00981
40 hours vs. 50 hours
![Page 56: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/56.jpg)
General ComputationsAlthough we originally built TensorFlow for our uses around deep neural networks, it’s actually quite flexible
Wide variety of machine learning and other kinds of numeric computations easily expressible in the computation graph model
![Page 57: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/57.jpg)
phones single machines (CPU and/or GPUs) …
distributed systems of 100sof machines and/or GPU cards
Runs on Variety of Platforms
custom ML hardware
![Page 58: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/58.jpg)
Trend: Much More Heterogeneous hardwareGeneral purpose CPU performance scaling has slowed significantly
Specialization of hardware for certain workloads will be more important
![Page 59: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/59.jpg)
Tensor Processing UnitCustom machine learning ASIC
In production use for >16 months: used on every search query, used for AlphaGo match, ...
See Google Cloud Platform blog: Google supercharges machine learning tasks with TPU custom chip,by Norm Jouppi, May, 2016
![Page 60: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/60.jpg)
Long Short-Term Memory (LSTMs):Make Your Memory Cells Differentiable[Hochreiter & Schmidhuber, 1997]
MX YMX Y
WRITE? READ?
FORGET?
W R
F
Sigmoids
![Page 61: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/61.jpg)
Example: LSTM [Hochreiter et al, 1997][Gers et al, 1999]
Enableslong termdependencies to flow
![Page 62: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/62.jpg)
Example: LSTM
for i in range(20): m, c = LSTMCell(x[i], mprev, cprev) mprev = m cprev = c
![Page 63: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/63.jpg)
Example: Deep LSTM
for i in range(20): for d in range(4): # d is depth input = x[i] if d is 0 else m[d-1] m[d], c[d] = LSTMCell(input, mprev[d], cprev[d]) mprev[d] = m[d] cprev[d] = c[d]
![Page 64: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/64.jpg)
Example: Deep LSTM
for i in range(20): for d in range(4): # d is depth input = x[i] if d is 0 else m[d-1] m[d], c[d] = LSTMCell(input, mprev[d], cprev[d]) mprev[d] = m[d] cprev[d] = c[d]
![Page 65: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/65.jpg)
Example: Deep LSTM
for i in range(20): for d in range(4): # d is depth with tf.device("/gpu:%d" % d): input = x[i] if d is 0 else m[d-1] m[d], c[d] = LSTMCell(input, mprev[d], cprev[d]) mprev[d] = m[d] cprev[d] = c[d]
![Page 66: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/66.jpg)
A B C D __ A B C
A B C D
GPU1
GPU2
GPU3
GPU4
A B C D
GPU5
GPU6
1000 LSTM cells2000 dims pertimestep
2000 x 4 = 8k dims persentence
80k softmax by1000 dimsThis is very big!
Split softmax into4 GPUs
![Page 67: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/67.jpg)
A B C D __ A B C
A B C D 80k softmax by1000 dimsThis is very big!
Split softmax into4 GPUs
1000 LSTM cells2000 dims pertimestep
2000 x 4 = 8k dims persentence
GPU1
GPU2
GPU3
GPU4
A B C D
GPU5
GPU6
![Page 68: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/68.jpg)
A B C D __ A B C
A B C D 80k softmax by1000 dimsThis is very big!
Split softmax into4 GPUs
1000 LSTM cells2000 dims pertimestep
2000 x 4 = 8k dims persentence
GPU1
GPU2
GPU3
GPU4
A B C D
GPU5
GPU6
![Page 69: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/69.jpg)
A B C D __ A B C
A B C D 80k softmax by1000 dimsThis is very big!
Split softmax into4 GPUs
1000 LSTM cells2000 dims pertimestep
2000 x 4 = 8k dims persentence
GPU1
GPU2
GPU3
GPU4
A B C D
GPU5
GPU6
![Page 70: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/70.jpg)
A B C D __ A B C
A B C D 80k softmax by1000 dimsThis is very big!
Split softmax into4 GPUs
1000 LSTM cells2000 dims pertimestep
2000 x 4 = 8k dims persentence
GPU1
GPU2
GPU3
GPU4
A B C D
GPU5
GPU6
![Page 71: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/71.jpg)
A B C D __ A B C
A B C D 80k softmax by1000 dimsThis is very big!
Split softmax into4 GPUs
1000 LSTM cells2000 dims pertimestep
2000 x 4 = 8k dims persentence
GPU1
GPU2
GPU3
GPU4
A B C D
GPU5
GPU6
![Page 72: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/72.jpg)
A B C D __ A B C
A B C D 80k softmax by1000 dimsThis is very big!
Split softmax into4 GPUs
1000 LSTM cells2000 dims pertimestep
2000 x 4 = 8k dims persentence
GPU1
GPU2
GPU3
GPU4
A B C D
GPU5
GPU6
![Page 73: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/73.jpg)
A B C D __ A B C
A B C D 80k softmax by1000 dimsThis is very big!
Split softmax into4 GPUs
1000 LSTM cells2000 dims pertimestep
2000 x 4 = 8k dims persentence
GPU1
GPU2
GPU3
GPU4
A B C D
GPU5
GPU6
![Page 74: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/74.jpg)
A B C D __ A B C
A B C D 80k softmax by1000 dimsThis is very big!
Split softmax into4 GPUs
1000 LSTM cells2000 dims pertimestep
2000 x 4 = 8k dims persentence
GPU1
GPU2
GPU3
GPU4
A B C D
GPU5
GPU6
![Page 75: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/75.jpg)
A B C D __ A B C
A B C D 80k softmax by1000 dimsThis is very big!
Split softmax into4 GPUs
1000 LSTM cells2000 dims pertimestep
2000 x 4 = 8k dims persentence
GPU1
GPU2
GPU3
GPU4
A B C D
GPU5
GPU6
![Page 76: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/76.jpg)
A B C D __ A B C
A B C D 80k softmax by1000 dimsThis is very big!
Split softmax into4 GPUs
1000 LSTM cells2000 dims pertimestep
2000 x 4 = 8k dims persentence
GPU1
GPU2
GPU3
GPU4
A B C D
GPU5
GPU6
![Page 77: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/77.jpg)
A B C D __ A B C
A B C D 80k softmax by1000 dimsThis is very big!
Split softmax into4 GPUs
1000 LSTM cells2000 dims pertimestep
2000 x 4 = 8k dims persentence
GPU1
GPU2
GPU3
GPU4
A B C D
GPU5
GPU6
![Page 78: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/78.jpg)
What are some ways thatdeep learning is having
a significant impact at Google?
All of these examples implemented using TensorFlowor our predecessor system
![Page 79: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/79.jpg)
“How cold is it outside?”
DeepRecurrent
Neural NetworkAcoustic Input Text Output
Reduced word errors by more than 30%
Speech Recognition
Google Research Blog - August 2012, August 2015
![Page 80: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/80.jpg)
The Inception Architecture (GoogLeNet, 2014)
Going Deeper with Convolutions
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, Andrew Rabinovich
ArXiv 2014, CVPR 2015
![Page 81: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/81.jpg)
Team Year Place Error (top-5)
XRCE (pre-neural-net explosion) 2011 1st 25.8%
Supervision (AlexNet) 2012 1st 16.4%
Clarifai 2013 1st 11.7%
GoogLeNet (Inception) 2014 1st 6.66%
Andrej Karpathy (human) 2014 N/A 5.1%
BN-Inception (Arxiv) 2015 N/A 4.9%
Inception-v3 (Arxiv) 2015 N/A 3.46%
Neural Nets: Rapid Progress in Image Recognition
ImageNet challenge classification task
![Page 82: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/82.jpg)
“ocean”Deep
ConvolutionalNeural Network
Your Photo
Automatic Tag
Search personal photos without tags.
Google Photos Search
Google Research Blog - June 2013
![Page 83: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/83.jpg)
Google Photos Search
![Page 84: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/84.jpg)
Reuse same model for completely different problems
Same basic model structuretrained on different data,
useful in completely different contexts
Example: given image → predict interesting pixels
![Page 85: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/85.jpg)
![Page 86: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/86.jpg)
![Page 87: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/87.jpg)
We have tons of vision problems
Image search, StreetView, Satellite Imagery, Translation, Robotics, Self-driving Cars,
www.google.com/sunroof
![Page 88: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/88.jpg)
MEDICAL IMAGING
Very good results using similar model for detecting diabetic retinopathy in retinal images
![Page 89: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/89.jpg)
“Seeing” Go
![Page 90: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/90.jpg)
Score for doc,query
pair
DeepNeural
NetworkQuery & document features
Query: “car parts for sale”,
Doc: “Rebuilt transmissions …”
Launched in 2015Third most important search ranking signal (of 100s)
RankBrain in Google Search Ranking
Bloomberg, Oct 2015: “Google Turning Its Lucrative Web Search Over to AI Machines”
![Page 91: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/91.jpg)
Sequence-to-Sequence Model
A B C
v
D __ X Y Z
X Y Z Q
Input sequence
Target sequence
[Sutskever & Vinyals & Le NIPS 2014]
Deep LSTM
![Page 92: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/92.jpg)
Sequence-to-Sequence Model: Machine Translation
v
Input sentence
Target sentence
[Sutskever & Vinyals & Le NIPS 2014] How
Quelle est taille?votre <EOS>
![Page 93: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/93.jpg)
Sequence-to-Sequence Model: Machine Translation
v
Input sentence
Target sentence
[Sutskever & Vinyals & Le NIPS 2014] How
Quelle est taille?votre <EOS>
tall
How
![Page 94: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/94.jpg)
Sequence-to-Sequence Model: Machine Translation
v
Input sentence
Target sentence
[Sutskever & Vinyals & Le NIPS 2014] How tall are
Quelle est taille?votre <EOS> How tall
![Page 95: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/95.jpg)
Sequence-to-Sequence Model: Machine Translation
v
Input sentence
Target sentence
[Sutskever & Vinyals & Le NIPS 2014] How tall you?are
Quelle est taille?votre <EOS> How aretall
![Page 96: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/96.jpg)
Sequence-to-Sequence Model: Machine Translation
v
Input sentence
[Sutskever & Vinyals & Le NIPS 2014]
At inference time:Beam search to choose most
probableover possible output sequences
Quelle est taille?votre <EOS>
![Page 97: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/97.jpg)
April 1, 2009: April Fool’s Day joke
Nov 5, 2015: Launched Real Product
Feb 1, 2016: >10% of mobile Inbox replies
Smart Reply
![Page 98: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/98.jpg)
Small Feed-Forward
Neural Network
Incoming Email
ActivateSmart Reply?
yes/no
Smart Reply Google Research Blog- Nov 2015
![Page 99: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/99.jpg)
Small Feed-Forward
Neural Network
Incoming Email
ActivateSmart Reply?
Deep Recurrent
Neural Network
Generated Replies
yes/no
Smart Reply Google Research Blog- Nov 2015
![Page 100: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/100.jpg)
Image Captioning
W __ A young girl
A young girl asleep[Vinyals et al., CVPR 2015]
![Page 101: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/101.jpg)
Model: A close up of a child holding a stuffed animal.
Human: A young girl asleep on the sofa cuddling a stuffed bear.
Model: A baby is asleep next to a teddy bear.
Image Captions Research
![Page 102: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/102.jpg)
![Page 103: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/103.jpg)
Combining Vision with Robotics
“Deep Learning for Robots: Learning from Large-Scale Interaction”, Google Research Blog, March, 2016
“Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection”, Sergey Levine, Peter Pastor, Alex Krizhevsky, & Deirdre Quillen,Arxiv, arxiv.org/abs/1603.02199
![Page 104: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/104.jpg)
How Can You Get Started with Machine Learning?Three ways, with varying complexity:
(1) Use a Cloud-based API (Vision, Speech, etc.)(2) Use an existing model architecture, and
retrain it or fine tune on your dataset(3) Develop your own machine learning models
for new problems
More flexible,
but more effort
required
![Page 105: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/105.jpg)
Use Cloud-based APIs
cloud.google.com/translate
cloud.google.com/speech
cloud.google.com/vision
cloud.google.com/text
![Page 106: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/106.jpg)
Use Cloud-based APIs
cloud.google.com/translate
cloud.google.com/speech
cloud.google.com/vision
cloud.google.com/text
![Page 107: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/107.jpg)
Google Cloud Vision APIhttps://cloud.google.com/vision/
![Page 108: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/108.jpg)
Google Cloud MLScaled service for training and inference w/TensorFlow
![Page 109: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/109.jpg)
A Few TensorFlow Community Examples(From more than 2000 results for ‘tensorflow’ on GitHub)
● DQN: github.com/nivwusquorum/tensorflow-deepq
● NeuralArt: github.com/woodrush/neural-art-tf
● Char RNN: github.com/sherjilozair/char-rnn-tensorflow
● Keras ported to TensorFlow: github.com/fchollet/keras
● Show and Tell: github.com/jazzsaxmafia/show_and_tell.tensorflow
● Mandarin translation: github.com/jikexueyuanwiki/tensorflow-zh
...
![Page 110: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/110.jpg)
A Few TensorFlow Community Examples(From more than 2000 2100 results for ‘tensorflow’ on GitHub)
● DQN: github.com/nivwusquorum/tensorflow-deepq
● NeuralArt: github.com/woodrush/neural-art-tf
● Char RNN: github.com/sherjilozair/char-rnn-tensorflow
● Keras ported to TensorFlow: github.com/fchollet/keras
● Show and Tell: github.com/jazzsaxmafia/show_and_tell.tensorflow
● Mandarin translation: github.com/jikexueyuanwiki/tensorflow-zh
...
![Page 112: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/112.jpg)
github.com/woodrush/neural-art-tf
![Page 113: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/113.jpg)
github.com/sherjilozair/char-rnn-tensorflow
![Page 115: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/115.jpg)
github.com/jazzsaxmafia/show_and_tell.tensorflow
![Page 116: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/116.jpg)
github.com/jikexueyuanwiki/tensorflow-zh
![Page 117: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/117.jpg)
What Does the Future Hold?Deep learning usage will continue to grow and accelerate:
● Across more and more fields and problems:○ robotics, self-driving vehicles, ...○ health care○ video understanding○ dialogue systems○ personal assistance○ ...
![Page 118: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/118.jpg)
ConclusionsDeep neural networks are making significant strides in understanding:In speech, vision, language, search, robotics, …
If you’re not considering how to use deep neural nets to solve your vision or understanding problems, you almost certainly should be
![Page 119: With TensorFlow Google Brain team Large-Scale Deep Learning …ordonez/ppt/deepnet-tensorflow.pdf · With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many](https://reader033.fdocuments.us/reader033/viewer/2022050523/5fa6bb6cd7978e3410398f85/html5/thumbnails/119.jpg)
Further Reading● Dean, et al., Large Scale Distributed Deep Networks, NIPS 2012,
research.google.com/archive/large_deep_networks_nips2012.html. ● Mikolov, Chen, Corrado & Dean. Efficient Estimation of Word Representations in Vector
Space, NIPS 2013, arxiv.org/abs/1301.3781.● Sutskever, Vinyals, & Le, Sequence to Sequence Learning with Neural Networks, NIPS,
2014, arxiv.org/abs/1409.3215. ● Vinyals, Toshev, Bengio, & Erhan. Show and Tell: A Neural Image Caption Generator.
CVPR 2015. arxiv.org/abs/1411.4555● TensorFlow white paper, tensorflow.org/whitepaper2015.pdf (clickable links in bibliography)
g.co/brain (We’re hiring! Also check out Brain Residency program at g.co/brainresidency)www.tensorflow.orgresearch.google.com/people/jeffresearch.google.com/pubs/BrainTeam.html
Questions?