Deep Learning with Coherent Nanophotonic Circuits Neural Networks (ANN) 11/8/2017 3 Deep Learning...
Transcript of Deep Learning with Coherent Nanophotonic Circuits Neural Networks (ANN) 11/8/2017 3 Deep Learning...
![Page 1: Deep Learning with Coherent Nanophotonic Circuits Neural Networks (ANN) 11/8/2017 3 Deep Learning with Coherent Nanophotonic Circuits Breakthroughs in deep learning: • Natural Language](https://reader031.fdocuments.us/reader031/viewer/2022020921/5ae363ce7f8b9a495c8d29f1/html5/thumbnails/1.jpg)
Deep Learning with Coherent Nanophotonic Circuits
Yichen Shen, Nicholas Harris, Dirk Englund, Marin Soljacic
Massachusetts Institute of Technology
@ Berkeley, Oct. 2017
11/8/2017 1Deep Learning with Coherent Nanophotonic Circuits
![Page 2: Deep Learning with Coherent Nanophotonic Circuits Neural Networks (ANN) 11/8/2017 3 Deep Learning with Coherent Nanophotonic Circuits Breakthroughs in deep learning: • Natural Language](https://reader031.fdocuments.us/reader031/viewer/2022020921/5ae363ce7f8b9a495c8d29f1/html5/thumbnails/2.jpg)
11/8/2017Deep Learning with Coherent
Nanophotonic Circuits2
Neuromorphic Computing
Biological Neural Networks Artificial Neural Networks
![Page 3: Deep Learning with Coherent Nanophotonic Circuits Neural Networks (ANN) 11/8/2017 3 Deep Learning with Coherent Nanophotonic Circuits Breakthroughs in deep learning: • Natural Language](https://reader031.fdocuments.us/reader031/viewer/2022020921/5ae363ce7f8b9a495c8d29f1/html5/thumbnails/3.jpg)
Artificial Neural Networks (ANN)
11/8/2017 3Deep Learning with Coherent
Nanophotonic Circuits
Breakthroughs in deep learning: • Natural Language
Processing (NLP)• Game Playing (Go, Atari)• Autonomous Vehicles• Control• Ad Placement• Researches (drug discovery,
material study)• Etc.
![Page 4: Deep Learning with Coherent Nanophotonic Circuits Neural Networks (ANN) 11/8/2017 3 Deep Learning with Coherent Nanophotonic Circuits Breakthroughs in deep learning: • Natural Language](https://reader031.fdocuments.us/reader031/viewer/2022020921/5ae363ce7f8b9a495c8d29f1/html5/thumbnails/4.jpg)
Basic Algorithm of ANN
Deep Learning with Coherent
Nanophotonic Circuits411/8/2017
Matrix Multiplication:
Nonlinear Activation:
…
… …
…
1
2
9
![Page 5: Deep Learning with Coherent Nanophotonic Circuits Neural Networks (ANN) 11/8/2017 3 Deep Learning with Coherent Nanophotonic Circuits Breakthroughs in deep learning: • Natural Language](https://reader031.fdocuments.us/reader031/viewer/2022020921/5ae363ce7f8b9a495c8d29f1/html5/thumbnails/5.jpg)
Hardware and Data Enable Deep Learning
11/8/2017Deep Learning with Coherent
Nanophotonic Circuits5
![Page 6: Deep Learning with Coherent Nanophotonic Circuits Neural Networks (ANN) 11/8/2017 3 Deep Learning with Coherent Nanophotonic Circuits Breakthroughs in deep learning: • Natural Language](https://reader031.fdocuments.us/reader031/viewer/2022020921/5ae363ce7f8b9a495c8d29f1/html5/thumbnails/6.jpg)
The Need for SpeedMore Data Bigger Models More need for Computation
11/8/2017Deep Learning with Coherent
Nanophotonic Circuits6
But Moore’s Law is no-longer providing more computation…
The Market:
On clouds:Millions of high power AI processors ($10,000 each) in data centers by 2020
On premise:Billions of compact AI processors needed due to the rise of autonomouse driving, AR and IoT.
Von Neumann ASIC/FPGA Optical Processing
![Page 7: Deep Learning with Coherent Nanophotonic Circuits Neural Networks (ANN) 11/8/2017 3 Deep Learning with Coherent Nanophotonic Circuits Breakthroughs in deep learning: • Natural Language](https://reader031.fdocuments.us/reader031/viewer/2022020921/5ae363ce7f8b9a495c8d29f1/html5/thumbnails/7.jpg)
11/8/2017Deep Learning with Coherent
Nanophotonic Circuits7
Optical AI Computing
![Page 8: Deep Learning with Coherent Nanophotonic Circuits Neural Networks (ANN) 11/8/2017 3 Deep Learning with Coherent Nanophotonic Circuits Breakthroughs in deep learning: • Natural Language](https://reader031.fdocuments.us/reader031/viewer/2022020921/5ae363ce7f8b9a495c8d29f1/html5/thumbnails/8.jpg)
In Deep Learning
Key Operation is dense M x V
11/8/2017Deep Learning with Coherent
Nanophotonic Circuits8
ajbi
Wij= x
In Optics, Matrix Multiplication
is very common & (usually)
consumes no energy !
Convolution / FFT
Matrix
Multiplication
![Page 9: Deep Learning with Coherent Nanophotonic Circuits Neural Networks (ANN) 11/8/2017 3 Deep Learning with Coherent Nanophotonic Circuits Breakthroughs in deep learning: • Natural Language](https://reader031.fdocuments.us/reader031/viewer/2022020921/5ae363ce7f8b9a495c8d29f1/html5/thumbnails/9.jpg)
Programmable Nanophotonic Processors
11/8/2017Deep Learning with Coherent
Nanophotonic Circuits9
100µm
S U ( 4 ) C o r e D M M C
a
b
c
Phase Shifter
Waveguide
Transm
issio
n
Voltage2
O p t i c a l I n t e r f e r e n c e U n i t ( O I U )
MZI
Detectors
Input Modes
φi✓i
J. Mower et al, Physical Reviews A, 92, 032322 (2015)Carolan, Jacques, et al. "Universal linear optics." Science 349.6249 (2015): 711-716.
![Page 10: Deep Learning with Coherent Nanophotonic Circuits Neural Networks (ANN) 11/8/2017 3 Deep Learning with Coherent Nanophotonic Circuits Breakthroughs in deep learning: • Natural Language](https://reader031.fdocuments.us/reader031/viewer/2022020921/5ae363ce7f8b9a495c8d29f1/html5/thumbnails/10.jpg)
ANN does NOT require high resolution
11/8/2017Deep Learning with Coherent
Nanophotonic Circuits10
Sze et al, arXiv:1703.09039 (2017)
![Page 11: Deep Learning with Coherent Nanophotonic Circuits Neural Networks (ANN) 11/8/2017 3 Deep Learning with Coherent Nanophotonic Circuits Breakthroughs in deep learning: • Natural Language](https://reader031.fdocuments.us/reader031/viewer/2022020921/5ae363ce7f8b9a495c8d29f1/html5/thumbnails/11.jpg)
Deep Learning Inference is “Passive”
11/8/2017Deep Learning with Coherent
Nanophotonic Circuits11
Once the Optical Neural Network is trained, no
need to update the weights frequently…
![Page 12: Deep Learning with Coherent Nanophotonic Circuits Neural Networks (ANN) 11/8/2017 3 Deep Learning with Coherent Nanophotonic Circuits Breakthroughs in deep learning: • Natural Language](https://reader031.fdocuments.us/reader031/viewer/2022020921/5ae363ce7f8b9a495c8d29f1/html5/thumbnails/12.jpg)
Deep Learning is very parallelizable
11/8/2017Deep Learning with Coherent
Nanophotonic Circuits12
Multiple wavelengths can be used to
simultaneously execute batch of data
![Page 13: Deep Learning with Coherent Nanophotonic Circuits Neural Networks (ANN) 11/8/2017 3 Deep Learning with Coherent Nanophotonic Circuits Breakthroughs in deep learning: • Natural Language](https://reader031.fdocuments.us/reader031/viewer/2022020921/5ae363ce7f8b9a495c8d29f1/html5/thumbnails/13.jpg)
11/8/2017Deep Learning with Coherent
Nanophotonic Circuits13
Coherent Optical Neural Networks (ONN)
X1
X2
X3
X4
h1(1)
h2(1)
h3(1)
h4(1)
Z (1) = W0X
h1(i)
h2(i)
h3(i)
h4(i)
h1(n)
h2(n)
h3(n)
h4(n)
Y1
Y2
Y3
Y4
h( i ) = f (Z ( i ) ) Y = Wn h(n )
Input LayerHidden Layers
Output Layer
Layer i Layer n
Optical Input Optical Output
Layer 1X Y
a
b
c
x i n xou t
Waveguide
0
0
Optical Interference Unit Optical Nonlinearity Unit
0
0
d
S(n)V (n) U (n)
Vowel X
M (n)M (1) = U (1)S(1)V (1) M (2) M (n− 1)
fNL()fNL()
Photonic Integrated Circuit
fNL
![Page 14: Deep Learning with Coherent Nanophotonic Circuits Neural Networks (ANN) 11/8/2017 3 Deep Learning with Coherent Nanophotonic Circuits Breakthroughs in deep learning: • Natural Language](https://reader031.fdocuments.us/reader031/viewer/2022020921/5ae363ce7f8b9a495c8d29f1/html5/thumbnails/14.jpg)
Optical Vowel Recognition
(4d 4 classes)
11/8/2017Deep Learning with Coherent
Nanophotonic Circuits14
![Page 15: Deep Learning with Coherent Nanophotonic Circuits Neural Networks (ANN) 11/8/2017 3 Deep Learning with Coherent Nanophotonic Circuits Breakthroughs in deep learning: • Natural Language](https://reader031.fdocuments.us/reader031/viewer/2022020921/5ae363ce7f8b9a495c8d29f1/html5/thumbnails/15.jpg)
11/8/2017Deep Learning with Coherent Nanophotonic Circuits 15
Y. Shen and N. Harris et al, Nature Photonics, 11, 441-446 (2017)
da
bLaser OIU Detectors Computer
U1S1V1
Tra
nsm
issio
nOIU1
OIU2 CPU OIU
3OIU
4
U1S1V1fSA ( Iin)
Input Output
Instance
Insta
nce
60µm
Optical Interference Unit
SU(4) Core DMMC
Dq Df
c
V2Dq
18µm
Directional Coupler
Phase ShifterSimulated Optical Nonlinearity
![Page 16: Deep Learning with Coherent Nanophotonic Circuits Neural Networks (ANN) 11/8/2017 3 Deep Learning with Coherent Nanophotonic Circuits Breakthroughs in deep learning: • Natural Language](https://reader031.fdocuments.us/reader031/viewer/2022020921/5ae363ce7f8b9a495c8d29f1/html5/thumbnails/16.jpg)
Experimental Result
11/8/2017Deep Learning with Coherent
Nanophotonic Circuits16
Simulation Result: 165/180=91.7%Experiment Result: 138/180=76.7%
VowelIdentified
VowelIdentified
Vo
wel
Sp
ok
en
Vo
wel
Sp
ok
en
![Page 17: Deep Learning with Coherent Nanophotonic Circuits Neural Networks (ANN) 11/8/2017 3 Deep Learning with Coherent Nanophotonic Circuits Breakthroughs in deep learning: • Natural Language](https://reader031.fdocuments.us/reader031/viewer/2022020921/5ae363ce7f8b9a495c8d29f1/html5/thumbnails/17.jpg)
11/8/2017Deep Learning with Coherent
Nanophotonic Circuits17
Fully Connected Neural Networks
Recurrent Neural Networks Convolutional Neural Networks
![Page 18: Deep Learning with Coherent Nanophotonic Circuits Neural Networks (ANN) 11/8/2017 3 Deep Learning with Coherent Nanophotonic Circuits Breakthroughs in deep learning: • Natural Language](https://reader031.fdocuments.us/reader031/viewer/2022020921/5ae363ce7f8b9a495c8d29f1/html5/thumbnails/18.jpg)
Recurrent Neural Networks
11/8/2017Deep Learning with Coherent
Nanophotonic Circuits18
Commonly used for Speech Recognition and Language Processing
![Page 19: Deep Learning with Coherent Nanophotonic Circuits Neural Networks (ANN) 11/8/2017 3 Deep Learning with Coherent Nanophotonic Circuits Breakthroughs in deep learning: • Natural Language](https://reader031.fdocuments.us/reader031/viewer/2022020921/5ae363ce7f8b9a495c8d29f1/html5/thumbnails/19.jpg)
Convolution Neural Networks
11/8/2017Deep Learning with Coherent
Nanophotonic Circuits19
Scott Skirlo and Yichen Shen et al, Manuscript in Preparation
![Page 20: Deep Learning with Coherent Nanophotonic Circuits Neural Networks (ANN) 11/8/2017 3 Deep Learning with Coherent Nanophotonic Circuits Breakthroughs in deep learning: • Natural Language](https://reader031.fdocuments.us/reader031/viewer/2022020921/5ae363ce7f8b9a495c8d29f1/html5/thumbnails/20.jpg)
Optical Convolutional Neural Network
11/8/2017Deep Learning with Coherent
Nanophotonic Circuits20
Scott Skirlo and Yichen Shen et al, Manuscript in Preparation
![Page 21: Deep Learning with Coherent Nanophotonic Circuits Neural Networks (ANN) 11/8/2017 3 Deep Learning with Coherent Nanophotonic Circuits Breakthroughs in deep learning: • Natural Language](https://reader031.fdocuments.us/reader031/viewer/2022020921/5ae363ce7f8b9a495c8d29f1/html5/thumbnails/21.jpg)
11/8/2017Deep Learning with Coherent
Nanophotonic Circuits21
Unified Buffer (localStorage SRAM)
ADC arrays
*modified block diagram from TPU architecture
![Page 22: Deep Learning with Coherent Nanophotonic Circuits Neural Networks (ANN) 11/8/2017 3 Deep Learning with Coherent Nanophotonic Circuits Breakthroughs in deep learning: • Natural Language](https://reader031.fdocuments.us/reader031/viewer/2022020921/5ae363ce7f8b9a495c8d29f1/html5/thumbnails/22.jpg)
Speed and Energy Efficiency Comparison with Electrical ANN
11/8/2017Deep Learning with Coherent
Nanophotonic Circuits22
NVIDIA TITAN X ONN (with thermal PS)
Architecture Von Neumann Neuromorphic
Power Consumption 1 kW 1-2 kW
Operation Speed 10 TFLOP 10,000 TFLOP
Y. Shen and N. Harris et al, Nature Photonics 11, 441 (2017)
![Page 23: Deep Learning with Coherent Nanophotonic Circuits Neural Networks (ANN) 11/8/2017 3 Deep Learning with Coherent Nanophotonic Circuits Breakthroughs in deep learning: • Natural Language](https://reader031.fdocuments.us/reader031/viewer/2022020921/5ae363ce7f8b9a495c8d29f1/html5/thumbnails/23.jpg)
11/8/2017Deep Learning with Coherent Nanophotonic Circuits 23
![Page 24: Deep Learning with Coherent Nanophotonic Circuits Neural Networks (ANN) 11/8/2017 3 Deep Learning with Coherent Nanophotonic Circuits Breakthroughs in deep learning: • Natural Language](https://reader031.fdocuments.us/reader031/viewer/2022020921/5ae363ce7f8b9a495c8d29f1/html5/thumbnails/24.jpg)
Some History on Optical Neural Networks
11/8/2017Deep Learning with Coherent
Nanophotonic Circuits24
![Page 25: Deep Learning with Coherent Nanophotonic Circuits Neural Networks (ANN) 11/8/2017 3 Deep Learning with Coherent Nanophotonic Circuits Breakthroughs in deep learning: • Natural Language](https://reader031.fdocuments.us/reader031/viewer/2022020921/5ae363ce7f8b9a495c8d29f1/html5/thumbnails/25.jpg)
Acknowledgement
11/8/2017Deep Learning with Coherent
Nanophotonic Circuits25
Nicholas HarrisPhD, EECSMIT
Scott SkirloPhD, Physics MIT
Prof. Marin SoljacicPhysics, MIT
Prof. Dirk EnglundEECS, MIT
Li JingPhD, Physics MIT
![Page 26: Deep Learning with Coherent Nanophotonic Circuits Neural Networks (ANN) 11/8/2017 3 Deep Learning with Coherent Nanophotonic Circuits Breakthroughs in deep learning: • Natural Language](https://reader031.fdocuments.us/reader031/viewer/2022020921/5ae363ce7f8b9a495c8d29f1/html5/thumbnails/26.jpg)
11/8/2017Deep Learning with Coherent
Nanophotonic Circuits26
Optical Convolutional Neural Network
Scott Skirlo and Yichen Shen et al, Manuscript in Preparation
![Page 27: Deep Learning with Coherent Nanophotonic Circuits Neural Networks (ANN) 11/8/2017 3 Deep Learning with Coherent Nanophotonic Circuits Breakthroughs in deep learning: • Natural Language](https://reader031.fdocuments.us/reader031/viewer/2022020921/5ae363ce7f8b9a495c8d29f1/html5/thumbnails/27.jpg)
Nonlinearity
11/8/2017Deep Learning with Coherent
Nanophotonic Circuits27
A. Selden, British Journal of Applied Physics 18 , 743 (1967)M. Soljacic, Physical Review E 66, 055601 (2002)Z. Cheng et al, IEEE Journal of Selected Topics in Quantum Electronics 20.1 (2014): 43-48.
Nonlinear Photonic System
Iin Iout=f(Iin)
Saturable Absorption Photodiode
For Deep Learning, the constraint on nonlinearity is weak