Joint Operations Command Enhancing Simulation Interoperability UNCLASSIFIED Allan Deacon JCTC.
Some slides and images are taken from: David Wolfe Corne...
Transcript of Some slides and images are taken from: David Wolfe Corne...
![Page 1: Some slides and images are taken from: David Wolfe Corne ...helper.ipam.ucla.edu/publications/eltut/eltut_14769.pdf · Noé and Nüske: SIAM MMS 11, 635-655 (2013) Nüske et al: JCTC](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3b987a692db516e66a9550/html5/thumbnails/1.jpg)
Introduction to Machine learning
Some slides and images are taken from:
David Wolfe Corne https://www.macs.hw.ac.uk/~dwcorne/Teaching/introdl.pptWikipediaGeoffrey A. Hinton
![Page 2: Some slides and images are taken from: David Wolfe Corne ...helper.ipam.ucla.edu/publications/eltut/eltut_14769.pdf · Noé and Nüske: SIAM MMS 11, 635-655 (2013) Nüske et al: JCTC](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3b987a692db516e66a9550/html5/thumbnails/2.jpg)
Examples 1
![Page 3: Some slides and images are taken from: David Wolfe Corne ...helper.ipam.ucla.edu/publications/eltut/eltut_14769.pdf · Noé and Nüske: SIAM MMS 11, 635-655 (2013) Nüske et al: JCTC](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3b987a692db516e66a9550/html5/thumbnails/3.jpg)
WaveNet
Examples 1
![Page 4: Some slides and images are taken from: David Wolfe Corne ...helper.ipam.ucla.edu/publications/eltut/eltut_14769.pdf · Noé and Nüske: SIAM MMS 11, 635-655 (2013) Nüske et al: JCTC](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3b987a692db516e66a9550/html5/thumbnails/4.jpg)
WaveNet
Examples 1
![Page 5: Some slides and images are taken from: David Wolfe Corne ...helper.ipam.ucla.edu/publications/eltut/eltut_14769.pdf · Noé and Nüske: SIAM MMS 11, 635-655 (2013) Nüske et al: JCTC](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3b987a692db516e66a9550/html5/thumbnails/5.jpg)
Examples 2
![Page 6: Some slides and images are taken from: David Wolfe Corne ...helper.ipam.ucla.edu/publications/eltut/eltut_14769.pdf · Noé and Nüske: SIAM MMS 11, 635-655 (2013) Nüske et al: JCTC](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3b987a692db516e66a9550/html5/thumbnails/6.jpg)
• Basics of Machine Learning
• Feedforward nets • Backpropagation • Convolutional networks
• Boltzmann machines • Deep Belief Nets
![Page 7: Some slides and images are taken from: David Wolfe Corne ...helper.ipam.ucla.edu/publications/eltut/eltut_14769.pdf · Noé and Nüske: SIAM MMS 11, 635-655 (2013) Nüske et al: JCTC](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3b987a692db516e66a9550/html5/thumbnails/7.jpg)
Supervised learning 1 – Function approximation
Shallow ML methods Deep ML methods
• Regression • Kernel Regression
x y = F(x)
• Feedforward networks • Deep tensor networks • ConvNets
![Page 8: Some slides and images are taken from: David Wolfe Corne ...helper.ipam.ucla.edu/publications/eltut/eltut_14769.pdf · Noé and Nüske: SIAM MMS 11, 635-655 (2013) Nüske et al: JCTC](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3b987a692db516e66a9550/html5/thumbnails/8.jpg)
Supervised learning 2 – Classification
![Page 9: Some slides and images are taken from: David Wolfe Corne ...helper.ipam.ucla.edu/publications/eltut/eltut_14769.pdf · Noé and Nüske: SIAM MMS 11, 635-655 (2013) Nüske et al: JCTC](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3b987a692db516e66a9550/html5/thumbnails/9.jpg)
Supervised learning 2 – Classification
Shallow ML methods Deep ML methods
• Support Vector Machines (SVMs) • Kernel SVMs
Inputs Classes
![Page 10: Some slides and images are taken from: David Wolfe Corne ...helper.ipam.ucla.edu/publications/eltut/eltut_14769.pdf · Noé and Nüske: SIAM MMS 11, 635-655 (2013) Nüske et al: JCTC](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3b987a692db516e66a9550/html5/thumbnails/10.jpg)
Unsupervised learning 1 – Learning distributions
![Page 11: Some slides and images are taken from: David Wolfe Corne ...helper.ipam.ucla.edu/publications/eltut/eltut_14769.pdf · Noé and Nüske: SIAM MMS 11, 635-655 (2013) Nüske et al: JCTC](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3b987a692db516e66a9550/html5/thumbnails/11.jpg)
Unsupervised learning 1 – Learning distributions
• Markov random fields
Shallow ML methods Deep ML methods
• Boltzmann machines • Deep Belief Nets • variational Autoencoders
![Page 12: Some slides and images are taken from: David Wolfe Corne ...helper.ipam.ucla.edu/publications/eltut/eltut_14769.pdf · Noé and Nüske: SIAM MMS 11, 635-655 (2013) Nüske et al: JCTC](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3b987a692db516e66a9550/html5/thumbnails/12.jpg)
Unsupervised learning 2 – Classification
Shallow ML methods Deep ML methods
• Clustering • Deep transformation + clustering
![Page 13: Some slides and images are taken from: David Wolfe Corne ...helper.ipam.ucla.edu/publications/eltut/eltut_14769.pdf · Noé and Nüske: SIAM MMS 11, 635-655 (2013) Nüske et al: JCTC](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3b987a692db516e66a9550/html5/thumbnails/13.jpg)
1. Linear least squares regression Example: Markov state models and Koopman models
2. Validation, cross-validation and hyperparameter selection
3. Regularization and sparsity Example: Sindy
4. Kernel ridge regression
Plan for the first talk
![Page 14: Some slides and images are taken from: David Wolfe Corne ...helper.ipam.ucla.edu/publications/eltut/eltut_14769.pdf · Noé and Nüske: SIAM MMS 11, 635-655 (2013) Nüske et al: JCTC](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3b987a692db516e66a9550/html5/thumbnails/14.jpg)
Linear least squares regression
![Page 15: Some slides and images are taken from: David Wolfe Corne ...helper.ipam.ucla.edu/publications/eltut/eltut_14769.pdf · Noé and Nüske: SIAM MMS 11, 635-655 (2013) Nüske et al: JCTC](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3b987a692db516e66a9550/html5/thumbnails/15.jpg)
Regression
![Page 16: Some slides and images are taken from: David Wolfe Corne ...helper.ipam.ucla.edu/publications/eltut/eltut_14769.pdf · Noé and Nüske: SIAM MMS 11, 635-655 (2013) Nüske et al: JCTC](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3b987a692db516e66a9550/html5/thumbnails/16.jpg)
Regression
![Page 17: Some slides and images are taken from: David Wolfe Corne ...helper.ipam.ucla.edu/publications/eltut/eltut_14769.pdf · Noé and Nüske: SIAM MMS 11, 635-655 (2013) Nüske et al: JCTC](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3b987a692db516e66a9550/html5/thumbnails/17.jpg)
Regression
![Page 18: Some slides and images are taken from: David Wolfe Corne ...helper.ipam.ucla.edu/publications/eltut/eltut_14769.pdf · Noé and Nüske: SIAM MMS 11, 635-655 (2013) Nüske et al: JCTC](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3b987a692db516e66a9550/html5/thumbnails/18.jpg)
Linear least squares
![Page 19: Some slides and images are taken from: David Wolfe Corne ...helper.ipam.ucla.edu/publications/eltut/eltut_14769.pdf · Noé and Nüske: SIAM MMS 11, 635-655 (2013) Nüske et al: JCTC](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3b987a692db516e66a9550/html5/thumbnails/19.jpg)
Normal equations
![Page 20: Some slides and images are taken from: David Wolfe Corne ...helper.ipam.ucla.edu/publications/eltut/eltut_14769.pdf · Noé and Nüske: SIAM MMS 11, 635-655 (2013) Nüske et al: JCTC](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3b987a692db516e66a9550/html5/thumbnails/20.jpg)
Normal equations
Solution methods:1)
do not literally do this!
![Page 21: Some slides and images are taken from: David Wolfe Corne ...helper.ipam.ucla.edu/publications/eltut/eltut_14769.pdf · Noé and Nüske: SIAM MMS 11, 635-655 (2013) Nüske et al: JCTC](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3b987a692db516e66a9550/html5/thumbnails/21.jpg)
Normal equations
Solution methods:2) Cholesky decomposition of CXX (still numerically unstable, but sometimes useful)
3) Orthogonolization of X (e.g. via Householder reflections) — stable
4) SVD of X to perform the pseudoinversion — stable
![Page 22: Some slides and images are taken from: David Wolfe Corne ...helper.ipam.ucla.edu/publications/eltut/eltut_14769.pdf · Noé and Nüske: SIAM MMS 11, 635-655 (2013) Nüske et al: JCTC](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3b987a692db516e66a9550/html5/thumbnails/22.jpg)
Markov state models and Koopman models of molecular dynamics
![Page 23: Some slides and images are taken from: David Wolfe Corne ...helper.ipam.ucla.edu/publications/eltut/eltut_14769.pdf · Noé and Nüske: SIAM MMS 11, 635-655 (2013) Nüske et al: JCTC](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3b987a692db516e66a9550/html5/thumbnails/23.jpg)
Markov state models and Koopman models of molecular dynamics
![Page 24: Some slides and images are taken from: David Wolfe Corne ...helper.ipam.ucla.edu/publications/eltut/eltut_14769.pdf · Noé and Nüske: SIAM MMS 11, 635-655 (2013) Nüske et al: JCTC](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3b987a692db516e66a9550/html5/thumbnails/24.jpg)
Markov state models and Koopman models of molecular dynamics
![Page 25: Some slides and images are taken from: David Wolfe Corne ...helper.ipam.ucla.edu/publications/eltut/eltut_14769.pdf · Noé and Nüske: SIAM MMS 11, 635-655 (2013) Nüske et al: JCTC](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3b987a692db516e66a9550/html5/thumbnails/25.jpg)
Markov state models
![Page 26: Some slides and images are taken from: David Wolfe Corne ...helper.ipam.ucla.edu/publications/eltut/eltut_14769.pdf · Noé and Nüske: SIAM MMS 11, 635-655 (2013) Nüske et al: JCTC](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3b987a692db516e66a9550/html5/thumbnails/26.jpg)
Koopman models
Koopman model:
Eigenvalues and eigenvectors of K have been used to perform dimension reduction in:
VAC (variational approach of conformation dynamics)
EDMD (Extended dynamic mode decomposition)
TICA (time-lagged independent component analysis)
Wu et al: JCP 146, 154104 (2017)
Noé and Nüske: SIAM MMS 11, 635-655 (2013)Nüske et al: JCTC 10, 1739-1752. (2014)
Williams, Kevrekidis and Rowley: J. Nonlinear Sci. 25, 1307-1346 (2015)
Perez-Hernandez et al: JCP 139, 015102 (2013)Schwantes and Pande: JCTC 9, 2000-2009 (2013)
![Page 27: Some slides and images are taken from: David Wolfe Corne ...helper.ipam.ucla.edu/publications/eltut/eltut_14769.pdf · Noé and Nüske: SIAM MMS 11, 635-655 (2013) Nüske et al: JCTC](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3b987a692db516e66a9550/html5/thumbnails/27.jpg)
Validation, cross-validation and hyperparameter selection
![Page 28: Some slides and images are taken from: David Wolfe Corne ...helper.ipam.ucla.edu/publications/eltut/eltut_14769.pdf · Noé and Nüske: SIAM MMS 11, 635-655 (2013) Nüske et al: JCTC](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3b987a692db516e66a9550/html5/thumbnails/28.jpg)
Validation, cross-validation and hyperparameter selection
![Page 29: Some slides and images are taken from: David Wolfe Corne ...helper.ipam.ucla.edu/publications/eltut/eltut_14769.pdf · Noé and Nüske: SIAM MMS 11, 635-655 (2013) Nüske et al: JCTC](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3b987a692db516e66a9550/html5/thumbnails/29.jpg)
Validation
![Page 30: Some slides and images are taken from: David Wolfe Corne ...helper.ipam.ucla.edu/publications/eltut/eltut_14769.pdf · Noé and Nüske: SIAM MMS 11, 635-655 (2013) Nüske et al: JCTC](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3b987a692db516e66a9550/html5/thumbnails/30.jpg)
Validation
![Page 31: Some slides and images are taken from: David Wolfe Corne ...helper.ipam.ucla.edu/publications/eltut/eltut_14769.pdf · Noé and Nüske: SIAM MMS 11, 635-655 (2013) Nüske et al: JCTC](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3b987a692db516e66a9550/html5/thumbnails/31.jpg)
Validation
![Page 32: Some slides and images are taken from: David Wolfe Corne ...helper.ipam.ucla.edu/publications/eltut/eltut_14769.pdf · Noé and Nüske: SIAM MMS 11, 635-655 (2013) Nüske et al: JCTC](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3b987a692db516e66a9550/html5/thumbnails/32.jpg)
Cross-validation
![Page 33: Some slides and images are taken from: David Wolfe Corne ...helper.ipam.ucla.edu/publications/eltut/eltut_14769.pdf · Noé and Nüske: SIAM MMS 11, 635-655 (2013) Nüske et al: JCTC](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3b987a692db516e66a9550/html5/thumbnails/33.jpg)
Cross-validation
![Page 34: Some slides and images are taken from: David Wolfe Corne ...helper.ipam.ucla.edu/publications/eltut/eltut_14769.pdf · Noé and Nüske: SIAM MMS 11, 635-655 (2013) Nüske et al: JCTC](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3b987a692db516e66a9550/html5/thumbnails/34.jpg)
Hyperparameter selection
![Page 35: Some slides and images are taken from: David Wolfe Corne ...helper.ipam.ucla.edu/publications/eltut/eltut_14769.pdf · Noé and Nüske: SIAM MMS 11, 635-655 (2013) Nüske et al: JCTC](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3b987a692db516e66a9550/html5/thumbnails/35.jpg)
Regularization and sparsity
![Page 36: Some slides and images are taken from: David Wolfe Corne ...helper.ipam.ucla.edu/publications/eltut/eltut_14769.pdf · Noé and Nüske: SIAM MMS 11, 635-655 (2013) Nüske et al: JCTC](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3b987a692db516e66a9550/html5/thumbnails/36.jpg)
L2 / Tihonov regularization / Ridge regression
![Page 37: Some slides and images are taken from: David Wolfe Corne ...helper.ipam.ucla.edu/publications/eltut/eltut_14769.pdf · Noé and Nüske: SIAM MMS 11, 635-655 (2013) Nüske et al: JCTC](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3b987a692db516e66a9550/html5/thumbnails/37.jpg)
Sparsity-inducing regularization
L0 (too hard)
L1 (LASSO)
Elastic net
Good approach in practice: - Use L1 regularization and annihilate small elements with thresholding.- Select regularization hyperparameters with cross-validation.
![Page 38: Some slides and images are taken from: David Wolfe Corne ...helper.ipam.ucla.edu/publications/eltut/eltut_14769.pdf · Noé and Nüske: SIAM MMS 11, 635-655 (2013) Nüske et al: JCTC](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3b987a692db516e66a9550/html5/thumbnails/38.jpg)
Sindy (sparse identification of nonlinear dynamics)
Brunton, Proctor and Kutz: PNAS 113, 3932–3937 (2016)
![Page 39: Some slides and images are taken from: David Wolfe Corne ...helper.ipam.ucla.edu/publications/eltut/eltut_14769.pdf · Noé and Nüske: SIAM MMS 11, 635-655 (2013) Nüske et al: JCTC](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3b987a692db516e66a9550/html5/thumbnails/39.jpg)
Kernel ridge regression
![Page 40: Some slides and images are taken from: David Wolfe Corne ...helper.ipam.ucla.edu/publications/eltut/eltut_14769.pdf · Noé and Nüske: SIAM MMS 11, 635-655 (2013) Nüske et al: JCTC](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3b987a692db516e66a9550/html5/thumbnails/40.jpg)
Kernel ridge regression
Reminder: ridge regression
Kernel formulation
![Page 41: Some slides and images are taken from: David Wolfe Corne ...helper.ipam.ucla.edu/publications/eltut/eltut_14769.pdf · Noé and Nüske: SIAM MMS 11, 635-655 (2013) Nüske et al: JCTC](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3b987a692db516e66a9550/html5/thumbnails/41.jpg)
Kernel ridge regression
Kernel formulation
![Page 42: Some slides and images are taken from: David Wolfe Corne ...helper.ipam.ucla.edu/publications/eltut/eltut_14769.pdf · Noé and Nüske: SIAM MMS 11, 635-655 (2013) Nüske et al: JCTC](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3b987a692db516e66a9550/html5/thumbnails/42.jpg)
Kernel ridge regression
Kernel formulation
![Page 43: Some slides and images are taken from: David Wolfe Corne ...helper.ipam.ucla.edu/publications/eltut/eltut_14769.pdf · Noé and Nüske: SIAM MMS 11, 635-655 (2013) Nüske et al: JCTC](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3b987a692db516e66a9550/html5/thumbnails/43.jpg)
Kernel ridge regression
Key ideas
![Page 44: Some slides and images are taken from: David Wolfe Corne ...helper.ipam.ucla.edu/publications/eltut/eltut_14769.pdf · Noé and Nüske: SIAM MMS 11, 635-655 (2013) Nüske et al: JCTC](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3b987a692db516e66a9550/html5/thumbnails/44.jpg)
Example: polynomial Kernel