Microscopic Advances with Large-Scale Learning: Stochastic...
Transcript of Microscopic Advances with Large-Scale Learning: Stochastic...
![Page 1: Microscopic Advances with Large-Scale Learning: Stochastic ...alipunjani/pdf/NIPS14-MLCB-Presentation.pdf · Microscopic Advances with Large-Scale Learning: Stochastic Optimization](https://reader033.fdocuments.us/reader033/viewer/2022051920/600cfe86fd8f456b9665dab7/html5/thumbnails/1.jpg)
Microscopic Advances with Large-Scale Learning: Stochastic Optimization for Cryo-EM
Ali Punjani, Marcus Brubaker
University of Toronto Department of Computer Science
![Page 2: Microscopic Advances with Large-Scale Learning: Stochastic ...alipunjani/pdf/NIPS14-MLCB-Presentation.pdf · Microscopic Advances with Large-Scale Learning: Stochastic Optimization](https://reader033.fdocuments.us/reader033/viewer/2022051920/600cfe86fd8f456b9665dab7/html5/thumbnails/2.jpg)
Structure Determination
} Macromolecules
} Protein structure determines function
} Traditional approaches:
} X-ray Crystallography
} NMR Spectroscopy
![Page 3: Microscopic Advances with Large-Scale Learning: Stochastic ...alipunjani/pdf/NIPS14-MLCB-Presentation.pdf · Microscopic Advances with Large-Scale Learning: Stochastic Optimization](https://reader033.fdocuments.us/reader033/viewer/2022051920/600cfe86fd8f456b9665dab7/html5/thumbnails/3.jpg)
Electron Cryo-Microscopy (Cryo-EM)
} No crystals needed, large molecules and complexes
Low dose electron beam
Particles in unknown 3D pose
Ice
Transfer Function
Corrupted Noisy Integral Projections Film/CCD
Computational Task: Recover 3D Electron Density
![Page 4: Microscopic Advances with Large-Scale Learning: Stochastic ...alipunjani/pdf/NIPS14-MLCB-Presentation.pdf · Microscopic Advances with Large-Scale Learning: Stochastic Optimization](https://reader033.fdocuments.us/reader033/viewer/2022051920/600cfe86fd8f456b9665dab7/html5/thumbnails/4.jpg)
Cryo-EM Image Formation
} Challenges for reconstruction: } Destructive CTF } Low SNR } Unknown pose
Low dose electron beam
Particles in unknown 3D pose
Ice
Transfer Function
Corrupted Noisy Integral Projections Film/CCD
Corruption by CTF
=
2D Particle Images
![Page 5: Microscopic Advances with Large-Scale Learning: Stochastic ...alipunjani/pdf/NIPS14-MLCB-Presentation.pdf · Microscopic Advances with Large-Scale Learning: Stochastic Optimization](https://reader033.fdocuments.us/reader033/viewer/2022051920/600cfe86fd8f456b9665dab7/html5/thumbnails/5.jpg)
Cryo-EM Image Formation
K
p(I|✓,R, t,V) = N (I|StC✓PRV,�2I)
I
✓Rt
V
![Page 6: Microscopic Advances with Large-Scale Learning: Stochastic ...alipunjani/pdf/NIPS14-MLCB-Presentation.pdf · Microscopic Advances with Large-Scale Learning: Stochastic Optimization](https://reader033.fdocuments.us/reader033/viewer/2022051920/600cfe86fd8f456b9665dab7/html5/thumbnails/6.jpg)
Cryo-EM Image Formation
K
p(I|✓,R, t,V) = N (I|StC✓PRV,�2I)
I
✓Rt
V
Linear
Voxels
Integral Projection
![Page 7: Microscopic Advances with Large-Scale Learning: Stochastic ...alipunjani/pdf/NIPS14-MLCB-Presentation.pdf · Microscopic Advances with Large-Scale Learning: Stochastic Optimization](https://reader033.fdocuments.us/reader033/viewer/2022051920/600cfe86fd8f456b9665dab7/html5/thumbnails/7.jpg)
Cryo-EM Image Formation
K
p(I|✓,R, t,V) = N (I|StC✓PRV,�2I)
I
✓Rt
V
p(I|✓,R, t, V) = N (I|StC✓PRV,�2I)
In Fourier Domain:
Diagonal
Linear
Voxels
Fourier Coefficients
Integral Projection
Slicing
![Page 8: Microscopic Advances with Large-Scale Learning: Stochastic ...alipunjani/pdf/NIPS14-MLCB-Presentation.pdf · Microscopic Advances with Large-Scale Learning: Stochastic Optimization](https://reader033.fdocuments.us/reader033/viewer/2022051920/600cfe86fd8f456b9665dab7/html5/thumbnails/8.jpg)
Marginalization for Latent Variables
K
I
✓Rt
V
p(I|✓, V) =Z
R2
Z
SO(3)p(I|✓,R, t, V)p(R)p(t)dRdt
![Page 9: Microscopic Advances with Large-Scale Learning: Stochastic ...alipunjani/pdf/NIPS14-MLCB-Presentation.pdf · Microscopic Advances with Large-Scale Learning: Stochastic Optimization](https://reader033.fdocuments.us/reader033/viewer/2022051920/600cfe86fd8f456b9665dab7/html5/thumbnails/9.jpg)
Marginalization for Latent Variables
K
I
✓Rt
V
p(I|✓, V) =Z
R2
Z
SO(3)p(I|✓,R, t, V)p(R)p(t)dRdt
} Numerical Quadrature
⇡MX
j=1
wjp(I|✓,Rj , tj , V)
![Page 10: Microscopic Advances with Large-Scale Learning: Stochastic ...alipunjani/pdf/NIPS14-MLCB-Presentation.pdf · Microscopic Advances with Large-Scale Learning: Stochastic Optimization](https://reader033.fdocuments.us/reader033/viewer/2022051920/600cfe86fd8f456b9665dab7/html5/thumbnails/10.jpg)
Maximum-a-Posteriori Estimation
K
I
✓Rt
V
p(V|D) / p(V)KY
i=1
p(Ii|✓i, V)
} Point Estimates for R, t: Projection Matching
} Expectation-Maximization: RELION (Scheres 2012)
![Page 11: Microscopic Advances with Large-Scale Learning: Stochastic ...alipunjani/pdf/NIPS14-MLCB-Presentation.pdf · Microscopic Advances with Large-Scale Learning: Stochastic Optimization](https://reader033.fdocuments.us/reader033/viewer/2022051920/600cfe86fd8f456b9665dab7/html5/thumbnails/11.jpg)
Optimization Problem
K
I
✓Rt
V
p(V|D) / p(V)KY
i=1
p(Ii|✓i, V)
argmin
V�
KX
i=1
⇣log p(˜I|✓, ˜V) +K�1
log p(V)⌘
![Page 12: Microscopic Advances with Large-Scale Learning: Stochastic ...alipunjani/pdf/NIPS14-MLCB-Presentation.pdf · Microscopic Advances with Large-Scale Learning: Stochastic Optimization](https://reader033.fdocuments.us/reader033/viewer/2022051920/600cfe86fd8f456b9665dab7/html5/thumbnails/12.jpg)
Stochastic Optimization for Cryo-EM
argmin
V�
KX
i=1
⇣log p(˜I|✓, ˜V) +K�1
log p(V)⌘
} Expensive to compute objective with large K
} Stochastic Optimization:
} Approximate objective with subset of images
} Update based on approximate gradient
} Various Algorithms (vary by update rule)
} Advantages: speed, random initialization
![Page 13: Microscopic Advances with Large-Scale Learning: Stochastic ...alipunjani/pdf/NIPS14-MLCB-Presentation.pdf · Microscopic Advances with Large-Scale Learning: Stochastic Optimization](https://reader033.fdocuments.us/reader033/viewer/2022051920/600cfe86fd8f456b9665dab7/html5/thumbnails/13.jpg)
Experiments: Datasets
} Real Dataset: } 46K Images of ATP Synthase from Thermus Thermophilius
} Low SNR and known CTF parameters
![Page 14: Microscopic Advances with Large-Scale Learning: Stochastic ...alipunjani/pdf/NIPS14-MLCB-Presentation.pdf · Microscopic Advances with Large-Scale Learning: Stochastic Optimization](https://reader033.fdocuments.us/reader033/viewer/2022051920/600cfe86fd8f456b9665dab7/html5/thumbnails/14.jpg)
Experiments: Datasets
} Synthetic Dataset: } 50,000 Projections of known artificial density
} Low SNR and realistic CTF parameters
![Page 15: Microscopic Advances with Large-Scale Learning: Stochastic ...alipunjani/pdf/NIPS14-MLCB-Presentation.pdf · Microscopic Advances with Large-Scale Learning: Stochastic Optimization](https://reader033.fdocuments.us/reader033/viewer/2022051920/600cfe86fd8f456b9665dab7/html5/thumbnails/15.jpg)
Experiments: Seven Methods
} Vanilla Stochastic Gradient Descent (SGD)
} Momentum Methods:
} Classical Momentum
} Nesterov’s Accelerated Gradient
} Adaptive Methods: } AdaGrad
} TONGA
} Quasi-Second Order Methods:
} Online L-BFGS
} Hessian Free
![Page 16: Microscopic Advances with Large-Scale Learning: Stochastic ...alipunjani/pdf/NIPS14-MLCB-Presentation.pdf · Microscopic Advances with Large-Scale Learning: Stochastic Optimization](https://reader033.fdocuments.us/reader033/viewer/2022051920/600cfe86fd8f456b9665dab7/html5/thumbnails/16.jpg)
Experiments: Results
} Identical random initialization in all experiments
![Page 17: Microscopic Advances with Large-Scale Learning: Stochastic ...alipunjani/pdf/NIPS14-MLCB-Presentation.pdf · Microscopic Advances with Large-Scale Learning: Stochastic Optimization](https://reader033.fdocuments.us/reader033/viewer/2022051920/600cfe86fd8f456b9665dab7/html5/thumbnails/17.jpg)
Experiments: Results
} Simplest Method
![Page 18: Microscopic Advances with Large-Scale Learning: Stochastic ...alipunjani/pdf/NIPS14-MLCB-Presentation.pdf · Microscopic Advances with Large-Scale Learning: Stochastic Optimization](https://reader033.fdocuments.us/reader033/viewer/2022051920/600cfe86fd8f456b9665dab7/html5/thumbnails/18.jpg)
Experiments: Results
} Momentum Method
![Page 19: Microscopic Advances with Large-Scale Learning: Stochastic ...alipunjani/pdf/NIPS14-MLCB-Presentation.pdf · Microscopic Advances with Large-Scale Learning: Stochastic Optimization](https://reader033.fdocuments.us/reader033/viewer/2022051920/600cfe86fd8f456b9665dab7/html5/thumbnails/19.jpg)
Experiments: Results
} Adaptive Step-size
![Page 20: Microscopic Advances with Large-Scale Learning: Stochastic ...alipunjani/pdf/NIPS14-MLCB-Presentation.pdf · Microscopic Advances with Large-Scale Learning: Stochastic Optimization](https://reader033.fdocuments.us/reader033/viewer/2022051920/600cfe86fd8f456b9665dab7/html5/thumbnails/20.jpg)
Experiments: Results
} Quasi-second order
![Page 21: Microscopic Advances with Large-Scale Learning: Stochastic ...alipunjani/pdf/NIPS14-MLCB-Presentation.pdf · Microscopic Advances with Large-Scale Learning: Stochastic Optimization](https://reader033.fdocuments.us/reader033/viewer/2022051920/600cfe86fd8f456b9665dab7/html5/thumbnails/21.jpg)
Experiments: Results
} Qualitatively Similar
} Reasonable in one pass through data
![Page 22: Microscopic Advances with Large-Scale Learning: Stochastic ...alipunjani/pdf/NIPS14-MLCB-Presentation.pdf · Microscopic Advances with Large-Scale Learning: Stochastic Optimization](https://reader033.fdocuments.us/reader033/viewer/2022051920/600cfe86fd8f456b9665dab7/html5/thumbnails/22.jpg)
Experiments: Results
![Page 23: Microscopic Advances with Large-Scale Learning: Stochastic ...alipunjani/pdf/NIPS14-MLCB-Presentation.pdf · Microscopic Advances with Large-Scale Learning: Stochastic Optimization](https://reader033.fdocuments.us/reader033/viewer/2022051920/600cfe86fd8f456b9665dab7/html5/thumbnails/23.jpg)
Experiments: Results
![Page 24: Microscopic Advances with Large-Scale Learning: Stochastic ...alipunjani/pdf/NIPS14-MLCB-Presentation.pdf · Microscopic Advances with Large-Scale Learning: Stochastic Optimization](https://reader033.fdocuments.us/reader033/viewer/2022051920/600cfe86fd8f456b9665dab7/html5/thumbnails/24.jpg)
Experiments: Comparison
Projection Matching RELION (E-M) Proposed Approach
3 Hours – 1 Epochs
![Page 25: Microscopic Advances with Large-Scale Learning: Stochastic ...alipunjani/pdf/NIPS14-MLCB-Presentation.pdf · Microscopic Advances with Large-Scale Learning: Stochastic Optimization](https://reader033.fdocuments.us/reader033/viewer/2022051920/600cfe86fd8f456b9665dab7/html5/thumbnails/25.jpg)
Experiments: Comparison
Projection Matching
24 Hours – 5 Epochs
RELION (E-M)
24 Hours – 5 Epochs
Proposed Approach
3 Hours – 1 Epochs
![Page 26: Microscopic Advances with Large-Scale Learning: Stochastic ...alipunjani/pdf/NIPS14-MLCB-Presentation.pdf · Microscopic Advances with Large-Scale Learning: Stochastic Optimization](https://reader033.fdocuments.us/reader033/viewer/2022051920/600cfe86fd8f456b9665dab7/html5/thumbnails/26.jpg)
Experiments: Comparison
Projection Matching
24 Hours – 5 Epochs
RELION (E-M)
24 Hours – 5 Epochs
Proposed Approach
3 Hours – 1 Epochs
![Page 27: Microscopic Advances with Large-Scale Learning: Stochastic ...alipunjani/pdf/NIPS14-MLCB-Presentation.pdf · Microscopic Advances with Large-Scale Learning: Stochastic Optimization](https://reader033.fdocuments.us/reader033/viewer/2022051920/600cfe86fd8f456b9665dab7/html5/thumbnails/27.jpg)
Experiments: Comparison
} Random Initialization is difficult for other methods
Projection Matching
24 Hours – 5 Epochs
RELION (E-M)
24 Hours – 5 Epochs
Proposed Approach
3 Hours – 1 Epochs
![Page 28: Microscopic Advances with Large-Scale Learning: Stochastic ...alipunjani/pdf/NIPS14-MLCB-Presentation.pdf · Microscopic Advances with Large-Scale Learning: Stochastic Optimization](https://reader033.fdocuments.us/reader033/viewer/2022051920/600cfe86fd8f456b9665dab7/html5/thumbnails/28.jpg)
Conclusions
} Introduced Cryo-EM Structure Determination
} Stochastic Optimization solution
} Simple methods are best
} State of the art speed and robustness
![Page 29: Microscopic Advances with Large-Scale Learning: Stochastic ...alipunjani/pdf/NIPS14-MLCB-Presentation.pdf · Microscopic Advances with Large-Scale Learning: Stochastic Optimization](https://reader033.fdocuments.us/reader033/viewer/2022051920/600cfe86fd8f456b9665dab7/html5/thumbnails/29.jpg)
Recent Progress
} Higher resolution reconstructions
} Importance Sampling: 100,000x speedup
![Page 30: Microscopic Advances with Large-Scale Learning: Stochastic ...alipunjani/pdf/NIPS14-MLCB-Presentation.pdf · Microscopic Advances with Large-Scale Learning: Stochastic Optimization](https://reader033.fdocuments.us/reader033/viewer/2022051920/600cfe86fd8f456b9665dab7/html5/thumbnails/30.jpg)
Recent Progress
} Higher resolution reconstructions
} Importance Sampling: 100,000x speedup
} Forward: } Heterogeneous mixtures of particles } Better priors } Video exposure