Fast Robustness Quantification with Variational...
Transcript of Fast Robustness Quantification with Variational...
![Page 1: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/1.jpg)
Fast Robustness Quantification with Variational Bayes
ITT Career Development Assistant Professor,
MIT
Tamara Broderick
With: Ryan Giordano, Rachael Meager, Jonathan Huggins, Michael I. Jordan
![Page 2: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/2.jpg)
• Bayesian inference • Complex, modular models; posterior distribution
• Have to express prior beliefs in a distribution: challenges • Time-consuming; subjective; complex models
1
![Page 3: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/3.jpg)
• Bayesian inference • Complex, modular models; posterior distribution
• Have to express prior beliefs in a distribution: challenges • Time-consuming; subjective; complex models
1
![Page 4: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/4.jpg)
• Bayesian inference • Complex, modular models; posterior distribution
• Have to express prior beliefs in a distribution: challenges • Time-consuming; subjective; complex models
1
![Page 5: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/5.jpg)
• Bayesian inference • Complex, modular models; posterior distribution
• Have to express prior beliefs in a distribution: challenges • Time-consuming; subjective; complex models
1
p(✓|x) /✓ p(x|✓)p(✓)
![Page 6: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/6.jpg)
• Bayesian inference • Complex, modular models; posterior distribution
• Have to express prior beliefs in a distribution: challenges • Time-consuming; subjective; complex models
1
p(✓|x) /✓ p(x|✓)p(✓)
![Page 7: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/7.jpg)
• Bayesian inference • Complex, modular models; posterior distribution
• Have to express prior beliefs in a distribution: challenges • Time-consuming; subjective; complex models
1
p(✓|x) /✓ p(x|✓)p(✓)
![Page 8: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/8.jpg)
• Bayesian inference • Complex, modular models; posterior distribution
• Have to express prior beliefs in a distribution: challenges • Time-consuming; subjective; complex models
1
p(✓|x) /✓ p(x|✓)p(✓)
![Page 9: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/9.jpg)
• Bayesian inference • Complex, modular models; posterior distribution
• Have to express prior beliefs in a distribution: challenges • Time-consuming; subjective; complex models
1
p(✓|x) /✓ p(x|✓)p(✓)
![Page 10: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/10.jpg)
• Bayesian inference • Complex, modular models; posterior distribution
• Have to express prior beliefs in a distribution: challenges • Time-consuming; subjective; complex models
1
p(✓|x) /✓ p(x|✓)p(✓)
![Page 11: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/11.jpg)
• Bayesian inference • Complex, modular models; posterior distribution
• Have to express prior beliefs in a distribution: challenges • Time-consuming; subjective; complex models
1
p(✓|x) /✓ p(x|✓)p(✓)
Bayes Theorem
![Page 12: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/12.jpg)
• Bayesian inference • Complex, modular models; posterior distribution
• Have to express prior beliefs in a distribution: challenges • Time-consuming; subjective; complex models
1
p(✓|x) /✓ p(x|✓)p(✓)
Bayes Theorem
![Page 13: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/13.jpg)
• Bayesian inference • Complex, modular models; posterior distribution
• Have to express prior beliefs in a distribution: challenges • Time-consuming; subjective; complex models
1
p(✓|x) /✓ p(x|✓)p(✓)
Bayes Theorem
![Page 14: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/14.jpg)
• Bayesian inference • Complex, modular models; posterior distribution
• Have to express prior beliefs in a distribution: challenges • Time-consuming; subjective; complex models
1
p(✓|x) /✓ p(x|✓)p(✓)
Bayes Theorem
![Page 15: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/15.jpg)
• Bayesian inference • Complex, modular models; posterior distribution
• Have to express prior beliefs in a distribution: challenges • Time-consuming; subjective; complex models
Robustness quantification
1
p(✓|x) /✓ p(x|✓)p(✓)
Bayes Theorem
![Page 16: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/16.jpg)
• Robustness • Global & local • Rarely used • Approximation,
MCMC • Our solution: linear
response variational Bayes
• Bayesian inference • Complex, modular models; posterior distribution
• Have to express prior beliefs in a distribution: challenges • Time-consuming; subjective; complex models
Robustness quantification
1
p(✓|x) /✓ p(x|✓)p(✓)
Bayes Theorem
![Page 17: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/17.jpg)
• Robustness • Global & local • Rarely used • Approximation,
MCMC • Our solution: linear
response variational Bayes
• Bayesian inference • Complex, modular models; posterior distribution
• Have to express prior beliefs in a distribution: challenges • Time-consuming; subjective; complex models
Robustness quantification
1
p(✓|x) /✓ p(x|✓)p(✓)
Bayes Theorem
![Page 18: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/18.jpg)
• Robustness • Global & local • Rarely used • Approximation,
MCMC • Our solution: linear
response variational Bayes
• Bayesian inference • Complex, modular models; posterior distribution
• Have to express prior beliefs in a distribution: challenges • Time-consuming; subjective; complex models
Robustness quantification
1
p(✓|x) /✓ p(x|✓)p(✓)
Bayes Theorem
![Page 19: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/19.jpg)
• Bayesian inference • Complex, modular models; posterior distribution
• Have to express prior beliefs in a distribution: challenges • Time-consuming; subjective; complex models
Robustness quantification
1
p(✓|x) /✓ p(x|✓)p(✓)
• Robustness • Global & local • Rarely used • Approximation
MCMC • Our solution: linear
response variational Bayes
Bayes Theorem
![Page 20: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/20.jpg)
• Bayesian inference • Complex, modular models; posterior distribution
• Have to express prior beliefs in a distribution: challenges • Time-consuming; subjective; complex models
Robustness quantification
1
p(✓|x) /✓ p(x|✓)p(✓)
• Robustness • Global & local • Rarely used • Approximation,
MCMC • Our solution: linear
response variational Bayes
Bayes Theorem
![Page 21: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/21.jpg)
• Bayesian inference • Complex, modular models; posterior distribution
• Have to express prior beliefs in a distribution: challenges • Time-consuming; subjective; complex models
Robustness quantification
1
p(✓|x) /✓ p(x|✓)p(✓)
• Robustness • Global & local • Rarely used • Approximation,
MCMC • Our solution: linear
response variational Bayes
Bayes Theorem
![Page 22: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/22.jpg)
• Bayesian inference • Complex, modular models; posterior distribution
• Have to express prior beliefs in a distribution: challenges • Time-consuming; subjective; complex models
Robustness quantification
1
p(✓|x) /✓ p(x|✓)p(✓)
• Robustness • Global & local • Rarely used • Approximation,
MCMC • Our solution: linear
response variational Bayes
Bayes Theorem
![Page 23: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/23.jpg)
• Variational Bayes as an alternative to MCMC • Challenges of VB • Accurate uncertainties from VB • Accurate robustness quantification from VB • Big idea: derivatives/perturbations are easy in VB
2
Robustness quantification
![Page 24: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/24.jpg)
• Variational Bayes as an alternative to MCMC • Challenges of VB • Accurate uncertainties from VB • Accurate robustness quantification from VB • Big idea: derivatives/perturbations are easy in VB
2
Robustness quantification
![Page 25: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/25.jpg)
• Variational Bayes as an alternative to MCMC • Challenges of VB • Accurate uncertainties from VB • Accurate robustness quantification from VB • Big idea: derivatives/perturbations are easy in VB
2
Robustness quantification
![Page 26: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/26.jpg)
• Variational Bayes as an alternative to MCMC • Challenges of VB • Accurate uncertainties from VB • Accurate robustness quantification from VB • Big idea: derivatives/perturbations are easy in VB
2
Robustness quantification
![Page 27: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/27.jpg)
• Variational Bayes as an alternative to MCMC • Challenges of VB • Accurate uncertainties from VB • Accurate robustness quantification from VB • Big idea: derivatives/perturbations are easy in VB
2
Robustness quantification
![Page 28: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/28.jpg)
• Variational Bayes as an alternative to MCMC • Challenges of VB • Accurate uncertainties from VB • Accurate robustness quantification from VB • Big idea: derivatives/perturbations are easy in VB
2
Robustness quantification
![Page 29: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/29.jpg)
• Variational Bayes (VB) • Approximation for
posterior • Minimize Kullback-Liebler
(KL) divergence:
p(✓|x)
KL(qkp(·|x))
• VB practical success • point estimates and prediction • fast
p(✓|x)q(✓)
q⇤(✓)
Variational Bayes
3
![Page 30: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/30.jpg)
• Variational Bayes (VB) • Approximation for
posterior • Minimize Kullback-Liebler
(KL) divergence:
p(✓|x)
KL(qkp(·|x))
• VB practical success • point estimates and prediction • fast
p(✓|x)q(✓)
q⇤(✓)
Variational Bayes
3
![Page 31: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/31.jpg)
• Variational Bayes (VB) • Approximation for
posterior • Minimize Kullback-Liebler
(KL) divergence:
p(✓|x)
KL(qkp(·|x))
• VB practical success • point estimates and prediction • fast
p(✓|x)q(✓)
q⇤(✓)
Variational Bayes
3
![Page 32: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/32.jpg)
• Variational Bayes (VB) • Approximation for
posterior • Minimize Kullback-Liebler
(KL) divergence:
p(✓|x)
KL(qkp(·|x))
• VB practical success • point estimates and prediction • fast
q⇤(✓)
Variational Bayes
3
![Page 33: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/33.jpg)
• Variational Bayes (VB) • Approximation for
posterior • Minimize Kullback-Liebler
(KL) divergence:
p(✓|x)
KL(qkp(·|x))
• VB practical success • point estimates and prediction • fast
q⇤(✓)
Variational Bayes
q(✓)
3
![Page 34: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/34.jpg)
q(✓)
• Variational Bayes (VB) • Approximation for
posterior • Minimize Kullback-Liebler
(KL) divergence:
p(✓|x)
KL(qkp(·|x))
• VB practical success • point estimates and prediction • fast
q⇤(✓)
Variational Bayes
p(✓|x)
3
![Page 35: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/35.jpg)
• Variational Bayes (VB) • Approximation for
posterior • Minimize Kullback-Liebler
(KL) divergence:
p(✓|x)
KL(qkp(·|x))
• VB practical success • point estimates and prediction • fast
q⇤(✓)
p(✓|x)
q⇤(✓)
Variational Bayes
3
![Page 36: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/36.jpg)
• Variational Bayes (VB) • Approximation for
posterior • Minimize Kullback-Liebler
(KL) divergence:
p(✓|x)
KL(qkp(·|x))
• VB practical success • point estimates and prediction • fast
q⇤(✓)
Variational Bayes
p(✓|x)
q⇤(✓)
3
![Page 37: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/37.jpg)
• Variational Bayes (VB) • Approximation for
posterior • Minimize Kullback-Liebler
(KL) divergence:
p(✓|x)
KL(qkp(·|x))
• VB practical success • point estimates and prediction • fast
q⇤(✓)
Variational Bayes
p(✓|x)
q⇤(✓)
3
![Page 38: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/38.jpg)
• Variational Bayes (VB) • Approximation for
posterior • Minimize Kullback-Liebler
(KL) divergence:
p(✓|x)
KL(qkp(·|x))
• VB practical success • point estimates and prediction • fast
q⇤(✓)
Variational Bayes
p(✓|x)
q⇤(✓)
3
![Page 39: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/39.jpg)
• Variational Bayes (VB) • Approximation for
posterior • Minimize Kullback-Liebler
(KL) divergence:
p(✓|x)
KL(qkp(·|x))
• VB practical success • point estimates and prediction • fast
q⇤(✓)
Variational Bayes
p(✓|x)
q⇤(✓)
3
![Page 40: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/40.jpg)
• Variational Bayes (VB) • Approximation for
posterior • Minimize Kullback-Liebler
(KL) divergence:
p(✓|x)
KL(qkp(·|x))
• VB practical success • point estimates and prediction • fast
q⇤(✓)
[Broderick, Boyd, Wibisono, Wilson, Jordan 2013]
Variational Bayes
p(✓|x)
q⇤(✓)
3
![Page 41: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/41.jpg)
• Variational Bayes (VB) • Approximation for
posterior • Minimize Kullback-Liebler
(KL) divergence:
p(✓|x)
KL(qkp(·|x))
• VB practical success • point estimates and prediction • fast, streaming, distributed
q⇤(✓)
[Broderick, Boyd, Wibisono, Wilson, Jordan 2013]
Variational Bayes
p(✓|x)
q⇤(✓)
3
![Page 42: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/42.jpg)
• Variational Bayes
!
• Mean-field variational Bayes (MFVB)
!
• Underestimates variance (sometimes severely)
• No covariance estimates
What about uncertainty?
[Bishop 2006]
q(✓) =JY
j=1
q(✓j)
KL(q||p(·|x)) =Z
✓q(✓) log
q(✓)
p(✓|x)d✓
✓1
✓2
4
![Page 43: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/43.jpg)
• Variational Bayes
!
• Mean-field variational Bayes (MFVB)
!
• Underestimates variance (sometimes severely)
• No covariance estimates
What about uncertainty?
q(✓) =JY
j=1
q(✓j)
4
![Page 44: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/44.jpg)
• Variational Bayes
!
• Mean-field variational Bayes (MFVB)
!
• Underestimates variance (sometimes severely)
• No covariance estimates
What about uncertainty?
[Bishop 2006]
q(✓) =JY
j=1
q(✓j)
✓1
✓2p(✓|x)
4
![Page 45: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/45.jpg)
• Variational Bayes
!
• Mean-field variational Bayes (MFVB)
!
• Underestimates variance (sometimes severely)
• No covariance estimates
What about uncertainty?
[Bishop 2006]
q(✓) =JY
j=1
q(✓j)
KL(q||p(·|x)) =Z
✓q(✓) log
q(✓)
p(✓|x)d✓
✓1
✓2p(✓|x)
4
![Page 46: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/46.jpg)
• Variational Bayes
!
• Mean-field variational Bayes (MFVB)
!
• Underestimates variance (sometimes severely)
• No covariance estimates
What about uncertainty?
[Bishop 2006]
q(✓) =JY
j=1
q(✓j)
KL(q||p(·|x)) =Z
✓q(✓) log
q(✓)
p(✓|x)d✓
✓1
✓2p(✓|x)
4
![Page 47: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/47.jpg)
• Variational Bayes
!
• Mean-field variational Bayes (MFVB)
!
• Underestimates variance (sometimes severely)
• No covariance estimates
What about uncertainty?
[Bishop 2006]
q(✓) =JY
j=1
q(✓j)
KL(q||p(·|x)) =Z
✓q(✓) log
q(✓)
p(✓|x)d✓
✓1
✓2p(✓|x)
q⇤(✓)
4
![Page 48: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/48.jpg)
• Variational Bayes
!
• Mean-field variational Bayes (MFVB)
!
• Underestimates variance (sometimes severely)
• No covariance estimates
What about uncertainty?
[Bishop 2006]
q(✓) =JY
j=1
q(✓j)
KL(q||p(·|x)) =Z
✓q(✓) log
q(✓)
p(✓|x)d✓
✓1
✓2p(✓|x)
q⇤(✓)
4
![Page 49: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/49.jpg)
• Variational Bayes
!
• Mean-field variational Bayes (MFVB)
!
• Underestimates variance (sometimes severely)
• No covariance estimates
What about uncertainty?
[Bishop 2006]
q(✓) =JY
j=1
q(✓j)
KL(q||p(·|x)) =Z
✓q(✓) log
q(✓)
p(✓|x)d✓
✓1
✓2p(✓|x)
q⇤(✓)
4
![Page 50: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/50.jpg)
• Variational Bayes
!
• Mean-field variational Bayes (MFVB)
!
• Underestimates variance (sometimes severely)
• No covariance estimates
What about uncertainty?
q(✓) =JY
j=1
q(✓j)
KL(q||p(·|x)) =Z
✓q(✓) log
q(✓)
p(✓|x)d✓
✓1
✓2
[MacKay 2003; Bishop 2006; Wang, Titterington 2004; Turner, Sahani 2011]
p(✓|x)
q⇤(✓)
4
![Page 51: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/51.jpg)
• Variational Bayes
!
• Mean-field variational Bayes (MFVB)
!
• Underestimates variance (sometimes severely)
• No covariance estimates
What about uncertainty?
q(✓) =JY
j=1
q(✓j)
KL(q||p(·|x)) =Z
✓q(✓) log
q(✓)
p(✓|x)d✓
✓1
✓2
[MacKay 2003; Bishop 2006; Wang, Titterington 2004; Turner, Sahani 2011]
p(✓|x)
q⇤(✓)
[Dunson 2014; Bardenet, Doucet, Holmes 2015]4
![Page 52: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/52.jpg)
• Cumulant-generating function
!
• True posterior covariance vs MFVB covariance
!
• “Linear response” !
• The LRVB approximation
V :=d2
dtT dtCq⇤(t)
����t=0
Linear response
mean =d
dtC(t)
����t=0
⌃ :=d2
dtT dtC
p(·|x)(t)
����t=0
log pt(✓) := log p(✓|x) + t
T✓ � C(t), MFVB q⇤t
⌃ =d
dtTEpt✓
����t=0
⇡ d
dtTEq⇤t ✓
����t=0
=: ⌃
[Bishop 2006]
C(t) := logEetT ✓
p(✓|x)
q⇤(✓)
5
![Page 53: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/53.jpg)
• Cumulant-generating function
!
• True posterior covariance vs MFVB covariance
!
• “Linear response” !
• The LRVB approximation
V :=d2
dtT dtCq⇤(t)
����t=0
Linear response
mean =d
dtC(t)
����t=0
⌃ :=d2
dtT dtC
p(·|x)(t)
����t=0
log pt(✓) := log p(✓|x) + t
T✓ � C(t), MFVB q⇤t
⌃ =d
dtTEpt✓
����t=0
⇡ d
dtTEq⇤t ✓
����t=0
=: ⌃
[Bishop 2006]
C(t) := logEetT ✓
p(✓|x)
q⇤(✓)
5
![Page 54: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/54.jpg)
• Cumulant-generating function
!
• True posterior covariance vs MFVB covariance
!
• “Linear response” !
• The LRVB approximation
V :=d2
dtT dtCq⇤(t)
����t=0
Linear response
mean =d
dtC(t)
����t=0
⌃ :=d2
dtT dtC
p(·|x)(t)
����t=0
log pt(✓) := log p(✓|x) + t
T✓ � C(t), MFVB q⇤t
⌃ =d
dtTEpt✓
����t=0
⇡ d
dtTEq⇤t ✓
����t=0
=: ⌃
[Bishop 2006]
C(t) := logEetT ✓
p(✓|x)
q⇤(✓)
5
![Page 55: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/55.jpg)
• Cumulant-generating function
!
• True posterior covariance vs MFVB covariance
!
• “Linear response” !
• The LRVB approximation
V :=d2
dtT dtCq⇤(t)
����t=0
Linear response
mean =d
dtC(t)
����t=0
⌃ :=d2
dtT dtC
p(·|x)(t)
����t=0
log pt(✓) := log p(✓|x) + t
T✓ � C(t), MFVB q⇤t
⌃ =d
dtTEpt✓
����t=0
⇡ d
dtTEq⇤t ✓
����t=0
=: ⌃
[Bishop 2006]
C(t) := logEetT ✓
p(✓|x)
q⇤(✓)
5
![Page 56: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/56.jpg)
• Cumulant-generating function
!
• True posterior covariance vs MFVB covariance
!
• “Linear response” !
• The LRVB approximation
V :=d2
dtT dtCq⇤(t)
����t=0
Linear response
mean =d
dtC(t)
����t=0
⌃ :=d2
dtT dtC
p(·|x)(t)
����t=0
log pt(✓) := log p(✓|x) + t
T✓ � C(t), MFVB q⇤t
⌃ =d
dtTEpt✓
����t=0
⇡ d
dtTEq⇤t ✓
����t=0
=: ⌃
[Bishop 2006]
C(t) := logEetT ✓
p(✓|x)
q⇤(✓)
p(✓|x)
[Bishop 2006]5
![Page 57: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/57.jpg)
• Cumulant-generating function
!
• True posterior covariance vs MFVB covariance
!
• “Linear response” !
• The LRVB approximation
V :=d2
dtT dtCq⇤(t)
����t=0
Linear response
mean =d
dtC(t)
����t=0
⌃ :=d2
dtT dtC
p(·|x)(t)
����t=0
log pt(✓) := log p(✓|x) + t
T✓ � C(t), MFVB q⇤t
⌃ =d
dtTEpt✓
����t=0
⇡ d
dtTEq⇤t ✓
����t=0
=: ⌃
[Bishop 2006]
C(t) := logEetT ✓
p(✓|x)
5
![Page 58: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/58.jpg)
• Cumulant-generating function
!
• True posterior covariance vs MFVB covariance
!
• “Linear response” !
• The LRVB approximation
V :=d2
dtT dtCq⇤(t)
����t=0
Linear response
mean =d
dtC(t)
����t=0
⌃ :=d2
dtT dtC
p(·|x)(t)
����t=0
log pt(✓) := log p(✓|x) + t
T✓ � C(t), MFVB q⇤t
⌃ =d
dtTEpt✓
����t=0
⇡ d
dtTEq⇤t ✓
����t=0
=: ⌃
[Bishop 2006]
C(t) := logEetT ✓
p(✓|x)
q⇤(✓)
[Bishop 2006]
p(✓|x)
q⇤(✓)
5
![Page 59: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/59.jpg)
• Cumulant-generating function
!
• True posterior covariance vs MFVB covariance
!
• “Linear response” !
• The LRVB approximation
V :=d2
dtT dtCq⇤(t)
����t=0
Linear response
mean =d
dtC(t)
����t=0
⌃ :=d2
dtT dtC
p(·|x)(t)
����t=0
log pt(✓) := log p(✓|x) + t
T✓ � C(t), MFVB q⇤t
⌃ =d
dtTEpt✓
����t=0
⇡ d
dtTEq⇤t ✓
����t=0
=: ⌃
[Bishop 2006]
C(t) := logEetT ✓
p(✓|x)
q⇤(✓)
[Bishop 2006]
p(✓|x)
q⇤(✓)
5
![Page 60: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/60.jpg)
• Cumulant-generating function
!
• True posterior covariance vs MFVB covariance
!
• “Linear response” !
• The LRVB approximation
V :=d2
dtT dtCq⇤(t)
����t=0
Linear response
mean =d
dtC(t)
����t=0
⌃ :=d2
dtT dtC
p(·|x)(t)
����t=0
log pt(✓) := log p(✓|x) + t
T✓ � C(t), MFVB q⇤t
⌃ =d
dtTEpt✓
����t=0
⇡ d
dtTEq⇤t ✓
����t=0
=: ⌃
[Bishop 2006]
C(t) := logEetT ✓
p(✓|x)
q⇤(✓)
[Bishop 2006]
p(✓|x)
q⇤(✓)
5
![Page 61: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/61.jpg)
• Cumulant-generating function
!
• True posterior covariance vs MFVB covariance
!
• “Linear response” !
• The LRVB approximation
V :=d2
dtT dtCq⇤(t)
����t=0
Linear response
mean =d
dtC(t)
����t=0
⌃ :=d2
dtT dtC
p(·|x)(t)
����t=0
log pt(✓) := log p(✓|x) + t
T✓ � C(t), MFVB q⇤t
⌃ =d
dtTEpt✓
����t=0
⇡ d
dtTEq⇤t ✓
����t=0
=: ⌃
[Bishop 2006]
C(t) := logEetT ✓
p(✓|x)
q⇤(✓)
[Bishop 2006]
p(✓|x)
q⇤(✓)
5
![Page 62: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/62.jpg)
• Cumulant-generating function
!
• True posterior covariance vs MFVB covariance
!
• “Linear response” !
• The LRVB approximation
V :=d2
dtT dtCq⇤(t)
����t=0
Linear response
mean =d
dtC(t)
����t=0
⌃ :=d2
dtT dtC
p(·|x)(t)
����t=0
log pt(✓) := log p(✓|x) + t
T✓ � C(t), MFVB q⇤t
⌃ =d
dtTEpt✓
����t=0
⇡ d
dtTEq⇤t ✓
����t=0
=: ⌃
[Bishop 2006]
C(t) := logEetT ✓
p(✓|x)
q⇤(✓)
[Bishop 2006]
p(✓|x)
q⇤(✓)
5
![Page 63: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/63.jpg)
• Cumulant-generating function
!
• True posterior covariance vs MFVB covariance
!
• “Linear response” !
• The LRVB approximation
V :=d2
dtT dtCq⇤(t)
����t=0
Linear response
mean =d
dtC(t)
����t=0
⌃ :=d2
dtT dtC
p(·|x)(t)
����t=0
log pt(✓) := log p(✓|x) + t
T✓ � C(t), MFVB q⇤t
⌃ =d
dtTEpt✓
����t=0
⇡ d
dtTEq⇤t ✓
����t=0
=: ⌃
[Bishop 2006]
C(t) := logEetT ✓
p(✓|x)
q⇤(✓)
[Bishop 2006]
p(✓|x)
q⇤(✓)
5
![Page 64: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/64.jpg)
• Cumulant-generating function
!
• True posterior covariance vs MFVB covariance
!
• “Linear response” !
• The LRVB approximation
V :=d2
dtT dtCq⇤(t)
����t=0
Linear response
mean =d
dtC(t)
����t=0
⌃ :=d2
dtT dtC
p(·|x)(t)
����t=0
log pt(✓) := log p(✓|x) + t
T✓ � C(t), MFVB q⇤t
⌃ =d
dtTEpt✓
����t=0
⇡ d
dtTEq⇤t ✓
����t=0
=: ⌃
[Bishop 2006]
C(t) := logEetT ✓
p(✓|x)
q⇤(✓)
[Bishop 2006]
p(✓|x)
q⇤(✓)
5
![Page 65: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/65.jpg)
• Cumulant-generating function
!
• True posterior covariance vs MFVB covariance
!
• “Linear response” !
• The LRVB approximation
V :=d2
dtT dtCq⇤(t)
����t=0
Linear response
mean =d
dtC(t)
����t=0
⌃ :=d2
dtT dtC
p(·|x)(t)
����t=0
log pt(✓) := log p(✓|x) + t
T✓ � C(t), MFVB q⇤t
⌃ =d
dtTEpt✓
����t=0
⇡ d
dtTEq⇤t ✓
����t=0
=: ⌃
[Bishop 2006]
C(t) := logEetT ✓
p(✓|x)
q⇤(✓)
[Bishop 2006]
p(✓|x)
q⇤(✓)
5
![Page 66: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/66.jpg)
• Cumulant-generating function
!
• True posterior covariance vs MFVB covariance
!
• “Linear response” !
• The LRVB approximation
V :=d2
dtT dtCq⇤(t)
����t=0
Linear response
mean =d
dtC(t)
����t=0
⌃ :=d2
dtT dtC
p(·|x)(t)
����t=0
log pt(✓) := log p(✓|x) + t
T✓ � C(t), MFVB q⇤t
⌃ =d
dtTEpt✓
����t=0
⇡ d
dtTEq⇤t ✓
����t=0
=: ⌃
[Bishop 2006]
C(t) := logEetT ✓
p(✓|x)
q⇤(✓)
[Bishop 2006]
p(✓|x)
q⇤(✓)
5
![Page 67: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/67.jpg)
• Cumulant-generating function
!
• True posterior covariance vs MFVB covariance
!
• “Linear response” !
• The LRVB approximation
V :=d2
dtT dtCq⇤(t)
����t=0
Linear response
mean =d
dtC(t)
����t=0
⌃ :=d2
dtT dtC
p(·|x)(t)
����t=0
log pt(✓) := log p(✓|x) + t
T✓ � C(t), MFVB q⇤t
[Bishop 2006]
C(t) := logEetT ✓
p(✓|x)
q⇤(✓)
[Bishop 2006]
p(✓|x)
q⇤(✓)
⌃ =d
dtT
d
dtC
p(·|x)(t)
�����t=05
![Page 68: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/68.jpg)
⌃ =d
dtTEpt✓
����t=0
⇡ d
dtTEq⇤t ✓
����t=0
=: ⌃
• Cumulant-generating function
!
• True posterior covariance vs MFVB covariance
!
• “Linear response” !
• The LRVB approximation
V :=d2
dtT dtCq⇤(t)
����t=0
Linear response
mean =d
dtC(t)
����t=0
⌃ :=d2
dtT dtC
p(·|x)(t)
����t=0
log pt(✓) := log p(✓|x) + t
T✓ � C(t), MFVB q⇤t
[Bishop 2006]
C(t) := logEetT ✓
p(✓|x)
q⇤(✓)
[Bishop 2006]
p(✓|x)
q⇤(✓)
5
![Page 69: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/69.jpg)
⌃ =d
dtTEpt✓
����t=0
⇡ d
dtTEq⇤t ✓
����t=0
=: ⌃
• Cumulant-generating function
!
• True posterior covariance vs MFVB covariance
!
• “Linear response” !
• The LRVB approximation
V :=d2
dtT dtCq⇤(t)
����t=0
Linear response
mean =d
dtC(t)
����t=0
⌃ :=d2
dtT dtC
p(·|x)(t)
����t=0
log pt(✓) := log p(✓|x) + t
T✓ � C(t), MFVB q⇤t
[Bishop 2006]
C(t) := logEetT ✓
p(✓|x)
q⇤(✓)
[Bishop 2006]
p(✓|x)
q⇤(✓)
5
![Page 70: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/70.jpg)
⌃ =d
dtTEpt✓
����t=0
⇡ d
dtTEq⇤t ✓
����t=0
=: ⌃
• Cumulant-generating function
!
• True posterior covariance vs MFVB covariance
!
• “Linear response” !
• The LRVB approximation
V :=d2
dtT dtCq⇤(t)
����t=0
Linear response
mean =d
dtC(t)
����t=0
⌃ :=d2
dtT dtC
p(·|x)(t)
����t=0
log pt(✓) := log p(✓|x) + t
T✓ � C(t), MFVB q⇤t
[Bishop 2006]
C(t) := logEetT ✓
p(✓|x)
q⇤(✓)
[Bishop 2006]
p(✓|x)
q⇤(✓)
5
![Page 71: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/71.jpg)
⌃ =d
dtTEpt✓
����t=0
⇡ d
dtTEq⇤t ✓
����t=0
=: ⌃
• Cumulant-generating function
!
• True posterior covariance vs MFVB covariance
!
• “Linear response” !
• The LRVB approximation
V :=d2
dtT dtCq⇤(t)
����t=0
Linear response
mean =d
dtC(t)
����t=0
⌃ :=d2
dtT dtC
p(·|x)(t)
����t=0
log pt(✓) := log p(✓|x) + t
T✓ � C(t), MFVB q⇤t
[Bishop 2006]
C(t) := logEetT ✓
p(✓|x)
q⇤(✓)
[Bishop 2006]
p(✓|x)
q⇤(✓)
5
![Page 72: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/72.jpg)
• Cumulant-generating function
!
• True posterior covariance vs MFVB covariance
!
• “Linear response” !
• The LRVB approximation
V :=d2
dtT dtCq⇤(t)
����t=0
Linear response
mean =d
dtC(t)
����t=0
⌃ :=d2
dtT dtC
p(·|x)(t)
����t=0
log pt(✓) := log p(✓|x) + t
T✓ � C(t), MFVB q⇤t
⌃ =d
dtTEpt✓
����t=0
⇡ d
dtTEq⇤t ✓
����t=0
=: ⌃
[Bishop 2006]
C(t) := logEetT ✓
p(✓|x)
q⇤(✓)
5
![Page 73: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/73.jpg)
• LRVB covariance estimate
• Suppose exponential family with mean parametrization
LRVB estimator⌃ :=
d
dtTEq⇤t ✓
����t=0
qt mt
6
![Page 74: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/74.jpg)
• LRVB covariance estimate
• Suppose exponential family with mean parametrization
LRVB estimator⌃ :=
d
dtTEq⇤t ✓
����t=0
qt mt
6
![Page 75: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/75.jpg)
• LRVB covariance estimate
• Suppose exponential family with mean parametrization
LRVB estimator⌃ :=
d
dtTEq⇤t ✓
����t=0
qt mt
= (I � V H)�1V
6
![Page 76: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/76.jpg)
• LRVB covariance estimate
• Suppose exponential family with mean parametrization
LRVB estimator
⌃ =
✓@2KL
@m@mT
����m=m⇤
◆�1
⌃ :=d
dtTEq⇤t ✓
����t=0
qt mt
= (I � V H)�1V
6
![Page 77: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/77.jpg)
• LRVB covariance estimate
• Suppose exponential family with mean parametrization
LRVB estimator
⌃ =
✓@2KL
@m@mT
����m=m⇤
◆�1
⌃ :=d
dtTEq⇤t ✓
����t=0
qt mt
= (I � V H)�1V
6
![Page 78: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/78.jpg)
• LRVB covariance estimate
• Suppose exponential family with mean parametrization
LRVB estimator
⌃ =
✓@2KL
@m@mT
����m=m⇤
◆�1
⌃ :=d
dtTEq⇤t ✓
����t=0
qt mt
= (I � V H)�1V
6
![Page 79: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/79.jpg)
• LRVB covariance estimate
• Suppose exponential family with mean parametrization
• Symmetric and positive definite at local min of KL
• The LRVB assumption:
LRVB estimator
⌃ =
✓@2KL
@m@mT
����m=m⇤
◆�1
⌃ :=d
dtTEq⇤t ✓
����t=0
qt mt
= (I � V H)�1V
6
![Page 80: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/80.jpg)
• LRVB covariance estimate
• Suppose exponential family with mean parametrization
• Symmetric and positive definite at local min of KL
• The LRVB assumption: Ept✓ ⇡ Eq⇤t ✓
LRVB estimator
⌃ =
✓@2KL
@m@mT
����m=m⇤
◆�1
⌃ :=d
dtTEq⇤t ✓
����t=0
qt mt
= (I � V H)�1V
6
![Page 81: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/81.jpg)
• LRVB covariance estimate
• Suppose exponential family with mean parametrization
• Symmetric and positive definite at local min of KL
• The LRVB assumption: Ept✓ ⇡ Eq⇤t ✓p(✓|x)
q⇤(✓)
[Bishop 2006]
LRVB estimator
⌃ =
✓@2KL
@m@mT
����m=m⇤
◆�1
⌃ :=d
dtTEq⇤t ✓
����t=0
qt mt
= (I � V H)�1V
6
![Page 82: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/82.jpg)
• LRVB covariance estimate
• Suppose exponential family with mean parametrization
• Symmetric and positive definite at local min of KL
• The LRVB assumption: Ept✓ ⇡ Eq⇤t ✓p(✓|x)
q⇤(✓)
• LRVB estimate is exact when MFVB gives exact mean (e.g. multivariate normal)
[Bishop 2006]
LRVB estimator
⌃ =
✓@2KL
@m@mT
����m=m⇤
◆�1
⌃ :=d
dtTEq⇤t ✓
����t=0
qt mt
= (I � V H)�1V
6
![Page 83: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/83.jpg)
Microcredit Experiment• Simplified from Meager (2015) • K microcredit trials (Mexico, Mongolia, Bosnia, India,
Morocco, Philippines, Ethiopia) • Nk businesses in kth site (~900 to ~17K) • Profit of nth business at kth site:
!
!• Priors and hyperpriors:
7
![Page 84: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/84.jpg)
Microcredit Experiment• Simplified from Meager (2015) • K microcredit trials (Mexico, Mongolia, Bosnia, India,
Morocco, Philippines, Ethiopia) • Nk businesses in kth site (~900 to ~17K) • Profit of nth business at kth site:
!
!• Priors and hyperpriors:
7
![Page 85: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/85.jpg)
Microcredit Experiment• Simplified from Meager (2015) • K microcredit trials (Mexico, Mongolia, Bosnia, India,
Morocco, Philippines, Ethiopia) • Nk businesses in kth site (~900 to ~17K) • Profit of nth business at kth site:
!
!• Priors and hyperpriors:
7
![Page 86: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/86.jpg)
Microcredit Experiment• Simplified from Meager (2015) • K microcredit trials (Mexico, Mongolia, Bosnia, India,
Morocco, Philippines, Ethiopia) • Nk businesses in kth site (~900 to ~17K) • Profit of nth business at kth site:
!
!• Priors and hyperpriors:
7
![Page 87: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/87.jpg)
Microcredit Experiment• Simplified from Meager (2015) • K microcredit trials (Mexico, Mongolia, Bosnia, India,
Morocco, Philippines, Ethiopia) • Nk businesses in kth site (~900 to ~17K) • Profit of nth business at kth site:
!
!• Priors and hyperpriors:
7
![Page 88: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/88.jpg)
Microcredit Experiment• Simplified from Meager (2015) • K microcredit trials (Mexico, Mongolia, Bosnia, India,
Morocco, Philippines, Ethiopia) • Nk businesses in kth site (~900 to ~17K) • Profit of nth business at kth site:
!
!• Priors and hyperpriors:
yknindep⇠ N (µk + Tkn⌧k,�
2k)
profit
7
![Page 89: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/89.jpg)
Microcredit Experiment• Simplified from Meager (2015) • K microcredit trials (Mexico, Mongolia, Bosnia, India,
Morocco, Philippines, Ethiopia) • Nk businesses in kth site (~900 to ~17K) • Profit of nth business at kth site:
!
!• Priors and hyperpriors:
yknindep⇠ N (µk + Tkn⌧k,�
2k)
profit
7
![Page 90: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/90.jpg)
Microcredit Experiment• Simplified from Meager (2015) • K microcredit trials (Mexico, Mongolia, Bosnia, India,
Morocco, Philippines, Ethiopia) • Nk businesses in kth site (~900 to ~17K) • Profit of nth business at kth site:
!
!• Priors and hyperpriors:
yknindep⇠ N (µk + Tkn⌧k,�
2k)
profit
7
![Page 91: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/91.jpg)
Microcredit Experiment• Simplified from Meager (2015) • K microcredit trials (Mexico, Mongolia, Bosnia, India,
Morocco, Philippines, Ethiopia) • Nk businesses in kth site (~900 to ~17K) • Profit of nth business at kth site:
!
!• Priors and hyperpriors:
yknindep⇠ N (µk + Tkn⌧k,�
2k)
profit
7
![Page 92: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/92.jpg)
Microcredit Experiment• Simplified from Meager (2015) • K microcredit trials (Mexico, Mongolia, Bosnia, India,
Morocco, Philippines, Ethiopia) • Nk businesses in kth site (~900 to ~17K) • Profit of nth business at kth site:
!
!• Priors and hyperpriors:
yknindep⇠ N (µk + Tkn⌧k,�
2k)
profit1 if microcredit
7
![Page 93: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/93.jpg)
Microcredit Experiment• Simplified from Meager (2015) • K microcredit trials (Mexico, Mongolia, Bosnia, India,
Morocco, Philippines, Ethiopia) • Nk businesses in kth site (~900 to ~17K) • Profit of nth business at kth site:
!
!• Priors and hyperpriors:
yknindep⇠ N (µk + Tkn⌧k,�
2k)
profit1 if microcredit
7
![Page 94: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/94.jpg)
Microcredit Experiment• Simplified from Meager (2015) • K microcredit trials (Mexico, Mongolia, Bosnia, India,
Morocco, Philippines, Ethiopia) • Nk businesses in kth site (~900 to ~17K) • Profit of nth business at kth site:
!
!• Priors and hyperpriors:
yknindep⇠ N (µk + Tkn⌧k,�
2k)
profit1 if microcredit
7
![Page 95: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/95.jpg)
Microcredit Experiment• Simplified from Meager (2015) • K microcredit trials (Mexico, Mongolia, Bosnia, India,
Morocco, Philippines, Ethiopia) • Nk businesses in kth site (~900 to ~17K) • Profit of nth business at kth site:
!
!• Priors and hyperpriors:
yknindep⇠ N (µk + Tkn⌧k,�
2k)
profit1 if microcredit
7
![Page 96: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/96.jpg)
Microcredit Experiment• Simplified from Meager (2015) • K microcredit trials (Mexico, Mongolia, Bosnia, India,
Morocco, Philippines, Ethiopia) • Nk businesses in kth site (~900 to ~17K) • Profit of nth business at kth site:
!
!• Priors and hyperpriors:
yknindep⇠ N (µk + Tkn⌧k,�
2k)
profit1 if microcredit
7
![Page 97: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/97.jpg)
Microcredit Experiment• Simplified from Meager (2015) • K microcredit trials (Mexico, Mongolia, Bosnia, India,
Morocco, Philippines, Ethiopia) • Nk businesses in kth site (~900 to ~17K) • Profit of nth business at kth site:
!
!• Priors and hyperpriors:
yknindep⇠ N (µk + Tkn⌧k,�
2k)
✓µk
⌧k
◆iid⇠ N
✓✓µ⌧
◆, C
◆
profit1 if microcredit
7
![Page 98: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/98.jpg)
Microcredit Experiment• Simplified from Meager (2015) • K microcredit trials (Mexico, Mongolia, Bosnia, India,
Morocco, Philippines, Ethiopia) • Nk businesses in kth site (~900 to ~17K) • Profit of nth business at kth site:
!
!• Priors and hyperpriors:
yknindep⇠ N (µk + Tkn⌧k,�
2k)
✓µk
⌧k
◆iid⇠ N
✓✓µ⌧
◆, C
◆
��2k
iid⇠ �(a, b)
profit1 if microcredit
7
![Page 99: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/99.jpg)
Microcredit Experiment• Simplified from Meager (2015) • K microcredit trials (Mexico, Mongolia, Bosnia, India,
Morocco, Philippines, Ethiopia) • Nk businesses in kth site (~900 to ~17K) • Profit of nth business at kth site:
!
!• Priors and hyperpriors:
yknindep⇠ N (µk + Tkn⌧k,�
2k)
✓µk
⌧k
◆iid⇠ N
✓✓µ⌧
◆, C
◆ ✓µ⌧
◆iid⇠ N
✓✓µ0
⌧0
◆,⇤�1
◆
��2k
iid⇠ �(a, b)
profit1 if microcredit
7C ⇠ Sep&LKJ(⌘, c, d)
![Page 100: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/100.jpg)
Microcredit Experiment
8
![Page 101: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/101.jpg)
Microcredit Experiment
MFV
B
8
![Page 102: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/102.jpg)
Microcredit Experiment
MFV
B
8
![Page 103: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/103.jpg)
Microcredit Experiment• One set of 2500
MCMC draws: 45 minutes
• All of MFVB optimization, LRVB uncertainties, all sensitivity measures: 58 seconds!
• Many other models and data sets: Mixture models, generalized linear mixed models, etc
MFV
B
8
![Page 104: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/104.jpg)
Microcredit Experiment• One set of 2500
MCMC draws: 45 minutes
• All of MFVB optimization, LRVB uncertainties, all sensitivity measures: 58 seconds!
• Many other models and data sets: Mixture models, generalized linear mixed models, etc
MFV
B
8
![Page 105: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/105.jpg)
Microcredit Experiment• One set of 2500
MCMC draws: 45 minutes
• All of MFVB optimization, LRVB uncertainties, all sensitivity measures: 58 seconds!
• Many other models and data sets: Mixture models, generalized linear mixed models, etc
MFV
B
LRVB,!MFVB
8
![Page 106: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/106.jpg)
Microcredit Experiment• One set of 2500
MCMC draws: 45 minutes
• All of MFVB optimization, LRVB uncertainties, all sensitivity measures: 58 seconds!
• Many other models and data sets: Mixture models, generalized linear mixed models, etc
MFV
B
LRVB,!MFVB
8
![Page 107: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/107.jpg)
Robustness quantification
• Variational Bayes as an alternative to MCMC • Challenges of VB • Accurate uncertainties from VB • Accurate robustness quantification from VB • Big idea: derivatives/perturbations are easy in VB
9
![Page 108: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/108.jpg)
Robustness quantification
• Variational Bayes as an alternative to MCMC • Challenges of VB • Accurate uncertainties from VB • Accurate robustness quantification from VB • Big idea: derivatives/perturbations are easy in VB
9
![Page 109: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/109.jpg)
Robustness quantification• Bayes Theorem
p(✓|x)/✓ p(x|✓)p(✓)
10
![Page 110: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/110.jpg)
Robustness quantification• Bayes Theoremp↵(✓) := p(✓|x,↵)
/✓ p(x|✓)p(✓|↵)
10
![Page 111: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/111.jpg)
Robustness quantification• Bayes Theoremp↵(✓) := p(✓|x,↵)
/✓ p(x|✓)p(✓|↵)
10
![Page 112: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/112.jpg)
Robustness quantification• Bayes Theoremp↵(✓) := p(✓|x,↵)
/✓ p(x|✓)p(✓|↵)
• Sensitivity
10
![Page 113: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/113.jpg)
Bayes Theorem
Robustness quantification• Bayes Theoremp↵(✓) := p(✓|x,↵)
/✓ p(x|✓)p(✓|↵)
• Sensitivity
10
![Page 114: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/114.jpg)
Bayes Theorem
Robustness quantification• Bayes Theoremp↵(✓) := p(✓|x,↵)
/✓ p(x|✓)p(✓|↵)
• Sensitivity
S :=dEp↵ [g(✓)]
d↵
����↵
�↵
10
![Page 115: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/115.jpg)
Bayes Theorem
Robustness quantification• Bayes Theoremp↵(✓) := p(✓|x,↵)
/✓ p(x|✓)p(✓|↵)
• Sensitivity
S :=dEp↵ [g(✓)]
d↵
����↵
�↵
10
![Page 116: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/116.jpg)
Bayes Theorem
Robustness quantification• Bayes Theoremp↵(✓) := p(✓|x,↵)
/✓ p(x|✓)p(✓|↵)
• Sensitivity
S :=dEp↵ [g(✓)]
d↵
����↵
�↵
10
![Page 117: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/117.jpg)
Bayes Theorem
Robustness quantification• Bayes Theoremp↵(✓) := p(✓|x,↵)
/✓ p(x|✓)p(✓|↵)
• Sensitivity
S :=dEp↵ [g(✓)]
d↵
����↵
�↵
10
![Page 118: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/118.jpg)
Bayes Theorem
Robustness quantification• Bayes Theoremp↵(✓) := p(✓|x,↵)
/✓ p(x|✓)p(✓|↵)
• Sensitivity
S :=dEp↵ [g(✓)]
d↵
����↵
�↵
⇡dEq⇤↵ [g(✓)]
d↵
����↵
�↵ =: S
10
![Page 119: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/119.jpg)
Bayes Theorem
Robustness quantification• Bayes Theoremp↵(✓) := p(✓|x,↵)
/✓ p(x|✓)p(✓|↵)
• Sensitivity
S :=dEp↵ [g(✓)]
d↵
����↵
�↵
⇡dEq⇤↵ [g(✓)]
d↵
����↵
�↵ =: S LRVB estimator
10
![Page 120: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/120.jpg)
Bayes Theorem
Robustness quantification• Bayes Theoremp↵(✓) := p(✓|x,↵)
/✓ p(x|✓)p(✓|↵)
• Sensitivity
S :=dEp↵ [g(✓)]
d↵
����↵
�↵
⇡dEq⇤↵ [g(✓)]
d↵
����↵
�↵ =: S LRVB estimator
• When in exponential familyq⇤↵
10
![Page 121: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/121.jpg)
Bayes Theorem
Robustness quantification• Bayes Theorem
S = A
✓@2KL
@m@mT
����m=m⇤
◆�1
B
p↵(✓) := p(✓|x,↵)/✓ p(x|✓)p(✓|↵)
• Sensitivity
S :=dEp↵ [g(✓)]
d↵
����↵
�↵
⇡dEq⇤↵ [g(✓)]
d↵
����↵
�↵ =: S LRVB estimator
• When in exponential familyq⇤↵
10
![Page 122: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/122.jpg)
C ⇠ Sep&LKJ(⌘, c, d)
Microcredit Experiment• Simplified from Meager (2015) • K microcredit trials (Mexico, Mongolia, Bosnia, India,
Morocco, Philippines, Ethiopia) • Nk businesses in kth site (~900 to ~17K) • Profit of nth business at kth site:
!
!• Priors and hyperpriors:
yknindep⇠ N (µk + Tkn⌧k,�
2k)
✓µk
⌧k
◆iid⇠ N
✓✓µ⌧
◆, C
◆ ✓µ⌧
◆iid⇠ N
✓✓µ0
⌧0
◆,⇤�1
◆
��2k
iid⇠ �(a, b)
profit1 if microcredit
11
![Page 123: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/123.jpg)
Microcredit Experiment
12
![Page 124: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/124.jpg)
Microcredit Experiment
MFV
B
12
![Page 125: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/125.jpg)
Microcredit Experiment
• Perturb Λ11: 0.03 ➔ 0.04
MFV
B
12
![Page 126: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/126.jpg)
Microcredit Experiment
• Perturb Λ11: 0.03 ➔ 0.04
Sensitivity
MFV
BLR
VB
12
![Page 127: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/127.jpg)
Microcredit Experiment
• Perturb Λ11: 0.03 ➔ 0.04
Sensitivity
MFV
BLR
VB
12
![Page 128: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/128.jpg)
Microcredit Experiment
13
![Page 129: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/129.jpg)
• Sensitivity of the expected microcredit effect (τ)
• Normalized to be on scale of standard deviations in τ
• E.g.
Microcredit Experiment
13
![Page 130: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/130.jpg)
• Sensitivity of the expected microcredit effect (τ)
• Normalized to be on scale of standard deviations in τ
• E.g.
Microcredit Experiment
13
![Page 131: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/131.jpg)
• Sensitivity of the expected microcredit effect (τ)
• Normalized to be on scale of standard deviations in τ
• E.g.
Microcredit Experiment
13
![Page 132: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/132.jpg)
• Sensitivity of the expected microcredit effect (τ)
• Normalized to be on scale of standard deviations in τ
• E.g.
Microcredit Experiment
13
![Page 133: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/133.jpg)
• Sensitivity of the expected microcredit effect (τ)
• Normalized to be on scale of standard deviations in τ
• E.g.
Microcredit Experiment
13
![Page 134: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/134.jpg)
• Sensitivity of the expected microcredit effect (τ)
• Normalized to be on scale of standard deviations in τ
• E.g.
Microcredit Experiment
StdDevq⌧ = 1.8
13
![Page 135: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/135.jpg)
• Sensitivity of the expected microcredit effect (τ)
• Normalized to be on scale of standard deviations in τ
• E.g.
Microcredit Experiment
StdDevq⌧ = 1.8Eq⌧ = 3.7
13
![Page 136: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/136.jpg)
• Sensitivity of the expected microcredit effect (τ)
• Normalized to be on scale of standard deviations in τ
• E.g.
Microcredit Experiment
StdDevq⌧ = 1.8Eq⌧ = 3.7= 2.06 ⇤ StdDevq⌧
13
![Page 137: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/137.jpg)
• Sensitivity of the expected microcredit effect (τ)
• Normalized to be on scale of standard deviations in τ
• E.g.
Microcredit Experiment
StdDevq⌧ = 1.8Eq⌧ = 3.7= 2.06 ⇤ StdDevq⌧
⇤12 + = 0.03
13
![Page 138: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/138.jpg)
• Sensitivity of the expected microcredit effect (τ)
• Normalized to be on scale of standard deviations in τ
• E.g.
Microcredit Experiment
StdDevq⌧ = 1.8Eq⌧ = 3.7= 2.06 ⇤ StdDevq⌧
⇤12 + = 0.03
13Eq⌧ < 1.0 ⇤ StdDevq⌧
![Page 139: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/139.jpg)
Conclusion• We provide linear response variational Bayes:
supplements MFVB for fast & accurate covariance estimate
• More from LRVB: fast & accurate robustness quantification
• Interested in your data and models: • Sensitivity to prior perturbations • Sensitivity to data perturbations
14
![Page 140: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/140.jpg)
T. Broderick, N. Boyd, A. Wibisono, A. C. Wilson, and M. I. Jordan. Streaming variational Bayes. NIPS, 2013. !R Giordano, T Broderick, and MI Jordan. Linear response methods for accurate covariance estimates from mean field variational Bayes. NIPS, 2015.!!R Giordano, T Broderick, and MI Jordan. Robust Inference with Variational Bayes. NIPS AABI Workshop, 2015. ArXiv:1512.02578.!! https://github.com/rgiordan/MicrocreditLRVB! !J. Huggins, T. Campbell, and T. Broderick. Core sets for scalable Bayesian logistic regression. Under review. ArXiv:1605.06423.
References
15
![Page 141: Fast Robustness Quantification with Variational …people.csail.mit.edu/tbroderick/files/broderick_2016...Fast Robustness Quantification with Variational Bayes ITT Career Development](https://reader035.fdocuments.us/reader035/viewer/2022081502/5ebd9a8e563fa54c976b61f5/html5/thumbnails/141.jpg)
References
R Bardenet, A Doucet, and C Holmes. On Markov chain Monte Carlo methods for tall data. arXiv, 2015.
CM Bishop. Pattern Recognition and Machine Learning.
D Dunson. Robust and scalable approach to Bayesian inference. Talk at ISBA 2014.
DJC MacKay. Information Theory, Inference, and Learning Algorithms. Cambridge University Press, 2003.
R Meager. Understanding the impact of microcredit expansions: A Bayesian hierarchical analysis of 7 randomised experiments. ArXiv:1506.06669, 2015.
RE Turner and M Sahani. Two problems with variational expectation maximisation for time-series models. In D Barber, AT Cemgil, and S Chiappa, editors, Bayesian Time Series Models, 2011.
B Wang and M Titterington. Inadequacy of interval estimates corresponding to variational Bayesian approximations. In AISTATS, 2004.
16