NONLINEAR REGRESSION I - Idiap
Transcript of NONLINEAR REGRESSION I - Idiap
![Page 1: NONLINEAR REGRESSION I - Idiap](https://reader031.fdocuments.us/reader031/viewer/2022012413/616d870afc2d5b20a12ee7ce/html5/thumbnails/1.jpg)
EE613Machine Learning for Engineers
NONLINEAR REGRESSION I
Sylvain CalinonRobot Learning & Interaction Group
Idiap Research InstituteDec. 12, 2019
![Page 2: NONLINEAR REGRESSION I - Idiap](https://reader031.fdocuments.us/reader031/viewer/2022012413/616d870afc2d5b20a12ee7ce/html5/thumbnails/2.jpg)
Outline
2
• Properties of multivariate Gaussian distributions:• Product of Gaussians• Linear transformation and combination• Conditional distribution• Gaussian estimate of a mixture of Gaussians
• Locally weighted regression (LWR)
• Gaussian mixture regression (GMR)
• Example of application:Dynamical movement primitives (DMP)
![Page 3: NONLINEAR REGRESSION I - Idiap](https://reader031.fdocuments.us/reader031/viewer/2022012413/616d870afc2d5b20a12ee7ce/html5/thumbnails/3.jpg)
Product of Gaussians:
Linear transformation and combination:
Conditional distribution:
Some very useful properties…
3
![Page 4: NONLINEAR REGRESSION I - Idiap](https://reader031.fdocuments.us/reader031/viewer/2022012413/616d870afc2d5b20a12ee7ce/html5/thumbnails/4.jpg)
Product of Gaussians
![Page 5: NONLINEAR REGRESSION I - Idiap](https://reader031.fdocuments.us/reader031/viewer/2022012413/616d870afc2d5b20a12ee7ce/html5/thumbnails/5.jpg)
Product of Gaussians - Motivating example
Product ofGaussians
(superposition)
(fusion)
center of the Gaussian
covariance matrix
precision matrix
![Page 6: NONLINEAR REGRESSION I - Idiap](https://reader031.fdocuments.us/reader031/viewer/2022012413/616d870afc2d5b20a12ee7ce/html5/thumbnails/6.jpg)
Product of Gaussians - Fusion of information
Using full weight matrices also include the special case
of using scalar weights
Scalar superposition
6
![Page 7: NONLINEAR REGRESSION I - Idiap](https://reader031.fdocuments.us/reader031/viewer/2022012413/616d870afc2d5b20a12ee7ce/html5/thumbnails/7.jpg)
t=0 t=1 t=2
Kalman filter as product of Gaussians
Product of Gaussians - Kalman filter
7
![Page 8: NONLINEAR REGRESSION I - Idiap](https://reader031.fdocuments.us/reader031/viewer/2022012413/616d870afc2d5b20a12ee7ce/html5/thumbnails/8.jpg)
Superposition Fusion
t=0 t=1 t=2t=0 t=1 t=2
![Page 9: NONLINEAR REGRESSION I - Idiap](https://reader031.fdocuments.us/reader031/viewer/2022012413/616d870afc2d5b20a12ee7ce/html5/thumbnails/9.jpg)
Linear transformation andcombination
9
![Page 10: NONLINEAR REGRESSION I - Idiap](https://reader031.fdocuments.us/reader031/viewer/2022012413/616d870afc2d5b20a12ee7ce/html5/thumbnails/10.jpg)
in a new situation…
Example exploiting linear transformation
and product properties
Coordinate system 1:This is where I expect
data to be located!
Coordinate system 2:This is where I expect
data to be located!
Product of linearly transformed Gaussians 10
![Page 11: NONLINEAR REGRESSION I - Idiap](https://reader031.fdocuments.us/reader031/viewer/2022012413/616d870afc2d5b20a12ee7ce/html5/thumbnails/11.jpg)
Conditional distribution
![Page 12: NONLINEAR REGRESSION I - Idiap](https://reader031.fdocuments.us/reader031/viewer/2022012413/616d870afc2d5b20a12ee7ce/html5/thumbnails/12.jpg)
12→ Linear regression from joint distribution
Conditional distribution
![Page 13: NONLINEAR REGRESSION I - Idiap](https://reader031.fdocuments.us/reader031/viewer/2022012413/616d870afc2d5b20a12ee7ce/html5/thumbnails/13.jpg)
13
Conditional distribution
![Page 14: NONLINEAR REGRESSION I - Idiap](https://reader031.fdocuments.us/reader031/viewer/2022012413/616d870afc2d5b20a12ee7ce/html5/thumbnails/14.jpg)
Conditional distribution - Geometric interpretation
14
![Page 15: NONLINEAR REGRESSION I - Idiap](https://reader031.fdocuments.us/reader031/viewer/2022012413/616d870afc2d5b20a12ee7ce/html5/thumbnails/15.jpg)
15
Conditional distribution - Resolution
![Page 16: NONLINEAR REGRESSION I - Idiap](https://reader031.fdocuments.us/reader031/viewer/2022012413/616d870afc2d5b20a12ee7ce/html5/thumbnails/16.jpg)
16
Conditional distribution - Resolution
![Page 17: NONLINEAR REGRESSION I - Idiap](https://reader031.fdocuments.us/reader031/viewer/2022012413/616d870afc2d5b20a12ee7ce/html5/thumbnails/17.jpg)
Conditional distribution - Resolution
17
![Page 18: NONLINEAR REGRESSION I - Idiap](https://reader031.fdocuments.us/reader031/viewer/2022012413/616d870afc2d5b20a12ee7ce/html5/thumbnails/18.jpg)
18
Conditional distribution - Resolution
![Page 19: NONLINEAR REGRESSION I - Idiap](https://reader031.fdocuments.us/reader031/viewer/2022012413/616d870afc2d5b20a12ee7ce/html5/thumbnails/19.jpg)
19
Conditional distribution - Resolution
![Page 20: NONLINEAR REGRESSION I - Idiap](https://reader031.fdocuments.us/reader031/viewer/2022012413/616d870afc2d5b20a12ee7ce/html5/thumbnails/20.jpg)
20
Conditional distribution - Summary
![Page 21: NONLINEAR REGRESSION I - Idiap](https://reader031.fdocuments.us/reader031/viewer/2022012413/616d870afc2d5b20a12ee7ce/html5/thumbnails/21.jpg)
Gaussian estimate of a mixture of Gaussians
21
![Page 22: NONLINEAR REGRESSION I - Idiap](https://reader031.fdocuments.us/reader031/viewer/2022012413/616d870afc2d5b20a12ee7ce/html5/thumbnails/22.jpg)
22
Gaussian estimate of a mixture of Gaussians
![Page 23: NONLINEAR REGRESSION I - Idiap](https://reader031.fdocuments.us/reader031/viewer/2022012413/616d870afc2d5b20a12ee7ce/html5/thumbnails/23.jpg)
Gaussian estimate of a mixture of Gaussians
23
![Page 24: NONLINEAR REGRESSION I - Idiap](https://reader031.fdocuments.us/reader031/viewer/2022012413/616d870afc2d5b20a12ee7ce/html5/thumbnails/24.jpg)
Locally weighted regression(LWR)
Python notebooks: demo_LWR.ipynb
Matlab codes: demo_LWR01.m
24
![Page 25: NONLINEAR REGRESSION I - Idiap](https://reader031.fdocuments.us/reader031/viewer/2022012413/616d870afc2d5b20a12ee7ce/html5/thumbnails/25.jpg)
Previous lecture on linear regression
25
Color darknessproportional
to weight
![Page 26: NONLINEAR REGRESSION I - Idiap](https://reader031.fdocuments.us/reader031/viewer/2022012413/616d870afc2d5b20a12ee7ce/html5/thumbnails/26.jpg)
Locally weighted regression (LWR)
26
![Page 27: NONLINEAR REGRESSION I - Idiap](https://reader031.fdocuments.us/reader031/viewer/2022012413/616d870afc2d5b20a12ee7ce/html5/thumbnails/27.jpg)
Locally weighted regression (LWR)
![Page 28: NONLINEAR REGRESSION I - Idiap](https://reader031.fdocuments.us/reader031/viewer/2022012413/616d870afc2d5b20a12ee7ce/html5/thumbnails/28.jpg)
Locally weighted regression (LWR)
28
![Page 29: NONLINEAR REGRESSION I - Idiap](https://reader031.fdocuments.us/reader031/viewer/2022012413/616d870afc2d5b20a12ee7ce/html5/thumbnails/29.jpg)
Locally weighted regression (LWR)
29
LWR can be used for local least squares polynomial
fitting by changing the definition of the inputs.
![Page 30: NONLINEAR REGRESSION I - Idiap](https://reader031.fdocuments.us/reader031/viewer/2022012413/616d870afc2d5b20a12ee7ce/html5/thumbnails/30.jpg)
Locally weighted regression (LWR)
30
![Page 31: NONLINEAR REGRESSION I - Idiap](https://reader031.fdocuments.us/reader031/viewer/2022012413/616d870afc2d5b20a12ee7ce/html5/thumbnails/31.jpg)
Locally weighted regression (LWR)
31
![Page 32: NONLINEAR REGRESSION I - Idiap](https://reader031.fdocuments.us/reader031/viewer/2022012413/616d870afc2d5b20a12ee7ce/html5/thumbnails/32.jpg)
Gaussian mixture regression(GMR)
Python notebooks: demo_GMR.ipynb
Matlab codes: demo_GMR01.m
demo_GMR_polyFit01.m
32
![Page 33: NONLINEAR REGRESSION I - Idiap](https://reader031.fdocuments.us/reader031/viewer/2022012413/616d870afc2d5b20a12ee7ce/html5/thumbnails/33.jpg)
Gaussian mixture regression (GMR)
33
![Page 34: NONLINEAR REGRESSION I - Idiap](https://reader031.fdocuments.us/reader031/viewer/2022012413/616d870afc2d5b20a12ee7ce/html5/thumbnails/34.jpg)
34
Gaussian mixture regression (GMR)
• Gaussian mixture regression (GMR) is a nonlinear regression technique that does not model the regression function directly, but instead first models the joint probability density of input-output data in the form of a Gaussian mixture model (GMM).
• The computation relies on linear transformation and conditioning properties of multivariate normal distributions.
• GMR provides a regression approach in which multivariate output distributions can be computed in an online manner, with a computation time independent of the number of datapoints used to train the model, by exploiting the learned joint density model.
• In GMR, both input and output variables can be multivariate, and after learning, any subset of input-output dimensions can be selected for regression. This can for example be exploited to handle different sources of missing data, where expectations on the remaining dimensions can be computed as a multivariate distribution.
![Page 35: NONLINEAR REGRESSION I - Idiap](https://reader031.fdocuments.us/reader031/viewer/2022012413/616d870afc2d5b20a12ee7ce/html5/thumbnails/35.jpg)
Gaussian mixture regression (GMR)
35
![Page 36: NONLINEAR REGRESSION I - Idiap](https://reader031.fdocuments.us/reader031/viewer/2022012413/616d870afc2d5b20a12ee7ce/html5/thumbnails/36.jpg)
Gaussian mixture regression (GMR)
36
![Page 37: NONLINEAR REGRESSION I - Idiap](https://reader031.fdocuments.us/reader031/viewer/2022012413/616d870afc2d5b20a12ee7ce/html5/thumbnails/37.jpg)
37
Gaussian mixture regression (GMR)
![Page 38: NONLINEAR REGRESSION I - Idiap](https://reader031.fdocuments.us/reader031/viewer/2022012413/616d870afc2d5b20a12ee7ce/html5/thumbnails/38.jpg)
GMR can cover a large range of regression approaches!
Gaussian mixture regression (GMR)
Nadaraya-Watsonkernel regression
Least squareslinear regression
38
![Page 39: NONLINEAR REGRESSION I - Idiap](https://reader031.fdocuments.us/reader031/viewer/2022012413/616d870afc2d5b20a12ee7ce/html5/thumbnails/39.jpg)
GMR for smooth piecewise polynomial fitting
39
![Page 40: NONLINEAR REGRESSION I - Idiap](https://reader031.fdocuments.us/reader031/viewer/2022012413/616d870afc2d5b20a12ee7ce/html5/thumbnails/40.jpg)
[Calinon, Guenter and Billard, IEEE Trans. on SMC-B 37(2), 2007]
Gaussian mixture regression - Examples
[Hersch, Guenter, Calinon and Billard, IEEE Trans. on Robotics 24(6), 2008]
With expectation-maximization (EM): (maximizing log-likelihood)
[Khansari-Zadeh and Billard, IEEE Trans. on Robotics 27(5), 2011]
With quadratic programming solver:(maximizing log-likelihood s.t. stability constraints)
![Page 41: NONLINEAR REGRESSION I - Idiap](https://reader031.fdocuments.us/reader031/viewer/2022012413/616d870afc2d5b20a12ee7ce/html5/thumbnails/41.jpg)
Example of application:
Dynamical movement primitives (DMP)
Python notebooks: demo_DMP.ipynb
demo_DMP_GMR.ipynb
Matlab codes: demo_DMP01.m
demo_DMP_GMR01.m41
![Page 42: NONLINEAR REGRESSION I - Idiap](https://reader031.fdocuments.us/reader031/viewer/2022012413/616d870afc2d5b20a12ee7ce/html5/thumbnails/42.jpg)
Dynamical movement primitives (DMP)
Spring-damper system
42
![Page 43: NONLINEAR REGRESSION I - Idiap](https://reader031.fdocuments.us/reader031/viewer/2022012413/616d870afc2d5b20a12ee7ce/html5/thumbnails/43.jpg)
(ideally damped)
(critically damped)
(underdamped)
(overdamped)
Dynamical movement primitives (DMP)
43
![Page 44: NONLINEAR REGRESSION I - Idiap](https://reader031.fdocuments.us/reader031/viewer/2022012413/616d870afc2d5b20a12ee7ce/html5/thumbnails/44.jpg)
Dynamical movement primitives (DMP)
44
![Page 45: NONLINEAR REGRESSION I - Idiap](https://reader031.fdocuments.us/reader031/viewer/2022012413/616d870afc2d5b20a12ee7ce/html5/thumbnails/45.jpg)
Learning of and retrieval of
Dynamical movement primitives with GMR
45
![Page 46: NONLINEAR REGRESSION I - Idiap](https://reader031.fdocuments.us/reader031/viewer/2022012413/616d870afc2d5b20a12ee7ce/html5/thumbnails/46.jpg)
References
LWR
C. G. Atkeson, A. W. Moore, and S. Schaal. Locally weighted learning for control. Artificial Intelligence Review, 11(1-5):75–113, 1997
W.S. Cleveland. Robust locally weighted regression and smoothing scatterplots. American Statistical Association 74(368):829–836, 1979
GMR
Z. Ghahramani and M. I. Jordan. Supervised learning from incomplete data via an EM approach. In Advances in Neural Information Processing Systems (NIPS), volume 6, pages 120–127, 1994
S. Calinon. Mixture models for the analysis, edition, and synthesis of continuous time series. Mixture Models and Applications, Springer, 2019
DMP
A. Ijspeert, J. Nakanishi and S. Schaal. Learning Control Policies For Movement Imitation and Movement recognition. NIPS’2003
A. Ijspeert, J. Nakanishi, P. Pastor, H. Hoffmann, and S. Schaal. Dynamical movement primitives: Learning attractor models for motor behaviors. Neural Computation, 25(2):328–373, 2013 46
![Page 47: NONLINEAR REGRESSION I - Idiap](https://reader031.fdocuments.us/reader031/viewer/2022012413/616d870afc2d5b20a12ee7ce/html5/thumbnails/47.jpg)
Appendix
47
![Page 48: NONLINEAR REGRESSION I - Idiap](https://reader031.fdocuments.us/reader031/viewer/2022012413/616d870afc2d5b20a12ee7ce/html5/thumbnails/48.jpg)
t=0 t=1 t=2
Kalman filter with feedback gains
Kalman filter as product of Gaussians
Kalman filter