Multivariate Dyadic Regression Trees for Sparse Learning Problems Xi Chen Machine Learning...
-
Upload
nathaniel-stewart -
Category
Documents
-
view
217 -
download
2
Transcript of Multivariate Dyadic Regression Trees for Sparse Learning Problems Xi Chen Machine Learning...
![Page 1: Multivariate Dyadic Regression Trees for Sparse Learning Problems Xi Chen Machine Learning Department Carnegie Mellon University (joint work with Han Liu)](https://reader030.fdocuments.us/reader030/viewer/2022032612/56649f215503460f94c3974e/html5/thumbnails/1.jpg)
Multivariate Dyadic Regression Trees for Sparse Learning Problems
Xi ChenMachine Learning Department
Carnegie Mellon University(joint work with Han Liu)
![Page 2: Multivariate Dyadic Regression Trees for Sparse Learning Problems Xi Chen Machine Learning Department Carnegie Mellon University (joint work with Han Liu)](https://reader030.fdocuments.us/reader030/viewer/2022032612/56649f215503460f94c3974e/html5/thumbnails/2.jpg)
Content
Experimental Results
Statistical Property
Multivariate Regression and Dyadic Regression Tree
Tree Learning Algorithm
Multivariate Dyadic Regression Tree for Sparse Learning
![Page 3: Multivariate Dyadic Regression Trees for Sparse Learning Problems Xi Chen Machine Learning Department Carnegie Mellon University (joint work with Han Liu)](https://reader030.fdocuments.us/reader030/viewer/2022032612/56649f215503460f94c3974e/html5/thumbnails/3.jpg)
Multivariate Regression Model
Multivariate Regression Model
Predictors Responses
Estimate : Minimize the L2-risk
Empirical Risk Minimization
![Page 4: Multivariate Dyadic Regression Trees for Sparse Learning Problems Xi Chen Machine Learning Department Carnegie Mellon University (joint work with Han Liu)](https://reader030.fdocuments.us/reader030/viewer/2022032612/56649f215503460f94c3974e/html5/thumbnails/4.jpg)
Tree Based Method
Estimation using tree based methodsWhy trees? Simplicity of Design Good Interpretability Easy Implementation Good Practical Performance
![Page 5: Multivariate Dyadic Regression Trees for Sparse Learning Problems Xi Chen Machine Learning Department Carnegie Mellon University (joint work with Han Liu)](https://reader030.fdocuments.us/reader030/viewer/2022032612/56649f215503460f94c3974e/html5/thumbnails/5.jpg)
Tree Based Method
CART (Classification and Regression Tree)[Breiman 1984]
No. of terminal nodesHard to be theoretically analyzed!
![Page 6: Multivariate Dyadic Regression Trees for Sparse Learning Problems Xi Chen Machine Learning Department Carnegie Mellon University (joint work with Han Liu)](https://reader030.fdocuments.us/reader030/viewer/2022032612/56649f215503460f94c3974e/html5/thumbnails/6.jpg)
Dyadic Decision/Regression Tree
Dyadic Split[Scott 2004]
![Page 7: Multivariate Dyadic Regression Trees for Sparse Learning Problems Xi Chen Machine Learning Department Carnegie Mellon University (joint work with Han Liu)](https://reader030.fdocuments.us/reader030/viewer/2022032612/56649f215503460f94c3974e/html5/thumbnails/7.jpg)
Sparse Model
Lower Minimax Rate of Convergence of the risk
Slow
Fast
Sparse Model
![Page 8: Multivariate Dyadic Regression Trees for Sparse Learning Problems Xi Chen Machine Learning Department Carnegie Mellon University (joint work with Han Liu)](https://reader030.fdocuments.us/reader030/viewer/2022032612/56649f215503460f94c3974e/html5/thumbnails/8.jpg)
Regression Tree
Piecewise Constant
Piecewise Linear
Piecewise Polynomial
Gamma-Ray Burst 845
![Page 9: Multivariate Dyadic Regression Trees for Sparse Learning Problems Xi Chen Machine Learning Department Carnegie Mellon University (joint work with Han Liu)](https://reader030.fdocuments.us/reader030/viewer/2022032612/56649f215503460f94c3974e/html5/thumbnails/9.jpg)
Multivariate Dyadic Regression Tree (MDRT)
Active Set
Rule 1
Rule 2
Multivariate Dyadic Regression Tree (MDRT) Variable Selection
![Page 10: Multivariate Dyadic Regression Trees for Sparse Learning Problems Xi Chen Machine Learning Department Carnegie Mellon University (joint work with Han Liu)](https://reader030.fdocuments.us/reader030/viewer/2022032612/56649f215503460f94c3974e/html5/thumbnails/10.jpg)
Multivariate Dyadic Regression Tree
Regularization Parameter
Fine partitionSparse Model
Lower degree poly
![Page 11: Multivariate Dyadic Regression Trees for Sparse Learning Problems Xi Chen Machine Learning Department Carnegie Mellon University (joint work with Han Liu)](https://reader030.fdocuments.us/reader030/viewer/2022032612/56649f215503460f94c3974e/html5/thumbnails/11.jpg)
Statistical Property
Assumption 1:
Assumption 2:
Convergence Rate
Minimax Rate
![Page 12: Multivariate Dyadic Regression Trees for Sparse Learning Problems Xi Chen Machine Learning Department Carnegie Mellon University (joint work with Han Liu)](https://reader030.fdocuments.us/reader030/viewer/2022032612/56649f215503460f94c3974e/html5/thumbnails/12.jpg)
Tree Learning Algorithm
Loss:
Minimize the cost
![Page 13: Multivariate Dyadic Regression Trees for Sparse Learning Problems Xi Chen Machine Learning Department Carnegie Mellon University (joint work with Han Liu)](https://reader030.fdocuments.us/reader030/viewer/2022032612/56649f215503460f94c3974e/html5/thumbnails/13.jpg)
Tree Learning Algorithm
Tree-growing stage
Pruning-back stage
Randomized
Greedy
![Page 14: Multivariate Dyadic Regression Trees for Sparse Learning Problems Xi Chen Machine Learning Department Carnegie Mellon University (joint work with Han Liu)](https://reader030.fdocuments.us/reader030/viewer/2022032612/56649f215503460f94c3974e/html5/thumbnails/14.jpg)
Experimental Results
Methods Compared
Methods
Greedy MDRT with M=1 MDRT(G, M=1)
Randomized MDRT with M=1 MDRT(R, M=1)
Greedy MDRT with M=0 MDRT(G, M=0)
Randomized MDRT with M=0 MDRT(R, M=0)
Classification and Regression Tree CART
Piecewise LinearPiecewise Constant
![Page 15: Multivariate Dyadic Regression Trees for Sparse Learning Problems Xi Chen Machine Learning Department Carnegie Mellon University (joint work with Han Liu)](https://reader030.fdocuments.us/reader030/viewer/2022032612/56649f215503460f94c3974e/html5/thumbnails/15.jpg)
Generalized Nonlinear Model
Experimental Results
Synthetic Data
Linear Model
Additive Model
![Page 16: Multivariate Dyadic Regression Trees for Sparse Learning Problems Xi Chen Machine Learning Department Carnegie Mellon University (joint work with Han Liu)](https://reader030.fdocuments.us/reader030/viewer/2022032612/56649f215503460f94c3974e/html5/thumbnails/16.jpg)
Experimental Results
![Page 17: Multivariate Dyadic Regression Trees for Sparse Learning Problems Xi Chen Machine Learning Department Carnegie Mellon University (joint work with Han Liu)](https://reader030.fdocuments.us/reader030/viewer/2022032612/56649f215503460f94c3974e/html5/thumbnails/17.jpg)
Experimental Results
Real Data (MSE)
10 artificial variables from Unif(0,1)
15 artificial variables from Unif(0,1)
Never selected in 20 runs for M=1
![Page 18: Multivariate Dyadic Regression Trees for Sparse Learning Problems Xi Chen Machine Learning Department Carnegie Mellon University (joint work with Han Liu)](https://reader030.fdocuments.us/reader030/viewer/2022032612/56649f215503460f94c3974e/html5/thumbnails/18.jpg)
Conclusion
Multivariate Regression Tree Model Dyadic Split A novel penalization term Theoretically, achieve nearly optimal minimax
rate for (α,C) smooth function Empirically, conduct variable selection for sparse
models Efficient computation tree learning algorithm
Extensions Classification Trees Forest Extensions
![Page 19: Multivariate Dyadic Regression Trees for Sparse Learning Problems Xi Chen Machine Learning Department Carnegie Mellon University (joint work with Han Liu)](https://reader030.fdocuments.us/reader030/viewer/2022032612/56649f215503460f94c3974e/html5/thumbnails/19.jpg)