Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces...
-
date post
21-Dec-2015 -
Category
Documents
-
view
223 -
download
2
Transcript of Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces...
![Page 1: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/1.jpg)
Object Orie’d Data Analysis, Last Time
• Mildly Non-Euclidean Spaces
• Strongly Non-Euclidean Spaces– Tree spaces
– No Tangent Plane
• Classification - Discrimination– Mean Difference (Centroid) method
– Fisher Linear Discrimination• Graphical Viewpoint (nonparametric)
• Maximum Likelihood Derivation (Gaussian based)
![Page 2: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/2.jpg)
Classification - Discrimination
Background: Two Class (Binary) version:
Using “training data” from
Class +1 and Class -1
Develop a “rule” for
assigning new data to a Class
Canonical Example: Disease Diagnosis
• New Patients are “Healthy” or “Ill”
• Determined based on measurements
![Page 3: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/3.jpg)
Classification - Discrimination
Important Distinction: Classification vs. Clustering
Classification:Class labels are known,
Goal: understand differencesClustering:
Goal: Find class labels (to be similar)Both are about clumps of similar data,
but much different goals
![Page 4: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/4.jpg)
Classification - Discrimination
Important Distinction:
Classification vs. Clustering
Useful terminology:
Classification: supervised learning
Clustering: unsupervised learning
![Page 5: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/5.jpg)
Fisher Linear Discrimination
Graphical Introduction (non-Gaussian):
![Page 6: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/6.jpg)
Classical Discrimination
Above derivation of FLD was:
• Nonstandard
• Not in any textbooks(?)
• Nonparametric (don’t need Gaussian
data)
• I.e. Used no probability distributions
• More Machine Learning than Statistics
![Page 7: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/7.jpg)
Classical DiscriminationSummary of FLD vs. GLR:• Tilted Point Clouds Data
– FLD good– GLR good
• Donut Data– FLD bad– GLR good
• X Data– FLD bad– GLR OK, not great
Classical Conclusion: GLR generally better
(will see a different answer for HDLSS data)
![Page 8: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/8.jpg)
Classical DiscriminationFLD Generalization II (Gen. I was GLR)
Different prior probabilitiesMain idea:
Give different weights to 2 classes• I.e. assume not a priori equally likely• Development is “straightforward”• Modified likelihood• Change intercept in FLD• Won’t explore further here
![Page 9: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/9.jpg)
Classical DiscriminationFLD Generalization III
Principal Discriminant Analysis• Idea: FLD-like approach to > two
classes• Assumption: Class covariance
matrices are the same (similar)(but not Gaussian, same situation as for
FLD)• Main idea:
Quantify “location of classes” by their means
k ,...,, 21
![Page 10: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/10.jpg)
Classical DiscriminationPrincipal Discriminant Analysis (cont.)Simple way to find “interesting
directions” among the means:PCA on set of means
i.e. Eigen-analysis of “between class covariance matrix”
Where
Aside: can show: overall
tB MM
)()1(1 k
kM
wB nkn
![Page 11: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/11.jpg)
Classical DiscriminationPrincipal Discriminant Analysis (cont.)
But PCA only works like Mean
Difference,
Expect can improve bytaking covariance into account.
Blind application of above ideas
suggests eigen-analysis of:Bw 1
![Page 12: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/12.jpg)
Classical DiscriminationPrincipal Discriminant Analysis (cont.)There are:• smarter ways to compute
(“generalized eigenvalue”)• other representations
(this solves optimization prob’s)Special case: 2 classes,
reduces to standard FLDGood reference for more: Section 3.8
of:Duda, Hart & Stork (2001)
![Page 13: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/13.jpg)
Classical DiscriminationSummary of Classical Ideas:• Among “Simple Methods”
– MD and FLD sometimes similar– Sometimes FLD better– So FLD is preferred
• Among Complicated Methods– GLR is best– So always use that
• Caution:– Story changes for HDLSS settings
![Page 14: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/14.jpg)
HDLSS Discrimination
Recall main HDLSS issues:
• Sample Size, n < Dimension, d
• Singular covariance matrix
• So can’t use matrix inverse
• I.e. can’t standardize (sphere) the data
(requires root inverse covariance)
• Can’t do classical multivariate analysis
![Page 15: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/15.jpg)
HDLSS DiscriminationAn approach to non-invertible
covariances:• Replace by generalized inverses• Sometimes called pseudo inverses• Note: there are several• Here use Moore Penrose inverse• As used by Matlab (pinv.m)• Often provides useful results
(but not always)Recall Linear Algebra Review…
![Page 16: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/16.jpg)
Recall Linear Algebra
Eigenvalue Decomposition:
For a (symmetric) square matrix
Find a diagonal matrix
And an orthonormal matrix
(i.e. )
So that: , i.e.
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
![Page 17: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/17.jpg)
Recall Linear Algebra (Cont.)
Eigenvalue Decomp. solves matrix problems:
• Inversion:
• Square Root:
• is positive (nonn’ve, i.e. semi) definite
all
t
d
BBX
1
11
1
0
0
t
d
BBX
2/1
2/11
2/1
0
0
X
0)(i
![Page 18: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/18.jpg)
Recall Linear Algebra (Cont.)
Moore-Penrose Generalized Inverse:
For
tr BBX
000
0
0
0
00
1
11
0,,0,, 11 drr
![Page 19: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/19.jpg)
Recall Linear Algebra (Cont.)
Easy to see this satisfies the definition of
Generalized (Pseudo) Inverse
•
•
• symmetric
• symmetric
XXXX
XXXXXX
XX
![Page 20: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/20.jpg)
Recall Linear Algebra (Cont.)
Moore-Penrose Generalized Inverse:
Idea: matrix inverse on non-null space of the corresponding linear transformation
Reduces to ordinary inverse, in full rank case,i.e. for r = d, so could just always use this
Tricky aspect:“>0 vs. = 0” & floating point arithmetic
![Page 21: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/21.jpg)
HDLSS DiscriminationApplication of Generalized Inverse to
FLD:
Direction (Normal) Vector:
Intercept:
Have replaced by
)2()1(ˆ XXn wFLD
)2()1(
21
21
XXFLD
1ˆ w wˆ
![Page 22: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/22.jpg)
HDLSS DiscriminationToy Example: Increasing Dimension
data vectors:
• Entry 1:
Class +1: Class –1:
• Other Entries:
• All Entries Independent
Look through dimensions,
1,2.2N
20 nn
1,2.2N
1,0N
1000,,2,1 d
![Page 23: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/23.jpg)
HDLSS Discrimination
Increasing Dimension Example
Proj. on
Opt’l Dir’n
Proj. on
FLD Dir’n
Proj. on
both Dir’ns
![Page 24: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/24.jpg)
HDLSS Discrimination
Add a 2nd Dimension (noise)
Same Proj. on
Opt’l Dir’n
Axes same
as dir’ns
Now See 2
Dim’ns
![Page 25: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/25.jpg)
HDLSS Discrimination
Add a 3rd Dimension (noise)
Project on
2-d subspace
generated
by optimal
dir’n & by
FLD dir’n
![Page 26: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/26.jpg)
HDLSS DiscriminationMovie Through Increasing Dimensions
Link
![Page 27: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/27.jpg)
HDLSS DiscriminationFLD in Increasing Dimensions:• Low dimensions (d = 2-9):
– Visually good separation– Small angle between FLD and Optimal– Good generalizability
• Medium Dimensions (d = 10-26):– Visual separation too good?!?– Larger angle between FLD and Optimal– Worse generalizability– Feel effect of sampling noise
![Page 28: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/28.jpg)
HDLSS DiscriminationFLD in Increasing Dimensions:• High Dimensions (d=27-37):
– Much worse angle– Very poor generalizability– But very small within class variation– Poor separation between classes– Large separation / variation ratio
![Page 29: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/29.jpg)
HDLSS DiscriminationFLD in Increasing Dimensions:• At HDLSS Boundary (d=38):
– 38 = degrees of freedom(need to estimate 2 class means)
– Within class variation = 0 ?!?– Data pile up, on just two points– Perfect separation / variation ratio?– But only feels microscopic noise aspects
So likely not generalizable– Angle to optimal very large
![Page 30: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/30.jpg)
HDLSS Discrimination
FLD in Increasing Dimensions:
• Just beyond HDLSS boundary (d=39-
70):
– Improves with higher dimension?!?
– Angle gets better
– Improving generalizability?
– More noise helps classification?!?
![Page 31: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/31.jpg)
HDLSS DiscriminationFLD in Increasing Dimensions:• Far beyond HDLSS boun’ry (d=70-
1000):– Quality degrades– Projections look terrible
(populations overlap)– And Generalizability falls apart, as well– Math’s worked out by Bickel & Levina
(2004)– Problem is estimation of d x d
covariance matrix
![Page 32: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/32.jpg)
HDLSS DiscriminationSimple Solution:
Mean Difference (Centroid) Method• Recall not classically recommended
– Usually no better than FLD– Sometimes worse
• But avoids estimation of covariance• Means are very stable• Don’t feel HDLSS problem
![Page 33: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/33.jpg)
HDLSS DiscriminationMean Difference (Centroid) Method
Same Data,
Movie overdim’s
Link
![Page 34: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/34.jpg)
HDLSS DiscriminationMean Difference (Centroid) Method• Far more stable over dimensions• Because is likelihood ratio solution
(for known variance - Gaussians)• Doesn’t feel HDLSS boundary• Eventually becomes too good?!?
Widening gap between clusters?!?• Careful: angle to optimal grows• So lose generalizabilityHDLSS data present some odd effects…
![Page 35: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/35.jpg)
Maximal Data Piling
Strange FLD effect at HDLSS boundary:
Data Piling:
For each class,
all data
project to
single
value
![Page 36: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/36.jpg)
Maximal Data PilingWhat is happening?
• Hard to imagine
• Since our intuition is 3-dim’al
• Came from our ancestors…
Try to understand data piling with
some simple examples
![Page 37: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/37.jpg)
Maximal Data PilingSimple example (Ahn & Marron 2005):
in
Let be the hyperplane:
• Generated by Class +1
• Which has dimension = 1
• I.e. line containing the 2 points
Similarly, let be the hyperplane
• Generated by Class -1
2 nn 3
~
~
![Page 38: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/38.jpg)
Maximal Data PilingSimple example: in
Let be
• Parallel shifts of
• So that they pass through the origin
• Still have dimension 1
• But now are subspaces
2 nn 3
,
~,~
![Page 39: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/39.jpg)
Maximal Data PilingSimple example: in2 nn 3
![Page 40: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/40.jpg)
Maximal Data PilingSimple example: in
Construction 1:
Let be
• Subspace generated by
• Two dimensional
• Shown as cyan plane
2 nn 3
&
![Page 41: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/41.jpg)
Maximal Data PilingSimple example: in
Construction 1 (cont.):
Let be
• Direction orthogonal to
• One dimensional
• Makes Class +1 Data project to one point
• And Class -1 Data project to one point
• Called Maximal Data Piling Direction
2 nn 3
MDPv
![Page 42: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/42.jpg)
Maximal Data PilingSimple example: in
Construction 2:
Let be
• Subspaces orthogonal to
(respectively)
• Projection collapses Class +1
• Projection collapses Class -1
• Both are 2-d (planes)
2 nn 3
&
&
![Page 43: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/43.jpg)
Maximal Data PilingSimple example: in
Construction 2 (cont.):
Let intersection of be
• Same Maximal Data Piling Direction
• Projection collapses both
Class +1 and Class -1
• Intersection of 2-d (planes) is 1-d dir’n
2 nn 3
& MDPv
![Page 44: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/44.jpg)
Maximal Data PilingGeneral Case: in with
Let be
• Hyperplanes generated by Classes
• Of Dimensions (resp.)
Let be
• Parallel subspaces
• I.e. shifts to origin
• Of Dimensions (resp.)
nn , d
~&~
1,1 nn
nnd
&
1,1 nn
![Page 45: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/45.jpg)
Maximal Data PilingGeneral Case: in with
Let be
• Orthogonal Hyperplanes
• Of Dim’ns (resp.)
• Where
– Proj’n in Dir’ns Collapse Class +1
– Proj’n in Dir’ns Collapse Class -1
• Expect intersection
nn , d
&
1,1 ndnd
nnd
2 nnd
![Page 46: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/46.jpg)
Maximal Data PilingGeneral Case: in with
Can show (Ahn & Marron 2005):
• Most dir’ns in intersection collapse all to 0
• But there is a direction,
• Where Classes collapse to different points
• Unique in subspace generated by the data
• Called Maximal Data Piling Direction
nn , d nnd
MDPv
![Page 47: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/47.jpg)
Maximal Data PilingMovie Through Increasing Dimensions
Link
![Page 48: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/48.jpg)
Maximal Data PilingFLD in Increasing Dimensions:• Sub HDLSS dimensions (d = 1-37):
– Looks similar to FLD?!?– Reason for this?
• At HDLSS Boundary (d = 38):– Again similar to FLD…
![Page 49: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/49.jpg)
Maximal Data PilingFLD in Increasing Dimensions:• For HDLSS dimensions (d = 39-1000):
– Always have data piling– Gap between grows for larger n– Even though noise increases?– Angle (gen’bility) first improves, d = 39–
180– Then worsens, d = 200-1000– Eventually noise dominates– Trade-off is where gap near optimal
diff’nce
![Page 50: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/50.jpg)
Maximal Data PilingHow to compute ?Can show (Ahn & Marron 2005):
Recall FLD formula:
Only difference is global vs. within classCovariance Estimates!
MDPv
XXv w
FLD
1ˆ
XXvMDP1ˆ
![Page 51: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/51.jpg)
Maximal Data Piling
Historical Note:
• Discovery of MDP
• Came from a programming error
• Forgetting to use within class
covariance
• In FLD…
![Page 52: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/52.jpg)
Maximal Data PilingVisual similarity of & ?
Can show (Ahn & Marron 2005), for d<n:
I.e. directions are the same!
• How can this be?
• Note lengths are different…
• Study from transformation viewpoint
MDPv FLDv
FLDFLDMDPMDP vvvv //
![Page 53: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/53.jpg)
Maximal Data PilingRecall
Transfo’
view of
FLD:
![Page 54: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/54.jpg)
Maximal Data PilingInclude Corres-pondingMDP Transfo’:
Both giveSameResult!
![Page 55: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/55.jpg)
Maximal Data PilingDetails:FLD, sep’ing plane normal vectorWithin Class, PC1 PC2Global, PC1 PC2
![Page 56: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/56.jpg)
Maximal Data Piling
Acknowledgement:
• This viewpoint
• I.e. insight into why FLD = MDP
(for low dim’al data)
• Suggested by Daniel Peña
![Page 57: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/57.jpg)
Maximal Data Piling
Fun e.g:
rotate
from PCA
to MDP
dir’ns
![Page 58: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/58.jpg)
Maximal Data PilingMDP for other class labellings:
• Always exists
• Separation bigger for natural clusters
• Could be used for clustering– Consider all directions
– Find one that makes largest gap
– Very hard optimization problem
– Over 2n possible directions
![Page 59: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/59.jpg)
Maximal Data PilingA point of terminology (Ahn & Marron
2005):
MDP is “maximal” in 2 senses:
1. # of data piled
2. Size of gap
(within subspace
gen’d by data)
![Page 60: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/60.jpg)
Maximal Data PilingRecurring, over-arching, issue:
HDLSS space is a weird place
![Page 61: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/61.jpg)
Kernel EmbeddingAizerman, Braverman and Rozoner
(1964) • Motivating idea:
Extend scope of linear discrimination,By adding nonlinear components to data
(embedding in a higher dim’al space)
• Better use of name:nonlinear discrimination?
![Page 62: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/62.jpg)
Kernel EmbeddingToy Examples:In 1d, linear separation splits the
domaininto only 2 parts
xx :
![Page 63: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/63.jpg)
Kernel EmbeddingBut in the
“quadratic embedded
domain”,
linear separation can give 3 parts
22 :, xxx
![Page 64: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/64.jpg)
Kernel EmbeddingBut in the quadratic embedded domain
Linear separation can give 3 parts• original data space lies in 1d manifold• very sparse region of • curvature of manifold gives:
better linear separation• can have any 2 break points
(2 points line)
22 :, xxx
2
![Page 65: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/65.jpg)
Kernel EmbeddingStronger effects for higher order polynomial embedding:
E.g. for cubic,
linear separation can give 4 parts (or fewer)
332 :,, xxxx
![Page 66: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/66.jpg)
Kernel EmbeddingStronger effects - high. ord. poly.
embedding:• original space lies in 1-d manifold,
even sparser in • higher d curvature gives:
improved linear separation• can have any 3 break points
(3 points plane)?• Note: relatively few
“interesting separating planes”
3
![Page 67: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/67.jpg)
Kernel EmbeddingGeneral View: for original data
matrix:
add rows:
i.e. embed inHigherDimensionalspace
dnd
n
xx
xx
1
111
nn
dnd
n
dnd
n
xxxx
xx
xx
xx
xx
212111
221
21
211
1
111
![Page 68: Object Orie’d Data Analysis, Last Time Mildly Non-Euclidean Spaces Strongly Non-Euclidean Spaces –Tree spaces –No Tangent Plane Classification - Discrimination.](https://reader030.fdocuments.us/reader030/viewer/2022032704/56649d595503460f94a399fe/html5/thumbnails/68.jpg)
Kernel EmbeddingEmbeddedFisher Linear Discrimination:
Choose Class 1, for any when:
in embedded space.• image of class boundaries in original
space is nonlinear• allows more complicated class regions• Can also do Gaussian Lik. Rat. (or
others) • Compute image by classifying points
from original space
dx 0
)2()1(1)2()1()2()1(10 ˆ2
1ˆ XXXXXXx wwt