Harmonic analysis on graphs and Lie groups : quadratic functionals, Riesz transforms and Besov
Harmonic Analysis on Graphs Global vs. Multiscale Approaches
Transcript of Harmonic Analysis on Graphs Global vs. Multiscale Approaches
![Page 1: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/1.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Harmonic Analysis on GraphsGlobal vs. Multiscale Approaches
Boaz Nadler
Weizmann Institute of Science, Rehovot, Israel
July 2011
Joint work withMatan Gavish (WIS/Stanford), Ronald Coifman (Yale), ICML 10'
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 2: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/2.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Challenge: Organization / Understanding of Data
In many �elds massive amounts of data collected or generated,
EXAMPLES:
�nancial data, multi-sensor data, simulations, documents,web-pages, images, video streams, medical data, astrophysical data,etc.
Need to organize / understand their structureInference / Learning from data
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 3: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/3.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Challenge: Organization / Understanding of Data
In many �elds massive amounts of data collected or generated,
EXAMPLES:
�nancial data, multi-sensor data, simulations, documents,web-pages, images, video streams, medical data, astrophysical data,etc.
Need to organize / understand their structureInference / Learning from data
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 4: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/4.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Classical vs. Modern Data Analysis Setup and Tasks
Classical Setup:
- Data typically in a (low-dimensional) Euclidean Space.- Small to medium sample sizes (n < 1000)- Either all data unlabeled (unsupervised) or all labeled (supervised).
Modern Setup:
- high-dimensional data, or data encoded as a graph.- Huge datasets, with n = 106 samples or more. Few labeled data.
The well developed and understood standard tools of statisticsnot always applicable
Question: Harmonic Analysis in such settings
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 5: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/5.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Classical vs. Modern Data Analysis Setup and Tasks
Classical Setup:
- Data typically in a (low-dimensional) Euclidean Space.- Small to medium sample sizes (n < 1000)- Either all data unlabeled (unsupervised) or all labeled (supervised).
Modern Setup:
- high-dimensional data, or data encoded as a graph.- Huge datasets, with n = 106 samples or more. Few labeled data.
The well developed and understood standard tools of statisticsnot always applicable
Question: Harmonic Analysis in such settings
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 6: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/6.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Classical vs. Modern Data Analysis Setup and Tasks
Classical Setup:
- Data typically in a (low-dimensional) Euclidean Space.- Small to medium sample sizes (n < 1000)- Either all data unlabeled (unsupervised) or all labeled (supervised).
Modern Setup:
- high-dimensional data, or data encoded as a graph.- Huge datasets, with n = 106 samples or more. Few labeled data.
The well developed and understood standard tools of statisticsnot always applicable
Question: Harmonic Analysis in such settings
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 7: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/7.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Classical vs. Modern Data Analysis Setup and Tasks
Classical Setup:
- Data typically in a (low-dimensional) Euclidean Space.- Small to medium sample sizes (n < 1000)- Either all data unlabeled (unsupervised) or all labeled (supervised).
Modern Setup:
- high-dimensional data, or data encoded as a graph.- Huge datasets, with n = 106 samples or more. Few labeled data.
The well developed and understood standard tools of statisticsnot always applicable
Question: Harmonic Analysis in such settings
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 8: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/8.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Harmonic Analysis and Learning
In the past 20 years (multiscale) harmonic analysis had profoundimpact on statistics and signal processing.
Well developed theory, long tradition:
geometry of space X ⇒ bases for {f : X → R}
Simplest Example: X = [0, 1]
Construct (multiscale) basis {Ψi}, that allows control of |⟨f ,Ψi ⟩|for some (smooth) class of functions f .
Theorem: ψ smooth wavelet, ψℓ,k - wavelet basis, f : [0, 1] → R,then
|f (x)− f (y)| < C |x − y |α ⇔ |⟨f , ψℓ,k⟩| ≤ C ′2−ℓ(α+1/2)
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 9: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/9.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Harmonic Analysis and Learning
In the past 20 years (multiscale) harmonic analysis had profoundimpact on statistics and signal processing.
Well developed theory, long tradition:
geometry of space X ⇒ bases for {f : X → R}
Simplest Example: X = [0, 1]
Construct (multiscale) basis {Ψi}, that allows control of |⟨f ,Ψi ⟩|for some (smooth) class of functions f .
Theorem: ψ smooth wavelet, ψℓ,k - wavelet basis, f : [0, 1] → R,then
|f (x)− f (y)| < C |x − y |α ⇔ |⟨f , ψℓ,k⟩| ≤ C ′2−ℓ(α+1/2)
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 10: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/10.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Harmonic Analysis and Learning
In the past 20 years (multiscale) harmonic analysis had profoundimpact on statistics and signal processing.
Well developed theory, long tradition:
geometry of space X ⇒ bases for {f : X → R}
Simplest Example: X = [0, 1]
Construct (multiscale) basis {Ψi}, that allows control of |⟨f ,Ψi ⟩|for some (smooth) class of functions f .
Theorem: ψ smooth wavelet, ψℓ,k - wavelet basis, f : [0, 1] → R,then
|f (x)− f (y)| < C |x − y |α ⇔ |⟨f , ψℓ,k⟩| ≤ C ′2−ℓ(α+1/2)
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 11: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/11.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Setting
Harmonic analysis wisdom for f : Rd → R
I Expand f in orthonormal basis {ψi} (e.g. wavelet)
f =∑i
⟨f , ψi ⟩ψi
I Shrink / estimate coe�cients
I Useful when f well approximated by a few terms(�fast coe�cent decay� / sparsity)
Can this work on general datasets?
Need data-adaptive basis {ψi} for space of functions f : X → R
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 12: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/12.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Setting
Harmonic analysis wisdom for f : Rd → R
I Expand f in orthonormal basis {ψi} (e.g. wavelet)
f =∑i
⟨f , ψi ⟩ψi
I Shrink / estimate coe�cients
I Useful when f well approximated by a few terms(�fast coe�cent decay� / sparsity)
Can this work on general datasets?
Need data-adaptive basis {ψi} for space of functions f : X → R
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 13: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/13.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Setting
Harmonic analysis wisdom for f : Rd → R
I Expand f in orthonormal basis {ψi} (e.g. wavelet)
f =∑i
⟨f , ψi ⟩ψi
I Shrink / estimate coe�cients
I Useful when f well approximated by a few terms(�fast coe�cent decay� / sparsity)
Can this work on general datasets?
Need data-adaptive basis {ψi} for space of functions f : X → R
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 14: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/14.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Setting
Harmonic analysis wisdom for f : Rd → R
I Expand f in orthonormal basis {ψi} (e.g. wavelet)
f =∑i
⟨f , ψi ⟩ψi
I Shrink / estimate coe�cients
I Useful when f well approximated by a few terms(�fast coe�cent decay� / sparsity)
Can this work on general datasets?
Need data-adaptive basis {ψi} for space of functions f : X → R
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 15: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/15.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Setting
Harmonic analysis wisdom for f : Rd → R
I Expand f in orthonormal basis {ψi} (e.g. wavelet)
f =∑i
⟨f , ψi ⟩ψi
I Shrink / estimate coe�cients
I Useful when f well approximated by a few terms(�fast coe�cent decay� / sparsity)
Can this work on general datasets?
Need data-adaptive basis {ψi} for space of functions f : X → R
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 16: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/16.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Harmonic Analysis on Graphs
Setup: We are given dataset X = {x1, . . . , xN}with similarity / a�nity matrix Wi ,j
Goal: Statistical inference of smooth f : X → RI Denoise f
I SSL / Regression / classi�cation: extend f from X ⊂ X to X
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 17: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/17.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Semi-Supervised Learning
In many applications - easy to collect lots of unlabeled data, BUTlabeling the data is expensive.
Question: Given (small) labeled set X ⊂ X = {xi , yi} (y = f (x)),construct f to label rest of X .
Key Assumption:
function f (x) has some smoothness w.r.t graph a�nities Wi ,j .
otherwise - unlabeled data is useless or may even harm prediction.
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 18: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/18.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Semi-Supervised Learning
In many applications - easy to collect lots of unlabeled data, BUTlabeling the data is expensive.
Question: Given (small) labeled set X ⊂ X = {xi , yi} (y = f (x)),construct f to label rest of X .
Key Assumption:
function f (x) has some smoothness w.r.t graph a�nities Wi ,j .
otherwise - unlabeled data is useless or may even harm prediction.
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 19: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/19.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
The Graph Laplacian
In most previous approaches, key object is
GRAPH LAPLACIAN:
L = D −W
where Dii =∑
j Wij ,
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 20: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/20.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Global Graph Laplacian Based SSL Methods
Zhu, Ghahramani, La�erty, SSL using Gaussian �elds and harmonicfunctions [ICML, 2003] , Azran [ICML 2007]
f (x) = arg minf (xi )=yi
1
n2
n∑i ,j=1
W i ,j(fi − fj)2 = argmin
1
n2fTLf
Y. Bengio, O. Delalleau, N. Le Roux, [2006]D. Zhou, O. Bousquet, T. Navin Lal, J. Weston, B. Scholkopf[NIPS 2004]
f (x) = argmin
1
n2
n∑i ,j=1
Wi ,j(fi − fj)2 + λ
ℓ∑i=1
(fi − yi )2
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 21: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/21.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Global Graph-Laplacian Based SSL
Belkin & Niyogi (2003): Given similarity matrix W �nd �rst feweigenvectors of Graph Laplacian, Lej = (D −W )ej = λjej .Expand
f =
p∑j=1
ajej
Estimate coe�cients aj from labeled data.
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 22: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/22.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Statistical Analysis of Laplacian Regularization
[N., Srebro, Zhou, NIPS 09']
Theorem: In the limit of large unlabeled data from Euclideanspace, with Wi ,j = K (∥xi − xj∥), Graph Laplacian RegularizatonMethods
f (x) = argmin
1
n2
n∑i ,j=1
Wi ,j(fi − fj)2 + λ
ℓ∑i=1
(fi − yi )2
are weill posed for underlying data in 1-d, but are not well posed fordata in dimension d ≥ 2.In particular, in limit of in�nite unlabeled data, f (x) → const at allunlabeled x .
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 23: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/23.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Eigenvector-Fourier Methods
Belkin, Niyogi [2003], suggested a di�erent approach based on theGraph Laplacian L = W − D:
Given similarity W , �nd �rst few eigenvectors (W − D)ej = λjej ,Expand
y(x) =
p∑j=1
ajej
Find coe�cients aj by least squares,
(a1, . . . , ap) = argminl∑
j=1
(yj − y(xj))2.
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 24: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/24.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Toy example
Example: USPS benchmark
I X is USPS (ML benchmark) as 1500 vectors in R16×16 = R256
I A�nity Wi ,j = exp(−∥xi − xj∥2
)I f : X → {1,−1} is the class label.
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 25: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/25.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Toy example
Example: USPS benchmark
I X is USPS (ML benchmark) as 1500 vectors in R16×16 = R256
I A�nity Wi ,j = exp(−∥xi − xj∥2
)I f : X → {1,−1} is the class label.
2 4 6 8 10 12 14 16
2
4
6
8
10
12
14
16
2 4 6 8 10 12 14 16
2
4
6
8
10
12
14
16
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 26: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/26.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Toy example
Example: USPS benchmark
I X is USPS (ML benchmark) as 1500 vectors in R16×16 = R256
I A�nity Wi ,j = exp(−∥xi − xj∥2
)
I f : X → {1,−1} is the class label.
2 4 6 8 10 12 14 16
2
4
6
8
10
12
14
16
2 4 6 8 10 12 14 16
2
4
6
8
10
12
14
16
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 27: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/27.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Toy example
Example: USPS benchmark
I X is USPS (ML benchmark) as 1500 vectors in R16×16 = R256
I A�nity Wi ,j = exp(−∥xi − xj∥2
)I f : X → {1,−1} is the class label.
2 4 6 8 10 12 14 16
2
4
6
8
10
12
14
16
2 4 6 8 10 12 14 16
2
4
6
8
10
12
14
16
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 28: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/28.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Toy example: visualization by kernel PCA
−0.015 −0.01 −0.005 0 0.005 0.01 0.015 0.02 0.025
−0.02
−0.01
0
0.01
0.02−0.025
−0.02
−0.015
−0.01
−0.005
0
0.005
0.01
0.015
−1
−0.8
−0.6
−0.4
−0.2
0
0.2
0.4
0.6
0.8
1
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 29: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/29.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Toy example: Prior art?
Generalizing Fourier: The Graph Laplacian eigenbasis
I Take (W − D)ψi = λiψi where Di ,i =∑
j Wi ,j
I (Belkin, Niyogi 2004) and others
In Euclidean setting
I Typical coe�cient decay rate in Fourier basis: polynomial
I But in wavelet bases: exponential
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 30: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/30.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Toy example: Prior art?
Generalizing Fourier: The Graph Laplacian eigenbasis
I Take (W − D)ψi = λiψi where Di ,i =∑
j Wi ,j
I (Belkin, Niyogi 2004) and others
In Euclidean setting
I Typical coe�cient decay rate in Fourier basis: polynomial
I But in wavelet bases: exponential
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 31: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/31.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Toy example: Prior art?
Generalizing Fourier: The Graph Laplacian eigenbasis
I Take (W − D)ψi = λiψi where Di ,i =∑
j Wi ,j
I (Belkin, Niyogi 2004) and others
In Euclidean setting
I Typical coe�cient decay rate in Fourier basis: polynomial
I But in wavelet bases: exponential
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 32: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/32.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Toy example: Prior art?
Generalizing Fourier: The Graph Laplacian eigenbasis
I Take (W − D)ψi = λiψi where Di ,i =∑
j Wi ,j
I (Belkin, Niyogi 2004) and others
In Euclidean setting
I Typical coe�cient decay rate in Fourier basis: polynomial
I But in wavelet bases: exponential
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 33: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/33.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Toy example: Prior art?
Generalizing Fourier: The Graph Laplacian eigenbasis
I Take (W − D)ψi = λiψi where Di ,i =∑
j Wi ,j
I (Belkin, Niyogi 2004) and others
In Euclidean setting
I Typical coe�cient decay rate in Fourier basis: polynomial
I But in wavelet bases: exponential
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 34: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/34.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Toy example: Prior art?
Generalizing Fourier: The Graph Laplacian eigenbasis
I Take (W − D)ψi = λiψi where Di ,i =∑
j Wi ,j
I (Belkin, Niyogi 2004) and others
In Euclidean setting
I Typical coe�cient decay rate in Fourier basis: polynomial
I But in wavelet bases: exponential
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 35: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/35.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Toy example: Prior art?
Generalizing Fourier: The Graph Laplacian eigenbasis
I Take (W − D)ψi = λiψi where Di ,i =∑
j Wi ,j
I (Belkin, Niyogi 2004) and others
In Euclidean setting
I Typical coe�cient decay rate in Fourier basis: polynomial
I But in wavelet bases: exponential
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 36: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/36.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Toy example: Graph Laplacian Eigenbasis
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 37: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/37.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Toy example: Graph Laplacian Eigenbasis
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 38: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/38.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Toy example: Graph Laplacian Eigenbasis
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 39: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/39.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Challenge
Challenge: build multiscale "wavelet-like" bases on X
1. Construct adaptive multiscale basis on general dataset
2. Investigate: coe�cient ⟨f , ψi ⟩ decay rate ⇔ f smooth
Previous Works- Di�fusion Wavelets [Coifman and Maggioni]- Multiscale Methods for Data on Graphs [Jansen, Nason,Silverman]- Wavelets on Graphs via Spectral Graph Theory [Hammond et al.]
Our Work- (Relavitely) Simple Construction-Accompanying Theory.
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 40: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/40.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Challenge
Challenge: build multiscale "wavelet-like" bases on X
1. Construct adaptive multiscale basis on general dataset
2. Investigate: coe�cient ⟨f , ψi ⟩ decay rate ⇔ f smooth
Previous Works- Di�fusion Wavelets [Coifman and Maggioni]- Multiscale Methods for Data on Graphs [Jansen, Nason,Silverman]- Wavelets on Graphs via Spectral Graph Theory [Hammond et al.]
Our Work- (Relavitely) Simple Construction-Accompanying Theory.
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 41: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/41.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Challenge
Challenge: build multiscale "wavelet-like" bases on X
1. Construct adaptive multiscale basis on general dataset
2. Investigate: coe�cient ⟨f , ψi ⟩ decay rate ⇔ f smooth
Previous Works- Di�fusion Wavelets [Coifman and Maggioni]- Multiscale Methods for Data on Graphs [Jansen, Nason,Silverman]- Wavelets on Graphs via Spectral Graph Theory [Hammond et al.]
Our Work- (Relavitely) Simple Construction-Accompanying Theory.
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 42: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/42.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Challenge
Challenge: build multiscale "wavelet-like" bases on X
1. Construct adaptive multiscale basis on general dataset
2. Investigate: coe�cient ⟨f , ψi ⟩ decay rate ⇔ f smooth
Previous Works- Di�fusion Wavelets [Coifman and Maggioni]- Multiscale Methods for Data on Graphs [Jansen, Nason,Silverman]- Wavelets on Graphs via Spectral Graph Theory [Hammond et al.]
Our Work- (Relavitely) Simple Construction-Accompanying Theory.
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 43: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/43.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Challenge
Challenge: build multiscale "wavelet-like" bases on X
1. Construct adaptive multiscale basis on general dataset
2. Investigate: coe�cient ⟨f , ψi ⟩ decay rate ⇔ f smooth
Previous Works- Di�fusion Wavelets [Coifman and Maggioni]- Multiscale Methods for Data on Graphs [Jansen, Nason,Silverman]- Wavelets on Graphs via Spectral Graph Theory [Hammond et al.]
Our Work- (Relavitely) Simple Construction-Accompanying Theory.
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 44: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/44.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Challenge
Challenge: build multiscale "wavelet-like" bases on X
1. Construct adaptive multiscale basis on general dataset
2. Investigate: coe�cient ⟨f , ψi ⟩ decay rate ⇔ f smooth
Previous Works- Di�fusion Wavelets [Coifman and Maggioni]- Multiscale Methods for Data on Graphs [Jansen, Nason,Silverman]- Wavelets on Graphs via Spectral Graph Theory [Hammond et al.]
Our Work- (Relavitely) Simple Construction-Accompanying Theory.
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 45: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/45.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Toy example: intriguing experiment
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 46: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/46.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Toy example: intriguing experiment
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 47: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/47.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Toy example: intriguing experiment
0 500 1000 150010
−5
10−4
10−3
10−2
10−1
100
101
102
sorted abs(coefficients) on log10 scale
eigenbasis
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 48: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/48.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Toy example: intriguing experiment
0 500 1000 150010
−18
10−16
10−14
10−12
10−10
10−8
10−6
10−4
10−2
100
102
sorted abs(coefficients) on log10 scale
eigenbasishaar−like basis
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 49: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/49.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Toy example: intriguing experiment
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 50: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/50.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Toy example: intriguing experiment
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 51: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/51.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Challenge and Result
Challenge: build "wavelet" bases on X
1. Construct adaptive multiscale basis on general dataset
2. Investigate: coe�cient ⟨f , ψi ⟩ decay rate ⇔ f smooth
Key Results
1. Partition tree on X induces �wavelet� Haar-like bases
2. Assuming �Balanced� tree
f smooth ⇐⇒ fast coe�cient decay
3. Novel SSL scheme with learning guarantees assuming smoothfunctions on tree.
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 52: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/52.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Challenge and Result
Challenge: build "wavelet" bases on X
1. Construct adaptive multiscale basis on general dataset
2. Investigate: coe�cient ⟨f , ψi ⟩ decay rate ⇔ f smooth
Key Results
1. Partition tree on X induces �wavelet� Haar-like bases
2. Assuming �Balanced� tree
f smooth ⇐⇒ fast coe�cient decay
3. Novel SSL scheme with learning guarantees assuming smoothfunctions on tree.
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 53: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/53.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Challenge and Result
Challenge: build "wavelet" bases on X
1. Construct adaptive multiscale basis on general dataset
2. Investigate: coe�cient ⟨f , ψi ⟩ decay rate ⇔ f smooth
Key Results
1. Partition tree on X induces �wavelet� Haar-like bases
2. Assuming �Balanced� tree
f smooth ⇐⇒ fast coe�cient decay
3. Novel SSL scheme with learning guarantees assuming smoothfunctions on tree.
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 54: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/54.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Challenge and Result
Challenge: build "wavelet" bases on X
1. Construct adaptive multiscale basis on general dataset
2. Investigate: coe�cient ⟨f , ψi ⟩ decay rate ⇔ f smooth
Key Results
1. Partition tree on X induces �wavelet� Haar-like bases
2. Assuming �Balanced� tree
f smooth ⇐⇒ fast coe�cient decay
3. Novel SSL scheme with learning guarantees assuming smoothfunctions on tree.
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 55: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/55.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Challenge and Result
Challenge: build "wavelet" bases on X
1. Construct adaptive multiscale basis on general dataset
2. Investigate: coe�cient ⟨f , ψi ⟩ decay rate ⇔ f smooth
Key Results
1. Partition tree on X induces �wavelet� Haar-like bases
2. Assuming �Balanced� tree
f smooth ⇐⇒ fast coe�cient decay
3. Novel SSL scheme with learning guarantees assuming smoothfunctions on tree.
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 56: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/56.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Challenge and Result
Challenge: build "wavelet" bases on X
1. Construct adaptive multiscale basis on general dataset
2. Investigate: coe�cient ⟨f , ψi ⟩ decay rate ⇔ f smooth
Key Results
1. Partition tree on X induces �wavelet� Haar-like bases
2. Assuming �Balanced� tree
f smooth ⇐⇒ fast coe�cient decay
3. Novel SSL scheme with learning guarantees assuming smoothfunctions on tree.
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 57: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/57.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Challenge and Result
Challenge: build "wavelet" bases on X
1. Construct adaptive multiscale basis on general dataset
2. Investigate: coe�cient ⟨f , ψi ⟩ decay rate ⇔ f smooth
Key Results
1. Partition tree on X induces �wavelet� Haar-like bases
2. Assuming �Balanced� tree
f smooth ⇐⇒ fast coe�cient decay
3. Novel SSL scheme with learning guarantees assuming smoothfunctions on tree.
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 58: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/58.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Challenge and Result
Challenge: build "wavelet" bases on X
1. Construct adaptive multiscale basis on general dataset
2. Investigate: coe�cient ⟨f , ψi ⟩ decay rate ⇔ f smooth
Key Results
1. Partition tree on X induces �wavelet� Haar-like bases
2. Assuming �Balanced� tree
f smooth ⇐⇒ fast coe�cient decay
3. Novel SSL scheme with learning guarantees assuming smoothfunctions on tree.
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 59: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/59.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
The Haar Basis on [0, 1]
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 60: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/60.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
The Haar Basis on [0, 1]
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 61: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/61.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
The Haar Basis on [0, 1]
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 62: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/62.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
The Haar Basis on [0, 1]
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 63: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/63.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
The Haar Basis on [0, 1]
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 64: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/64.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
The Haar Basis on [0, 1]
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 65: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/65.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
The Haar Basis on [0, 1]
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 66: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/66.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
The Haar Basis on [0, 1]
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 67: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/67.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
The Haar Basis on [0, 1]
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 68: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/68.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
The Haar Basis on [0, 1]
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 69: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/69.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
The Haar Basis on [0, 1]
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 70: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/70.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Hierarchical partition of X
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 71: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/71.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Hierarchical partition of X
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 72: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/72.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Hierarchical partition of X
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 73: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/73.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Hierarchical partition of X
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 74: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/74.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Hierarchical partition of X
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 75: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/75.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Partition Tree (Dendrogram)
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 76: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/76.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Partition Tree (Dendrogram)
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 77: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/77.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Partition Tree (Dendrogram)
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 78: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/78.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Partition Tree (Dendrogram)
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 79: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/79.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Partition Tree (Dendrogram)
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 80: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/80.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Partition Tree (Dendrogram)
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 81: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/81.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Partition Tree (Dendrogram)
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 82: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/82.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Partition Tree ⇒ Haar-like basis
[Lee, N., Wasserman, AOAS 08']Simple Observation:
Hierarchical Tree → Mutli-Resolution Analysis of Space of Functions→ Haar-like multiscale basis
V = {f | f : X → R}V ℓ = {f | f constant at partitions at level ℓ}
V 1 ⊂ V 2 ⊂ . . .V L = V
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 83: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/83.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Partition Tree ⇒ Haar-like basis
[Lee, N., Wasserman, AOAS 08']Simple Observation:
Hierarchical Tree → Mutli-Resolution Analysis of Space of Functions→ Haar-like multiscale basis
V = {f | f : X → R}V ℓ = {f | f constant at partitions at level ℓ}
V 1 ⊂ V 2 ⊂ . . .V L = V
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 84: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/84.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Partition Tree ⇒ Haar-like basis
[Lee, N., Wasserman, AOAS 08']Simple Observation:
Hierarchical Tree → Mutli-Resolution Analysis of Space of Functions→ Haar-like multiscale basis
V = {f | f : X → R}V ℓ = {f | f constant at partitions at level ℓ}
V 1 ⊂ V 2 ⊂ . . .V L = V
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 85: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/85.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Partition Tree ⇒ Haar-like basis
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 86: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/86.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Partition Tree ⇒ Haar-like basis
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 87: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/87.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Partition Tree ⇒ Haar-like basis
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 88: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/88.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Partition Tree ⇒ Haar-like basis
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 89: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/89.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Partition Tree ⇒ Haar-like basis
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 90: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/90.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Partition Tree ⇒ Haar-like basis
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 91: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/91.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Partition Tree ⇒ Haar-like basis
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 92: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/92.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Partition Tree ⇒ Haar-like basis
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 93: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/93.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Partition Tree ⇒ Haar-like basis
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 94: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/94.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Partition Tree ⇒ Haar-like basis
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 95: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/95.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Partition Tree ⇒ Haar-like basis
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 96: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/96.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Toy example: Haar-like basis function
−0.015−0.01
−0.0050
0.0050.01
0.0150.02
0.025−0.02
−0.01
0
0.01
0.02−0.025
−0.02
−0.015
−0.01
−0.005
0
0.005
0.01
0.015
−0.06
−0.04
−0.02
0
0.02
0.04
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 97: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/97.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Smoothness ⇐⇒Coe�cient decay
How to de�ne smoothness ?
I Partition tree induces tree metric d(xi , xj) [ultrametric]
I Measure smoothness of f : X → R in tree metric
Theorem:
Let f : X → R. Then
|f (xi )− f (xj)| ≤ Cd(xi , xj)α ⇔ |⟨f , ψℓ,i ⟩| ≤ C ′q−ℓ(α+1/2) .
where- q measures the tree balance
- α measures function smoothness w.r.t. tree
Particular example of Space of Homogeneous Type [Coifman &Weiss]
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 98: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/98.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Smoothness ⇐⇒Coe�cient decay
How to de�ne smoothness ?
I Partition tree induces tree metric d(xi , xj) [ultrametric]
I Measure smoothness of f : X → R in tree metric
Theorem:
Let f : X → R. Then
|f (xi )− f (xj)| ≤ Cd(xi , xj)α ⇔ |⟨f , ψℓ,i ⟩| ≤ C ′q−ℓ(α+1/2) .
where- q measures the tree balance
- α measures function smoothness w.r.t. tree
Particular example of Space of Homogeneous Type [Coifman &Weiss]
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 99: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/99.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Smoothness ⇐⇒Coe�cient decay
How to de�ne smoothness ?
I Partition tree induces tree metric d(xi , xj) [ultrametric]
I Measure smoothness of f : X → R in tree metric
Theorem:
Let f : X → R. Then
|f (xi )− f (xj)| ≤ Cd(xi , xj)α ⇔ |⟨f , ψℓ,i ⟩| ≤ C ′q−ℓ(α+1/2) .
where- q measures the tree balance
- α measures function smoothness w.r.t. tree
Particular example of Space of Homogeneous Type [Coifman &Weiss]
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 100: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/100.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Smoothness ⇐⇒Coe�cient decay
How to de�ne smoothness ?
I Partition tree induces tree metric d(xi , xj) [ultrametric]
I Measure smoothness of f : X → R in tree metric
Theorem:
Let f : X → R. Then
|f (xi )− f (xj)| ≤ Cd(xi , xj)α ⇔ |⟨f , ψℓ,i ⟩| ≤ C ′q−ℓ(α+1/2) .
where- q measures the tree balance
- α measures function smoothness w.r.t. tree
Particular example of Space of Homogeneous Type [Coifman &Weiss]
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 101: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/101.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Application: SSL
Given dataset X with weighted graph G , similarity matrix W ,labeled points {xi , yi},
1. Construct a balanced hierarchical tree of graph
2. Construct corresponding Haar-like basis
3. Estimate coe�cients from labeled points
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 102: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/102.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Toy Example: Benchmarks
0 20 40 60 80 1000
5
10
15
20
25
30
# of labeled points (out of 1500)
Tes
t Err
or (
%)
Laplacian EigenmapsLaplacian Reg.Adaptive ThresholdHaar−like basisState of the art
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 103: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/103.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Toy Example: MNIST 8 vs. {3,4,5,7}
0 20 40 60 80 1000
5
10
15
20
25
# of labeled points (out of 1000)
Tes
t Err
or (
%)
Laplacian EigenmapsLaplacian Reg.Adaptive ThresholdHaar−like basis
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 104: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/104.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Summary
1. A balanced partition tree induces a useful �wavelet� basis
2. Designing the basis allows a theory connectingCoe�cients decay, smoothness and learnability
3. �Wavelet arsenal� becomes available: wavelet shrinkage, etc
4. Interesting harmonic analysis. Promising for data analysis?
5. Computational experiments motivate theory and vice-versa
The End
www.wisdom.weizmann.ac.il/∼nadler
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 105: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/105.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Summary
1. A balanced partition tree induces a useful �wavelet� basis
2. Designing the basis allows a theory connectingCoe�cients decay, smoothness and learnability
3. �Wavelet arsenal� becomes available: wavelet shrinkage, etc
4. Interesting harmonic analysis. Promising for data analysis?
5. Computational experiments motivate theory and vice-versa
The End
www.wisdom.weizmann.ac.il/∼nadler
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 106: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/106.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Summary
1. A balanced partition tree induces a useful �wavelet� basis
2. Designing the basis allows a theory connectingCoe�cients decay, smoothness and learnability
3. �Wavelet arsenal� becomes available: wavelet shrinkage, etc
4. Interesting harmonic analysis. Promising for data analysis?
5. Computational experiments motivate theory and vice-versa
The End
www.wisdom.weizmann.ac.il/∼nadler
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 107: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/107.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Summary
1. A balanced partition tree induces a useful �wavelet� basis
2. Designing the basis allows a theory connectingCoe�cients decay, smoothness and learnability
3. �Wavelet arsenal� becomes available: wavelet shrinkage, etc
4. Interesting harmonic analysis. Promising for data analysis?
5. Computational experiments motivate theory and vice-versa
The End
www.wisdom.weizmann.ac.il/∼nadler
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 108: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/108.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Summary
1. A balanced partition tree induces a useful �wavelet� basis
2. Designing the basis allows a theory connectingCoe�cients decay, smoothness and learnability
3. �Wavelet arsenal� becomes available: wavelet shrinkage, etc
4. Interesting harmonic analysis. Promising for data analysis?
5. Computational experiments motivate theory and vice-versa
The End
www.wisdom.weizmann.ac.il/∼nadler
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 109: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/109.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Summary
1. A balanced partition tree induces a useful �wavelet� basis
2. Designing the basis allows a theory connectingCoe�cients decay, smoothness and learnability
3. �Wavelet arsenal� becomes available: wavelet shrinkage, etc
4. Interesting harmonic analysis. Promising for data analysis?
5. Computational experiments motivate theory and vice-versa
The End
www.wisdom.weizmann.ac.il/∼nadler
Boaz Nadler Multiscale Harmonic Analysis on Graphs
![Page 110: Harmonic Analysis on Graphs Global vs. Multiscale Approaches](https://reader036.fdocuments.us/reader036/viewer/2022071602/613d5baa736caf36b75c61a0/html5/thumbnails/110.jpg)
. . . . . .
Semi-Supervised LearningGlobal (Graph Based) Approaches
Multiscale Methods
Summary
1. A balanced partition tree induces a useful �wavelet� basis
2. Designing the basis allows a theory connectingCoe�cients decay, smoothness and learnability
3. �Wavelet arsenal� becomes available: wavelet shrinkage, etc
4. Interesting harmonic analysis. Promising for data analysis?
5. Computational experiments motivate theory and vice-versa
The End
www.wisdom.weizmann.ac.il/∼nadler
Boaz Nadler Multiscale Harmonic Analysis on Graphs