Post on 18-Jan-2018
description
NIPS 2000 Workshop on Kernel methods
Frame, Reproducing Kernel and Learning
Alain RakotomamonjyStéphane Canu
http://asi.insa-rouen.fr/~arakotomAlain.Rakoto,Stephane.Canu@insa-rouen.fr
Perception, Systèmes et InformationInsa de Rouen,76801 St Etienne du RouvrayFrance
NIPS 2000 Workshop on Kernel methods
Motivations
Wavelet-based approximation (wavelet or ridgelet networks) are regularization networks?
Construction of multiresolution scheme of approximation
kernel adapted to the structures of function to be learned
NIPS 2000 Workshop on Kernel methods
Motivations Ctd.
Frame based framework for learning
Approximating highly oscillating structure Without losing
regularity in smooth region
NIPS 2000 Workshop on Kernel methods
Road Map
Introduction on FrameFrom Frame to KernelsFrom Frame kernels to learningConclusions and perspectives
NIPS 2000 Workshop on Kernel methods
Frame : A definition
H : Hilbert Space dot product nn A sequence of elements of H
nn is a frame of H if there exists A,B > O s.t
n
2
H
2
Hn2
HfB,ffAHf
A,B are the frame bounds
H,
NIPS 2000 Workshop on Kernel methods
Frame : definition Ctd.
Frame intepretation
Frame allows stable representationas for all f in H
n
n*n,ff
Frame = "Basis" + linear dependency + redundancy
n*n being a dual frame of n in H
NIPS 2000 Workshop on Kernel methods
Particular cases of Frame
Tight FrameFrame with bounds s.t A=B
Orthonormal BasisA=B=1
Riesz BasisFrame elements are linearly independent
n
2
H
2
Hn f,f,Hf
]np[, n*p
n*n A
1
NIPS 2000 Workshop on Kernel methods
Examples of Frame
Tight Frame of IR2
Frame of L2(IR)
11 e
21
2 e23
2e
21
3 e23
2e
3
1n
22
n2 f
23,f,IRf
2Zn,jk,j )t(
j
jo
jk,j aanut
a
1)t( is an admissible wavelet
12
3
NIPS 2000 Workshop on Kernel methods
Road Map
Introduction on FrameFrom Frame to KernelsFrom Frame kernels to learningConclusions and perspectives
NIPS 2000 Workshop on Kernel methods
Frameable RKHS
Condition for having a RKHSSuppose H is a Hilbert space of function IRIR n
nnand a frame of H
Hn
n*n )t()(,t
The Reproducing Kernel is
n
n*n )t()s()t,s(K
H is a RKHS if Htt fM)t(ft.s0M,t,Hf
On a frameable Hilbert Space, this is equivalent to
NIPS 2000 Workshop on Kernel methods
Construction of Frameable RKHS
A Practical way to build a RKHSF is a Hilbert Space of function IRIR:f n
N..1nn A finite set of F elements such that
Fn,N...1n
M)t(t,N...1n,IRM n
Fn ,,span is a RKHS with {n} as frame elements
NIPS 2000 Workshop on Kernel methods
Example of Frameable RKHS
frameable RKHS included in L2(IR)i : L2 function (e.g i is a wavelet) span {i}i=1…N is a RKHS
span a RKHS with kernel
3
1ii
*i )t()s()t,s(K
3 wavelets at same scale jExample
NIPS 2000 Workshop on Kernel methods
Road Map
Introduction on FrameFrom Frame to KernelsFrom Frame kernels to learningConclusions and perspectives
NIPS 2000 Workshop on Kernel methods
Semiparametric EstimationContext Learning from training set (xi,yi)i=1..N
One looks for the minimizer of the risk functional 2
H
N
1iii f)x(f,yC
in a space H + span{i}i=1…m H being a RKHS
m
1j
N
1iiHijj
* )x,x(Kb)x(a)x(f
Under general conditions,
Semiparametric framework
span{i}i=1…m : parametric hypothesis space
NIPS 2000 Workshop on Kernel methods
Semiparametric EstimationParametric hyp. space is a frameable
RKHSP is a frameable RKHS spanned by {n}, with P H, H RHKS
PPH PPKKK
2
P
N
1iii f)x(f,yC
One looks for the minimizer in H of
m
1j
N
1iiPijj
* )x,x(Kb)x(a)x(f
As spaces are orthogonal, backfitting is sufficient for estimating f*
Semiparametric estimation on H with P as a parametric hyp. space
NIPS 2000 Workshop on Kernel methods
Semiparametric Estimation
H= P + N P : Frameable RKHS, N : Frameable RKHS
N: "unknown component" to be regularized
P : "known component" not to regularized
Frame view point H frameable
H defined by kernel K
H
NPH KN=KH-KP
P N : due to linear dependency of frame
P : Frameable RKHS
NIPS 2000 Workshop on Kernel methods
Multiscale approximation
H a frameable RKHS
1m...1iFHH 1i1ii
m1m10 HHHH And any space Hi or Fi is a RKHS
Hi : Trend Spaces Fi : Details Spaces
H is splitted in different spaces {Fi}i=1…m-1 and H0
NIPS 2000 Workshop on Kernel methods
Multiscale Approximation Ctd.
At each step j, trend obtained at step j-1 is decomposed in trend and details
H
H2 F2
H1 F1
H0 F0
ii y,x
k
k,2k,2d k
ik,2k,2 )x,x(Kc Resid.
k
k,1k,1d k
ik,1k,1 )x,x(Kc Resid.
k
k,0k,0d k
ik,0k,0 )x,x(Kc Resid.
f*
NIPS 2000 Workshop on Kernel methods
Multiscale Approximation Ctd.
ValidityAt each step, representer Theorem
Hypothesis must be verified
Solution
Trend
kkk
Details
1m
0j
N
1iijj,i
*
0
)x(d)x,x(Kc)x(f
NIPS 2000 Workshop on Kernel methods
Illustration on toy problem
)2t(5))2t(5sin(
)5t())5t(sin()xsin()x(s
Function to be learnedData xi : N points from the random sampling of [0, 10]
),0(N)x(sy ii
Algorithm - SVM Regression- Multiscale Regularisation networks on Frameable RKHS
Sin/Sinc based kernelWavelet based kernel
NIPS 2000 Workshop on Kernel methods
Results
SVM Wavelet Kernel Sinc Kernel
L2 error
1 ± 0.096
1 ± 0.028
0.9297 ± 0.312
0.8280± 0.025
0.5115 ± 0.098
0.7252± 8.022
N=902 Results are averagerad over 300 experiments and
normalized with regards to SVM performance
NIPS 2000 Workshop on Kernel methods
Plots of typical results
NIPS 2000 Workshop on Kernel methods
Road Map
Introduction on FrameFrom Frame to KernelsFrom Frame kernels to learningConclusions and perspectives
NIPS 2000 Workshop on Kernel methods
Summary new design of kernel based on frame
elements algorithm for multiscale learning
But no explicit definition of kernel
Time-consuming
NIPS 2000 Workshop on Kernel methods
Future work
Multidimensional extensionTight Frame of multidimensional wavelet
Using a priori knowledge on the learning problem
How to choose the frame elements?Theoretical justification and analysis
of multiscale approximation
NIPS 2000 Workshop on Kernel methods