VARIABLES INDEPENDENT VARIABLES DEPENDENT VARIABLES CONTROLLED EXPERIMENTS Analyzing Experiments.
Towards Independent Subspace Analysis in Controlled ...
Transcript of Towards Independent Subspace Analysis in Controlled ...
Towards Independent Subspace Analysis inControlled Dynamical Systems
Zoltán Szabó, András Lorincz
Neural Information Processing Group,Department of Information Systems,
Eötvös Loránd University,Budapest, Hungary
ICA Research Network 2008
Acknowledgements:
Zoltán Szabó, András Lorincz nipg.inf.elte.hu
Tools to Integrate-1 (Independent Subspace Analysis)
Cocktail party problemGeneralization of ICA:
multidimensional components,groups of ’people/music bands’
Hidden, independent, multidimensional processes – NOCONTROL.
Zoltán Szabó, András Lorincz nipg.inf.elte.hu
Tools to Integrate-2 (D-optimal Identification ofDynamical Systems)
Problem: estimate the parameters of a fully observablecontrolled dynamical system by the ‘optimal’ choice of thecontrol.
’Parameters’: dynamics, noise.’Optimal’: in information theoretical sense → D-optimality.
Synonyms: active learning, optimal experimental design.
For ARX models: QP in Bayesian framework.
Controlled dynamical system – FULLY OBSERVABLE.
Zoltán Szabó, András Lorincz nipg.inf.elte.hu
Motivation
Goal: integrate the former methodologieshidden multidimensional sources,optimal design in controlled systems.
Zoltán Szabó, András Lorincz nipg.inf.elte.hu
Motivation
Goal: integrate the former methodologieshidden multidimensional sources,optimal design in controlled systems.
EEG data analysis: recognition + prediction, e.g., epileptic
Zoltán Szabó, András Lorincz nipg.inf.elte.hu
Motivation
Goal: integrate the former methodologieshidden multidimensional sources,optimal design in controlled systems.
EEG data analysis: recognition + prediction, e.g., epileptic
EU-FP7: interested in partnership
Zoltán Szabó, András Lorincz nipg.inf.elte.hu
Contents
1 Independent Subspace Analysis2 D-optimal ARX Identification
︸ ︷︷ ︸
↓
3 D-optimal Hidden ARX Identification4 Illustrations
Zoltán Szabó, András Lorincz nipg.inf.elte.hu
Independent Subspace Analysis (ISA/MICA)
ISA equations: Observation x is linear mixture ofindependent multidimensional components:
x(t) = As(t),
s(t) = [s1(t); . . . ; sM(t)],
wheresm(t) ∈ R
dm are i.i.d. sampled random variables in time,I(s1, . . . , sM) = 0,mixing matrix A ∈ R
D×D is invertible, with D := dim(s).
Goal: s. Specially for ∀dm = 1: ICA.
Ambiguities: permutation, linear (/orth.) transformation.
Zoltán Szabó, András Lorincz nipg.inf.elte.hu
D-optimal ARX Identification
Observation equation (ARX model; u: control, e: noise):
s(t + 1) = Fs(t) + Bu(t + 1) + e(t + 1).
Task: ’efficient’ estimation ofsystem parameters: Θ = [F, B, parameters(e)], ornoise: e
by the ’optimal’ choice of control u.
Optimality (D-optimal/’InfoMax’):
Jpar (ut+1) := I(Θ, st+1|st , st−1, . . . , ut+1, ut , . . .) → maxut+1∈U
, or
Jnoise(ut+1) := I(et+1, st+1|st , st−1, . . . , ut+1, ut , . . .) → maxut+1∈U
.
Result (Póczos & Lorincz, 2008): In the Bayesian setting,optimization of J can be reduced to QP.
Zoltán Szabó, András Lorincz nipg.inf.elte.hu
D-optimal Hidden ARX Identification
State (s) + observation (x) equation:
s(t + 1) = Fs(t) + Bu(t + 1) + e(t + 1), (1)
x(t) = As(t). (2)
Assumptions: I(e1, . . . , eM) = 0 – hidden non-Gaussianindependent multiD componentsTrick: reduce the problem to the fully observable case(d-dep. CLT) + ISA:
x(t + 1) = [AFA−1]x(t) + [AB]u(t + 1) + [Ae(t + 1)],x – ’fully observable tool’ → [AFA−1], [AB], Ae,Ae – ISA→ A} ⇒ F, B, e.
Note: for higher order ARX systems the same idea holds.
Zoltán Szabó, András Lorincz nipg.inf.elte.hu
Databases, Performance Measure, Questions
Databases (3D-geom, ABC, celebrities):
Performance measure: Amari-index (r ) ∈ [0, 1], 0–perfect.Questions:
1 Dependence on δu = |Ucontrol|,2 Dependence on J = deg(Bcontrol[z]),3 Dependencies on I = deg(FAR[z]) and λ:
FAR[z] =
I−1∏
i=0
(I − λOiz) (|λ| < 1, λ ∈ R, Oi : RND orth.).
Zoltán Szabó, András Lorincz nipg.inf.elte.hu
Illustrations: Dependence on δu = |Ucontrol|
Decline of the estimation error: power-law [r(T ) ∝ T−c (c > 0)]
1 2 5 10 20 50
10−2
10−1
100
3D−geom (I=3, J=3, λ=0.95)
Number of samples (T)
Am
ari−
inde
x (r
)
δu:0.01
δu:0.1
δu:0.2
δu:0.5
δu:1
x1031 2 5 10 20 50
10−2
10−1
100
I=3, J=3, δu=0.2, λ=0.95
Number of samples (T)A
mar
i−in
dex
(r)
3D−geomABCcelebrities
x103
Zoltán Szabó, András Lorincz nipg.inf.elte.hu
Illustration: 3D-geom
observation:
Zoltán Szabó, András Lorincz nipg.inf.elte.hu
Illustration: 3D-geom
observation:
estimated innovation (input of ISA):
Zoltán Szabó, András Lorincz nipg.inf.elte.hu
Illustration: 3D-geom
observation:
estimated innovation (input of ISA):
Hinton-diagram, estimated components:
Zoltán Szabó, András Lorincz nipg.inf.elte.hu
Illustration: Dependencies on J = deg(Bcontrol[z]),I = deg(FAR[z])
Precise even for J = 50; I = 50 (ր, λ ց)
1 2 5 10 20 50
10−2
10−1
100
3D−geom (I=3, δu=0.2, λ=0.95)
Number of samples (T)
Am
ari−
inde
x (r
)
J:3J:5J:10J:20J:50
x1031 2 5 10 20 50
10−2
10−1
100
3D−geom (J=1, δu=0.2)
Number of samples (T)
Am
ari−
inde
x (r
)
I:5, λ:0.8I:5, λ:0.85I:5, λ:0.9I:10, λ:0.6I:10, λ:0.65I:10, λ:0.7I:50, λ:0.25I:50, λ:0.3
x103
Zoltán Szabó, András Lorincz nipg.inf.elte.hu
Summary
Integration of two methodologies:hidden independent multidimensional sources,optimal design in controlled dynamical systems.
Numerical experiences:Decline of the estimation error follows power-law.Robust against:
the order of the AR process,temporal memory of the control.
Possibility to apply ICA, ISA, . . . in controlled systems.
Zoltán Szabó, András Lorincz nipg.inf.elte.hu
TYFYA!
Zoltán Szabó, András Lorincz nipg.inf.elte.hu
References
B. Póczos and A. Lorincz.
D-optimal Bayesian interrogation for parameter and noise identificationof recurrent neural networks.
2008.
(submitted; available at http://arxiv.org/abs/0801.1883).
V. V. Petrov.
Central limit theorem for m-dependent variables.
In Proceedings of the All-Union Conference on Probability Theory andMathematical Statistics, pages 38–44, 1958.
Z. Szabó.
Separation Principles in Independent Process Analysis.PhD thesis, Eötvös Loránd University, Budapest, 2008.
(submitted; available at nipg.inf.elte.hu: Download).
Zoltán Szabó, András Lorincz nipg.inf.elte.hu