1UAI2010 Tutorial, July 8, Catalina Island.

Non-Gaussian Methods for Learning Linear Structural

Equation ModelsEquation Models

Shohei Shimizu and Yoshinobu KawaharaOsaka Universityy

Special thanks to1

Special thanks to Aapo Hyvärinen, Patrik O. Hoyer and Takashi Washio.

2

AbstractAbstractLi t t l ti d l (li SEM )• Linear structural equation models (linear SEMs) can be used to model data generating processes of variablesof variables.

• We review a new approach to learn or estimateWe review a new approach to learn or estimate linear SEMs.

• The new estimation approach utilizes non-Gaussianity of data for model identification

d i l ti t h id i t fand uniquely estimates much wider variety of models.

33

OutlineOutline • Part I. Overview (70 min.) : Shohei

• Break (10 min.)

• Part II. Recent advances (40 min): Yoshi( )– Time series

Latent confounders– Latent confounders

4

Motivation (1/2)Motivation (1/2)• Suppose that data X was randomly generatedSuppose that data X was randomly generated

from either of the following two data generating processes:

Model 1: Model 2:

or11

exbxex

+==

22

12121

exexbx

=+=x1

x2

e1

e2

x1

x2

e1

e2

where and are latent variables (disturbances, errors).

21212 exbx += 22 exx2 e2 x2 e2

1e 2e ( , )

• We want to estimate or identify which model t d th d t X b d th d t X l

1 2

generated the data X based on the data X only.

5

Motivation (2/2)( )• We want to identify which model generated the

data X based on the data X onlydata X based on the data X only.

If 1 d 2 G i it i ll k th t• If x1 and x2 are Gaussian, it is well known that we cannot identify the data generating process.

M d l 1 d 2 ll fit d t

1e 2e

– Models 1 and 2 equally fit data.

If 1 d 2 G i i t ti• If x1 and x2 are non-Gaussian, an interesting result is obtained: We can identify which of Models 1 and 2 generated the data

1e 2e

Models 1 and 2 generated the data.

Thi t t i l i h h G i• This tutorial reviews how such non-Gaussian methods work.

Problem formulation

7

Basic problem setup (1/3)Basic problem setup (1/3)• Assume that the data generating process of g g p

continuous observed variables is graphically represented by a directed acyclic graph (DAG).

ixp y y g p ( )

– Acyclicity means that there are no directed cycles.

E l f di t dExample of a directed acyclic graph (DAG):

Example of a directed cyclic graph:

x3 e3 x3 e3

x1 e1

x2 e2

x1 e1

x2 e2x2 e2 x2 e2

is a parent of etc.1x3x

8

Basic problem setup (2/3)Basic problem setup (2/3)• Further assume linear relations of variables .ix• Then we obtain a linear acyclic SEM (Wright, 1921; Bollen,

1989):

eBxx +=ijiji exbx += ∑ or

where

iij

jiji ∑ofparents:

where– The are continuous latent variables that are not

determined inside the model which we call externalie

determined inside the model, which we call external influences (disturbances, errors).

– The are of non-zero variance and are independenteThe are of non zero variance and are independent.– The ‘path-coefficient’ matrix B = [ ] corresponds to a DAG.

ieijb

99

Example of linear acyclic SEMs• A three-variable linear acyclic SEM:

p y

⎥⎤

⎢⎡

⎥⎤

⎢⎡⎥⎤

⎢⎡

⎥⎤

⎢⎡ 111 5.100 exx

131 5.1 exx +=

⎥⎥⎥

⎦⎢⎢⎢

⎣

+⎥⎥⎥

⎦⎢⎢⎢

⎣⎥⎥⎥

⎦⎢⎢⎢

⎣

−=⎥⎥⎥

⎦⎢⎢⎢

⎣ 3

2

3

2

3

2

000003.1

ee

xx

xx

212 3.1ex

exx=

+−= or

44 344 21

B33 ex =

• B corresponds to the data-generating DAG:

x3 e3x3

x1

e3

e11.5 ijij xxb tofromedgedirected No0 ⇔=

b fddi dA0x2 e2

-1.3 ijij xxb tofromedgedirectedA 0 ⇔≠

10

Assumption of acyclicity• Acyclicity ensures existence of an ordering of

variables that makes B lower triangular withxvariables that makes B lower-triangular with zeros on the diagonal (Bollen, 1989).

ix

⎥⎥⎤

⎢⎢⎡

+⎥⎥⎤

⎢⎢⎡

⎥⎥⎤

⎢⎢⎡

=⎥⎥⎤

⎢⎢⎡

1

3

1

3

1

3

005.1000

ee

xx

xx 0

00 0

0⎥⎥⎤

⎢⎢⎡

+⎥⎥⎤

⎢⎢⎡

⎥⎥⎤

⎢⎢⎡−=⎥

⎥⎤

⎢⎢⎡

2

1

2

1

2

1

003.15.100

ee

xx

xx 00

⎥⎥⎦⎢

⎢⎣⎥

⎥⎦⎢

⎢⎣⎥⎥⎦⎢

⎢⎣ −⎥

⎥⎦⎢

⎢⎣ 222 03.10 exx

44 344 210

B⎥⎥⎦⎢

⎢⎣⎥

⎥⎦⎢

⎢⎣⎥⎥⎦⎢

⎢⎣⎥

⎥⎦⎢

⎢⎣ 333 000 exx

44 344 21

Bx3 e3

permBB:isorderingThex3

x1

e3

e11.5

ofancestoranbemay.g

213

xxxxxx <<

x2-1.3

e2 .versavicenotbut ,,ofancestor an bemay 213 xxx

11Assumption of independence between external influences

• It implies that there are no latent confounders (Spirtes et al. 2000)– A latent confounder is a latent variable that is a parent of morefA latent confounder is a latent variable that is a parent of more

than or equal to two observed variables:f

x1

x2f

e1’

e2’

• Such a latent confounder makes external influences

x2e2

fSuch a latent confounder makes external influences dependent (Part II):

x1 e1

f

x2 e2

1212Basic problem setup (3/3):Learning linear acyclic SEMs

• Assume that data X is randomly sampled from a linear acyclic SEM (with no latentfrom a linear acyclic SEM (with no latent confounders):

eBxx +=x1 e1

21bx2 e2

• Goal: Estimate the path-coefficient matrix B by observing data X only!observing data X only!– B corresponds to the data-generating DAG.

Problems: Identifiability problems of

con entional methodsconventional methods

1414

Under what conditions B is identifiable?

• `B is identifiable’ `B is uniquely determined or estimated from p(x)’

≡estimated from p(x) .

S• Linear acylic SEM:

Bx1 e1

beBxx +=x2 e2

21b

– B and p(e) induce p(x). If p(x) are different for different B then B is uniquely– If p(x) are different for different B, then B is uniquely determined.

15

Conventional estimation principle: Causal Markov condition

• If the data-generating model is a (linear) acyclic SEM causal Markov conditionacyclic SEM, causal Markov condition holds :

E h b d i bl i i i d d t f itx– Each observed variable xi is independent of its non-descendants in the DAG conditional on its

t

ix

parents (Pearl & Verma, 1991) :

( ) ( )∏=p

iii xxpp

1

ofparents|x=i 1

1616Conventional methods based on

causal Markov condition• Methods based on conditional independencies

(Spirtes & Glymour, 1991)

M li li SEM i t f– Many linear acyclic SEMs give a same set of conditional independences and equally fit data.

• Scoring methods based on Gaussianity(Chickering 2002)(Chickering, 2002)

– Many linear acyclic SEMs give a same Gaussian distribution and equally fit datadistribution and equally fit data.

• In many cases the path coefficient matrix B is• In many cases, the path-coefficient matrix B is not uniquely determined.

1717

Example• Two models with Gaussian e1 and :1e 2e

11 ex = 121 8.0 exx +=Model 1: Model 2:

x1 e1 x1 e1

212 8.0 exx += 22 ex =x2 e2 x2 e2

• Both introduce no conditional independence:

( ) ( ) 1varvar 21 == xx( ) ( ) ,021 == eEeE

• Both introduce no conditional independence:

( ) 08.0,cov 21 ≠=xx

• Both induce the same Gaussian distribution:

( )21

⎞⎛ ⎤⎡⎤⎡⎤⎡ 8010x⎟⎟⎠

⎞⎜⎜⎝

⎛⎥⎦

⎤⎢⎣

⎡⎥⎦

⎤⎢⎣

⎡⎥⎦

⎤⎢⎣

⎡18.08.01

00

~2

1 Nxx

A solution:A solution: Non-Gaussian approachpp

1919

A new direction: Non-Gaussian approach

• Non-Gaussian data in many applications:– Neuroinformatics (Hyvarinen et al 2001); BioinformaticsNeuroinformatics (Hyvarinen et al., 2001); Bioinformatics

(Sogawa et al., ICANN2010); Social sciences (Micceri, 1989); Economics (Moneta, Entner, et al., 2010)

• Utilize non-Gaussianity for model identification:– Bentler (1983); Mooijaart (1985); Dodge and Rousson (2001)Bentler (1983); Mooijaart (1985); Dodge and Rousson (2001)

• The path-coefficient matrix B is uniquely p q yestimated if ei are non-Gaussian. – Shimizu, Hoyer, Hyvarinen and Kerminen (JMLR, 2006)

ie

2020Illustrative example: G i G iGaussian vs non-Gaussian

Gaussian Non-Gaussian(uniform)

Model 1:x2x2

Model 1:x1 e1 x1x111

80ex

+=

M d l 2

x2 e2212 8.0 exx +=

Model 2:

x1

x2

x1 e1

x2

121 8.0 exx +=x1

x2 e2x1

22 ex =

( ) ( ) 1varvar 21 == xx( ) ( ) ,021 == eEeE

2121Linear Non-Gaussian A li M d l LiNGAMAcyclic Model: LiNGAM(Shimizu, Hyvarinen, Hoyer & Kerminen, JMLR, 2006)

• Non-Gaussian version of linear acyclic SEM:

eBxx +=exbx += ∑ or eBxx +=iij

jiji exbx += ∑ofparents:

or

whereThe external influence variables (disturbancese– The external influence variables (disturbances, errors) are

• of non-zero variance

ie

of non-zero variance.• non-Gaussian and mutually independent.

22

Identifiability of LiNGAM modelIdentifiability of LiNGAM model

• LiNGAM model can be shown to beidentifiableidentifiable.– B is uniquely estimated.

• To see the identifiability, helpful to review y, pindependent component analysis (ICA) (Hyvarinen et al., 2001).

23Independent Component Analysis (ICA)(ICA) (Jutten & Herault, 1991; Comon, 1994)

• Observed random vector x is modeled byObserved random vector x is modeled by

Asx =∑=p

jiji sax or

where

Asx∑=j

jiji1

– The mixing matrix A = [ ] is square and is of full column rank.The latent ariables (independent components)s

ija

– The latent variables (independent components) are of non-zero variance, non-Gaussian and mutually independent.

is

• Then, A is identifiable up to permutation P and scaling D of the columns:scaling D of the columns:

APDA =ica

2424Estimation of ICA• Most of estimation methods estimate

(Hyvarinen et al., 2001):1−= AW

( y , )

sWAsx 1−==

• Most of the methods minimize mutual information (or its approximation) of estimated independent components:

xWs ica=ˆ• W is estimated up to permutation P and scaling D of the rows:

( )1−== PDAPDWWi

• Consistent and computationally efficient algorithms:

( )PDAPDWWica

– Fixed point (FastICA) (Hyvarinen,99); Gradient-based (Amari, 98)

– Semiparametric: no specific distributional assumption

Back to LiNGAM model

2626Identifiability of LiNGAM (1/3):ICA hi h lf f id tifi tiICA achieves half of identification

• LiNGAM model is ICA• LiNGAM model is ICA.– Observed variables are linear combinations of

non Gaussian independent external influences :ix

enon-Gaussian independent external influences :

eBIxeBxx −−=⇔+= 1)(ie

eWAe 1−== ( )BIW −=

• ICA gives .– P: unknown permutation matrix

)( BIPDPDWW −==icaP: unknown permutation matrix

– D: unknown scaling (diagonal) matrix

• Need to determine P and D to identify B.

2727Identifiability of LiNGAM (2/3):

No permutation indeterminacy (1/6)• ICA gives .

– P : permutation matrix; D: scaling (diagonal) matrix)( BIPDPDWW −==ica

• We want to find such a permutation matrix that cancels the permutation i e :IPP =

PPpermutation , i.e., :IPP =

IDWPDWPWP ==ica

P

• We can show (Shimizu et al., UAI05) (illustrated in the next slides):– If i e no permutation is made on the rows ofIPP =

I=

DW– If , i.e., no permutation is made on the rows of , has no zero in the diagonal (obvious by definition).

If i id ti l t ti i d th

IPP =icaWP

DW

– If , i.e., any nonidentical permutation is made on the rows of , has a zero in the diagonal.

IPP ≠icaWPDW

28

Identifiability of LiNGAM (2/3):No permutation indeterminacy (2/6)

• By definition has all unities in the diagonal.– The diagonal elements of B are all zeros.

BIW −=

• Acyclicity ensures existence of an ordering of variables that makes lower triangular and then isBIWBthat makes lower-triangular, and then is also lower-triangular.

BIW −=B

• So, WLG, can be assumed to be lower-triangular:W

⎥⎥⎤

⎢⎢⎡

= 01*001

W0 0

0 No zeros in the diagonal!

⎥⎥⎦⎢

⎢⎣ 1**

0W in the diagonal!

29Identifiability of LiNGAM (2/3):N t ti i d t i (3/6)No permutation indeterminacy (3/6)

• Premultiplying by a scaling (diagonal) matrix D does NOT change the zero/non-zero pattern of :

WW

⎥⎥⎤

⎢⎢⎡

=11

0*00

dd

DW 0

0 0

⎥⎥⎤

⎢⎢⎡

= 01*001

W0 0

0

⎥⎥⎦⎢

⎢⎣

=

33

22

**0

ddDW 0

⎥⎥⎦⎢

⎢⎣

=1**01W 0

No zeros in the diagonal!

3030

Identifiability of LiNGAM (2/3):No permutation indeterminacy (4/6)

• Any other permutation of the rows of changes the zero/non-zero pattern of and

DWDWg p

brings zero in the diagonal:

⎥⎥⎤

⎢⎢⎡

= 11

2212 00

0*d

dDWP

⎥⎥⎤

⎢⎢⎡

=11

0*00

dd

DW 00 0 0

00⎥⎥⎦⎢

⎢⎣ 33

11

** d⎥⎥⎦⎢

⎢⎣

=

33

22

**0

ddDW 0 00

Exchanging 1stExchanging 1and 2nd rows Zero in the diagonal!

3131

Identifiability of LiNGAM (2/3):No permutation indeterminacy (5/6)

• Any other permutation of the rows of changes the zero/non-zero pattern of and

DWDWg p

brings zero in the diagonal:

⎥⎥⎤

⎢⎢⎡ 11

0*00

dd

DW ⎥⎥⎤

⎢⎢⎡

= 0*** 33

13 dd

DWP00 0

0⎥⎥⎦⎢

⎢⎣

=

33

22

**0*

ddDW

⎥⎥⎦⎢

⎢⎣

=000

11

22

ddDWP0 0

0Exchanging 1st

0Exchanging 1st

and 3rd rows Zero in the diagonal!

3232

Identifiability of LiNGAM (2/3):No permutation indeterminacy (6/6)

• We can find correct by finding that gives no th di l f WP

P Pzero on the diagonal of (Shimizu et al., UAI05).icaWP

• Thus, we can solve the permutation indeterminacy and obtain: y

( )BIDDWPDWPWP −=== ( )BIDDWPDWPWP −===ica

I=

3333

Identifiability of LiNGAM (3/3): No scaling indeterminacy

• Now we have: B)D(IWP −=ica

• Then, ( )icaWPD diag=

• Divide each row of by its corresponding icaWP y p gdiagonal element to get , i.e., :

icaBI − B

( ) BIB)D(IDWPWP −=−= −− 11diag icaica

3434

Estimation of LiNGAM model

1 ICA-LiNGAM algorithm1. ICA LiNGAM algorithm2. DirectLiNGAM algorithm

3535

Two estimation algorithmsg• ICA-LiNGAM algorithm

(Shi i H H i & K i JMLR 2006)(Shimizu, Hoyer, Hyvarinen & Kerminen, JMLR, 2006)

• DirectLiNGAM algorithm (Shimizu Hyvarinen Kawahara & Washio UAI09)(Shimizu, Hyvarinen, Kawahara & Washio, UAI09)

• Both estimate an ordering of variables that makes gthe path-coefficient matrix B to be lower-triangular.– Acyclicity ensures existence of such an ordering.

⎤⎡ OA full DAG

permpermperm exx +⎥⎦

⎤⎢⎣

⎡=

31

Ox3x1

321

permBx2

redundant edges

3636

Once such an ordering is found…

• Many existing (covariance-based) methods can do:– Pruning the redundant path-coefficients

• Sparse methods like weighted lasso (Zou, 2006）

– Finding significant path-coefficients • Testing, bootstrapping (Shimizu et al., 2006; Hyvarinen et al. 2010)

x3x1 x3x1exx +⎥⎤

⎢⎡

= O0*

A full DAG

x2 x2

permpermperm exx +⎥⎦

⎢⎣

=321

00*

* *

permB

3737

1. Outline of ICA-LiNGAM algorithmOut e o C G a go t(Shimizu, Hoyer, Hyvarinen, & Kerminen, JMLR, 2006)

1 Estimate B by ICA 2. Pruning1. Estimate B by ICA + permutation

g

x2x1x1 x2

x3x3x3 23b13b

Redundant edgesRedundant edges

3838

ICA-LiNGAM algorithm (1/2): Step 1. Estimation of B

1. Perform ICA (here, FastICA (Hyvarinen, 1999)) to obtain an estimate of

B)PD(IPDWW −==ica

2. Find a permutation that makes the diagonal elements of as large (non-zero) as possible in b l t l

icaWP ˆP

absolute value:

( )WPP

P ˆ1minˆ = Hungarian alg.

(Kuhn, 1955)

3 N li h f th t

( )iiicaWPP ( , )

WP ˆˆ3. Normalize each row of , then we get an estimate of I-B and .

icaWPB̂

3939ICA-LiNGAM algorithm (2/2): St 2 P iStep 2. Pruning

• Find such an ordering of variables that makes• Find such an ordering of variables that makes estimated B be as close to be lower-triangular as possiblepossible.– Find a permutation matrix Q that minimizes the sum of the

elements in the upper triangular part of permuted :B̂elements in the upper triangular part of permuted :

( )∑≤

=ji

ijT 2ˆminˆ QBQQ

Q

B

– Approximate algorithm for large variables (Hoyer et al., ICA06)

≤ ji

x2x1

0 1 3

0.1

55x1 x2

x3x3x30.1

0.1 3 0.1 3-0.01

4040

Basic properties of ICA-LiNGAM algorithm

• ICA-LiNGAM algorithm = ICA + permutationsComputationally efficient with the help of well developed– Computationally efficient with the help of well-developed ICA techniques.

• Potential problems ICA is an iterative search method:– ICA is an iterative search method:

• May get stuck in a local optimum if the initial guess or step size is badly chosensize is badly chosen.

– The permutation algorithms are not scale-invariant:• May provide different estimates for different scales ofMay provide different estimates for different scales of

variables.

41

Estimation of LiNGAM model

1 ICA-LiNGAM algorithm1. ICA LiNGAM algorithm2. DirectLiNGAM algorithm

4242

2. DirectLiNGAM algorithm2. DirectLiNGAM algorithm(Shimizu, Hyvarinen, Kawahara & Washio, UAI2009)

• Alternative estimation method without ICA– Estimates an ordering of variables that makes path-g p

coefficient matrix B to be lower-triangular.

⎤⎡A full DAG

permpermperm exx +⎥⎦

⎤⎢⎣

⎡= O

x3x1

A full DAG

⎦⎣ 321

permB x2Redundant edges

• Many existing (covariance-based) methods can do further pruning or finding significant pathdo further pruning or finding significant path coefficients (Zou, 2006; Shimizu et al., 2006; Hyvarinen et al. 2010)

43Basic idea (1/2) :A i bl b t thAn exogenous variable can be at the

top of a right orderingtop of a right ordering • An exogenous variable is a variable with no jxg

parents (Bollen, 1989), here .– The corresponding row of B has all zeros.

3xp g

• So, an exogenous variable can be at the top ofsuch an ordering that makes B lower-triangularwith zeros on the diagonal.

⎥⎥⎤

⎢⎢⎡

+⎥⎥⎤

⎢⎢⎡

⎥⎥⎤

⎢⎢⎡

=⎥⎥⎤

⎢⎢⎡ 333

0051000

ee

xx

xx

000

00x3 x1 x2

⎥⎥⎦⎢

⎢⎣

+⎥⎥⎦⎢

⎢⎣⎥⎥⎦⎢

⎢⎣ −

=⎥⎥⎦⎢

⎢⎣ 2

1

2

1

2

1

03.10005.1

ee

xx

xx 0 0

0

x3 x1 x2

44Basic idea (2/2): R tRegress exogenous out

• Compute the residuals regressing the other

3x( ) )21(3 =ir• Compute the residuals regressing the other

variables on exogenous :The residuals form a LiNGAM model

( ) )2,1( =iri

3x)2,1( =ixi( ) ( )33 and rr– The residuals form a LiNGAM model.

– The ordering of the residuals is equivalent to that of corresponding original variables.

21 and rr

p g g

⎥⎤

⎢⎡

⎥⎤

⎢⎡⎥⎤

⎢⎡

⎥⎤

⎢⎡ 333 000 exx 0 0 000

⎤⎡⎤⎡⎤⎡⎤⎡

⎥⎥⎥

⎦⎢⎢⎢

⎣

+⎥⎥⎥

⎦⎢⎢⎢

⎣⎥⎥⎥

⎦⎢⎢⎢

⎣ −=

⎥⎥⎥

⎦⎢⎢⎢

⎣ 2

1

2

1

2

1

03.10005.1

ee

xx

xx 0 0

0 ⎥⎦

⎤⎢⎣

⎡+⎥

⎦

⎤⎢⎣

⎡⎥⎦

⎤⎢⎣

⎡−

=⎥⎦

⎤⎢⎣

⎡

2

1)3(

2

)3(1

)3(2

)3(1

03.100

ee

rr

rr 0 0

0⎦⎣⎦⎣⎦⎣⎦⎣ 222 ⎦⎣⎦⎣

)3(2r

)3(1rx3 x1 x2

• Exogenous implies ` can be at the second top’.)3(1r 1x

45

Outline of DirectLiNGAM• Iteratively find exogenous variables until all the

variables are ordered:variables are ordered:1. Find an exogenous variable .

P t t th t f th d i3x

x– Put at the top of the ordering. – Regress out.

2 Fi d id l h )3(r

3x3x

2. Find an exogenous residual, here .– Put at the second top of the ordering.

R t

)(1r

1x)3(– Regress out.

3. Put at the third top of the ordering and terminate. Th ti t d d i i

)3(1r

2x<<The estimated ordering is .213 xxx <<

Step. 1 Step. 2 Step. 3)3(

2r)3(

1rx3 x1 x2 )1,3(2r

46-1Identification of an exogenous variable (two variable cases)

ii) is NOT exogenousi) is exogenous)( 11 ex = 1xii) is NOT exogenous.i) is exogenous.

( )011 =

bbex

)( 11 ex 1x( )122121 0bxbx ≠+= 1e

( )02121212 ≠+= bexbx 22 ex =

)var(),cov(,on Regressing

112

2)1(

2

12

xx

xxxr

xx

−=1

122

)1(2

12

),cov(,on Regressing

xxxxr

xx

−=

( ))var(

var)var(

),cov(1

)var(

2122

1212

1

xxbx

xxxb

x

−⎭⎬⎫

⎩⎨⎧−=1212

11

22 )var(xbx

xx

xr

−=1e

)var()var( 11 xx ⎭⎩2e=

t.independenNOTareand )1(21 rxt.independenareand )1(

21 rx

46-2Need to use Darmois-Skitovitch’ ththeorem (Darmois, 1953; Skitovitch, 1953)

ii) is NOT exogenous1x( )1212121 0bexbx ≠⋅+=

Darmois-Skitovitch’ theorem:

Define two variables and as

ii) is NOT exogenous.1x

1x1

2x22 ex =Define two variables and as

∑∑ ==p

jj

p

jj eaxeax 2211 ,

1x 2x

112

2)1(

2

21

)var(),cov(,on Regressing

xx

xxxr

xx

−=

∑∑== j

jjj

jj11

where are independent d i bl

je

( )1

22

1212

1

)var(var

)var(),cov(1

)var(

ex

xxx

xxb

x

−⎭⎬⎫

⎩⎨⎧−=

random variables.

If there exists a non-Gaussian ie 12b

11 )var()var( xx ⎭⎩for which , and are dependent.

021 ≠iiaa1x 2x

t.independenNOTareand )1(21 rx

47

Identification of an exogenous variable (p variable cases)

• Lemma 1: and its residual ( )j

jii

ji x

xxxr

)cov( ,−=jxLemma 1: and its residual

are independent for all is exogenous

jj

ii xx

x)var(j

ji ≠ jx⇔are independent for all is exogenous

I ti id tif i bl

ji ≠ jx⇔

• In practice, we can identify an exogenous variable by finding a variable that is most independent of it id lits residuals.

48

Independence measures• Evaluate independence between a variable and a

residual by a nonlinear correlation:

p

residual by a nonlinear correlation:

( ){ } ( )tanhcorr )( =grgx j

• Taking the sum over all the residuals we get:

( ){ } ( )tanh,corr =grgx ij

• Taking the sum over all the residuals, we get:

( ){ } ( ){ }∑ += jj rxgrgxT )()( corrcorr

C hi ti t d ll

( ){ } ( ){ }∑≠

+=ji

ijij rxgrgxT ,corr,corr

• Can use more sophisticated measures as well (Bach & Jordan, 2002; Gretton et al., 2005; Kraskov et al., 2004).– Kernel-based independence measure (Bach & Jordan 2002)Kernel based independence measure (Bach & Jordan, 2002)

often gives more accurate estimates (Sogawa et al., IJCNN10).

49

Important properties of DirectLiNGAM

• DirectLiNGAM repeats:Least squares simple linear regression– Least squares simple linear regression

– Evaluation of pairwise independence between each variable and its residualsvariable and its residuals.

• No algorithmic parameters like stepsize, initial g p p ,guesses, convergence criteria.

• Guaranteed convergence to the right solution in a fixed number of steps (the number of variables) if the data strictly follows the model.

50Estimation of LiNGAM model: Summary (1/2)

• Two estimation algorithms:ICA LiNGAM: Estimation using ICA– ICA-LiNGAM: Estimation using ICA

• Pros. Fast• Cons Possible local optimum; Not scale invariant• Cons. Possible local optimum; Not scale-invariant

– DirectLiNGAM: Alternative estimation without ICA• Pros. Guaranteed convergence; Scale-invariant• Cons. Less fast

– Cf. Neither needs faithfulness (Shimizu et al., JMLR, 2006; Hoyer personal comm July 2010)2006; Hoyer, personal comm., July, 2010).

51Estimation of LiNGAM model: Summary (2/2)

• Experimental comparison of the two algorithms: (Sogawa et al., IJCNN2010)

• Sample size: Both need at least 1000 sample size for more than 10 variables.S l bilit B th l 100 i bl Th• Scalability: Both can analyze 100 variables. The performances depend on the sample size etc., of course!

• Scale invariance: ICA-LiNGAM is less robust for changing g gscales of variables.

• Local optima?: For less than 10 variables ICA LiNGAM often a bit– For less than 10 variables, ICA-LiNGAM often a bit better.

– For more than 10 variables, DirectLiNGAM often better perhaps because the problem of local optima might become more serious?

52

Testing andTesting and Reliability evaluationy

53

Testing testable assumptionsg p• Non-Gaussianity of external influences :iey

– Gaussianity testsi

• Could detect violations of some assumptions:– Local test

• Independence of external influences• Conditional independencies between observed

variables (causal Markov condition)

ie

xvariables (causal Markov condition)• Linearity

– Overall fit of the model assumptions

ix

Overall fit of the model assumptions• Chi-square test using 3rd and/or 4th-order moments

(Shimizu & Kano, 2008) • Still under development

54

Reliability evaluationReliability evaluation• Need to evaluate statistical reliability of LiNGAMy

results:– Sample fluctuations– Smaller non-Gaussianity makes the model closer to

be NOT identifiable.

• Reliability evaluation by bootstrapping:(Komatsu Shimizu & Shimodaira ICANN2010; Hyvarinen et al JMLR 2010)(Komatsu, Shimizu & Shimodaira, ICANN2010; Hyvarinen et al., JMLR, 2010)

– If either the sample size is small or the magnitude of non-Gaussianity is small, LiNGAM would give very different results for bootstrap samples.

55

Extensions

56

Extensions (a partial list)Extensions (a partial list)R l i th ti f LiNGAM d l• Relaxing the assumptions of LiNGAM model:– Acyclic Cyclic (Lacerda et al., UAI2008)

– Single homogenous population heterogeneous population (Shimizu et al., ICONIP2007)

– i.i.d. sampling time structures (Part II.) (Hyvarinen et al, JMLR, 2010; Kawahara, S & Washio, 2010)

– No latent confounders Allow latents (Part II.) (Hoyer et al., IJAR, 08; Kawahara, Bollen et al., 2010)

– Linear Non-linear (Hoyer et al., NIPS08; Zhang & Hyvarinen, UAI09; Tilmann, Gretton & Spirtes, NIPS09)

57

Application areas so far

58

Non-Gaussian SEMs have been applied to…

• Neuroinformatics– Brain connectivity analysis (Hyvarinen et al., JMLR, 2010)y y ( y )

• Bioinformatics– Gene network estimation (Sogawa et al ICANN2010)Gene network estimation (Sogawa et al., ICANN2010)

• Economics (Wan & Tan, 2009; Moneta, Entner, Hoyer & Coad, 2010)

• Genetics (Ozaki & Ando, 2009)

• Environmental sciences (Niyogi et al., 2010)( y g , )

• Physics (Kawahara, Shimizu & Washio, 2010)

S i l• Sociology (Kawahara, Bollen, Shimizu & Washio, 2010)

59

Final summary of Part IFinal summary of Part IU f G i it i li SEM i• Use of non-Gaussianity in linear SEMs is useful for model identification.

• Non-Gaussian data is encountered in many applicationsapplications.

• The non-Gaussian approach can be a goodThe non Gaussian approach can be a good option.

• Links to codes and papers: http://homepage.mac.com/shoheishimizu/lingampapers.html

60

FAQs

61

Q. My data is Gaussian. LiNGAM will not be useful.

• A. You’re right. Try Gaussian methods.

• Comment: Hoyer et al. (UAI2008) showed: `To hat e tent one can identif the model for aTo what extent one can identify the model for a mixture of Gaussian and non-Gaussian external influence variables’influence variables’.

62

Q. I applied LiNGAM, but the result is not reasonable to background knowledge.

• A. You might first want to check: Some model assumptions might be violated– Some model assumptions might be violated.

Try other extensions of LiNGAMor non-parametric methods PC or FCI etcor non parametric methods PC or FCI etc. (Spirtes et al., 2000).

– Small sample size or small non-GaussianityTry bootstrap to see if the result is reliableTry bootstrap to see if the result is reliable.

Background knowledge might be wrong– Background knowledge might be wrong.

63

Q. Relation to causal Markov condition?

• A. The following 3 estimation principles are equivalent (Zhang & Hyvarinen ECML09; Hyvarinen et al JMLR 2010):equivalent (Zhang & Hyvarinen, ECML09; Hyvarinen et al., JMLR, 2010):1. Maximize independence between external influences .2 Minimize the sum of entropies of external influences

iee2. Minimize the sum of entropies of external influences .

3. Causal Markov condition (Each variable is independent of its non-descendants in the DAG conditional on its

ie

its non-descendants in the DAG conditional on its parents) AND maximization of independence between the parents of each variable and its corresponding p p gexternal influences .ie

64

Q. I am a psychometrician and am more interested in latent factors.• A. Shimizu, Hoyer, and Hyvarinen. (2009)

proposes LiNGAM for latent factors:proposes LiNGAM for latent factors:

dBff += -- LiNGAM for latent factors

eGfx += -- Measurement model

65

Others• Q. Prior knowledge?

– It is possible to incorporate prior knowledge. The accuracy of p p p g yDirectLiNGAM is often greatly improved even if the amount of prior knowledge is not so large (Inazumi et al., LVA/ICA2010).

• Q. Sparse LiNGAM?– Zhang et al. (ICA09) and Hyvarinen et al. (JMLR, 2010).– ICA + adaptive Lasso (Zou 2006)ICA + adaptive Lasso (Zou, 2006).

• Q. Bayesian approach?H d H tti (UAI09) H d Wi th (NIPS09)– Hoyer and Hyttinen (UAI09); Henao and Winther. (NIPS09).

• Q. The idea can be applied to discrete variables?Q. The idea can be applied to discrete variables?– One proposal by Peters et al. (AISTATS2010).– Comment: if your discrete variables are close to be continuous, e.g.,

ordinal scales with many points LiNGAM might workordinal scales with many points, LiNGAM might work.

66

Q Nonlinear extensions?• A Several nonlinear SEMs have been proposed:

Q. Nonlinear extensions?• A. Several nonlinear SEMs have been proposed:

– DAG; No latent confounders.

( ) ij

iiji ex jfx +=∑ ofparent -- Imoto et al. (2002)1.

( ) iiii

j

exfx += ofparents -- Hoyer et al. (NIPS08)2.

( )( )iiiii exffx += − ofparents1,12, -- Zhang et al. (UAI09)3.

• For two variable cases, unique identification possible except several combinations of nonlinearities and distributions (Hoyer et al., NIPS08; Zhang & Hyvarinen, UAI09).

67

Nonlinear extensions (continued)

• Proposals to aim at computational efficiency (Mooij et al., ICML09; Tilmann et al., NIPS09; Zhang & Hyvarinen, ECML09; UAI09).

• Pros: – Nonlinear models are more general than linear models.g

• Cons:– Computationally demanding.

• Current: at most 7 or 8 variables.• Perhaps assumption of Gaussian external influences might• Perhaps, assumption of Gaussian external influences might

help: Imoto et al. (2002) analyzes 100 variables.

– More difficult to allow other possible violations of LiNGAMassumptions, latent confounders etc.

68

Q. My data follows neither such linear SEMs nor such nonlinear SEMs as you

have talked.

• A Try non-parametric methods e g

have talked.

A. Try non parametric methods, e.g.,– DAG: PC (Spirtes & Glymour, 1991)

– DAG with latent confounders: FCI (Spirtes et al 1995)DAG with latent confounders: FCI (Spirtes et al., 1995).

( )exfx ofparents=

( ) f

( )iiii exfx ,ofparents=

• Probably you get an (probably large) class of equivalence models rather than a single model, b h ld b h b lbut that would be the best you currently can.

69

Q Deterministic relations?Q. Deterministic relations?

• A. LiNGAM is not applicable.

• See Daniusis et al. (UAI2010) for a method to analyze deterministic relationsanalyze deterministic relations.

70

Extra slides

Very brief review ofVery brief review of Linear SEMs and causalityy

See Pearl (2000) for detailsSee Pearl (2000) for details.

72

We can give a causal interpretation to each path-coefficient bij (Pearl, 2000).ijb

• Example:

11

exbxex

+= x1 e1

221b

32323

21212

exbxexbx

+=+=

x3

x2

e3

e232b

32b

• Causal interpretation of :32bCausa e p e a o o,constant atoconstant afromchangeyouIf 102 ccx

32

( ).bychangewill)(then 01323 ccbxE −

73

`Change from to ’2x 1c0cg• It means `Fix at regardless of the variables 2x 0c

2 10

that previously determines and change the constant from to ’ (Pearl, 2000).1c0c

2x

• `Fix at regardless of the variables that determines ’ is denoted by .2x ( )02 cxdo =

2x 0c

– In SEM, `Replace the function determining with a constant ’:

Mutilated model0c

2x

11 ex =11 ex =

Mutilated model with do(x2=c0):( )02 cxdo =

21212

11

exbxexbx

+=+=

02

11

bcxex

+=

32323 exbx +=32323 exbx +=

(Assumption of invariance: the other functions don’t change)

74

Average causal effect: Definition (Rubin, 1974; Pearl, 2000)

• Average causal effect of x2 on x3 when changing x2 from c0 to c1:2x

2x 3x0c 1cchanging x2 from c0 to c1:

( )( ) ( )( )023123 || cxdoxEcxdoxE =−= c0c1

2 0c 1c

0cc01c

Computed based on the mutilated models

( )0132 ccb −= 32b– Computed based on the mutilated models

with do(x2=c1) or do(x2=c0) .( )12 cxdo = ( )02 cxdo =

• Average causal effects can be computed based on estimated path-coefficient matrix Bon estimated path coefficient matrix B.

75Cf. Conducting experiments with d i t i th d trandom assignment is a method to

estimate average causal effectsestimate average causal effects• Example: a drug trialp g• Random assignment of subjects into two groups

fixes to or regardless of the variables that2x 1c 0cfixes to or regardless of the variables that determines :– Experimental group with (take the drug)

2 1c 0c2x

12 cx =Experimental group with (take the drug) – Control group with (placebo)02 cx =

12 cx

( )( ) ( )( )023123 || cxdoxEcxdoxE =−=

for the exp. group for the control group-)( 3xE )( 3xE