Bayesian Structural Equation Modeling:

An Overview and Some Recent Results

Sik-Yum Lee

IMPS 2011, Hong Kong

1. INTRODUCTION

2. BAYESIAN APPROACH

Estimation

Model Comparison

Applications

3. BAYESIAN ANALYSIS OF A LONGITUDINAL SEM

4. BAYESIAN ANALYSIS OF A NON-PARAMETRIC SEM

5. CONCLUSION

1 / 40

1. INTRODUCTION

Basic Structural Equation Model (SEM):

Measurement Equation:

yi = µ+ Λωi + εi ; i = 1, · · · , n

ωi =

ηiξi

Structural Equation: ηi = Πηi + Γξi + δi

Regression models with latent variables.

As latent variables are random, regression methods cannot apply.

2 / 40

Covariance Structure Analysis: Analyze the covariance matrix of y,

Σ(θ), based on the sample covariance matrix S.

Difficulties in analyzing complex SEMs and/or data structures:

For examples:

• NSEM with a nonlinear structural equation.

• Two level NSEM with ordered categorical data.

• Mixtures NSEMs, etc.

3 / 40

Advantages of the Bayesian Approach:

a) A more flexible approach to deal with complex situations.

b) Utilizes useful prior information (if available).

c) Achieves reliable results with small/moderate sample sizes

(Scheines, Hoijtink & Boomsma, 1999; Dunson, 2000;

Lee & Song, 2004).

d) Gives direct estimates of latent variables.

4 / 40

2. BAYESIAN APPROACH

2.1 Bayesian Estimation

M: The model of interest.

θ: Vector of unknown parameters in M.

D0: Observed data; continuous and/or discrete data.

Du: Unobserved data; missing data, hidden continuous values

underlying ordered categorical data, etc.

Ω: Various types of latent variables.

The main issue is to estimate θ.

5 / 40

Treat θ as random with prior pdf p(θ)

p(θ|D0) posterior density of θ given D0, under M:

behavior of θ under the given data.

Posterior Analysis

p(θ|D0) ∝ p(D0|θ)p(θ)

log p(θ|D0) ∝ log p(D0|θ) + log p(θ)

∝ log likelihood + log prior

Use information available from the data D0, and prior information

from p(θ).

6 / 40

Prior Distribution

Non-informative prior distributions: p(θ) proportional to a constant

or has an extremely large variance. No prior information available.

Informative prior distributions: Used when some prior knowledge is

available.

If p(θ) is chosen, s.t. p(θ) and p(θ|D0) are of the same form, then

p(θ) is called a conjugate distribution.

7 / 40

Conjugate prior distributions for parameters in SEMs:

µD= N[µ0,Σ0]

ΨεkD= Inverted Gamma(α0εk , β0εk).

(Similarly for Ψδ)

ΛkD= N[Λ0k ,H0k ], kth row of Λ

cov(ξ) = ΦD= IW [R0, ρ0],

where µ0, Σ0, α0εk , β0εk , Λ0k , H0k , R0, ρ0 are hyperparam-

eters whose values are specified based on the prior information or

knowledge.

8 / 40

Bayesian estimate of θ: The mean of p(θ|D0).

Usually not in closed form.

Simulate a large sample of θ = θ(1), · · · ,θ(T ) from p(θ|D0), and

get the Bayesian estimate as

θ = T−1T∑t=1

θ(t)

How to get this sample?

9 / 40

Usually, it is hard to simulate observations from p(θ|D0)

Data Augmentation: Augment D0 with Du and Ω.

Consider the joint posterior p(θ,Du,Ω|D0).

Simulate (θ,Du,Ω) from p(θ,Du,Ω|D0) via some Markov chain

Monte Carlo (MCMC) methods in statistical computing.

10 / 40

Gibbs Sampler: Begin with any starting values θ(0), D(0)u , Ω(0).

At the j th iteration with θ(j), D(j)u , Ω(j); simulate

θ(j+1) from p(θ|D(j)u ,Ω(j),D0),

D(j+1)u from p(Du|θ(j+1),Ω(j),D0),

Ω(j+1) from p(Ω|θ(j+1),D(j+1)u ,D0).

Need to derive various components in the full conditional distribu-

tions p(θ|Du,Ω,D0), p(Du|θ,Ω,D0), and p(Ω|θ,Du,D0).

11 / 40

Full conditional distributions are easier to deal with than p(θ|D0).

i. Given Ω, SEM is the regression model.

ii. Given Du, e.g. the hidden continuous values,

difficulties related to ordered categorical variables can be solved,

etc.

Most of the full conditional distributions: Normal, Gamma, etc.

For non-standard conditional distributions, some standard MCMC

methods (MH algorithm) can be used to draw observations.

12 / 40

Check convergence: Parallel sequences generated with starting val-

ues mixed well together.

bb[5] chains 1:3

iteration

1 1000 2000 3000 4000

-0.5

0.0

0.5

1.0

1.5

a

bb[15] chains 1:3

iteration

1 1000 2000 3000 4000

-2.0

-1.0

0.0

1.0

b

13 / 40

After achieving convergence, say after J iterations,

(θ(J+1),D(J+1)u ,Ω(J+1)) · · · can be regarded as observations

from p(θ,Du,Ω|D0).

Collect (θ(t),D(t)u ,Ω(t)), t = J + 1, · · · , J + T for statistical

inference.

θ = T−1T∑t=1

θ(t), Ω = T−1T∑t=1

Ω(t).

WinBUGS (Spiegelhalter et al. 2003) gives Bayesian estimates of

parameters and latent variables for many SEMs.

14 / 40

2.2 Bayesian Model Comparison

D0: Given data set.

M1, M2: Competing SEMs with θ1 and θ2, respectively.

(i) Bayes Factor: B10 = p(D0|M1)p(D0|M0)

.

Select M0 if B10 < 1;

Select M1 if B10 > 1.

See Kass & Raftery (1995) for more detailed interpretation.

p(D0|Mk) =

∫p(θk ,D0|Mk)dθk =

∫p(D0|θk ,Mk)p(θk)dθk .

Computed via path sampling (see, Lee, 2007).

15 / 40

(ii) Deviance Information Criterion (DIC, Spiegelhalter et al., 2002):

DICk = − 2

T

T∑t=1

log p(D0|θ(t)k ,Mk) + 2dk

dk = dimension of θk

Model with smaller DIC value is selected.

For many SEMs, DIC values are available in WinBUGS.

16 / 40

(iii) Lν Measure:

Let Drep0 = Drep

1 , · · · ,Drepn be a response from a replicate exper-

iment having from the same p(D0|θ). The Lν measure (Ibrahim,

Chen & Sinha, 2001) is defined by

Lν(D0) =n∑

i=1

tr

cov(Drep

i |D0)

+ ν

n∑i=1

tr

[E (Drep

i |D0)−Di

E (Drep

i |D0)−Di

′]

Expectation is taken w.r.t. Drep0 |D0, computed from simulated

observations in estimation.

Model with smaller Lν is selected.

ν usually taken as 0.5.

See Song et al. (2011) for more details and some applications.

17 / 40

2.4 Applications to SEMs or Related Models

a. SEMs with mixed continuous and discrete data:

Shi & Lee (2000, JRSSB), Dunson (2000, JRSSB),

van Onna (2002, Psym), Dunson & Herring (2005, Biostatistics),

Song et al. (2009, Stat. Med.)

b. SEMs with missing data (MAR/Non-ignorable):

Song & Lee (2002, Psym), Song & Lee (2009, Stat. Med.)

c. Nonlinear SEMs:

Arminger & Muthen (1997, Psym), Lee & Song (2003, Psym)

d. Multilevel SEMs:

Ansari & Jedidi (2000, Psym),

Ansari, Jedidi & Duke (2002, Psym), Song & Lee (2004, BJMSP)

18 / 40

e. Mixture SEMs:

Jedidi, Jagpal & DeSarbo (1997, Marketing Sci.),

Zhu & Lee (2001, Psym), Cai, Song & Hser (2010, Stat. Med.)

f. Item Response Models:

Hoijtink & Molenaar (1997, Psym), Patz & Junker (1997, JEBS),

Begnin & Glas (2001, Psym), Fox & Glas (2001, Psym),

Miyazaki & Hoshino (2009, Psym)

g. Bayesian Semiparametric SEMs:

Lee, Lu & Song (2007, Stat. Med.),

Song, Lee & Xia (2009, Stat. Med.), Yang & Dunson (2010, Psym)

19 / 40

h. Longitudinal SEMs:

Dunson (2003, JASA), Song, et al. (2011, SEM)

i. SEM with a Non-parametric Structural Equation:

Song & Lu (2010, J. Comp. Graphical Stat.)

j. Transformation SEMs:

Song & Lu (2011)

Data Augmentation, MCMC methods

20 / 40

3. BAYESIAN ANALYSIS OF A LONGITUDINAL SEM

3.1 Motivation

SEM1 SEM2 · · · SEMT

t = 1 t = 2 · · · t = T

For investigating various behaviors that are

i. invariant over time,

ii. dynamically change over time.

21 / 40

3.2 The Model

Let ugt be an observed random vector for the g th (g = 1, · · · ,G )

individual measured at time point t (t = 1, · · · ,T ). A two-level

model for ugt is defined by: (Song et al., 2010)

ugt = yg + vgt , (1)

yg : second level random vector (independent of t) for accounting

characteristics invariant with t

vgt : first level random vector for accounting characteristics that

are changed dynamically with t

22 / 40

The Second level model

For assessing invariant characteristics over time:

yg = A0cg0 + Λ0ωg0 + εg0, g = 1, · · · ,G (2)

cg0: vector of covariates

ωg0: vector of latent variables

εg0: vector of residual errors

A0 and Λ0: matrices of unknown coefficients

ωg0 is i.i.d N[0,Φ0]

εg0 is independent of ωg0, and i.i.d N[0,Ψ0], Ψ0 is diagonal

Not depending on t, invariant over time23 / 40

The first-level dynamic model

For g = 1, · · · ,G , t = 1, · · · ,T , the measurement equation is:

vgt = Atcgt + Λtωgt + εgt , (3)

where the definitions of At , cgt , Λt , ωgt , and εgt are similar to

those given in equation (2), except here they are defined at time t

nested within the individual g .

Here, εgt are independently dist. as N[0,Ψt ], and Ψt is diagonal.

24 / 40

Let ωgt = (η′gt , ξ′gt)′. The relationships among ηgt and ξgt ,

covariates, and latent vectors at previous times are studied through

the following structural equation:

ηgt = B0dg0+Btdgt+ΓtFt(ηg1, · · · ,ηg ,t−1, ξg1, · · · , ξg ,t−1, ξgt)+δgt ,

(4)

B0, Bt , Γt : matrices of unknown coefficients

dg0: covariates that are invariant with time

dgt : covariates variant with time

Ft : differentiable vector-valued function of ξgt at the current

time t, and ηg1, · · · ,ηg ,t−1, ξg1, · · · , ξg ,t−1 at previous times

δgt : residual errors, independently distributed as N[0,Ψδt ]

ξg = (ξ′g1, · · · , ξ′gT )′D= N[0,Φ]

25 / 40

Some simple examples of Ft(ηg1, · · · ,ηg ,t−1, ξg1, · · · , ξg ,t−1, ξgt):

For t = 1, · · · ,T , ωgt = (ηgt , ξ1gt , ξ2gt)′,

ηgt =γη1ηg1 + · · ·+ γη,t−1ηg ,t−1 + γt1ξ1gt + γt2ξ2gt + γt3ξ1gtξ2gt + δgt ,

ηgt =γ1ηg ,t−1 + γ2ξ1g ,t−1 + γ3ξ2g ,t−1 + γ4ξ1gt + γ5ξ2gt + γ6ξ21gt

+ γ7ξ22gt + γ8ξ1gtξ2gt + δgt .

May add covariates.

26 / 40

Let µgt = Xgt ,Zgt, Xgt continuous, Zgt ordered categorical.

D0 = (X0,Z0), observed continuous data X0,

observed ordered categorical data Z0.

Du=missing data, hidden continuous values underlying Z0.

Ωg = y1, · · · , yG,

Ω0 = ω10, · · · ,ωG0,

Ω1 = ω1, · · · ,ωG, ωg = ωg1, · · · , ωgT, ωgt =

ηgtξgt

,

θ =[A0,B0,Λ0,Φ0,Ψ0

,

(At ,Λt ,Ψt ,Bt ,Γt), t = 1, · · · ,T, Φ

],

α = unknown thresholds.

27 / 40

Data Augmentation: Augment D0 with Du, Ωg , Ω0, Ω1.

Consider p(θ,α,Du,Ωg ,Ω0,Ω1|D0).

Gibbs sampler: Simulate:

p(α,Du|θ,Ωg ,Ω0,Ω1,D0)

p(θ|α,Du,Ωg ,Ω0,Ω1,D0)

p(Ωg |θ,α,Du,Ω0,Ω1,D0)

p(Ω0|θ,α,Du,Ωg ,Ω1,D0)

p(Ω1|θ,α,Du,Ωg ,Ω0,D0)

See Song et al. (2010) for technical details and numerical results.

28 / 40

4. BAYESIAN ANALYSIS OF A SEM WITH A NON-

PARAMETRIC STRUCTURAL EQUATION

4.1 Motivation

For SEMs, model with a given specific parametric structural equation

may not be realistic.

Consider: η1 = axi + γ1ξi1 + γ2ξi2 + γ3ξi3 + δi

Regression — observed variables — data

SEM — latent variables — ?

Desirable to develop model with its structural equation formulated

through unspecific function of latent variables and covariates.

29 / 40

4.2 The Model (Song & Lu, 2010)

Measurement Equation : yi = Aci + Λωi + εi , i = 1, · · · , n (5)

ωi =

ηiξi

,ηi : r × 1 vector of outcome l.v.

ξi : s × 1 vector of explanatory l.v.D= N[0,Φ]

Structural Equation: for the j th element ηij in ηi : For j = 1, · · · , r :

ηij = gj1(xi1) + · · ·+ gjd(xid) + fj1(ξi1) + · · ·+ fjs(ξis) + δj , (6)

xi1, · · · , xid : covariates

gj1, · · · , gjd : unspecified functions

fj1, · · · , fjs : unspecified functions

30 / 40

For any unspecified general function f (x) with a continuous 2nd

order derivative, it can be modeled by a sum of B-splines basis with

K knots in the domain of x as (see De Boor, 1978):

f (x) =K∑

k=1

βkBk(x),

K : number of knots (usually 10-30)

βk : unknown parameter

Bk(x): B-spline of appropriate order

See examples B-splines in the text book (De Boor, 1978)

31 / 40

The basic idea for modeling (6) is: (suppress subscript j)

ηi =

Kb1∑k=1

b1kBx1k(xi1) + · · ·+

Kbd∑k=1

bdkBxdk(xid)

+

K1∑k=1

β1kB1k(ξi1) + · · ·+Ks∑k=1

βskBsk(ξis) + δ.

(7)

K ’s: number of knots

b1k , · · · , bdK1b, · · · , · · · , bdKbd

: unknown parameters

β1k , · · · , β1K1 , · · · , · · · , βsKs : unknown parameters

Bxmk , Bmk : modified B-splines

32 / 40

Let

Y = (y1, · · · , yn)

Ω = (ω1, · · · ,ωn)

θ: vector consists of all unknown parameters

Posterior Analysis: Consider the joint posterior distribution with

conjugate priors. Apply Gibbs sampler (with MH algorithm):

a) draw Ω from p(Ω|θ,Y)

b) draw θ from p(θ|Ω,Y)

Only rough ideas.

See Song & Lu (2010) for much more details in formulating the

model and solving the difficulties in the posterior analysis.

33 / 40

4.3 A Simulation Study

Measurement equation:

yi = a + Λωi + εi ,

where a = (a1, · · · , a12)′, ωi = (ηi , ξi1, ξi2, ξi3)′, and

Λ′ =

1 λ21 λ31 0 0 0 0 0 0 0 0 0

0 0 0 1 λ52 λ62 0 0 0 0 0 0

0 0 0 0 0 0 1 λ83 λ93 0 0 0

0 0 0 0 0 0 0 0 0 1 λ11,4 λ12,4

,

where ξD= N[0,Φ], and

aj = 0.5, ψj = 0.3, for j = 1, · · · , 12

λ21 = λ31 = λ52 = λ62 = λ83 = λ93 = λ11,4 = λ12,4 = 0.8,

φii = 1.0, φik = 0.2, for all i , k , i 6= k ; ψδ = 0.334 / 40

Structural equation:

ηi = g(xi ) + f1(ξi1) + f2(ξi2) + f3(ξi3) + δi ,

where

g(x) = (x/2)3

f1(ξ) = sin(1.5ξ)− ξ

f2(ξ) = 1.65− exp(ξ)

f3(ξ) = −0.5 + exp(2ξ)/[1 + exp(2ξ)]

35 / 40

n = 300, K = 20, number of replications=100.

Par. True Bias RMS Par. True Bias RMS

λ21 0.8 -0.001 0.018 φ11 1.0 -0.057 0.110

λ31 0.8 -0.001 0.019 φ12 0.2 -0.022 0.070

λ52 0.8 0.022 0.044 φ13 0.2 -0.010 0.064

λ62 0.8 0.021 0.048 φ22 1.0 -0.037 0.113

λ83 0.8 0.022 0.050 φ23 0.2 -0.003 0.058

λ93 0.8 0.023 0.048 φ33 1.0 -0.025 0.116

λ11,4 0.8 0.007 0.049 ψδ 0.3 0.020 0.040

λ12,4 0.8 0.016 0.046

36 / 40

The solid curves represent the true curves, the dashed and dot-dashed

curves respectively represent the estimated posterior means and the 5%-

and 95%-pointwise quantiles on the basis of 100 replications.37 / 40

Remarks:

• The estimated curves correctly capture the true functional

relationships.

• The parameter estimates are accurate: all bias close to zero,

most RMS values are below 0.05.

• The sensitivity analysis shows that the Bayesian results are ro-

bust to different prior inputs.

• For analysis of some real-world data sets, see Song & Lu (2010).

38 / 40

5. CONCLUSION

a) The Bayesian approach with data augmentation and MCMC

methods is flexible for analyzing complex SEMs.

b) Future research/work

(i) Dynamic SEM with non-ignorable missing data

(ii) SEMs with structural equation involving nonparametric

function of interaction of latent variables; such as

ηi = f1(ξi1) + f2(ξi2) + f3(ξi1ξi2).

(iii) Dynamic SEM with nonparametric structural equation.

(iv) Development of use-friendly software for applied researchers.

39 / 40

Thank you!

40 / 40