Multi-level AnalysisRecognizing the Problem

Maureen Smith, MD PhDDepts. of Population Health Sciences

and Family MedicineUniversity of Wisconsin-Madison

Target Audience

Those of you likely to sit down in front of a computer and try to do this!

Make sure you have pencil and paper.

Goals Today

• Understand how – Data and multi-level modeling relate– Underlying concepts are ubiquitous– What the typical output means– Much trouble you can get into

• We will not– Spend a lot of time of statistical tests– Spend a lot of time of software

A day in the life of a researcher

• We have data– ID (observation #)– X (variable 1)– Y (variable 2)

• We want to use the value of X to explain the value of Y

ID X Y

1 60 3

2 75 6

3 81 10

4 70 7

5 65 5

Welcome to the fantasy world of linear regression

• A simple model

yi = intercept + slope(xi) + errori indicates observations

(1…N)

• Assumptions– Linearity– Independence– Normality– Constant variance

Reality check

• How often are observations truly independent from one another?– Dot indicates geographic

location of teenager– Orange or green indicates

hair color

• Do these teenagers look independent?

Clustering

(Artificial or natural)

1) Clustering introduced in sampling

• Multistage sampling– Circles represent city blocks– Blocks randomly sampled– All persons in block surveyed to

determine attitudes

• Persons in one block are more like their neighbors than persons who live in another block

• Nesting or clustering of data – Persons within blocks

Block 1 Block 2

Block 3Block 4

Not all blocks are selected

Effect of sample design on errors

• Errors in linear regression– Assume independence– Each person => info– Each person worth “1”

• If clustering occurs– Obs not independent– Each person => less info– Each person worth < “1”

Block 1 Block 2

Block 3Block 4

Simple linear regression won’t work!

• Violates assumption of independence

• If don’t account for it– Standard errors are too small – Makes coefficients look more significant– “You think there is more information in the

data than actually exists”

How much information is lost?“Design Effect”

• If designing a study using multistage sampling, need to increase sample size to account for loss of information

• Design effect– Each observation is “worth less”– Need to estimate your “effective” sample size– Used for sample size calculations in multi-stage sampling

Neffective = Nn Design effect

Questions – Pair up!

• Multi-stage sample design– City blocks N= 3– Persons N=26

• Design effect = 2

1. What is the effective sample size?

2. What sample size would you use in your power calculations?

Block 1 Block 2

Block 3Block 4

2) Clustering introduced naturally

• Analyze costs of care for hospitalized patients

• Patients in one hospital are more alike than patients in another hospital

• Nesting or clustering of data– Patients within hospitals

Hospital 1Hospital 2

Hospital 3

Hospital 4

Effect of natural clusters on errors

• Same effect on errors– Obs not independent– Each person => less info– Each person worth < “1”

• Simple linear regression won’t work!

Hospital 1Hospital 2

Hospital 3

Hospital 4

Accounting for Clustering

Do we care?

What do we do?

• First question - do we care?– Is clustering a nuisance?

OR– Is clustering an interesting phenomenon?

• Leads to different analytic strategies

If clustering is a nuisance

• Example - Multi-stage sampling– Don’t care how people vary within city

blocks versus between city blocks– Artificially imposed by the sampling design– Not interested in measuring it– Just want to correct for it

• Use analytic strategies that correct for clustering

How to correct errors for clustering

• Robust estimates of variance– Stata “, robust cluster (____)”– SAS empirical estimates of variance

• Programs that account for complex survey design (weights, strata, clusters)– Stata “svy” commands– SAS “survey___” commands

• Other strategies

If clustering is interesting

• Example - examine costs for hospitalized patients

• Split out the variation in costs– How much variation due to differences in patients?– How much variation due to differences in hospitals?

• Examine factors that explain variation in costs– Characteristics of patients– Characteristics of hospitals

• Analytic strategy = Multi-level modeling!

Questions

1. Identify 3 patient characteristics that might explain variation in costs

2. Identify 3 hospital characteristics that might explain variation in costs

3. Do you think more of the variation in costs is explained by the patient or the hospital?

Representing Clustering in a Model

(Multi-level models)

(Hierarchical linear models)

(Random effects models)

The concept of “levels”

• Our example – 2 levels– Micro = patients (N=26)

• Micro-level = “units”

– Macro = hospitals (N=3)• Macro-level = “groups”

• At each level– Patient characteristics– Hospital characteristics

Hospital 1Hospital 2

Hospital 3

Hospital 4

Data Structure - Patient

Patient ID

Hospital ID

Age(X)

Cost (Y)

1 1 60 3

2 1 75 6

3 2 81 10

4 2 70 7

5 2 65 5

• Y represents a patient characteristic– Cost (thousands of $)

• X represents a patient characteristic– Age– Note – understand possible

mechanism at each step– “Older patients are sicker

and tend to cost more”

Patient-level data ( = “unit-level data”)

Simple Linear Regression

yi = a + bxi + ei

• i indexes patients (i=1 to N)

• Relates x to y

• Both variables are patient characteristics

• Remember the assumptions

Questions

costi = a + b(agei) + ei

1. Is there a problem with this model when applied to these data?

2. If so, what?

Patient ID

Hospital ID

Age (X)

Cost (Y)

1 1 60 3

2 1 75 6

3 2 81 10

4 2 70 7

5 2 65 5

The Problem

• Does not account for the clustering of patients within hospitals– Data have a

structure that is not represented

– ei - Assumption of independence is not met

Patient ID

Hospital ID

Age (X)

Cost (Y)

1 1 60 3

2 1 75 6

3 2 81 10

4 2 70 7

5 2 65 5

Do we care?

• If clustering is nuisance => Stata robust option

• If clustering is interesting => Multilevel model

Data Structure - Hospital

Hospital ID

Beds (W)

1 10

2 65

Hospital-level data ( = “group-level data”)

• W represents a hospital characteristic– # of beds in the hospital

• Possible mechanism = “Bigger hospitals do things more expensively”– More technology– More high-cost specialists

Combined Data Structure

Hospital ID

Patient ID

Age (X)

Cost (Y)

1 1 60 3

1 2 75 6

2 3 81 10

2 4 70 7

2 5 65 5

Patient-level data

+Hospital

IDBeds (W)

1 10

2 65

Hospital-level data

= ?

Combined Data Structure

Patient ID

Hospital ID

Age (X)

Cost (Y)

Beds (W)

1 1 60 3 10

2 1 75 6 10

3 2 81 10 65

4 2 70 7 65

5 2 65 5 65

Patient- and hospital-level data

• Age (X) and Cost (Y)– Variation between

patients

• Beds (W)– Only variation

between hospitals– No variation within

hospitals

WARNING – Equations coming up!

Remember - In multi-level modeling …

SUBSCRIPTS ARE YOUR FRIENDS!

Simple Linear Regression(one approach to modeling this data structure)

yij = a + bxij + dwj + eij

• j indexes hospitals (j=1 to N)• i indexes patients within hospitals (i=1 to nj)

costij = a + b(ageij) + d(bedsj) + eij

• Frequently used

Questions

1. Is there a problem with this model when applied to these data?

2. If so, what?

Patient ID

Hospital ID

Age (X)

Cost (Y)

Beds (W)

1 1 60 3 10

2 1 75 6 10

3 2 81 10 65

4 2 70 7 65

5 2 65 5 65

costij = a + b(ageij) + d(bedsj) + eij

The Problem, Part 2

• You must assume that all of the data structure is represented by the explanatory variables

• Unlikely this will account for the clustering of patients within hospitals– Assumes that all clustering within hospitals is explained

by the number of beds in the hospital (W)– If “beds” does not explain all clustering, then

assumption of independence is not met for eij

How do we represent the clustering?

• Let the regression coefficients vary from group to group

yij = aj + bjxij + dwj + eij

• Groups j can have higher or lower values of aj and bj

• Why not create dj?

Starting simple – random intercept

• Model the clustering between groups– Let the intercept only (aj) vary from group to group

– Take out all group-level variables (W)

yij = aj + bxij + eij

• Groups j - higher or lower values of aj only

• Assumes some groups tend to have, on average, higher or lower values of Y

Question

yij = aj + bxij + eij

1. Why take the group-level variable (W) out of this model?

2. Must W be taken out of the model?

How do we want to model variation between groups?

• W – a “partial” way to model variation between groups– If included, it will pick up part of the variation between

groups– “Part of the variation in costs between hospitals will be

explained by the number of beds in the hospital”

• Goal of a random intercept model– Model the actual structure of the data– Let groups vary, on average, in Y– “Let the hospitals vary, on average, in cost”

How do we actually do it?

yij = aj + bxij + eij

• Split aj into (a0 + uj)

yij = a0 + uj + bxij + eij

• a0 = average intercept (constant)• uj = deviation from the average intercept for group j = conditional on X, individuals in group j have Y values

that are uj higher than the overall average

• “Conditional on patient age, patients in Hospital j have costs that are uj higher than the average costs for all patients”

Representation as EquationsSingle Equation vs. Multiple Equation Representation

(1) yij = a0 + bxij + uj + eij

OR

(2) yij = aj + bxij + eij

aj = a0 + uj

• uj = deviation from the overall average for group j

What do we do with uj?Part 1 – Fixed effects

• Are groups j regarded as unique?– Do you want to draw conclusions about

each group?

TREAT AS “FIXED EFFECTS”

• Create j – 1 indicator variables (0/1)

• Leads to j – 1 regression parameters

Questions

1. For our data, what does this equation look like if uj is modeled as a fixed effect?

2. Are all indicator variables in a model also fixed effects?

Patient ID

Hospital ID

Age (X)

Cost (Y)

1 1 60 3

2 1 75 6

3 2 81 10

4 2 70 7

5 2 65 5

costij = a0 + b(ageij) + uj + eij

Modeling uj as a fixed effect(uj = “differences between hospitals”)

costij = a0 + b(ageij) + c(hosp2ij) + eij

• hosp2 = 0/1 – 1 = patient i in hospital 2, 0 = patient i in hospital 1

• Do we need index j? No – why?

costi = a0 + b(agei) + c(hosp2i) + ei

• What assumptions does this model make?

What do we do with uj?Part 2 – Random effects

• Three issues– Are groups regarded as sample from pop.?– Do you want to test the effect of group level variables

(remember W = # beds)?– Do you have small group sizes (2-50 or 100)?

TREAT AS “RANDOM EFFECTS”• Model uj explicitly• Additional assumption that uj is i.i.d.

– Groups (hospitals) considered exchangeable• Can include group-level explanatory variables (W)

Questions

1. For our data, what does this equation look like if uj is modeled as a random effect?

2. How would we include our hospital-level explanatory variable?

yij = a0 + b(xij) + uj + eij

Patient ID

Hospital ID

Age (X)

Cost (Y)

Beds (W)

1 1 60 3 10

2 1 75 6 10

3 2 81 10 65

4 2 70 7 65

5 2 65 5 65

Modeling uj as a random effect(uj = “differences between hospitals”)

costij = a0 + b(ageij) + uj + eij

• uj = deviation from the average cost for hospital j = estimated using HLM, SAS, Stata (get a number!)

costij = a0 + b(ageij) + d(bedsj) + uj + eij

• Uses the number of beds in the hospital to explain some of the variation in uj

• Question - what happens to uj if the number of beds explains all of the differences between hospitals?

Question

costij = a0 + b(ageij) + c(hosp2ij) + d(bedsj) + uj + eij

• Is this equation an example of random or fixed effects?

• What are the challenges in modeling this equation?

It is GARBAGE!!!!!

• Totally meaningless– Models uj as fixed effect as well as a random

effect with a hospital-level covariate

• Probably won’t run (if it does, don’t believe anything you see)

• Can’t model uj (variation in the average cost between hospitals) as both a random effect and a fixed effect at the same time!

What we have done so far

• We discussed– Clustering (artificial and natural)– Accounting for clustering

• Nuisance = robust estimates of variance• Interesting = multilevel models

– Representing clustering in simple model• Fixed effects• Random effects with group-level explanatory

variables

What we will do next

• Sitting down at the computer– Modeling random effects using SAS proc

mixed– Random coefficients other than the

intercept (briefly)

• Repeated measures

• Non-continuous outcomes

• Computer programs

SAS Proc Mixed

Sitting in front of the computer

(Major reality check!)

Random Effects Models for Continuous Outcomes

• SAS proc mixed – Powerful and dangerous– Poor documentation (use Singer 1998)– Defaults may not be appropriate– Single level representation of equations

• Stata xtreg – Good documentation– Defaults usually ok– Single level representation of equations

Example of a Hospital EpidemicRevised from Singer 1998

• 7,185 patients admitted to 160 hospitals

• 14-67 patients per hospital– Hospital with N=67 in Washington

DC• Patient-level outcome is the

severity of new disease (MATHACH)

• Question: what does an equation look like if hospital modeled as random effect?– Use single and multiple equation

representation

Patient ID Hospital

MATHACH (Y)

1 1 3

2 1 6

3 2 10

4 2 7

5 2 5

Modeling hospital as a random effect(uj = “differences between hospitals”)

MATHACHij = a0 + uj + eij

and

MATHACHij = aj + eij

aj = a0 + uj

• uj = deviation from the average MATHACH severity score for hospital j

• Write code to model using SAS proc mixed (single equation)

Code for SAS proc mixed(Described using Singer’s terminology)

Proc mixed noclprint covtest;

Class hospital;

Model MATHACH = /solution;

Random intercept /subject=hospital;

Model statement indicates fixed effects (in this case, only one fixed effect – the intercept

a0 – which is implied)

Indicates random effects (intercept and error term) and the specification of the level 2

units (hospitals). The error term eij is implied.

STOP!• Something seems strange!• How can an intercept be fixed and random?

• Proc mixed terminology derived from multiple equation representation

MATHACHij = aj + eij

aj = a0 + uj

• Fixed and random in proc mixed refer to the modeling of aj, not uj

• BUT decision to model hospitals as fixed or random applies to uj, not aj (earlier discussion)

Important Lesson(This is a source of major confusion)

• Use of the terminology (fixed vs. random) differs widely

• Even when the term in question (e.g., uj) is agreed upon, definitions not only differ but are often incompatible (Gelman 2004 Ann Stat)

• Solution– Don’t just throw terminology around– Draw the dataset, write down the equation(s) and

circle the terms that you want to model– Make sure you understand exactly where those

terms are on the computer output

Code for SAS proc mixed

Proc mixed noclprint covtest;

Class hospital;

Model MATHACH = /solution;

Random intercept /subject=hospital;

Fixed effects = a0 is implied (representing the fixed part of

the intercept).

Adds random effect for hospital = uj (representing the random part

of the intercept).Random effect for patient = eij is implied (represents error term).

MATHACHij = a0 + uj + eij

SAS OutputMATHACHij = a0 + uj + eij

Yikes!

More info needed

Question: What IS a random effect?

MATHACHij = a0 + uj + eij

• How do we describe it?

• What do we want to know?

• How do you model those subscripts?

A random effect is a random variable

• Random variables are described by distributions

• Start with the easy one (basic error term)– eij ~ N(0,2)

– The estimated parameter is 2

• Same approach to new term– uj ~ N(0,)

– The estimated parameter is

• Assume that eij and uj are independent

Interpreting this equation

• One estimated fixed effect– a0 describes average MATHACH score for the hospitals

• Two random effects (variances are estimated) describes variability in hospital means (variability in

average MATHACH between hospitals) describes variability in MATHACH within hospitals

• What is this equation called?

MATHACHij = a0 + uj + eij uj ~ N(0,) eij ~ N(0,2)

Code for SAS proc mixed

Proc mixed noclprint covtest;

Class hospital;

Model MATHACH = /solution;

Random intercept /subject=hospital;

Request hypothesis tests for variance and covariance

components ( & 2)

Identify categorical variables

/solution prints estimates and hypothesis tests for fixed effects (a0 in this case)

SAS Output

MATHACHij = a0 + uj + eij uj ~ N(0,) eij ~ N(0,2)

SAS Output

MATHACHij = a0 + uj + eij uj ~ N(0,) eij ~ N(0,2)

SAS Output

MATHACHij = a0 + uj + eij uj ~ N(0,) eij ~ N(0,2)

ICC - A cool thing

• Use the output to calculate the ICC (intraclass correlation coefficient)

• Figure out the portion of the total variance that occurs between (as opposed to within) hospitals– Bigger means more clustering

• ICC = / ( + 2) = 8.6 / (8.6 + 39.1) = 0.18

Adding predictor variables

• Patient-level covariate is SES

• Hospital-level covariate (MEANSES) is aggregate of patient SES

• MEANSES and SES are centered at the grand mean (mean of 0)

Patient ID

Hospital

SES (X)

MATHACH (Y)

MEANSES (W)

1 1 60 3 -2

2 1 -75 6 -2

3 2 81 10 9

4 2 -70 7 9

5 2 65 5 9

Question

• What does the equation look like if we add a hospital-level characteristic (MEANSES)?

• What do we expect might happen to the estimate of uj?

Solution

MATHACHij = a0 + d(MEANSESj) + uj + eij

uj ~ N(0,) eij ~ N(0,2)

Use the average SES in hospitals to explain some of the variation in MATHACH between hospitals

We would expect that the uj (represented by ) might decrease

Code for SAS proc mixed

Proc mixed noclprint covtest;

Class hospital;

Model MATHACH = MEANSES /solution ddfm=bw;

Random intercept /subject=hospital;

Added fixed effect for MEANSES (recall that fixed

effect for a0 is implied)

Keep random effect for hospital = uj (representing the random

part of the intercept).Random effect for patient = eij is implied (represents error term).

MATHACHij = a0 + d(MEANSESj) + uj + eij

Use between/within method for computing denominator degrees

of freedom (read about it)

SAS Output

Estimate d for MEANSES

Estimate

of decreasesfrom 8.6to 2.6

Interpreting the Numbers

a0 = 12.65 = ?

d = 5.86 = ?

uj = ?

eij = ?

Remember = 2.64

Remember 2 = 39.16

MATHACHij = a0 + d(MEANSESj) + uj + eij

uj ~ N(0,) eij ~ N(0,2)

Question: what is the real-world meaning of each term?What does each term say about MATHACH and hospitals?

Interpreting the OutputConditional Fixed Effects

• a0 = Intercept estimates average MATHACH among hospitals when all other predictors are 0– MEANSES is centered at the grand mean (has a

mean of 0)– Intercept is average MATHACH in a hospital of

average MEANSES

• d = estimated coefficient on MEANSES– 1 unit increase in average SES in a hospital

associated with a 5.86 units increase in MATHACH

MATHACHij = a0 + d(MEANSESj) + uj + eij

Interpreting the OutputConditional Random Effects

= describes variability in average MATHACH between hospitals after accounting for MEANSES– Decreased from 8.6 to 2.6– MEANSES explains a large portion of between-hospital

variation in MATHACH

2 = residual variability in MATHACH among patients within hospitals– Does not change significantly

MATHACHij = a0 + d(MEANSESj) + uj + eij

uj ~ N(0,) eij ~ N(0,2)

More Cool Things

Var. Comp.

a0 = 12.65d = not est. = 8.61 2 = 39.15

MATHACHij = a0 + d(MEANSESj) + uj + eij

uj ~ N(0,) eij ~ N(0,2)

Add MEANSES

a0 = 12.65

d = 5.86

= 2.64

2 = 39.16

Write down everything on this slide

Questions

• What percent of the explainable variation in mean hospital MATHACH scores is explained by MEANSES?

• What is the (residual) ICC among hospitals after accounting for MEANSES?

Solutions

• 69% of the explainable variation in average MATHACH between hospitals is explained by MEANSES

(8.61 – 2.64) / 8.61 = 0.69

• Residual ICC = 0.06 = 2.64 / (2.64 + 39.16)– Correlation between hospitals that have the

same average SES– Can you drop the random effect for hospitals?

Has a sufficient amount of uj been picked up by MEANSES?

Random Coefficients

Briefly!

How do we represent the clustering?

• Let the regression coefficients vary from group to group

yij = aj + bjxij + dwj + eij

• Groups j can have higher or lower values of aj and bj

Representation as EquationsMultiple Equation Representation

yij = aj + bjxij + eij

aj = a0 + uj

bj = b0 + vj

eij ~ N(0,2) uj ~ N(0,u) vj ~ N(0,v) cov (uj,vj) = uv

Doing Random Coefficients

• Can definitely do it

• Extremely complex to model

• Difficult to interpret (must be very careful about centering)

• Maureen’s opinion– Usually like hitting a gnat with a sledgehammer– Almost impossible to explain in the real world

Repeated Measures

One case where you might want to think about random coefficients

What do the data look like?

• Repeated measures of SBP over time are the level-1 units

• The time point of each measurement is a level-1 characteristic

• Each set of measurements clustered within level-2 units (patients)

• Age represents a level-2 characteristic

Obs #SBP (Y)

Time (X)

Patient

Age (W)

1 140 1 1 40

2 90 2 1 40

3 160 1 2 60

4 120 3 2 60

5 130 4 2 60

Question

• What is the equation to model SBP, accounting for clustering within patients?– As a function of

time?– Adding in patient

age?

Obs #SBP (Y)

Time (X)

Patient

Age (W)

1 140 1 1 40

2 90 2 1 40

3 160 1 2 60

4 120 3 2 60

5 130 4 2 60

Solutions

SBPij = a0 + b(TIMEij) + uj + eij

uj ~ N(0,) eij ~ N(0,2)

SBPij = a0 + b(TIMEij) + d(AGEj) + uj + eij

uj ~ N(0,) eij ~ N(0,2)

Code for SAS proc mixed

Proc mixed noclprint covtest;

Class patient;

Model SBP = TIME /solution ddfm=bw;

Random intercept /subject=patient type=un;

Specifies that you want the structure of the variance-covariance matrix of the

random effects to be unstructured (previously it was compound symmetry).

SBPij = a0 + b(TIMEij) + uj + eij

Non-continuous outcomes

Slightly more complex extensions

Basics are similar

• Generalized linear mixed models

• Move to Stata – SAS has multiple problems– Xt commands are useful– GLLAMM is great and flexible

• Skrondal & Rabe-Hesketh have good introductory paper

Choosing a Computer Program

• Decision points– Distribution of dependent variable– Easy of use and compatibility– Single vs. multiple equation representation

• Major choices– SAS vs. Stata– Specialized programs (HLM, MLWin)

Thank you!

(All those identifying symptoms of MATHACH, please proceed to the closest Washington DC hospital)