Brain Connectivity Analysis:from Unimodal to Multimodal

Sergey M. Plis

joint work with Rene Huster, Terran Lane, Vince Clark, Michael Weisend and Vince D. Calhoun

Contents

1 Definitions and Tools

2 Unimodal Connectivity Analysis: EEG

3 Contrasting modalities

4 Data Sharing Fusion

5 Validation through Interventions

1 / 27

Definitions and Tools

Outline

1 Definitions and Tools

2 Unimodal Connectivity Analysis: EEG

3 Contrasting modalities

4 Data Sharing Fusion

5 Validation through Interventions

1 / 27

Definitions and Tools

Introduction

Neuroimaging studies brain function

Advanced techniques produce immense amounts of data

Each with their strength and weaknesses

Our goal: causal relations among brain networks

2 / 27

Definitions and Tools

Functional Neuroimaging (fMRI)

functional Magnetic ResonanceImaging (fMRI)

Blood Oxygenation Level Dependent(BOLD) response

4D data (3D volumes evolving in time)

Advantage: Relatively well localizedDisadvantage: Slow sampling rate

3 / 27

Definitions and Tools

Functional Neuroimaging (MEG)

Magneto-EncephaloGraphy (MEG)

Electromagnetic phenomenonAdvantage: Instant reflection of theunderlying activity (ms resolution)Disadvantage: Uncertain spatiallocalization

4 / 27

Definitions and Tools

Common Underlying Phenomenon

Inverse problem

Functional connectivity

5 / 27

Definitions and Tools

Data Fusion: Source Analysis

6 / 27

Definitions and Tools

Single modality: Connectivity Analysis

MEG fMRI

7 / 27

Definitions and Tools

Connectivity Inference: Bayesian Networks

Pθ(X ) =

n∏

i=1

P(Xi |Pa(Xi); θ)

hippocam

pus

amygdala

hypothalamus

amygdala hypothalamus

hippocampus

child

parents

8 / 27

Definitions and Tools

Comparing the Results: Graph Characterization1

in-degree

out-degree

degree centrality

maximum degree

diameter

density

average path length

1Rubinov, M. et al. Neuroimage 52, 1059–1069 (2010).

9 / 27

Unimodal Connectivity Analysis: EEG

Outline

1 Definitions and Tools

2 Unimodal Connectivity Analysis: EEG

3 Contrasting modalities

4 Data Sharing Fusion

5 Validation through Interventions

9 / 27

Unimodal Connectivity Analysis: EEG

Stop-Go task

10 / 27

Unimodal Connectivity Analysis: EEG

What are the nodes?

11 / 27

Unimodal Connectivity Analysis: EEG

Contrasted sliding graphs metrics2

Higher clustering coefficient for the stop task:network consolidates for processing?

Shorter characteristic path-length for the stop task:network becomes more efficient?

2Grzegorczyk, M. et al. Machine Learning 71, 265–305 (2008).

12 / 27

Contrasting modalities

Outline

1 Definitions and Tools

2 Unimodal Connectivity Analysis: EEG

3 Contrasting modalities

4 Data Sharing Fusion

5 Validation through Interventions

12 / 27

Contrasting modalities Neuroimaging

Comparison Pipeline3

singleparadigmsubjects

+

MEG

fMRI

voxelcurrents

ROIaverages

ROIaverages

quantize structuresearch

analyzestructures

afnideconvolve

MNE projectto source space

average

compare

voxelHRFs

measured data trigger locked data ROI data discrete data BN graphs metric distributions

collect modalities: same subjects same paradigm

process modalities: place data in the same ROIs

pre-process data: align sampling rates and quantize

infer effective connectivity: Bayesian Nets

compare results: aggregate metric

3Plis, S. M. et al. Computers in Biology and Medicine 41, 1156–1165 (2011).

13 / 27

Contrasting modalities Neuroimaging

Comparison Pipeline3

singleparadigmsubjects

+

MEG

fMRI

voxelcurrents

ROIaverages

ROIaverages

quantize structuresearch

analyzestructures

afnideconvolve

MNE projectto source space

average

compare

voxelHRFs

measured data trigger locked data ROI data discrete data BN graphs metric distributions

collect modalities: same subjects same paradigm

process modalities: place data in the same ROIs

pre-process data: align sampling rates and quantize

infer effective connectivity: Bayesian Nets

compare results: aggregate metric

3Plis, S. M. et al. Computers in Biology and Medicine 41, 1156–1165 (2011).

13 / 27

Contrasting modalities Neuroimaging

Comparison Pipeline3

singleparadigmsubjects

+

MEG

fMRI

voxelcurrents

ROIaverages

ROIaverages

quantize structuresearch

analyzestructures

afnideconvolve

MNE projectto source space

average

compare

voxelHRFs

measured data trigger locked data ROI data discrete data BN graphs metric distributions

collect modalities: same subjects same paradigm

process modalities: place data in the same ROIs

pre-process data: align sampling rates and quantize

infer effective connectivity: Bayesian Nets

compare results: aggregate metric

3Plis, S. M. et al. Computers in Biology and Medicine 41, 1156–1165 (2011).

13 / 27

Contrasting modalities Neuroimaging

Comparison Pipeline3

singleparadigmsubjects

+

MEG

fMRI

voxelcurrents

ROIaverages

ROIaverages

quantize structuresearch

analyzestructures

afnideconvolve

MNE projectto source space

average

compare

voxelHRFs

measured data trigger locked data ROI data discrete data BN graphs metric distributions

collect modalities: same subjects same paradigm

process modalities: place data in the same ROIs

pre-process data: align sampling rates and quantize

infer effective connectivity: Bayesian Nets

compare results: aggregate metric

3Plis, S. M. et al. Computers in Biology and Medicine 41, 1156–1165 (2011).

13 / 27

Contrasting modalities Neuroimaging

Comparison Pipeline3

singleparadigmsubjects

+

MEG

fMRI

voxelcurrents

ROIaverages

ROIaverages

quantize structuresearch

analyzestructures

afnideconvolve

MNE projectto source space

average

compare

voxelHRFs

measured data trigger locked data ROI data discrete data BN graphs metric distributions

collect modalities: same subjects same paradigm

process modalities: place data in the same ROIs

pre-process data: align sampling rates and quantize

infer effective connectivity: Bayesian Nets

compare results: aggregate metric

3Plis, S. M. et al. Computers in Biology and Medicine 41, 1156–1165 (2011).

13 / 27

Contrasting modalities Neuroimaging

Processing

novel target

ME

GfM

RI

novel

targ

et

0

67

17

34

51

ROIs

0

67

17

34

51

ROIs

1 2 3 4 5 61 2 3 4 5 6

MEG fMRI

subjects

time (epochs) time (epochs)

quantization levels

14 / 27

Contrasting modalities Neuroimaging

Resulting ConnectivityR

OIs

novel

ME

G

target

RO

Is

ROIs ROIs

fMR

I

nove

ltarg

et

fMRI MEG

left hemisphere right hemisphere

marginal distributions of edges highest scoring networksin transverse view

15 / 27

Contrasting modalities Neuroimaging

Comparing the ResultsGraph Metrics Distributions

0.1

0.2

0.3

0 5 10 15 20 25

0.1

0.2

0.3

0 5 10 15 20 25

novel

# of children

MEG

pro

babi

lity

fMRI

pro

babi

lity

# of children

target

node-degree distribution

16 / 27

Contrasting modalities Neuroimaging

Comparing the ResultsGraph Metrics Distributions

0.1

0.2

0.3

0 5 10 15 20 25

0.1

0.2

0.3

0 5 10 15 20 25

novel

# of children

MEG

pro

babi

lity

fMRI

pro

babi

lity

# of children

target

noveltarget

max

degr

eecl

uste

ring

coeffi

cien

tav

erag

e pa

thle

ngth

MEG fMRI

graph metrics distributionsnode-degree distribution

16 / 27

Data Sharing Fusion

Outline

1 Definitions and Tools

2 Unimodal Connectivity Analysis: EEG

3 Contrasting modalities

4 Data Sharing Fusion

5 Validation through Interventions

16 / 27

Data Sharing Fusion

Why do fusion in dynamical settings?

temporal resolution affects causality4

fusion helps to avoid temporal inverse problem5

4Daunizeau, J. et al. Neuroimage 36, 69–87 (May 2007).

5Riera, J. J. et al. Neuroimage 21, 547–567 (Feb. 2004).

17 / 27

Data Sharing Fusion

Why do fusion in dynamical settings?

temporal resolution affects causality4

fusion helps to avoid temporal inverse problem5

4Daunizeau, J. et al. Neuroimage 36, 69–87 (May 2007).

5Riera, J. J. et al. Neuroimage 21, 547–567 (Feb. 2004).

17 / 27

Data Sharing Fusion Neuroimaging

Dynamic Bayesian Networks6

P(

Rt0:tTR,Mt0:tTR

,Bt0,tTR

)

=P(

Rt0

)

P(

Bt0 |Rt0

)

P(

BtTR|RtTR

)

TR∏

i=1

P(

Rti |Rti−1

)

TR∏

i=0

P(

Mti |Rti

)

m

0.065

0.032

-0.00021

-0.033

-0.066

m

0.081

0.053

0.026

-0.0019

-0.029

m

-0.089

-0.047

-0.005

0.037

0.079

circles - hidden

squares - observed6Murphy, K. PhD thesis (UC Berkeley, 2002).

18 / 27

Data Sharing Fusion Neuroimaging

Dynamic Bayesian Networks transition model

P(

Rt0:tTR,Mt0:tTR

,Bt0,tTR

)

=P(

Rt0

)

P(

Bt0 |Rt0

)

P(

BtTR|RtTR

)

TR∏

i=1

P(

Rti |Rti−1

)

TR∏

i=0

P(

Mti |Rti

)

Rt = kRt−1 + σRηt

circles - hidden

squares - observed

18 / 27

Data Sharing Fusion Neuroimaging

Dynamic Bayesian Networks MEG forward model6

P(

Rt0:tTR,Mt0:tTR

,Bt0,tTR

)

=P(

Rt0

)

P(

Bt0 |Rt0

)

P(

BtTR|RtTR

)

TR∏

i=1

P(

Rti |Rti−1

)

TR∏

i=0

P(

Mti |Rti

)

Mt = MFM(Rt) + σMηt ,

circles - hidden

squares - observed6Sarvas, J. eng. Phys Med Biol 32, 11–22 (1987).

18 / 27

Data Sharing Fusion Neuroimaging

Dynamic Bayesian Networks fMRI forward model6

P(

Rt0:tTR,Mt0:tTR

,Bt0,tTR

)

=P(

Rt0

)

P(

Bt0 |Rt0

)

P(

BtTR|RtTR

)

TR∏

i=1

P(

Rti |Rti−1

)

TR∏

i=0

P(

Mti |Rti

)

Bt = HFM(Rt) + σBηt

circles - hidden

squares - observed6Friston, K. J. et al. Neuroimage 12, 466–477 (2000).

18 / 27

Data Sharing Fusion Neuroimaging

Dynamic Bayesian Networks inference6

P(

Rt0:tTR,Mt0:tTR

,Bt0,tTR

)

=P(

Rt0

)

P(

Bt0 |Rt0

)

P(

BtTR|RtTR

)

TR∏

i=1

P(

Rti |Rti−1

)

TR∏

i=0

P(

Mti |Rti

)

Particle Filtering

circles - hidden

squares - observed

6(eds Doucet, A. et al.) (Springer-Verlag, Berlin, 2001).

18 / 27

Data Sharing Fusion Neuroimaging

Demonstration7

fMRI only MEG only fMRI+MEG

7Plis, S. M. et al. Frontiers in Neuroinformatics 4, 12 (2010).

19 / 27

Data Sharing Fusion Neuroimaging

Comparison: fMRI vs. fMRI+MEG

from sparseto constant activity

1000 runs per point

E =

1000∑

i=1

‖Ti − Mi‖2

‖Ti‖2

Event Related studies

fusion yields:

lower errors

stabler estimates

20 / 27

Data Sharing Fusion Neuroimaging

Comparison: fMRI vs. fMRI+MEG

from sparseto constant activity

1000 runs per point

E =

1000∑

i=1

‖Ti − Mi‖2

‖Ti‖2

Event Related studies

fusion yields:

lower errors

stabler estimates

20 / 27

Data Sharing Fusion Neuroimaging

Comparison: fMRI vs. fMRI+MEG

from sparseto constant activity

1000 runs per point

E =

1000∑

i=1

‖Ti − Mi‖2

‖Ti‖2

Event Related studies

fusion yields:

lower errors

stabler estimates

20 / 27

Data Sharing Fusion Neuroimaging

Comparison: fMRI vs. fMRI+MEG

from sparseto constant activity

1000 runs per point

E =

1000∑

i=1

‖Ti − Mi‖2

‖Ti‖2

Event Related studies

fusion yields:

lower errors

stabler estimates

20 / 27

Data Sharing Fusion Neuroimaging

Speed up and stability

BOLD (% of change)

21 / 27

Data Sharing Fusion Neuroimaging

Speed up and stability

BOLD (% of change) neural activity (a.u.)

fusion yields: ◦ lower variance and ◦ faster computation

21 / 27

Data Sharing Fusion Neuroimaging

Real data

same paradigm for fMRI and MEG

120 trials of an 8 Hz checkerboard reversal

MEG

1200 Hz

averaged

fMRI

interpolated to 1200 Hz

averaged

22 / 27

Data Sharing Fusion Neuroimaging

Real data results

fMRI only

23 / 27

Data Sharing Fusion Neuroimaging

Real data results

fMRI only MEG only

23 / 27

Data Sharing Fusion Neuroimaging

Real data results

fMRI only MEG only fMRI+MEG

23 / 27

Validation through Interventions

Outline

1 Definitions and Tools

2 Unimodal Connectivity Analysis: EEG

3 Contrasting modalities

4 Data Sharing Fusion

5 Validation through Interventions

23 / 27

Validation through Interventions

Does inferred connectivity reflect brain function?

Manipulation principle: learn by breaking parts of the system!

How to alter brain function without subjects complaining too loud?

Transcranial Direct Current Stimulation (tDCS): a noninvasive lowcurrent technique affecting firing thresholds of cortical neurons.

24 / 27

Validation through Interventions

Does inferred connectivity reflect brain function?

Manipulation principle: learn by breaking parts of the system!

How to alter brain function without subjects complaining too loud?

Transcranial Direct Current Stimulation (tDCS): a noninvasive lowcurrent technique affecting firing thresholds of cortical neurons.

24 / 27

Validation through Interventions

Does inferred connectivity reflect brain function?

Manipulation principle: learn by breaking parts of the system!

How to alter brain function without subjects complaining too loud?

Transcranial Direct Current Stimulation (tDCS): a noninvasive lowcurrent technique affecting firing thresholds of cortical neurons.

24 / 27

Validation through Interventions

Preliminary Results

0.1

0.2

0.3

0 5 10 15 20 25

0.1

0.2

0.3

0 5 10 15 20 25

novel

# of children

MEG

pro

babi

lity

fMRI

pro

babi

lity

# of children

target

25 / 27

Validation through Interventions

Preliminary Results

0.1

0.20.3

run1

pro

babili

ty(logscale

)

# of children

run2

# of children

run3

# of children

run4

# of children # of children

run5

0 5 10 15 20 25 30 35 40

# of children

0 5 10 15 20 25 30 35 40

# of children

0 5 10 15 20 25 30 35 40

# of children

0 5 10 15 20 25 30 35 40

# of children

distribution of edges in likely graphs (right hand stimulation)

0 5 10 15 20 25 30 35 40

0.1

0.20.3

# of children

pro

bability

(logscale

)sh

amfull

< 20 children

>= 20 children

25 / 27

Validation through Interventions

Summary

Goal: Combine modalities to infer function-induced networks

Results so far:Demonstrated useful results of sliding window treatmentDemonstrated pitfalls of single-modality connectivity estimation8

Demonstrated fMRI+MEG fusion in the DBN framework9

Future work:Causal structure fusionWhole brain DBN fusion frameworktDCS-based analysis framework for validation

8Plis, S. M. et al. Computers in Biology and Medicine 41, 1156–1165 (2011).

9Plis, S. M. et al. Frontiers in Neuroinformatics 4, 12 (2010).

26 / 27

Validation through Interventions

Summary

Goal: Combine modalities to infer function-induced networks

Results so far:Demonstrated useful results of sliding window treatmentDemonstrated pitfalls of single-modality connectivity estimation8

Demonstrated fMRI+MEG fusion in the DBN framework9

Future work:Causal structure fusionWhole brain DBN fusion frameworktDCS-based analysis framework for validation

8Plis, S. M. et al. Computers in Biology and Medicine 41, 1156–1165 (2011).

9Plis, S. M. et al. Frontiers in Neuroinformatics 4, 12 (2010).

26 / 27

Validation through Interventions

Summary

Goal: Combine modalities to infer function-induced networks

Results so far:Demonstrated useful results of sliding window treatmentDemonstrated pitfalls of single-modality connectivity estimation8

Demonstrated fMRI+MEG fusion in the DBN framework9

Future work:Causal structure fusionWhole brain DBN fusion frameworktDCS-based analysis framework for validation

8Plis, S. M. et al. Computers in Biology and Medicine 41, 1156–1165 (2011).

9Plis, S. M. et al. Frontiers in Neuroinformatics 4, 12 (2010).

26 / 27

Validation through Interventions

Summary

Goal: Combine modalities to infer function-induced networks

Results so far:Demonstrated useful results of sliding window treatmentDemonstrated pitfalls of single-modality connectivity estimation8

Demonstrated fMRI+MEG fusion in the DBN framework9

Future work:Causal structure fusionWhole brain DBN fusion frameworktDCS-based analysis framework for validation

8Plis, S. M. et al. Computers in Biology and Medicine 41, 1156–1165 (2011).

9Plis, S. M. et al. Frontiers in Neuroinformatics 4, 12 (2010).

26 / 27

Validation through Interventions

Summary

Goal: Combine modalities to infer function-induced networks

Results so far:Demonstrated useful results of sliding window treatmentDemonstrated pitfalls of single-modality connectivity estimation8

Demonstrated fMRI+MEG fusion in the DBN framework9

Future work:Causal structure fusionWhole brain DBN fusion frameworktDCS-based analysis framework for validation

8Plis, S. M. et al. Computers in Biology and Medicine 41, 1156–1165 (2011).

9Plis, S. M. et al. Frontiers in Neuroinformatics 4, 12 (2010).

26 / 27

Validation through Interventions

Summary

Goal: Combine modalities to infer function-induced networks

Results so far:Demonstrated useful results of sliding window treatmentDemonstrated pitfalls of single-modality connectivity estimation8

Demonstrated fMRI+MEG fusion in the DBN framework9

Future work:Causal structure fusionWhole brain DBN fusion frameworktDCS-based analysis framework for validation

8Plis, S. M. et al. Computers in Biology and Medicine 41, 1156–1165 (2011).

9Plis, S. M. et al. Frontiers in Neuroinformatics 4, 12 (2010).

26 / 27

Validation through Interventions

Summary

Goal: Combine modalities to infer function-induced networks

Results so far:Demonstrated useful results of sliding window treatmentDemonstrated pitfalls of single-modality connectivity estimation8

Demonstrated fMRI+MEG fusion in the DBN framework9

Future work:Causal structure fusionWhole brain DBN fusion frameworktDCS-based analysis framework for validation

8Plis, S. M. et al. Computers in Biology and Medicine 41, 1156–1165 (2011).

9Plis, S. M. et al. Frontiers in Neuroinformatics 4, 12 (2010).

26 / 27

Validation through Interventions

Summary

Goal: Combine modalities to infer function-induced networks

Results so far:Demonstrated useful results of sliding window treatmentDemonstrated pitfalls of single-modality connectivity estimation8

Demonstrated fMRI+MEG fusion in the DBN framework9

Future work:Causal structure fusionWhole brain DBN fusion frameworktDCS-based analysis framework for validation

8Plis, S. M. et al. Computers in Biology and Medicine 41, 1156–1165 (2011).

9Plis, S. M. et al. Frontiers in Neuroinformatics 4, 12 (2010).

26 / 27

Validation through Interventions

Summary

Goal: Combine modalities to infer function-induced networks

Results so far:Demonstrated useful results of sliding window treatmentDemonstrated pitfalls of single-modality connectivity estimation8

Demonstrated fMRI+MEG fusion in the DBN framework9

Future work:Causal structure fusionWhole brain DBN fusion frameworktDCS-based analysis framework for validation

8Plis, S. M. et al. Computers in Biology and Medicine 41, 1156–1165 (2011).

9Plis, S. M. et al. Frontiers in Neuroinformatics 4, 12 (2010).

26 / 27

Validation through Interventions

Thank you!

27 / 27