© S. C. Strother, 2006 Managing and Optimizing fMRI Pipelines Stephen C. Strother, Ph.D. Rotman...
-
Upload
gabriella-hudson -
Category
Documents
-
view
215 -
download
0
Transcript of © S. C. Strother, 2006 Managing and Optimizing fMRI Pipelines Stephen C. Strother, Ph.D. Rotman...
© S. C. Strother, 2006© S. C. Strother, 2006
Managing and Optimizing
fMRI Pipelines
Stephen C. Strother, Ph.D.Rotman Research Institute, Baycrest Centre
http://www.rotman-baycrest.on.ca/rotmansite/home.php
& Medical Biophysics, University of Toronto
© S. C. Strother, 2006© S. C. Strother, 2006
OverviewOverview• Background
−data-driven statistics, pipelines and meta-models
• fMRI File Management: NIfTI and the DFWG• Why optimize pipeline meta-models;
−the Functional Imaging Analysis Contest (FIAC) experience?
• Seven meta-model optimization frameworks• Results with the 7th framework: NPAIRS• Data-analysis choices in pipeline meta-models:
−General linear model (GLM)−Canonical variates analysis (CVA)
• Pipeline-driven, between-subject heterogeneity• Recap: What have we learnt?
© S. C. Strother, 2006© S. C. Strother, 2006
The Statistician and the ScientistThe Statistician and the Scientist
When asked, the statistician says that he’d like to When asked, the statistician says that he’d like to give one give one last lecture on his theory of statistics. lecture on his theory of statistics.
When the scientist is asked, he says, “I’d like to be When the scientist is asked, he says, “I’d like to be shot first!”shot first!”
Rob Tibshirani @ S-Plus Users conference Oct., 1999. Rob Tibshirani @ S-Plus Users conference Oct., 1999.
http://www-stat.stanford.edu/~tibshttp://www-stat.stanford.edu/~tibs
A statistician and a scientist are going to be executed, and the executioner asks each for their last request.
© S. C. Strother, 2006© S. C. Strother, 2006
Why Bother with Data-Driven Statistics?Why Bother with Data-Driven Statistics?
Philosophy• “All models (pipelines) are wrong, but some are useful!”
− “All models are wrong.” G.E. Box (1976) quoted by Marks Nester in, “An applied statistician’s creed,” Applied Statistics, 45(4):401-410, 1996.
• A goal is to quantify and optimize utility!
• “I believe in ignorance-based methods because humans have a lot of ignorance and we should play to our strong suit.”
− Eric Lander, Whitehead Institute, M.I.T.
• Minimize the number of modeling assumptions and/or test multiple hypotheses, i.e., strong inference!
© S. C. Strother, 2006© S. C. Strother, 2006
What are the most, 2nd most etc., important steps?
We need more research across multiple data sets!
fMRI Pipelines and Meta-modelsfMRI Pipelines and Meta-modelsReconstructed
fMRI Data
B0 Correction
Slice TimingAdjustment
MotionCorrection
Non-LinearWarping
Spatial & TemporalFiltering
Statistical AnalysisEngine
StatisticalMaps
Some Preprocessing
Steps
ExperimentalDesignMatrix
Rendering of Results on Anatomy
Data Modeling/Analysis
Optimisation Metrics– ROCs– p-values– AIC, BIC– Replication– Prediction– NPAIRS
Why are we not using more modern analysis techniques?
Why are we still focused on null-hypothesis testing?
We need better tools and education!Automated Software
Frameworks
Fiswidget GUIFiswidget GUIXNAT, LONI, Fiswidgets
© S. C. Strother, 2006© S. C. Strother, 2006
OverviewOverview• Background
−data-driven statistics, pipelines and meta-models
• fMRI File Management: NIfTI and the DFWG• Why optimize pipeline meta-models;
−the FIAC experience?
• Seven optimization frameworks• Results with the 7th framework: NPAIRS• Data-analysis choices in pipeline meta-models:
−General linear model (GLM)−canonical variates analysis (CVA)
• Pipeline-driven, between-subject heterogeneity• What have we learnt?
© S. C. Strother, 2006© S. C. Strother, 2006
NIfTI-DFWG-NIfTI-1.1NIfTI-DFWG-NIfTI-1.1
© S. C. Strother, 2006© S. C. Strother, 2006
NIfTI-DFWG-NIfTI-1.1NIfTI-DFWG-NIfTI-1.1
© S. C. Strother, 2006© S. C. Strother, 2006
NIfTI-DFWG-NIfTI-1.1NIfTI-DFWG-NIfTI-1.1
Leading candidates for NIfTI-2 are:MINC-2.0, Multi-frame DICOMXCEDE XML schema
© S. C. Strother, 2006© S. C. Strother, 2006
OverviewOverview• Background
−data-driven statistics, pipelines and meta-models
• fMRI File Management: NIfTI and the DFWG• Why optimize pipeline meta-models;
−the FIAC experience?
• Seven optimization frameworks• Results with the 7th framework: NPAIRS• Data-analysis choices in pipeline meta-models:
−General linear model (GLM)−canonical variates analysis (CVA)
• Pipeline-driven, between-subject heterogeneity• Recap: What have we learnt?
© S. C. Strother, 2006© S. C. Strother, 2006
Why Optimize Pipeline Meta-models?Why Optimize Pipeline Meta-models?Practice New insights into human brain function may
be obscured by poor and/or limited choices in the data-processing pipeline!
We don’t understand the relative importance of meta-model choices.
“Neuroscientific plausibility” of results is used to justify the meta-model choices made.
Systematic bias towards prevailing neuroscientific expectations, and against new discoveries and/or testing multiple hypotheses.
© S. C. Strother, 2006© S. C. Strother, 2006
The Functional Image Analysis Competition 1 The Functional Image Analysis Competition 1
Examine the perisylvian Examine the perisylvian language network using a language network using a repetition-priming design with repetition-priming design with spoken sentencesspoken sentences
3T whole-body Bruker3T whole-body Bruker
T2-weighted EPI, TR=2.5s, 30 T2-weighted EPI, TR=2.5s, 30 x 4 mm slicesx 4 mm slices
Epochs of 20s ON & 9s OFFEpochs of 20s ON & 9s OFF
2 x 2 design for 4 conditions2 x 2 design for 4 conditions
4 epochs/condition x 2 runs4 epochs/condition x 2 runs
Same Same sentence, sentence, same same speakerspeaker
Same sentence, Same sentence, different speakerdifferent speaker
Different Different sentence, sentence, same same speakerspeaker
Different sentence, Different sentence, different speakerdifferent speaker
© S. C. Strother, 2006© S. C. Strother, 2006
The Functional Image Analysis Competition 2The Functional Image Analysis Competition 2
Poline JB, Strother SC, Dehaene-Lambertz G, Egan GF, Lancaster JL. Poline JB, Strother SC, Dehaene-Lambertz G, Egan GF, Lancaster JL. Motivation and synthesis of Motivation and synthesis of the FIAC experiment: The reproducibility of fMRI results across expert analyses.the FIAC experiment: The reproducibility of fMRI results across expert analyses. (in press, special (in press, special issue Hum Brain Mapp)issue Hum Brain Mapp)
© S. C. Strother, 2006© S. C. Strother, 2006
The Functional Image Analysis Competition 3The Functional Image Analysis Competition 3
Abstract:Abstract: “… the FIAC … helped identify new activation “… the FIAC … helped identify new activation regions in the test-base data, and …, regions in the test-base data, and …, it illustrates the it illustrates the significant methods-driven variability that potentially exists significant methods-driven variability that potentially exists in the literature.in the literature. Variable results from different methods Variable results from different methods reported here should provide a cautionary note, and reported here should provide a cautionary note, and motivate the Human Brain Mapping community to explore motivate the Human Brain Mapping community to explore more thoroughly the methodologies they use for analysing more thoroughly the methodologies they use for analysing fMRI data.”fMRI data.”
Poline JB, Strother SC, Dehaene-Lambertz G, Egan GF, Lancaster JL. Motivation and synthesis of the FIAC experiment: The reproducibility of fMRI results across expert analyses. (in press, special issue Hum Brain Mapp)
© S. C. Strother, 2006© S. C. Strother, 2006
The Functional Image Analysis Competition 4The Functional Image Analysis Competition 4z=-12 z=2 z=5
3
1,4
21
3 3 31
3
The main effects of sentence repetition (in red) and of speaker repetition (in blue). 1: Meriaux et al, Madic; 2: Goebel et al, Brain voyager; 3: Beckman et al, FSL; and 4: Dehaene-Lambertz et al, SPM2.
© S. C. Strother, 2006© S. C. Strother, 2006
OverviewOverview• Background
−data-driven statistics, pipelines and meta-models
• fMRI File Management: NIfTI and the DFWG• Why optimize pipeline meta-models;
−the FIAC experience?
• Seven optimization frameworks• Results with the 7th framework: NPAIRS• Data-analysis choices in pipeline meta-models:
−General linear model (GLM)−canonical variates analysis (CVA)
• Pipeline-driven, between-subject heterogeneity• Recap: What have we learnt?
© S. C. Strother, 2006© S. C. Strother, 2006
Optimization Metric FrameworksOptimization Metric Frameworks
Simulation & ROC curvesSimulation & ROC curves1.1. Skudlarski P., et al., Skudlarski P., et al., NeuroimageNeuroimage. 9(3):311‑329, 1999.. 9(3):311‑329, 1999.2.2. Della-Maggiore V., et al., Della-Maggiore V., et al., NeuroimageNeuroimage 17:19–28, 2002. 17:19–28, 2002.3.3. Lukic AS., et al., Lukic AS., et al., IEEE Symp. Biomedical ImagingIEEE Symp. Biomedical Imaging, 2004., 2004.4.4. Beckmann CF & Smith SM. Beckmann CF & Smith SM. IEEE Trans. Med. Img.IEEE Trans. Med. Img. 23:137-152, 2004. 23:137-152, 2004.
Data-Driven:Data-Driven:
1.1. GLM DiagnosticsGLM Diagnostics1.1. SPMd, SPMd, Luo W-L, Nichols T. Luo W-L, Nichols T. NeuroImageNeuroImage 19:1014-32, 2003 19:1014-32, 2003
2.2. Minimize p-valuesMinimize p-values1.1. Hopfinger JB, et al., Hopfinger JB, et al., Neuroimage,Neuroimage, 11:326-333, 2000. 11:326-333, 2000.
2.2. Tanabe J, et al. Tanabe J, et al. Neuroimage,Neuroimage, 15:902-907, 2002. 15:902-907, 2002.
3.3. Model Selection:Model Selection: Classical hypothesis testing, maximum likelihood, Classical hypothesis testing, maximum likelihood, Akaike’s information criterion (AIC), Minimum DescriptionLength, Bayesian Akaike’s information criterion (AIC), Minimum DescriptionLength, Bayesian Information Criterion (BIC) & Model Evidence, Information Criterion (BIC) & Model Evidence, Cross ValidationCross Validation
4.4. Replication/ReproducibilityReplication/Reproducibilitya.a. Empirical ROCsEmpirical ROCs – mixed multinomial model – mixed multinomial model
1.1. Genovese CR., et al., Genovese CR., et al., Magnetic Resonance in MedicineMagnetic Resonance in Medicine, 38:497–507, 1997., 38:497–507, 1997.2.2. Maitra, R., et al., Maitra, R., et al., Magnetic Resonance in Medicine, 48, Magnetic Resonance in Medicine, 48, 62 –70, 2002.62 –70, 2002.3.3. Liou M., et al., Liou M., et al., J. Cog. NeuroscienceJ. Cog. Neuroscience, 15:935-945, 2003., 15:935-945, 2003.
b.b. Empirical ROCsEmpirical ROCs – lower bound on ROC – lower bound on ROC1.1. Nandy RR & Cordes D. Nandy RR & Cordes D. Magnetic Resonance in MedicineMagnetic Resonance in Medicine 49:1152–1162, 2003. 49:1152–1162, 2003.
5.5. Prediction Error/AccuracyPrediction Error/Accuracy1.1. Kustra R & Strother SC. Kustra R & Strother SC. IEEE Trans Med ImgIEEE Trans Med Img 20:376-387, 2001. 20:376-387, 2001.2.2. Carlson, T.A., et al., Carlson, T.A., et al., J Cog NeuroscienceJ Cog Neuroscience, 15:704–717, 2003., 15:704–717, 2003.3.3. Hanson,S.J., et al., Hanson,S.J., et al., NeuroImageNeuroImage 23:156– 166, 2004 23:156– 166, 2004
6.6. NPAIRS: Prediction + ReproducibilityNPAIRS: Prediction + Reproducibility1.1. Strother SC, et. al., Strother SC, et. al., NeuroimageNeuroimage 15:747-771, 2002. 15:747-771, 2002.
2.2. Kjems U, et al., et al., Kjems U, et al., et al., NeuroimageNeuroimage 15:772-786, 2002. 15:772-786, 2002.
3.3. Shaw ME, et. al. Shaw ME, et. al. NeuroimageNeuroimage 19:988-1001, 2003. 19:988-1001, 2003.
4.4. LaConte S, et. al. LaConte S, et. al. NeuroimageNeuroimage 18:18:10-23, 2003.10-23, 2003.
5.5. Strother SC, et. al., Strother SC, et. al., Neuroimage Neuroimage 23S1:S196-S207, 2004.23S1:S196-S207, 2004.
6.6. LaConte S, et. al., LaConte S, et. al., NeuroimageNeuroimage 26:317-329, 2005 26:317-329, 2005
© S. C. Strother, 2006© S. C. Strother, 2006
Optimization via SimulationsOptimization via SimulationsReceiver Operating Characteristic (ROC) Curves
PA = P(True positive)
= P(Truly active voxel
is classified as active)
= Sensitivity
PI = P(False positive)
= P(Inactive voxel
is classified as active)
= False alarm rateSkudlarski P, Skudlarski P, NeuroimageNeuroimage. 9(3):311‑329, 1999.. 9(3):311‑329, 1999.Della-Maggiore V, Della-Maggiore V, NeuroimageNeuroimage 17:19–28, 2002. 17:19–28, 2002.Lukic AS, Lukic AS, IEEE Symp. Biomedical ImagingIEEE Symp. Biomedical Imaging, 2004., 2004.Beckmann CF, Smith SM. Beckmann CF, Smith SM. IEEE Trans. Med. Img.IEEE Trans. Med. Img. 23:137-152, 2004. 23:137-152, 2004.
pAUC
© S. C. Strother, 2006© S. C. Strother, 2006
Optimization via SimulationsOptimization via Simulations
© S. C. Strother, 2006© S. C. Strother, 2006
Optimization Framework 1 (SPMd)Optimization Framework 1 (SPMd)
Massively univariate testing of GLM assumptions and data exploration:
• Luo W-L, Nichols T. Diagnosis and exploration of massively univariate neuroimaging models. NeuroImage 19:1014-32, 2003
• Zhang H, Luo W-L, Nichols TE. Diagnosis of Single Subject & Group fMRI Data with SPMd. Hum Brain Mapp (in press, special FIAC issue)
Example: The impact of high-pass filtering in a phantom.
© S. C. Strother, 2006© S. C. Strother, 2006
Optimization Framework 1 (SPMd)Optimization Framework 1 (SPMd)
Lund TE, Madsen KH, Sidaros K, Luo W-L, Nichols TE. Non-white noise in fMRI: Does modelling have an impact? Neuroimage 29:54 – 66, 2006
© S. C. Strother, 2006© S. C. Strother, 2006
Optimization Framework 2Optimization Framework 2
Minimize p-values or maximize SPM values, e.g.,
• Hopfinger JB, Buchel C, Holmes AP, Friston KJ, A study of analysis parameters that influence the sensitivity of event related fMRI analyses, Neuroimage, 11:326-333, 2000.
• Tanabe J, Miller D, Tregellas J, Freedman R, Meyer FG. Comparison of detrending methods for optimal fMRI preprocessing. Neuroimage, 15:902-907, 2002.
Does not imply a stronger likelihood of getting the same result in another replication of the same experiment!
© S. C. Strother, 2006© S. C. Strother, 2006
Optimization Framework 3Optimization Framework 3Model Selection: An attempt to formulate some
traditional problems in the methodology of science in a rigorous way.
Standard methods: (Classical hypothesis testing, maximum likelihood, Akaike’s information criterion (AIC), Minimum DescriptionLength, Bayesian Information Criterion (BIC) & Model Evidence, Cross Validation) compensate for errors in the estimation of model parameters.
All tradeoff fit with simplicity (least # parameters), but give simplicity different weights.
All favor more complex (less simple) models with more data.
Forster MR. Key concepts in model selection: Performance and Generalizability. J Math Psych 44:205-231, 2000
© S. C. Strother, 2006© S. C. Strother, 2006
Quantifying replication/reproducibility because:
• replication is a fundamental criterion for a result to be considered scientific;
• smaller p values do not necessarily imply a stronger likelihood of repeating the result;
• for “good scientific practice” it is necessary, but not sufficient, to build a measure of replication into the experimental design and data analysis;
• results are data-driven and avoid simulations.
Optimization Framework 4Optimization Framework 4
© S. C. Strother, 2006© S. C. Strother, 2006
Optimization Framework 4aOptimization Framework 4aData-Driven, Empirical ROCs:Data-Driven, Empirical ROCs:
• Genovese CR, Noll DC, Eddy WF. Genovese CR, Noll DC, Eddy WF. Estimating test-retest reliability Estimating test-retest reliability in functional MR imaging. I. Statistical methodology.in functional MR imaging. I. Statistical methodology. Magnetic Magnetic Resonance in MedicineResonance in Medicine, 38:497–507, 1997., 38:497–507, 1997.
• Maitra, R., Roys, S. R., & Gullapalli, R. P. Maitra, R., Roys, S. R., & Gullapalli, R. P. Test–retest reliability Test–retest reliability estimation of functional MRI data.estimation of functional MRI data. Magnetic Resonance in Magnetic Resonance in Medicine, 48, Medicine, 48, 62 –70, 2002.62 –70, 2002.
• Liou M, Su H-R, Lee J-D, Cheng PE, Huang C-C, Tsai C-H. Liou M, Su H-R, Lee J-D, Cheng PE, Huang C-C, Tsai C-H. Bridging Functional MR Images and Scientific Inference: Bridging Functional MR Images and Scientific Inference: Reproducibility Maps.Reproducibility Maps. J. Cog. NeuroscienceJ. Cog. Neuroscience, 15:935-945, 2003., 15:935-945, 2003.
V VV V(M - R ) (M - R )R R
A A I IV
Mλ P 1- P + 1- λ P 1- P
R
© S. C. Strother, 2006© S. C. Strother, 2006
Optimization Framework 4bOptimization Framework 4b
Data-Driven, Empirical ROCs:• Nandy RR, Cordes D. Novel ROC-Type Method for Testing the
Efficiency of Multivariate Statistical Methods in fMRI. Magnetic Resonance in Medicine 49:1152–1162, 2003.
P(Y) = P(voxel identified as active) P(Y/F) = P(inactive voxel identified as active) P(Y) vs. P(Y/F) is a lower bound for true ROC
• Two runs: −standard experimental AND −resting-state for P(Y/F).
• Assumes common noise structure for accurate P(Y/F).
© S. C. Strother, 2006© S. C. Strother, 2006
Is Replication a Sufficient Metric?Is Replication a Sufficient Metric?
A silly data analysis approach produces the value 1.0/voxel regardless of the input data!
Results are perfectly replicable; • no variance;
• completely useless because they are severely biased!
Must consider such bias-variance tradeoffs when measuring pipeline performance.
© S. C. Strother, 2006© S. C. Strother, 2006
Optimization Framework 5Optimization Framework 5
Prediction/CrossvalidationResampling
Stone, M. Cross-validatory choice and assessment of statistical predictions. J. R. Stat. Soc. B 36: 111–147. 1974
Hastie T, Tibshirani R, Friedman J. Hastie T, Tibshirani R, Friedman J. The The elements of statistical learning theory.elements of statistical learning theory. Springer-Verlag, New York, 2001Springer-Verlag, New York, 2001
© S. C. Strother, 2006© S. C. Strother, 2006
Optimization Framework 5Optimization Framework 5Prediction/Crossvalidation Resampling Papers
Principal Component AnalysisL. K. Hansen, et al. Neuroimage, vol. 9, no. 5, pp. 534-44, 1999.Prediction via GLM, Split-Half ReproducibilityJ. V. Haxby, et al. Science, vol. 293, no. 5539, pp. 2425-30, 2001.Linear Discriminant Analysis/Canonical Variates AnalysisR. Kustra and S. Strother, IEEE Trans Med Imaging, vol. 20, no. 5, pp. 376-87, 2001.T. A. Carlson, et al. J Cogn Neurosci, vol. 15, no. 5, pp. 704-17, 2003.J. D. Haynes and G. Rees, Nat Neurosci, vol. 8, no. 5, pp. 686-91, 2005.Y. Kamitani and F. Tong, Nat Neurosci, vol. 8, no. 5, pp. 679-85, 2005.A.J. O'Toole, et al. J Cogn Neurosci, vol. 17, no. 4, pp. 580-90, 2005.Support Vector Machines (and LDA)D. Cox and R. L. Savoy, Neuroimage, vol. 19, no. 2 Pt 1, pp. 261-70, 2003.S. LaConte, et al. Neuroimage, vol. 26, no. 2, pp. 317-29, 2005.J. Mourao-Miranda, et al. Neuroimage, vol. 28, no. 4, pp. 980-95, 2005.Artificial Neural NetworksB. Lautrup, et al. in Proceedings of the Workshop on Supercomputing in Brain Research: From
Tomography to Neural Networks, H. J. Hermann, et al., Eds. Ulich, Germany: World Scientific, pp. 137-144, 1994.
N. Morch, et al. Lecture Notes in Computer Science 1230, J. Duncan and G. Gindi, Eds. New York: Springer-Verlag, pp. 259-270, 1997.
S. J. Hanson, et al. Neuroimage, vol. 23, no. 1, pp. 156-66, 2004.S. M. Polyn, et al. Science, vol. 310, no. 5756, pp. 1963-6, 2005.
© S. C. Strother, 2006© S. C. Strother, 2006
Optimization Framework 6: NPAIRSOptimization Framework 6: NPAIRS
NPAIRS Uses NPAIRS Uses “split-half” resampling to combine: resampling to combine:• Prediction & Reproducibility Metrics
• PCA-based reproducibility measures of:PCA-based reproducibility measures of:− uncorrelated signal and noise SPMs;uncorrelated signal and noise SPMs;− reproducible SPMs (rSPM) on a Z-score scale;reproducible SPMs (rSPM) on a Z-score scale;− multivariate dimensionality.multivariate dimensionality.
• Combined prediction and reproducibility metrics for:Combined prediction and reproducibility metrics for:− data-driven ROC-like curves;data-driven ROC-like curves;− optimizing bias-variance tradeoffs of pipeline interactions.optimizing bias-variance tradeoffs of pipeline interactions.
• Other Measures:Other Measures:− empirical random effects correction;empirical random effects correction;− measures of individual observation influence.measures of individual observation influence.
© S. C. Strother, 2006© S. C. Strother, 2006
NPAIRS Metrics in Functional NPAIRS Metrics in Functional Neuroimaging StudiesNeuroimaging Studies
PETPET Strother SC, et. al., Hum Brain Mapp, 5:312-316, 1997. Strother SC, et. al., Hum Brain Mapp, 5:312-316, 1997. Frutiger S, et. al., Neuroimage 12:515-527, 2000.Frutiger S, et. al., Neuroimage 12:515-527, 2000. Muley SA, et. al., Neuroimage 13:185-195, 2001.Muley SA, et. al., Neuroimage 13:185-195, 2001. Shaw ME, et. al., Neuroimage 15:661-674, 2002.Shaw ME, et. al., Neuroimage 15:661-674, 2002. Strother SC, et. al., Neuroimage 15:747-771, 2002.Strother SC, et. al., Neuroimage 15:747-771, 2002. Kjems U, et al., et al., Neuroimage 15:772-786, 2002.Kjems U, et al., et al., Neuroimage 15:772-786, 2002.
fMRIfMRI Tegeler C, et. al. Hum Brain Mapp, 7:267-283, 1999.Tegeler C, et. al. Hum Brain Mapp, 7:267-283, 1999. Shaw ME, et. al. Neuroimage Shaw ME, et. al. Neuroimage 19:988-1001, 2003., 2003. LaConte S, et. al. Neuroimage LaConte S, et. al. Neuroimage 18:10-23, 2003. Strother SC, et. al., Neuroimage 23S1:S196-S207, 2004. LaConte S, et. al., Neuroimage 26:317-329, 2005. Chen X, et. al., Hum Brain Mapp (in press, special FIAC Chen X, et. al., Hum Brain Mapp (in press, special FIAC
issue)issue)
© S. C. Strother, 2006© S. C. Strother, 2006
NPAIRS: Split-half reSampling for NPAIRS: Split-half reSampling for Activation-Pattern Reproducibility MetricsActivation-Pattern Reproducibility Metrics
1 1 1 11 r 1+r 02 2 2 2r 1 1 1 0 1-r 1 1
2 2 2 2
© S. C. Strother, 2006© S. C. Strother, 2006
NPAIRS Split-Half Prediction and NPAIRS Split-Half Prediction and Reproducibility ResamplingReproducibility Resampling
© S. C. Strother, 2006© S. C. Strother, 2006
OverviewOverview• Background
−data-driven statistics, pipelines and meta-models
• fMRI File Management: NIfTI and the DFWG• Why optimize pipeline meta-models;
−the FIAC experience?
• Seven optimization frameworks• Results with the 7th framework: NPAIRS• Data-analysis choices in pipeline meta-models:
−General linear model (GLM)−canonical variates analysis (CVA)
• Pipeline-driven, between-subject heterogeneity• Recap: What have we learnt?
© S. C. Strother, 2006© S. C. Strother, 2006
PCA of data matrix:
Canonical Variates Analysis (CVA):
• Design matrix (G) “brain states” = discriminant classes.−prediction metric = posterior probability of class membership.−maximizes a multivariate signal-to-noise ratio:
(between-class, B)/(pooled within-class, W) covariance;
A Multivariate Model for NPAIRSA Multivariate Model for NPAIRS
Tt
T 1/ 2 T 1/ 2 1t t( ) ( ) svd G G EG WE E B
t vt x v
svd X E SU
© S. C. Strother, 2006© S. C. Strother, 2006
Optimization of fMRI Static Force fMRI Optimization of fMRI Static Force fMRI Sixteen subjects with 2 runs/subject Sixteen subjects with 2 runs/subject Acquisition:
• Whole-brain, interleaved 1.5T BOLD-EPI;• 30 slices = 1 whole-brain scan;• 1 oblique slice = 3.44 x 3.44 x 5 mm3;• TR/TE = 4000 ms/70 ms
Experimental Design:
Analyzed with NPAIRS, GLM and PCA/CVA:• Dropped initial non-equilibrium and state-transition scans;Dropped initial non-equilibrium and state-transition scans;• 2-class single-subject;2-class single-subject;• 11-class 16-subject, group analysis;11-class 16-subject, group analysis;• NPAIRS/CVA, GLM-CVA comparison across preprocessing pipelines.NPAIRS/CVA, GLM-CVA comparison across preprocessing pipelines.
© S. C. Strother, 2006© S. C. Strother, 2006
Preprocessing for Static ForcePreprocessing for Static Force All runs/subject(s) passed initial quality control:All runs/subject(s) passed initial quality control:
• movement (AIR 5) < 1 voxel;movement (AIR 5) < 1 voxel;• no artifacts in functional or structural scans;no artifacts in functional or structural scans;• no obvious outliers in PCA of centered data matrix.no obvious outliers in PCA of centered data matrix.
Alignment (AIR 5):Alignment (AIR 5):• Within-Subject:Within-Subject: across runs to 1st retained scan of run one; across runs to 1st retained scan of run one;
• Between-Subject:Between-Subject: 1 1st (Affine)st (Affine), 3, 3rdrd, 5, 5thth and 7 and 7thth order polynomials; order polynomials;
• Tri-linear and sinc (AIR 05) interpolation.Tri-linear and sinc (AIR 05) interpolation.
Temporal Detrending using GLM Cosine Basis (SPM):Temporal Detrending using GLM Cosine Basis (SPM):• None, None, • 0.5, (0.5,1.0), (0.5-1.5), (0.5-2.0), (0.5-2.5), (0.5-3.0) cosines/run.0.5, (0.5,1.0), (0.5-1.5), (0.5-2.0), (0.5-2.5), (0.5-3.0) cosines/run.
− (0.5-1.5) includes three GLM columns with 0.5, 1.0 and 1.5 cosines/run(0.5-1.5) includes three GLM columns with 0.5, 1.0 and 1.5 cosines/run
Spatial Smoothing with 2D Gaussian:Spatial Smoothing with 2D Gaussian:• None;None;• FWHM = 1, 1.5, 2, 3, 4, 6, 8 pixels (3.44 mm)FWHM = 1, 1.5, 2, 3, 4, 6, 8 pixels (3.44 mm)
− FWHM = 1.5 voxels = 0.52 mm; FWHM = 6 voxels = 21 mm.FWHM = 1.5 voxels = 0.52 mm; FWHM = 6 voxels = 21 mm.
© S. C. Strother, 2006© S. C. Strother, 2006
ROC-Like: Prediction vs. Reproducibility ROC-Like: Prediction vs. Reproducibility 2-Class Static Force, Single Subject A Bias-Variance Tradeoff.
As model complexity increases (i.e., As model complexity increases (i.e., #PCs 10 →100), prediction of #PCs 10 →100), prediction of design matrix’s class labels design matrix’s class labels improves and reproducibility improves and reproducibility
(i.e., activation SNR) decreases.(i.e., activation SNR) decreases.
Optimizing Performance.Like an ROC plot there is a single Like an ROC plot there is a single point, (1, 1), on this prediction vs. point, (1, 1), on this prediction vs. reproducibility plot with the best reproducibility plot with the best performance; at this location the performance; at this location the model has perfectly predicted the model has perfectly predicted the design matrix while extracting an design matrix while extracting an infinite SNR.infinite SNR.
LaConte S, et. al. LaConte S, et. al. Evaluating preprocessing Evaluating preprocessing choices in single-subject BOLD-fMRI choices in single-subject BOLD-fMRI studies using data-driven performance studies using data-driven performance metricsmetrics. . Neuroimage Neuroimage 18:10-23, 2003
© S. C. Strother, 2006© S. C. Strother, 2006
Prediction, Reproducibility, Dimensionality and Prediction, Reproducibility, Dimensionality and Canonical Variates (1.5 cos)Canonical Variates (1.5 cos)
© S. C. Strother, 2006© S. C. Strother, 2006
Differences in Scanner SmoothnessDifferences in Scanner Smoothness
Courtesy Lee Friedman, UNM & Functional BIRNCourtesy Lee Friedman, UNM & Functional BIRN
© S. C. Strother, 2006© S. C. Strother, 2006
OverviewOverview• Background
−data-driven statistics, pipelines and meta-models
• fMRI File Management: NIfTI and the DFWG• Why optimize pipeline meta-models;
−the FIAC experience?
• Seven optimization frameworks• Results with the 7th framework: NPAIRS• Data-analysis choices in pipeline meta-models:
−General linear model (GLM)−canonical variates analysis (CVA)
• Pipeline-driven, between-subject heterogeneity• Recap: What have we learnt?
© S. C. Strother, 2006© S. C. Strother, 2006
Pipeline Meta-models: Data Analysis 1Pipeline Meta-models: Data Analysis 1 Bias-variance tradeoffs as a function of finite sample size are a
critical issue because:• Traditional, inferential, statistical parameter estimation is only asymptotically
unbiased & minimum variance.• Non-traditional estimation may = better signal detection!
− smaller parameter variance in non-asymptotic samples;− asymptotically-biased, often no asymptotic, inferential framework leading to resampling
techniques!
Resampling• Favour parameter estimation with Bootstrap over null hypothesis testing with
permutations!• Bootstrap’s advantage: it can be combined with cross-validation to
simultaneously obtain prediction and parameter estimates.• Boostrap’s disadvantage: requires iid samples; more restrictive than
permutations.
© S. C. Strother, 2006© S. C. Strother, 2006
Pipeline Meta-models: Data Analysis 2Pipeline Meta-models: Data Analysis 2 Part of science by Strong Inference:Part of science by Strong Inference:
• for a scientifically interesting observation enumerate all alternative for a scientifically interesting observation enumerate all alternative hypotheses that can account for the observation, based on present hypotheses that can account for the observation, based on present knowledgeknowledge
− Jewett DL. What’s wrong with a single hypothesis. The Scientist, 19(21):10, 2005Jewett DL. What’s wrong with a single hypothesis. The Scientist, 19(21):10, 2005− Platt JR. Strong inference. Science, 146:347-353, 1964Platt JR. Strong inference. Science, 146:347-353, 1964
Comparing univariate GLM versus multivariate CVA Comparing univariate GLM versus multivariate CVA data analysis is a simple means of implementing multi-data analysis is a simple means of implementing multi-hypothesis tests in neuroimaging:hypothesis tests in neuroimaging:• test localizationist versus network theories of brain function!test localizationist versus network theories of brain function!• account for differences in data-analysis sensitivity and specificity!account for differences in data-analysis sensitivity and specificity!• test different interactions with preprocessing pipeline choices!test different interactions with preprocessing pipeline choices!
© S. C. Strother, 2006© S. C. Strother, 2006
Simple Motor-Task Replication at 4.0TSimple Motor-Task Replication at 4.0T
t-test Fisher Linear Discriminant = 2-class CVA
L R
C. Tegeler, S. C. Strother, J. R. Anderson, and S. G. Kim, "Reproducibility of BOLD-based functional MRI obtained at 4 T," Hum Brain Mapp, vol. 7, no. 4, pp. 267-83, 1999.
© S. C. Strother, 2006© S. C. Strother, 2006
Testing Meta-Model Differences: Static ForceTesting Meta-Model Differences: Static Force
Apply 9 different pipelines to each run of 16 static-force subjects:
− 4 x NPAIRS.CVA (optimised detrending, smoothing, # PCs);− 3 x NPAIRS.GLM (same paramters as 3 x NPAIRS.CVA);− FSL3.2.GLM (high-pass filtering, prewhitening, default HRF)− SPM2.GLM (high-pass filtering, prewhitening, default HRF)
9 statistical parametric images (SPI) x 2 runs = 18 SPIs/subject. perform NPAIRS, splitting on subjects for 18 x 16 SPIs.
© S. C. Strother, 2006© S. C. Strother, 2006
Testing Pipeline Differences: Static ForceTesting Pipeline Differences: Static Force
© S. C. Strother, 2006© S. C. Strother, 2006
Group-Preprocessing InteractionsGroup-Preprocessing Interactions
© S. C. Strother, 2006© S. C. Strother, 2006
Group-Preprocessing InteractionsGroup-Preprocessing Interactions
© S. C. Strother, 2006© S. C. Strother, 2006
OverviewOverview• Background
−data-driven statistics, pipelines and meta-models
• fMRI File Management: NIfTI and the DFWG• Why optimize pipeline meta-models;
−the FIAC experience?
• Seven optimization frameworks• Results with the 7th framework: NPAIRS• Data-analysis choices in pipeline meta-models:
−General linear model (GLM)−Canonical variates analysis (CVA)
• Pipeline-driven, between-subject heterogeneity• Reacp: What have we learnt?
© S. C. Strother, 2006© S. C. Strother, 2006
Subject-Specific Pipeline OptimizationSubject-Specific Pipeline Optimization
Shaw ME, et. al., Neuroimage 19:988-1001, 2003
© S. C. Strother, 2006© S. C. Strother, 2006
Subject-Specific Pipeline OptimizationSubject-Specific Pipeline Optimization
Shaw ME, et. al., Neuroimage 19:988-1001, 2003
© S. C. Strother, 2006© S. C. Strother, 2006
RecapRecap• Background
−data-driven statistics, pipelines and meta-models
• fMRI File Management: NIfTI and the DFWG• Why optimize pipeline meta-models;
−the Functional Imaging Analysis Contest (FIAC) experience?
• Seven meta-model optimization frameworks• Results with the 7th framework: NPAIRS• Data-analysis choices in pipeline meta-models:
−General linear model (GLM)−Canonical variates analysis (CVA)
• Pipeline-driven, between-subject heterogeneity• Recap: What have we learnt?
© S. C. Strother, 2006© S. C. Strother, 2006
AcknowledgementsAcknowledgements
Rotman Research InstituteRotman Research InstituteXu Chen, Ph.D.Xu Chen, Ph.D.
Anita Oder, B.Sc.Anita Oder, B.Sc.
Wayne Lee, B.Eng.Wayne Lee, B.Eng.
Cheryl Grady, Ph.D.Cheryl Grady, Ph.D.
Randy McIntosh, Ph.D.Randy McIntosh, Ph.D.
Principal Funding Sources: NIH Human Brain Project, P20-EB02013-Principal Funding Sources: NIH Human Brain Project, P20-EB02013-10 & P20-MH072580-01.10 & P20-MH072580-01.
© S. C. Strother, 2006© S. C. Strother, 2006
AcknowledgementsAcknowledgementsUniversity of MinnesotaUniversity of Minnesota
International Neuroimaging Consortium & International Neuroimaging Consortium & VAMC: http://neurovia.umn.edu/incweb
Jon R. Anderson, M.Sc., Sally Frutiger, Ph.D., Kelly Rehm, Ph.D., David Rottenberg, M.D., Jon R. Anderson, M.Sc., Sally Frutiger, Ph.D., Kelly Rehm, Ph.D., David Rottenberg, M.D.,
Kirt Schaper, M.Sc., John Sidtis, Ph.D., Jane Zhang, Ph.D.Kirt Schaper, M.Sc., John Sidtis, Ph.D., Jane Zhang, Ph.D.
Seong-Ge Kim, Ph.D., Essa Yacob, Ph.D., Seong-Ge Kim, Ph.D., Essa Yacob, Ph.D., CMRR & Biomed. Eng.CMRR & Biomed. Eng.
James Ashe, M.D., Ph.D., James Ashe, M.D., Ph.D., Neurology & VAMCNeurology & VAMC
Suraj A. Muley, M.D., Suraj A. Muley, M.D., Neurology & VAMCNeurology & VAMC
Emory UniversityEmory University University of TorontoUniversity of Toronto
Xiaoping Hu, Ph.D.Xiaoping Hu, Ph.D. Rafal Kustra, Ph.D.Rafal Kustra, Ph.D.
Stephen LaConte, Ph.D.Stephen LaConte, Ph.D.
Technical University of DenmarkTechnical University of Denmark Melbourne UniversityMelbourne University
Lars Kai Hansen, Ph.D.Lars Kai Hansen, Ph.D. Gary Egan, Ph.D.Gary Egan, Ph.D.
Finn Arup Nielsen, Ph.D.Finn Arup Nielsen, Ph.D. Marnie Shaw, Ph.D.Marnie Shaw, Ph.D.
Principal Funding Sources: NIH Human Brain Project, P20-EB02013-10 & P20-MH072580-01.Principal Funding Sources: NIH Human Brain Project, P20-EB02013-10 & P20-MH072580-01.
© S. C. Strother, 2006© S. C. Strother, 2006
Consensus-Model ROC ResultsSimple Signal Complex Signal
Hansen LK, Nielsen FA, Strother SC, Lange N. Consensus Inference in Neuroimaging. Neuroimage 13:1212-1218, 2001
© S. C. Strother, 2006© S. C. Strother, 2006
NPAIRS-CVA Static Force Results: f(Preprocessing)NPAIRS-CVA Static Force Results: f(Preprocessing)