Post on 30-Dec-2015
Parallel and Distributed Computing for Neuroinformatics
Allen D. Malony
University of Oregon
ProfessorDepartment of Computerand Information Science
DirectorNeuroInformatics Center
Computational Science Institute
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
Outline Neuroinformatics
Dynamic brain analysis problem NeuroInformatics Center (NIC) at UO
Neuroinformatics research at the NIC Dense-array EEG analysis (APECS, HiPerSat, Mc) Brain image segmentation Computational head modeling Ontologies and tool integration (NEMO, GEMINI) Parallel and distributed computing emphasis
ICONIC Grid HPC system at UO Cerebral Data Systems Oregon E-science
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
Neuroscience and Neuroinformatics
Application of computer and information science to the understanding of brain organization and function
Integration of information across many levels Physical and functional Gene to behavior Microscopic to macroscopic scales
Challenges in brain observation and modeling Structure and organization (imaging) Operational and functional dynamics (temporal/spatial) Physical, functional, and cognitive operation (models)
How to create and maintain of integrated views of the brain for both scientific and clinical purposes?
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
Human Brain Dynamics Analysis Problem
Understand functional operation of the human cortex Dynamic cortex activation Link to sensory/motor and cognitive activities Multiple experimental paradigms and methods Multiple research, clinical, and medical domains
Need for coupled/integrated modeling and analysis Multi-modal observation (electromagnetic, MR, optical) Physical brain models and theoretical cognitive models
Need for robust tools Complex analysis of large multi-model data Reasoning and interpretation of brain behavior
Problem solving environment for brain analysis
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
Neuroimaging Technology
Alternative techniques for analyzing brain dynamics Blood flow neuroimaging (PET, fMRI)
Good spatial resolution allows functional brain mapping
Temporal limitations to tracking of dynamic activities Electromagnetic measures (EEG/ERP, MEG)
msec temporal resolution can distinguish fast neurological components
Spatial resolution sub-optimal requires a mapping to cortical sources
Need integrated neuroimaging technology (How?) Achieve both good spatial and temporal resolution
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
Integrated Dynamic Brain Analysis
IndividualBrain Analysis
Structural /FunctionalMRI/PET
DenseArray EEG /
MEG
ConstraintAnalysis
Head Analysis
Source Analysis
Signal Analysis
Response Analysis
Experimentsubject
temporaldynamics
neuralconstraints
CorticalActivity Model
ComponentResponse Model
spatial patternrecognition
temporal patternrecognition
Cortical ActivityKnowledge Base
Component ResponseKnowledge Base
good spatialpoor temporal
poor spatialgood temporal neuroimaging
integration
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
Experimental Methodology and Tool Integration
source localization constrained to cortical surface
processed EEG
BrainVoyager
BESA
CT / MRI
EEG segmentedtissues
16x256bits permillisec(30MB/m)
mesh generation
EMSEInterpolator 3D
NetStation
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
NeuroInformatics Center (NIC) at UO Application of computational science methods to
human neuroscience problems Tools to help understand dynamic brain function Tools to help diagnosis brain-related disorders HPC simulation, large-scale data analysis, visualization
Integration of neuroimaging methods and technology Need for coupled modeling (EEG/ERP, MR analysis) Apply advanced statistical signal analysis (PCA, ICA) Develop computational brain models (FDM, FEM) Build source localization models (dipole, linear inverse) Optimize temporal and spatial resolution
Internet-based capabilities for brain analysis services, data archiving, and data mining
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
NIC Organization
Allen D. Malony, Director Don M. Tucker, Associate Director Sergei Turovets, Computational Physicist Bob Frank, Mathematician Dan Keith, Software Engineer (distributed systems grid) Chris Hoge, Software Engineer (computational) Ryan Martin / Brad Davidson, Systems administrators Gwen Frishkoff, Research Associate, Univ. Pittsburgh Kai Li, Ph.D. student (brain segmentation) Adnan Salman, Ph.D. student (computational modeling) Performance Research Lab (my other hat)
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
Brainwave Research 101
Electroencephalogram (EEG)
Electrodes (sensors) measure uV EEG time series analysis
Event-related potentials (ERP) link brain activity to sensory–motor, cognitive functions statistical average to increase signal-to-noise ratio (SNR)
Signal cleaning and component decomposition Localize (map) to neural sources
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
Electrical Geodesics Inc. (EGI)
EGI Geodesics Sensor Net Dense-array sensor technology
64/128/256 channels 256-channel GSN
AgCl plastic electrodes
Net Station Advanced EEG/ERP data analysis
and visualization Stereotactic EEG sensor registration State of the art technology Research / clinical products EGI/CDS medical services
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
Epilepsy
Epilepsy affects more than 5 million people yearly U.S., Europe, and Japan
EEG in epilepsy diagnosis Childhood and juvenile absence Idiopathic (genetic) Distinguish different types
EEG in presurgical planning Localize seizure onset Fast, safe, inexpensive Dense array improves accuracy Requires good source modeling
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
EEG Time Series - Progression of Absence Seizure
First full spike–wave
Pre-spike“buzz’
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
Topographic Waveforms – First Full Spike-Wave
350ms interval
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
Topographic Mapping of Spike-Wave Progression
Palette scaled for wave-and-spike interval (~350ms)-130 uV (dark blue) 75 uV (dark red)
1 millisecond temporal resolution is required Spatial density (256) to capture shifts in topography
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
Dense-array EEG signal analysis and decomposition Artifact cleaning and component analysis
Automatic brain image segmentation Brain tissue identification Cortex extraction
Computational head modeling Tissue conductivity estimation Source localization
Statistical analysis to detect brain states Discriminant analysis Pattern recognition
Electromagnetic databases and ontologies
Neuroinformatic Challenges
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
Applying ICA for EEG Blink Removal
Component analysis is used to separate EEG signals Independent component analysis (ICA)
Blinks are a major source of noise in EEG data Blink signals are separable from cognitive responses
Raw EEGFormatting
EEGpreprocessing
Event infoTime markersBlink events
Bad channel removalBaseline correction…
ICA Analysis
ICA algorithms - Infomax - FastICA - …ICA components
Identify blinksand remove
Blink templatesReconstitute EEG w/out blink data
ERP Analysis
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
Independent Component Analysis
Mixedsinusoids- raw EEG
Originalsinusoids
ICA
EEG waveform
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
Tool for EEG Data Decomposition (APECS)
Automated Protocol for Electromagnetic Component Separation (APECS)
Motivation EEG data cleaning (increases signal-to-noise (SNR)) EEG component separation (addresses superposition) Data preprocessing prior to source localization
Distinctive Features Implements different decomposition methods Multiple metrics for component classification Quantitative and qualitative criteria for evaluation
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
APECS Evaluation: Qualitative Criteria
Bad GoodOriginal
Spatio-temporal blink profiles
Blink-freebaseline
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
APECS Evaluation: Quantitative Criteria
Covariance between “baseline” (blink-free) and ICA-filtered data: Infomax and FastICA.
Infomax gives consistently better results. FastICA results are more variable.
ICA decompositions most successful when only one spatial projector is strongly correlated with blink “template” (spatial filter).
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
Parallelization of Component Analysis Algorithms
Dense-array EEG increases analysis complexity Long time measurements require more processing ICA algorithms are computationally challenging
Processing time and memory requirements Increase performance through ICA parallelization High-Performance Signal Analysis Toolkit (HiPerSAT)
Matlab ICA algorithms implemented in C++ Infomax: Matlab (runica.m) parallel (OpenMP) FastICA: Matlab (fastica.m) parallel (MPI)
Validate results with Matlab standard algorithms Evaluate accuracy and compare speedup
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
HiPerSAT Parallel Infomax
Requires multi-threading Over 3 times
faster thanMatlab 3-fold increase
on 4 processors Speedup falls after
4 processors Limits on parallelization
of loop matrix operations May be able to improve
with larger blocking
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
HiPerSAT Parallel FastICA
Linear speedup Over 130 times faster
than Matlab 8-fold increase
on 32 processors Performance
may allow finerprocessing
Tradeoff improvedaccuracy versus morecomplex processing andexecution time
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
Matlab Tool Integration
Many neuroimaging tools are based on Matlab EEGLAB (UC San Diego) BrainStorm (USC / Los Alamos National Lab) SPM (University College of London)
Matlab is mostly a closed computational environment EEG/MEG analysis can overwhelm Matlab
Limited to workstation processing resources Memory requirements high due to Matlab workspace
Desire to use Matlab as a client in distributed systems Matlab is not multi-threaded Complicates building concurrent for external interfaces
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
Mc: Matlab Concurrent
Matlab is built on top of a JVM Use for GUI and graphics
We can leverage JVM for concurrency and interaction How do we create concurrent tasks in Matlab?
Matlab programming semantics issue Create task abstraction Provide Matlab package for constructing tasks
Concurrent tasks interface with runtime layer Client task manager runs tasks on servers and monitors Server task executor schedules tasks on resources
Voila! Mc
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
Mc System Architecture
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
Mc Task Manager
Single background thread running in the JVM Coordinate several components:
File transfer Ganymed SSH2 implementation for Java
Matlab interactionMatlabControl class
Scripting with Jython dynamic access to objects in the JVM Matlab m-scripting through MatlabControl class
waits for free cycles when execution Matlab code
Responds to user or timer events Manages execution of tasks and status reporting
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
Task Manager Workflow
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
HiPerSAT, APECS, and Mc
Demonstrated APECS running HiPerSAT tasks 20 simultaneous HiPerSAT servers on a cluster Compare with sequential EEGLAB processing
ICA processing on 20 1-GB EEG files Performance speedup
5x for C++ optimization and 2-way parallelism 10x speedup for 20 concurrent HiPerSAT tasks
Also demonstrate for multiple Matlab clients Recently applied to EEG processing for imagery
analysis 100+ simultaneous tasks across 8 different platforms
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
Papers K. Glass, G. Frishkoff, R. Frank, C. Davey, J. Dien, A. Malony,
A Framework for Evaluating ICA Methods of Artifact Removal from Multichannel EEG, ICA Conference, Grenada, Spain, 2004.
R. Frank and G. Frishkoff, APECS: A Framework for Implementation and Evaluation of Blink Extraction from Multichannel EEG, Journal of Clinical Neurophysiology, to appear, 2006.
D. Keith, C. Hoge, A. Malony, and R. Frank, “Parallel ICA Methods for EEG Neuroimaging,” International Parallel and Distributed Processing Symposium (IPDPS 2006), May 2006.
C. Hoge, D. Keith, and A. Malony, “Client-side Task Support in Matlab for Concurrent Distributed Execution,” Austrian-Hungarian Workshop on Distributed and Parallel Systems (DAPSYS), September 2006.
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
Spatial and temporal dynamicsare important to observetogether
Linked cortical networks Fronto-thalamic circuit (executive control) Limbic circuit (episodic memory)
Problem of Superposition How many sources? Where are they located?
Can only infer locations using“scalp space” information
Animated Topography of Spike–Wave Dynamics
QuickTime™ and aAnimation decompressor
are needed to see this picture.
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
Addressing Superposition: Brain Electrical Fields
Brain electrical fields are dipolar Volume conduction
Depth and location indeterminacy Highly resistive skull (CSF: skull est. from 1:40 to 1:80) Left-hemisphere scalp field may be generated by a
right-hemisphere source Multiple sources superposition
Radial source Tangential sources one and two sources varying
depths
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
Source Localization
Mapping of scalp potentials to cortical generators Signal decomposition (addressing superposition) Anatomical source modeling (localization)
Source modeling Anatomical Constraints
Accurate head model and physics Computational head model formulation
Mathematical Constraints Criteria (e.g., “smoothness”) to constrain solution
Current solutions limited by Simplistic geometry Assumptions of conductivities
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
Brain Sources of Epileptic Seizure
Single time point source solution Need to identify sources for each msec time sample Visualize dynamics in “source space”
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
Dipole Sources in the Cortex
Scalp EEG is generated in the cortex
Interested in dipole location, orientation, and magnitude Cortical sheet gives
possible dipole locations Orientation is normal to
cortical surface Need to capture convoluted
geometry in 3D mesh From segmented MRI/CT
Linear superposition
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
Advanced Image Segmentation
Native MR gives high gray-to-white matter contrast
Image analysis techniques Edge detection, edge
merger, region growing Level set methods and
hybrid methods Knowledge-based
After segmentation, color contrasts tissue type
Registered segmented MRI
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
Network Flow Based Skull Stripping
Graph construction with LoG Nodes, edges, edge weights,
source terminals, sink terminals Identify non-brain terminals
Scalp, eyeballs, orbits Fourth ventricle, pons
Identify brain terminals Knowledge-based terminal
detection Min-cut (Max-flow)
Separate source from sink Sum of weights is minimum
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
Cortical Surface Extraction Pipeline
Initial segmentation of white matter Scale-space based analysis of LoG Incorporation of spatial knowledge
Cortical surface topology correction Minimize cortical surface are increase Based on network flow theory Based on Cauchy-Crofton formula
Cortical surface geometry refinement White matter shape knowledge
radius function constraints sheet constraints
Medial surface (skeleton) manipulation
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
Topology Correction in Cortex Extraction
Before topology correction
After topology correction
Extracted Cortex
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
Building Computational Brain Models
MRI segmentation of brain tissues Conductivity model
Measure head tissue conductivity Electrical impedance tomography
small currents are injectedbetween electrode pair
resulting potential measuredat remaining electrodes
Finite element forward solution Source inverse modeling
Explicit and implicit methods Bayesian methodology
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
Conductivity Modeling
Governing Equations ICS/BCS
Discretization
System of Algebraic Equations
Equation (Matrix) Solver
Approximate Solution
Continuous Solutions
Finite-DifferenceFinite-Element
Boundary-ElementFinite-Volume
Spectral
Discrete Nodal Values
TridiagonalADISOR
Gauss-SeidelGaussian elimination
(x,y,z,t)J (x,y,z,t)B (x,y,z,t)
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
Conductivity Optimization and Parallelization
Design as a conductivity search problem Master launches new inverse problems with guesses Inverse solvers run iterative forward calculations Compare solutions with measured results
Parallelization approach Forward solver
Alterative Direction Implicit (ADI) method Optimization search
Simplex or simulated annealing (Monte Carlo) Hybrid
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
Alternating Direction Implicit (ADI) Method
Finite difference method C++ with OpenMP for parallelization LAPACK for matrix operations
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
Conductivity Search Architecture
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
Conductivity Optimization Dynamics
Attempting todetermine tissueconductivity values Scalp Skull Brain CSF
Simplex algorithm Values converge
to minimal error Increase number of features in future work
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
Multi-Cluster Search Dynamics
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
Papers K. Li, A. Malony, and D. Tucker, “Automatic Brain MR Image
Segmentation with Relative Thresholding and Morphological Image Analysis,” International Conference on Computer Vision Theory and Applications (VISAPP), Setúbal, Portugal, February 2006.
K. Ki, A. Malony, and D. Tucker, “A Multiscale Morphological Approach to Topology Correction of Cortical Surfaces,” International Workshop on Medical Imaging and Augmented Reality (MIAR 2006), Sanghai, China, August 2006.
A. Salman, S. Turovets, A. Malony, J. Eriksen, and D. Tucker, “Computational Modeling of Human Head Conductivity,” International Conference on Computational Science (ICCS), best paper, May 2005.
A. Salman, S. Turovets, A. Malony, V. Vasilov, “Multi-Cluster, Mixed-Mode Computational Modeling of Human Head Conductivity,” International Workshop on OpenMP (IWOMP), June 2005.
S. Turovets, A. Salman, A. Malony, P. Poolman, C. Davey, D. Tucker, “Anatomically Constrained Conductivity Estimation of the Human Head in Vivo: Computational Procedure and Preliminary Experiments,” Electrical Impedance Tomography (EIT), July 2006.
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
Neural ElectroMagnetic Ontology (NEMO) How can brain electromagnetic (EEG and MEG) data be
compared and integrated across experiments and laboratories? Need a system for representation, storage, mining, dissemination Need standardization of methods for measure generation
Identification and labeling of components Patterns of interest
General agreement on criteria for component identification Patterns can be hard to identify Variability in techniques for measure generation
NEMO will address issue by providing Spatial and temporal ontology database Use for large-scale data representation, mining, and meta-analysis Components in average EEG and MEG (ERPs)
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
Electromagnetic Data Spaces and Representation
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
NEMO Architecture Composed of three modules
Database mining Inference engine Query (user) interface
Methods for measure generation Spatial ontologies and temporal ontologies Cognitive functional mapping
User interactions Query formulation Mapping-rule definitions
Scalable integration system Online repository for storing metadata
Spatio-temporal ontologies, database schema, mappings
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
NEMO System
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
ICONIC Grid
SMPServerIBM p655
GraphicsSMP
SGI Prism
Shared Storage System
Gbit Campus Backbone
NIC CIS CIS
Internet 2
SharedMemory
IBM p690
DistributedMemory
IBM JS20
CNI
DistributedMemory
Dell Pentium Xeon
NIC4x8 16 16 2x8 2x16
graphics workstations interactive, immersive viz other campus clusters
40 TerabytesTape
Backup112 totalprocessors
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
Computational Integrated Neuroimaging System
… …
raw
storageresources
virtualservices
compute resources
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
CIS Computational and Visualization Laboratory Support interdisciplinary computer science
Informatics Computational science
Resource development Phase 1 (complete)
NSF MRI grant ($1M) ICONIC Grid
Phase II (underway) Visualization Lab ($100K)
rear projection» 3D stereo and 2x2 tiled
3x4 tiled 24” LCD display
Phase III …
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
GEMINI Project
Grid-enabled Methods for Integrated NeuroImaging Dynamic neuroimaging algorithms and visualization Grid-based integration (processing and data sharing) High-end tool integration and environments Neuroinformatics data ontologies GEMINI is
A distributable set of programs and configuration files A set of web services and WSRF-accessible resources An internet-accessible service executing on systems Based on open technologies
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
GEMINI Model
Zones GEMINI resource
servers Tasks
Job representation Agents
Execution context Build on endpoint
references Supports workflow
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
GEMINI Architecture
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
Leveraging Internet, HPC, and Grid Computing
Telemedicine imaging and neurology Distributed EEG and MRI measurement and analysis Neurological medical services Shared brain data repositories Remote and rural imaging capabilities
Build on emerging web services and grid technology Leverage HPC compute and data centers Create institutional and industry partnerships
Electrical Geodesics, Inc. Cerebral Data Systems (UO partnership with EGI) Other industrial partnerships
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
Cerebral Data Systems Partnership between EGI and University of Oregon Develop and market neuroinformatics services
Neurological medical data transfer, storage, and analysis High-performance, expert EEG and MR analysis Telemedicine and distributed collaboration Shared brain repositories
Target markets Research and clinical Epilepsy diagnosis and pre-surgical planning MR image segmentation
Technology integration Internet, computional grids, and HPC
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
CDS Computational Server and Imaging Clients
University of Linz September 2006Parallel and Distributed Computing for Neuroinformatics
Oregon E-Science Grid
Region 4
Region 1Region 2
Region 3
Region 5
Internet 2 /National LambdaRailRegional networks
Institutional partnersRegional points of presence
PSI