Biomedical Informatics Research Network Pipelines & Processing: Tools & Toolkits David Rex, John...
-
Upload
matthew-collins -
Category
Documents
-
view
213 -
download
0
Transcript of Biomedical Informatics Research Network Pipelines & Processing: Tools & Toolkits David Rex, John...
Biomedical InformaticsResearch Network
Pipelines & Processing: Tools & Toolkits
David Rex, John Moreland
October 9, 2002
2nd Annual All Hands Meeting
Other Pipelines
Interactive Data/Visualization Pipelines
Data flow graph
Nodegroups
Nodes
Image viewers
Control panel
Other Pipelines - OpenDX
(4) Colorize(4) Colorize
(2) Smooth(2) Smooth(1) View(1) View
(3) Filter(3) Filter
Other Pipelines - Visualization
Utah SCIrun AVS Khoros
Kyma IRIS Explorer
Scalable Data/Visualization Toolkits
Data flow/pipelining needs good user interface tools• Server-side/batch pipelines (eg: LONI Pipeline)• Client-side/interactive pipelines (eg: OpenDX, AVS, etc)
Pipelines need command-line tools and/or plug-in modules
Tools and modules will need reusable toolkits/APIs to support common functionality
Reusable Components
API API
Tools
API
Modules
Motivation
CT
Cryosection
Classification
Support analysis, filtering, and compositing• Larger-than-core (and swap) data sets• Multi-modal and time-varying data• Multiple data sets simultaneously
And...• Do efficient data movement• Execute well on parallel architectures• Integrate easily w/existing applications & toolkits
Support BIRN project applications
Application
Data Grid Toolkit
Data Management
File Format Handling
SRB, ADR, etc.
Mesh Toolkit
Expression Tree Toolkit
Layered Toolkit Architecture
Manage an N-space data gridManage an N-space data grid
Cache pages for lazy I/OCache pages for lazy I/O
Support specific file formatsSupport specific file formats
Manage file storageManage file storage
Bind a coord. system to dataBind a coord. system to data
Orchestrate filter executionOrchestrate filter execution
Managing Data Grids
Manage a paged data grid (array-like)• An N-dimensional grid of cells• Spatial data & time-series• Arbitrary cell data content
Handle larger-than-core data• Transparently pages data in/out• Support from ADR & DataCutter• Compressed data (disk & memory)
Application
Data Grid Toolkit
Data Management
File Format Handling
SRB, ADR, etc.
Mesh Toolkit
Expression Tree Toolkit
Data Grid Toolkit
Random access (slow)• Get/set cells in any order
Structured access (faster)• Get/set cells in a pre-defined order
Data-order access (fastest)• Get/set cells in the data’s storage order
Pre-fetching Intelligently
5 13 2467
51 32 4 6 7 8 9
9 8
51 32 4 6 7 8 9
Application
Data Grid Toolkit
Data Management
File Format Handling
SRB, ADR, etc.
Mesh Toolkit
Expression Tree Toolkit
Data Grid Toolkit
Paging Intelligently
Neighborhood-aware paging• Page in nearby cells in N dimensions• Support convolution filtering, rendering, marching-
cubes, ...
0 1 2 3 4
5 6 7 8 9
10 11 12 13 14
15 16 17 18 19
20 21 22 23 24
Current center cellCurrent center cell
Keep neighboring cells Keep neighboring cells paged-in as wellpaged-in as well
Application
Data Grid Toolkit
Data Management
File Format Handling
SRB, ADR, etc.
Mesh Toolkit
Expression Tree Toolkit
Data Grid Toolkit
Filter windowFilter window
Using Coordinate Systems
Bind a coordinate system to a data grid• Euclidean, cylindrical, spherical, time-series, ...• Uniform, structured, unstructured
Handle coordinate system-based operations• Resampling with interpolation• Lazy-evaluation
Multiple file format handlers
Application
Data Grid Toolkit
Data Management
File Format Handling
SRB,ADR, etc.
Mesh Toolkit
Expression Tree Toolkit
Mesh Toolkit
Operating on Data
Define an expression tree for data operations• Leaf nodes are data sets, functions, ...• Interior nodes are composite, filter, ...• Transforms align overlapping data sets
Execute it to generate samples• Client defines the expression• Server on big iron executes it
Application
Data Grid Toolkit
Data Management
File Format Handling
SRB, ADR, etc.
Mesh Toolkit
Expression Tree Toolkit
Expression Tree Toolkit
ClientClient
ServerServer
Operating on Expressions
Expressions can be optimized• Re-order operators• Similar to optimizing compilers & databases
Sample order can be optimized• Re-order data accesses for better cache efficiency
Data can be staged & intermediate results cached
Application
Data Grid Toolkit
Data Management
File Format Handling
SRB, ADR, etc.
Mesh Toolkit
Expression Tree Toolkit
Expression Tree Toolkit
Combining Brain Data Sets
RGB to HSIScalar to RGB
Mask by Hue
ScalarScalarCT-scanCT-scan
Color Color CryosectionCryosection
Color Color SegmentationSegmentation
Extract Hue
Composite
512 x 512 x 230512 x 512 x 230 547 x 710 x 672547 x 710 x 672 547 x 710 x 672547 x 710 x 672
Application
Data Grid Toolkit
Data Management
File Format Handling
SRB, ADR, etc.
Mesh Toolkit
Expression Tree Toolkit
Expression Tree Toolkit
Combining Brain Data Sets
Cryosection
Application
Data Grid Toolkit
Data Management
File Format Handling
SRB, ADR, etc.
Mesh Toolkit
Expression Tree Toolkit
Expression Tree Toolkit
CT
Composited
And more toolkits...
Interactive imaging with...• Mitsubishi VolumePro cards• Point clouds & 3D texture mapping with graphics pipelines
High-quality imaging with VISTA...
Application
Data Grid Toolkit
Data Management
File Format Handling
SRB, ADR, etc.
Mesh Toolkit
Expression Tree Toolkit
Other Toolkits
VolumePro
PointCloud VISTA
3DTexture
Data-Visualization Pipeline
Get data from disk efficiently
Get data from disk efficiently
Manage data in memory efficiently
Manage data in memory efficiently
Compute on data efficiently
Compute on data efficiently
Visualize data efficiently
Visualize data efficiently
Computation Visualization
Data
SRB ServerSRB Server
MCAT (Metadata)
MCAT (Metadata)
ADR DataCutterADR DataCutter SRB ServerSRB Server
KeLP FloorPlanKeLP FloorPlan
Data
Data Orchestration
. . .
Data-Visualization Pipeline
Get data from disk efficiently
Get data from disk efficiently
Manage data in memory efficiently
Manage data in memory efficiently
Compute on data efficiently
Compute on data efficiently
Visualize data efficiently
Visualize data efficiently
Computation Visualization
Data
Data - Vis Toolkits
Data - Vis Toolkits
Interaction ToolsInteraction Tools
VISTA RendererVISTA Renderer
Data Orchestration
. . .