Towards a Digital Human Kathy Yelick EECS Department U.C. Berkeley.
Support for Adaptive Computations Applied to Simulation of Fluids in Biological Systems Kathy Yelick...
-
date post
19-Dec-2015 -
Category
Documents
-
view
217 -
download
2
Transcript of Support for Adaptive Computations Applied to Simulation of Fluids in Biological Systems Kathy Yelick...
![Page 1: Support for Adaptive Computations Applied to Simulation of Fluids in Biological Systems Kathy Yelick U.C. Berkeley.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d3e5503460f94a17ac8/html5/thumbnails/1.jpg)
Support for Adaptive Computations Applied to
Simulation of Fluids in Biological Systems
Kathy YelickU.C. Berkeley
![Page 2: Support for Adaptive Computations Applied to Simulation of Fluids in Biological Systems Kathy Yelick U.C. Berkeley.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d3e5503460f94a17ac8/html5/thumbnails/2.jpg)
Project Summary
• Provide easy-to-use, high performance tool for simulation of fluid flow in biological systems.– Using the Immersed Boundary Method
• Enable simulations on large-scale parallel machines. – Distributed memory machine including SMP
clusters
• Using Titanium, ADR, and KeLP with AMR• Specific demonstration problem: Simulation
of the heart model on Blue Horizon.
![Page 3: Support for Adaptive Computations Applied to Simulation of Fluids in Biological Systems Kathy Yelick U.C. Berkeley.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d3e5503460f94a17ac8/html5/thumbnails/3.jpg)
Outline
• Generic Immersed Boundary Method in Titanium (IBT)– Titanium status – Immersed Boundary method status
• Applications of the method– Heart model– Cochlea model
• Future Plans– Data analysis– Solvers
![Page 4: Support for Adaptive Computations Applied to Simulation of Fluids in Biological Systems Kathy Yelick U.C. Berkeley.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d3e5503460f94a17ac8/html5/thumbnails/4.jpg)
Titanium on Blue Horizon• Recent improvements:
– Support for PAPI (performance analysis)– Cache optimizations– Portable runtime layer (maintainable)– Faster LAPI-based implementation– Adapted to OS upgrade, ongoing issues
• Gcc does not on recent AIX’s• IBM C++ does not fully support templates
• Plans– Communication optimizations– Common runtime with UPC (possibly
CAF)
![Page 5: Support for Adaptive Computations Applied to Simulation of Fluids in Biological Systems Kathy Yelick U.C. Berkeley.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d3e5503460f94a17ac8/html5/thumbnails/5.jpg)
MPI vs. LAPI on the IBM SP
0
50
100
150
200
250
300
350
400
1 2 4 8
16
32
64
12
8
25
6
51
2
10
24
20
48
40
96
81
92
16
38
4
32
76
8
65
53
6
13
10
72
MB
/s IBM/MPI
IBM/LAPI
• LAPI bandwidth higher than MPI• Also better small-message overhead
– 9usec vs. 11usec
• Latest Titanium release leverages this
![Page 6: Support for Adaptive Computations Applied to Simulation of Fluids in Biological Systems Kathy Yelick U.C. Berkeley.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d3e5503460f94a17ac8/html5/thumbnails/6.jpg)
Immersed Boundary Method Structure
Fiber activation & force calculation
InterpolateVelocity
Navier-Stokes Solver
SpreadForce
4 steps in each timestep
Fiber Points
Interaction
Fluid Lattice
![Page 7: Support for Adaptive Computations Applied to Simulation of Fluids in Biological Systems Kathy Yelick U.C. Berkeley.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d3e5503460f94a17ac8/html5/thumbnails/7.jpg)
Immersed Boundary Method
• Recent Performance Improvements– Use of FFTW in Spectral solver
• 10x performance improvement on t3e• Use on BH still pending
– Use of scatter/gather communication• Copying bounding boxes is still faster• Depends on application and machine
– Load balancing• Alignment of fluid grid (in slabs) and fiber• Multigrid solver might offer more
possibilities
![Page 8: Support for Adaptive Computations Applied to Simulation of Fluids in Biological Systems Kathy Yelick U.C. Berkeley.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d3e5503460f94a17ac8/html5/thumbnails/8.jpg)
Load Balancing
Egg slicer Pizza cutter
0
50
100
150
200
250
300
350
400
450
0 5 10 15 20No. of processors.
MF
LO
PS
SpreadForce(Pizzacutter)
InterpolateVelocity(Pizzacutter)SpreadForce(Eggslicer)
InterpolateVelocity(Eggslicer)
Fluid grid is divided in slabs for 3D FFT
![Page 9: Support for Adaptive Computations Applied to Simulation of Fluids in Biological Systems Kathy Yelick U.C. Berkeley.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d3e5503460f94a17ac8/html5/thumbnails/9.jpg)
Application: Heart Simulation
05000
1000015000200002500030000350004000045000
Fiber Spreadforce
NS Solver Interpolatevelocity
Before
After
• Performance improvements over the last year
-64 node t3e ~= 2 node C90 ~= 1-node (8p) BH (probably)
Heart simulation on a Cray T3E
![Page 10: Support for Adaptive Computations Applied to Simulation of Fluids in Biological Systems Kathy Yelick U.C. Berkeley.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d3e5503460f94a17ac8/html5/thumbnails/10.jpg)
Heart Simulation
• Recent improvements– Support for heart input– Generate data for NYU visualization– Visualization is now OpenGL (ongoing)– Runs on
• Blue Horizon, NERSC SP, T3E, SGI, Millennium
• Short term plans– Finish checkpoint/restart– Larger runs– Complete performance model– Validation
![Page 11: Support for Adaptive Computations Applied to Simulation of Fluids in Biological Systems Kathy Yelick U.C. Berkeley.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d3e5503460f94a17ac8/html5/thumbnails/11.jpg)
Cochlea Model
• Model of the inner ear– Developed by Julian Bunn and Ed Givelberg
• Contains new features, e.g. membranes
• Implemented one of these last fall• Plan to have Givelberg here next year
![Page 12: Support for Adaptive Computations Applied to Simulation of Fluids in Biological Systems Kathy Yelick U.C. Berkeley.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d3e5503460f94a17ac8/html5/thumbnails/12.jpg)
Alpha Project Plans
• Support for new applications– Berkeley and Michigan
• Scaling– Berkeley
• Solvers– San Diego, Berkeley, LBNL
• Data analysis and model construction– Ohio, Maryland
![Page 13: Support for Adaptive Computations Applied to Simulation of Fluids in Biological Systems Kathy Yelick U.C. Berkeley.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d3e5503460f94a17ac8/html5/thumbnails/13.jpg)
Scallop: Multigrid Poisson Solver
• A latency tolerant elliptical solver library– Will be used to build Navier-Stokes Solver– Implemented in KeLP, with a simple interface
• Work by Scott Baden and Greg Balls– Based on Balls/Colella algorithm– 2D implementation in both KeLP and Titanium
• 3D Solver – Algorithm is complete– Implementation running, but performance
tuning is ongoing– Interface between Titanium and KeLP
developed
![Page 14: Support for Adaptive Computations Applied to Simulation of Fluids in Biological Systems Kathy Yelick U.C. Berkeley.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d3e5503460f94a17ac8/html5/thumbnails/14.jpg)
Elliptical solvers
• A finite-difference based solvers– Good for regular, block-structured domains
• Method of Local Corrections– Local solutions corrected by a coarse solution– Good accuracy, well-conditioned solutions
• Limited communication– Once to generate coarse grid values– Once to correct local solutions– Trades off extra computation for fewer
messages
![Page 15: Support for Adaptive Computations Applied to Simulation of Fluids in Biological Systems Kathy Yelick U.C. Berkeley.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d3e5503460f94a17ac8/html5/thumbnails/15.jpg)
KeLP implementation
• Advantages – abstractions available in C++– built in domain calculus– communication management– numerical kernels written in Fortran
• Simple interface– callable from other languages– no KeLP required in user code
![Page 16: Support for Adaptive Computations Applied to Simulation of Fluids in Biological Systems Kathy Yelick U.C. Berkeley.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d3e5503460f94a17ac8/html5/thumbnails/16.jpg)
Improved Heart Structure Model
• Current model is– Based on dog heart, textbook anatomy– Approximation by composing cones
• Building a more accurate model– Use modern imaging on human heart for model– Need to see individual fibers– Collaboration between
• Joel Saltz’s group and • Dr. Robert DePhilip in Anatomy Division of Biomedical
Informatics Dept. at Ohio State University
• Long term goal– Specialize model to patient using MRI data
![Page 17: Support for Adaptive Computations Applied to Simulation of Fluids in Biological Systems Kathy Yelick U.C. Berkeley.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d3e5503460f94a17ac8/html5/thumbnails/17.jpg)
Analysis of Cardiac Simulations
• Methods and tools to analyze 3D datasets from cardiac blood flow. – Outputs are velocity and pressure values on a
3D grid (1283) over many time steps.
• Characterize behavior under different pathological and physiological conditions. – Natural and artificial heart valves– Vary the initial values of
• Kinematic viscosity • Fluid density• Fiber stiffness and resting lengths
• Use parameter study to find conditions that may lead to aneurysm.
![Page 18: Support for Adaptive Computations Applied to Simulation of Fluids in Biological Systems Kathy Yelick U.C. Berkeley.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d3e5503460f94a17ac8/html5/thumbnails/18.jpg)
Analysis of Cardiac Simulations
• These queries require support for – spatial subsetting of one or more datasets – processing of the data of interest to
visualize and compare results from one or more datasets
– execution on high-performance machines and in a distributed environment.
• Implemented using DataCutter developed by Joel Saltz’s group.
![Page 19: Support for Adaptive Computations Applied to Simulation of Fluids in Biological Systems Kathy Yelick U.C. Berkeley.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d3e5503460f94a17ac8/html5/thumbnails/19.jpg)
Summary of Alpha Project Impact
• Several categories– Application development
• Heart and cochlea-component simulation
– Application-level package• Generic immersed boundary method• Parallel for shared and distributed memory• Enables new larger-scale simulations; finer grid
– Solver libraries• Method of Local Corrections• Improved scalability and load balance expected
– Data analysis• Building input data and analysis of results
applicationdata
softwaresystems