Evolution of the HemeLB Parallel Simulation Environment for Human Brain Bloodflow
-
Upload
university-college-london -
Category
Technology
-
view
130 -
download
0
Transcript of Evolution of the HemeLB Parallel Simulation Environment for Human Brain Bloodflow
2008 First journal publication
on HemeLB.
2010 Development of steering and visualization client.
2009 HemeLB run across sites
Using MPI-g.
2011 Improved domain
decomposition with ParMETIS.
2013 Multiscale 3D-1D couplingwith Python Navier Stokes.
2013 Up to 50% faster
Calculations with SSE.
2014 Better load balance using weighted decomposition.
2012 Support for 2nd order
accurate wall conditions.
2014 First comparison tests with
clinical data.
2014 HemeLB-Chaste coupling
to simulate vascularremodelling processes.
2013 More stable model
constructionwith CGAL.
2014Gained support for
implementing immersed boundary conditions.
2012 Performance prediction
Model.
2012 Improved scalability
With coalesced comms design pattern.
2011 Code reengineered forimproved accuracy and
stability.
2014Support for PT-Scotch,
Zoltan and ParMETIS indomain decomposition.
Performance improvements
Accuracy improvements
Scientific advances
New functionalities
2013 Framework for
convenient propertyextraction
Evolution of the HemeLB bloodflow simulation environment
The performance of HemeLB has improved considerably over the past 7 years:
Indeed, for smaller geometries we can now do production simulations (e.g., 5M time steps)in less than an hour. However, simulating a full Circle of Willis model on a relevant time scale (e.g., 25M time steps) will still take about a day.
Circle of Willisrun
(first trial)
HemeLB models sparse vasculature geometries as a lattice of fluid sites. These geometries contain bulk sites, wall sites and in/outlet sites. Both wall and in/outlet sites are generally more expensive to compute than bulk sites, leading to load imbalance among different processes.
To reduce this load imbalance, we assign weights to each lattice site (see right below) before we partition and distribute the domain among
the processes. Using this approach resulted we have managed to reduce the calculation load imbalance by up to 85%.
Highlight: Weighted Decomposition
Recommended readingContributorsMarco Mazzeo, Steven Manos, Gary Doctors – initial developersRupert Nash, Hywel Carver, James Hetherington, Timm Krueger – Main developers during 2010-2013.Miguel Bernabeu, Derek Groen, Sebastian Schmieschek – current HemeLB developers.Dan Holmes – Colloids code 2012.Jiri Jaros, David Abou Chacra – Performance optimizations during 2013.Jens Nielsen – CGAL setup tool optimizations 2013.Gregor Matura, Fang Chen – pre- and post-processing 2014.Aditya Jitta – Comparison against clinical data, 2013.
2013 Comparison of different
Rheology models.
2012 Comparison of different
wall conditions.
Cerebrovascular bloodflow Introducing HemeLB
Derek Groen, Miguel Bernabeu, Rupert Nash, Sebastian Schmieschek and Peter Coveney
2013 Support for
Colloidal particles
2010 Python model construction
tool developed.
● Stroke is the main cause of about 1.1M deaths per year in Europe. ● About 15% of these strokes are caused by bleeding in the brain, e.g. due to the rupture of brain aneurysms. ● These brain aneurysms frequently reside in arteries branching from the Circle of Willis.
● HemeLB is a feature-rich simulation environment for modelling blood flow in sparse geometries.● It relies on the lattice-Boltzmann method and is well suited for execution on large supercomputers.
HemeLB consists of several key components:● The main simulation code, which can be coupled to other codes.● The setup tool, for constructing 3 dimensional geometries from segmented angiography scans.● A Python-based framework for constructing initial conditions and analyzing output data.● An interactive steering and visualization tool.● A Python-based automation environment for deploying and executing the code on remote machines.
Circle of Willis with one diminshed artery. © Nevit Dilmen
Simulating bloodflow
Clinicians can conveniently measure blood pressure and flow velocities on patients at rest with limited resolution.
Simulations allow us to estimate and predict flow properties in other regimes as well. These include:● Flow velocity estimates for patients during exercise and other forms of activity.● Wall stress estimations under all these conditions (e.g., wall shear stress).● Flow properties in specific locations within a geometry, e.g. velocities and stresses in an aneurysm sac.
Both very high and very low wall shear stress have been associated with aneurysm formation and rupture.
Sample visualization of a HemeLB simulation.
1. Performance: JoCS, DOI: 10.1016/j.jocs.2013.03.0022. Weighted Decomposition: EASC 2014. Preprint available.3. Boundary Conditions: Phys. Rev. E 89, 023303 (2014).4. Clinical Validation: work in progress. Slides available.5. Multiscale: Interface Focus, DOI: 10.1098/ rsfs.2012.00876. Retinal blood flow: Interface (submitted) arXiv:1311.1640.
Access the source code at:http://ccs.chem.ucl.ac.uk/hemelb
For requesting preprints, please send an e-mail to Derek Groen ([email protected]).
Highlight: Comparison against clinical data
We have compared flow predictions from HemeLB with clinical measurements. We have done this work in collaboration with Fergus Robertson and Hoskote Chandrashekar from UCL Hospital.
● We obtained rotational angiography images of a middle cerebral artery, as well as velocity measurements (TCD) in 5 planes within this artery.● We imposed one plane as a velocity-based inlet in HemeLB, and ran the code to predict the velocities in the four other planes (we used pressure outlets).● See below for an overview of the geometry, and a comparison between the HemeLB flow predictions and the TCD measurements on the plane furthest away from the inlet.
● We are currently working to repeat this exercise with a second patient, and to improve our comparison techniques.
2012 Tools for automatic
compilation and execution on remote machines.
inlet @63mm.
v plane @49mm.
2014 Prediction of vascular
development in retinas.
2020SCIENCEwww.2020science.net
UKCOMESUK Consortium On Mesoscale
Engineering Sciences
Legend