UK HPC Review 05 Sept 2005 Paris, 31 March 2003 Give me a suitable title.... Peter Coveney Centre...
-
Upload
cecilia-jennings -
Category
Documents
-
view
213 -
download
0
Transcript of UK HPC Review 05 Sept 2005 Paris, 31 March 2003 Give me a suitable title.... Peter Coveney Centre...
UK HPC Review 05 Sept 2005Paris, 31 March 2003
Give me a suitable title.... Peter Coveney
Centre for Computational Science
University College London
UK HPC Review 05 Sept 2005
HPC and Grid activities (1)
Group general interests in HPC include scalable codes for Group general interests in HPC include scalable codes for capability computing in order to address grand challenge capability computing in order to address grand challenge problems in several areas of comptuational science through use problems in several areas of comptuational science through use of novel computational methods such as computational steering of novel computational methods such as computational steering and grid computingand grid computing
RealityGrid: RealityGrid: A £6M ($11) project funded by UK EPSRC. Initially A £6M ($11) project funded by UK EPSRC. Initially 2002-2005, now 2005-2009 (selected for Platform Grant)2002-2005, now 2005-2009 (selected for Platform Grant)..
118 PhD students over 8 years as special DTA award to the project8 PhD students over 8 years as special DTA award to the project
Through RealityGrid access to Through RealityGrid access to NSF PACI NSF PACI && NRAC NRAC computing awards computing awards
Spawned several sub-projects: Ongoing Joint US NSF-UK Spawned several sub-projects: Ongoing Joint US NSF-UK SPICE (2005-SPICE (2005-2006) 2006) andand TeraGyroid (2003) TeraGyroid (2003)
ESLEA - Exploitation of Switched Lightpaths for eScience ESLEA - Exploitation of Switched Lightpaths for eScience Applications. Applications. (£1.33M; 2005-2007). HPC component(£1.33M; 2005-2007). HPC component
Primary Primary UK project to join international effort in exploiting Lambda UK project to join international effort in exploiting Lambda networksnetworks
UK HPC Review 05 Sept 2005
HPC and Grid activities (2)
Partners in EPSRC e-Science Pilot Project funded Partners in EPSRC e-Science Pilot Project funded Integrative Integrative BiologyBiology (£2.4 M; 2004-2007): Deploying RealityGrid capabilities (£2.4 M; 2004-2007): Deploying RealityGrid capabilities for steering, high performance computing and grid deployment for steering, high performance computing and grid deployment of both monolithic and coupled applications in cardiac dynamics of both monolithic and coupled applications in cardiac dynamics and tumour growth.and tumour growth.
IntBioSim:IntBioSim: An integrated approach to multi-level biomolecular An integrated approach to multi-level biomolecular simulations. E-Science project in computational systems biology simulations. E-Science project in computational systems biology (£976K;2004-2007)(£976K;2004-2007)
EPSRC e-Science Best Practice project entitled EPSRC e-Science Best Practice project entitled Rapid Rapid Prototyping of Usable Grid MiddlewarePrototyping of Usable Grid Middleware (£331K; 2005-2006). (£331K; 2005-2006).
Robust application hosting in WSRF::LiteRobust application hosting in WSRF::Lite (£170K; 2004-2006). (£170K; 2004-2006). OMII project provides users with a general hosting environment OMII project provides users with a general hosting environment for grid-enabled applications that exploits WSRF capabilities. for grid-enabled applications that exploits WSRF capabilities.
UK HPC Review 05 Sept 2005
LB3D: Three dimensional Lattice-Boltzmann simulations• LB3D code is written in Fortran90
and parallelized using MPI
• Scales linearly on all available resources (CSAR, HPCx, Lemieux,
Linux/Itanium clusters)
• Fully steerable
• Uses parallel data format PHDF5
• Data produced during a single large scale simulation can exceed
hundreds of gigabytes to terabytes
• Simulations require supercomputers
• High end visualization hardware and parallel rendering software
(e.g. VTK) needed for data analysis
3D datasets showing snapshots from a simulation of spinodal decomposition: A binary mixture of water and oil phase separates. ‘Blue’ areas denote high water densities and ‘red’ visualizes the interface between both fluids.
UK HPC Review 05 Sept 2005
Computational steering
Initial condition: Random water/ surfactant mixture.
Self-assembly starts.
Rewind and restart from checkpoint.
Lamellar phase: surfactant bilayers between water layers.
Cubic micellar phase, low surfactant density gradient.
Cubic micellar phase, high surfactant density gradient.
UK HPC Review 05 Sept 2005
Known experimentally for several years
Many interesting properties, theoretical and practical
Discovered using LB3D in 2003;announced at DSFD2003
Liquid crystalline structure leads to many questions—role of defects in materials and fluid properties, etc
Liquid crystalline gyroid cubic mesophase
N. González-Segredo and P. V. Coveney, "Self-assembly of the gyroid cubic mesophase: lattice-Boltzmann simulations." Europhys. Lett., 65, 6, 795-801 (2004);
UK HPC Review 05 Sept 2005
•Funded by EPSRC (UK) & NSF (USA)-to federate the UK e-Science Grid and US TeraGrid
•Main objective was to deliver high impact science which it would not be possible to perform without the combined resources of the US and UK grids
•Study of defect dynamics in liquid crystalline surfactant systems using lattice-Boltzmann methods
•Four month project including work exhibited at Supercomputing 2003 and SC Global
TeraGyroid: A high performance computing grid project
UK HPC Review 05 Sept 2005
TeraGyroid Grid
VisualizationComputation
Starlight (Chicago)
Netherlight (Amsterdam)
BT provision
PSC
ANL
NCSA
Phoenix
Caltech
SDSC
UCL
Daresbury
Manchester
SJ4MB-NG
Network PoP
Access Grid nodeService Registry
production network
Dual-homed system
10 Gbps
2 x 1 Gbps
UK HPC Review 05 Sept 2005
Defects and their dynamics in gyroid phases
Developed analysis tools for identifying &tracking defects; classified defects
Studied time evolution of defects
Mechanical and rheological propertiesLattice sizes up to 10243
Huge data analysis challenge
UK HPC Review 05 Sept 2005
Spherical Pickering emulsion droplets = “colloidosomes”
http://www.deas.harvard.edu/projects/weitzlab also Hull group webpages
COLLOIDS IN BINARY SOLVENTS
Colloids in binary solvents
Spherical colloids superposed on lattice:
Surfaces are cut by boundary links
Particle centres move smoothly
Bounceback on links algorithm
N.-Q. Nguyen and A. Ladd, Phys Rev E 66, 046708 (2002)
Net force/torque on particle:
Sum of contributions from all boundary links
Used to update the colloid velocity
BBL causes no overall change in mass if flow incompressible
COLLOID HYDRODYNAMICS IN LB
LUDWIG
• MPI, Open MP, etc; 40,000 lines• Supercomputer + workstation compatible• Gold Star rating on HPCx
‘BIG SCIENCE’Algorithm design, coding, problem definition, parameter steering, data production, analysis, interpretation....
SYSTEM SIZES• 30/40 fluid/fluid interfaces across system; about 10,000 colloids•1283 lattice size maximum with colloids• larger with MPI on BlueGene/L expected Q3 2005
LIMITATIONS
•Careful parameter steering / testing needed to keep systematic errors in check.
• Large but finite range of parameters achievable: maintain hierarchy
UK HPC Review 05 Sept 2005
SPICE: Simulated Pore Interactive Computing
Environment
●Transport of biomolecules through protein pores not well understood.
● Fully atomistic simulations crucial to capture pore-protein interaction.
●Systems of this size and complexity are computationally intensive.
● Time scale of transport is typically tens of microseconds. Traditional computational approaches (MD) manage barely few nanoseconds.
DNA beginning its translocation through an alpha-hemolysin protein pore embedded in a lipid layer. There are 275,000 atoms in the system. Water is not shown
UK HPC Review 05 Sept 2005
• Grid approach facilitates more
effective and greater resource
utilization; permits novel analysis
approaches; uniform interface makes
utilization “easier”.
● Appropriate choice of algorithms is
required to make the problem amenable
to a grid-based solution.
SMD+ JE: Steered Molecular Dynamics
(SMD) to “pull DNA through the pore”
and Jarzynksi's Equation (JE) to compute
the equilibrium free energy profile from
the non-equilibrium pulling.
A view into the pore from the top end. Note the seven fold symmetry of the system.
SPICE: Grid Computing Using Novel Algorithms
UK HPC Review 05 Sept 2005
Step I: Understand structural features using static visualization
Step II: Interactive simulations for dynamic and energetic features Steered simulations: Visualizer used to apply forces; forces sent to simulation; simulation computes effect of forces; simulation sends updated information back to visualizer Haptic interaction: Use haptic device to feel feedback forces
Step III: Simulations to compute “optimal” parameters values. 75 simulations on 128/256 processors each. (100pN/A spring constant; 12.5 A/ns pulling velocity)
Step IV: Use computed “optimal” values to calculate full FEP along the cylindrical axis of the pore. Will requires 250,000 CPU hours (100 simulations of 2500 CPU hours) at least!!
Computing the Free Energy Profile (FEP)
UK HPC Review 05 Sept 2005
The infrastructure used similar to the TeraGyroid project -- Software & Hardware. Some resources (e.g UK-NGS) more mature. Grid middleware not much better!
High-end systems required to provide realtime interactivity. Simulation might stall due to “unreliable” sim-vis communication.
Advanced networks provide schedulable capacity and high QoS in terms of packet loss, jitter and latency. Better performance using UKLight
SPICE: Grid Infrastructure
UK HPC Review 05 Sept 2005
Calculating binding free energies
• Usually possible to only compute ∆∆Gbind
• Consider the thermodynamic cycle
+
∆GA
+
∆GB
∆G1
∆G2
•Free energy is a thermodynamic state function ∆G1 + ∆GB - ∆G2 - ∆GA = 0
∆∆Gbind = ∆GB - ∆GA
∆∆Gbind = ∆G2 - ∆G1
UK HPC Review 05 Sept 2005
Thermodynamic integration•The free energy change is calculated using the thermodynamic integral1
GAB =
€
∂H
∂λ0
1
∫ dλ
1. Run 10+ simulations, each at a distinct value of (e.g. 0.1,0.2…)
2. For each, compute
3. Plot as a function of and integrate the function using the trapezium rule
€
∂H
∂λ
€
∂H
∂λ
-100
-80
-60
-40
-20
0
20
0 0.5 1
Example TI plot
1. Leach, A. R. 2001 Molecular Modelling. Principles and Applications 2nd edition Edinburgh: Pearson Education Ltd.
UK HPC Review 05 Sept 2005
=0.1
=0.2
=0.3
…
=0.9
Starting conformation
t
Seed successive simulations
(10 sims, each 2ns)
Check for convergence
lambda
Combine and calculate integral
time
Use steering to launch, spawn and terminate - jobs
Run each independent job on the Grid
This calculation is ideally suited for a computational grid
STIMD
UK HPC Review 05 Sept 2005
A User’s PerspectiveNAMD2.5-
ReG
SGS
US TeraGrid
Steering client
Registry
find
publish
bindManchester
laptop/PDA/portal
UK NGS
1. Using the RealityGrid (ReG) launching wizard, the user launches a parallel NAMD2.5-ReG job on the US TeraGrid. The exchange of locations and data is facilitated by the Registry.
2. A second simulation is launched with a different value of . This is spawned from the first simulation and, for example, is displayed in a different tab of the ReG steering client.
3. The ensemble averages are plotted by the ReG steering client. Once a simulation has converged, it is terminated.
4…the second simulation is migrated
spawned
NAMD2.5-ReG
SGS
NAMD2.5-ReG
SGSSTIMD
UK HPC Review 05 Sept 2005
The SH2 calculation at the AHM2004 was a success
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
STIMD
UK HPC Review 05 Sept 2005
Immunoinformatics: Agonist or antagonistFree energy differences of Tax to P6A
UnboundPPG
21 Bound
PPG21
BindingPG
1
+
+
P1
P2
P1
P2BindingPG
2
TI TI
Experiment
Experiment
102K atoms
11 windows
1.2ns/window
256 procs
LeMieux (PSC)
30K CPU hours
120h Wallclock
53K atoms
11 windows
2ns/window
256 procs
LeMieux (PSC)
33K CPU hours
130h Wallclock
Wan, S., et al J. Im munol. 157, 1715-1723, 2005.
∆∆Gbind = -2.9 +/- 0.2 kcal/mol Experiment
∆∆Gbind = -1.9 +/- 0.4 kcal/mol TI
Possibly worlds largest TI calculation!!
UK HPC Review 05 Sept 2005
Immunological Synapse
330K atoms
Newton (CSAR):1ns: 16hours on
192 procs3100 CPU hours
~1 million atoms
Newton (CSAR):1ns: 48hours on 192 procs (estimated)
9300 CPU hours (estimated)
Turn immune responses:
Immunological synapse
UK HPC Review 05 Sept 2005
WEDS• WSRF::Lite Environment for Distributed Simulation
• Prototype application hosting environment built using WSRF::Lite (UoM)
• Provides a WSRF-compliant wrapper around an application.
• Services for discovering and instantiating applications and for file staging.
• Application binary remains unaltered.
• WEDS version 1.0 released (2005)
UK HPC Review 05 Sept 2005
Application launching using WEDS
machine_01(launcher)
machine_02 machine_03
...
machine_NN
multiprocessor
( OR )
UK HPC Review 05 Sept 2005
AHE (‘Application Hosting Environment’)
• Perl-based successor to WEDS; design incorporates experience gained with WEDS
• Provide a web services presentation of existing (‘legacy’) applications; no requirement to modify app. source or binary
• Delegates access to compute resources (Globus, OMII, Sun Grid Engine) to GridSAM services (OMII)
• Uses WS-Security; MyProxy for user credentials
• Registry and discovery services for locating hosted apps.