Post on 12-Jan-2016
HPC@sheffield -where do we go next?
John HardingDept. Materials Science and
Engineering, Sheffield
3
Simulation Scales
www.nsf.gov/news/speeches/bordogna/rochester/sld006.htm
Simulations and scales
4
For details seewww.prace-ri.eu
Grid computing – particle physics
Iceberg Current Configuration• Cluster based on Intel 6- core X5650
– 71 nodes: 12 cores and 24GB– 4 nodes: 12 cores and 48GB – 8 NVIDIA Tesla Fermi
M2070GPUs• Storage
– 45TB NFS mounted mirrored data store
• User quota system– 80TB Lustre based parallel file
store• Fastdata store a scratch area
data is deleted after 90 days
How do we use what we have at Sheffield Number of Active Users
0100200300400500600700800900
1000
2009 2010 2011 2012 2013 2014
Total CPU Time for Iceberg (Years)
0
200
400
600
800
1000
1200
1400
2009 2010 2011 2012 2013 2014
Mean Top100 Users usage
0
20000
40000
60000
80000
100000
120000
2009 2010 2011 2012 2013 2014
Year
Ho
urs mean top100 wallclock
mean top100 cpu
Mean User Usage
0
2000
4000
6000
8000
10000
12000
2009 2010 2011 2012 2013 2014
Year
Ho
urs mean user wallclock
mean user cpu
Iceberg main users
• Mechanical Engineering– Aerodynamics– Thermofluids
• Applied Maths– Solar Physics and Space Plasma Research
• Insigneo– Virtual Physiological Human Initiative
• Electrical Engineering– Device Electronics: nanoWires and NanoTubes – Lensless microscopy
Future of computing in Sheffield,SP2RC approach
Solar Physics and Space Plasma Research Centre (SP2RC)
Solar Wave Theory Group (SWAT)School of Mathematics and Statistics (SoMaS), Faculty of Science, UoS
Space System Laboratory (SSL)Automatic Control and Systems Engineering (ACSE), Faculty of Engineering, UoS
SP2RC at the UoS seeks to understand the nature of key plasma processes occurring in •the solar interior,•the atmosphere of the Sun, from photosphere to corona, •with particular attention devoted to the various coupling mechanisms of these apparently distinct regions.
Interfaces for photovoltaics
VBTFp EE
p-type Schottky barrier height.
Supercell Fermi energy.
Valence band top energy of bulk semi-conductor.
pgn E
n-type Schottky barrier height.
Semi-conductor band gap energy.
Interface φp/eV φn/eV φn/eV (expt.)
<111> 0.31 0.80 0.74<110> 0.50 0.61
11
K.T. Butler, P.E. Vullum, A.M. Muggerud, E. Cabrera and J.H. Harding, Phys. Rev. B 83 (2011) Art 235307
Biomechanics (Lacroix)
A new, bigger Iceberg
• Infrastructure to be added to existing Intel Nodes
• Cluster based on Intel Ivy Bridge 8- core E5-X2650v2– 71 nodes each with 16 cores and 64 GB memory– 8 nodes each with 16 cores and 256GB memory– 8 NVIDIA Tesla Fermi K40 GPUs
• Storage– 100TB Lustre based parallel file store
• Fastdata store a scratch area data is deleted after 90 days Network
• Expand Infiniband spine to facilitate further expansion and addition of nodes by research groups
• Update of NFS mounted data – TBD• Older AMD nodes to be decommissioned• Coming by June 2014.
Hardware Access Routes
• Research groups can 1. access the general facility – fair share using
system scheduler2. Purchase hardware and host on iceberg3. Purchase a number of compute nodes for a
specified of time
• Purchase route provides a dedicated access to a resource effectively ‘owned’ by the reseacher
Client Access to Visualisation Cluster
VirtualGL Client
Iceberg –Campus Compute Cloud
VirtualGL Server(NVIDIA GPU)
Applications Supported Using Graphical Processing Units
• Ansys• Matlab• Flame• Beast• Ansys Abaqus• Smoothed Particle Hydrodynamics (DualSPhysics)• Astronomy
– MHD Applications (SMAUG, Athena)– Nbody 6
N8 HPC• 316 nodes (5,056 cores)
with 4 GByte of RAM/core • 16 nodes (256 cores) with
16 GByte of RAM/core• Sandybridge architecture• Nodes connected by
infiniband. • 110 Tflop/sec peak. • 174Tb Lustre v2 parallel
file system • 109Tb NFS backed-up file
system www.n8hpc.org.uk
N8 HPC - objectives
• Seed engagement between industry and academia around research using e-infrastructure
• Develop skills in the use of e-infrastructure across the N8 partnership
• Share the asset of skills and equipment across the N8 partnership via the facilitation of networks of people
What people have done on N8
MySpine – Damian Lacroix; Sheffield
Studies of turbulenceShuisheng He
Biomolecules on surfacesShaun Hall
N8 HPC - access
• You need a N8 HPC Project Account. We ask that PIs are permanent members of staff at their institution
• Contact the local HPC representative at Sheffield – hpchub@sheffield.ac.uk or Iceberg admin.
• You will need to produce a project proposal for consideration by the local committee.
• You will be expected to produce a report saying what you have done with the resources.
ARCHER (www.archer.ac.uk)
What there is (Stage 1)
• ARCHER compute nodes contain two 2.7 GHz, 12-core E5-2697 v2 (Ivy Bridge) series processors (2632 nodes)
• Standard compute nodes on ARCHER have 64 GB of memory shared between the two processors. Smaller number of high-memory nodes with 128 GB of memory shared between the two processors (376 nodes)
• In all, 3008 nodes; 72,192 cores.
Getting access• ARCHER Access Through EPSRC and NERC
Grants – process similar to HECToR. Need to submit a technical assessment before the grant application and attach result to JES form
• ARCHER Instant Access – means what it says; limited amounts for pump-priming
• ARCHER Access Through the Resource Allocation Panel – stand-alone access request. Next closing date is May 14th (EPSRC remit)
• ARCHER access through a consortium – there may be one in your area; need to contact PI.
Some facilities available
Getting access
• PRACE publishes its Calls for Proposals for Project Access twice per year according to a fixed schedule:
• Call opens in February > Access is provided starting September of the same year.
• Call opens in September > Access is provided starting March of the next year.
• See www.prace-ri.eu for details.
Some issues• High Performance Computing is essential but does
not stand on its own. Simulations are simulations of a real world problem. They must not become divorced from that. All simulation has a multidisciplinary side.
• People can always use more computing power – but getting good, reliable software is even better (and much harder)
• Hardware is useless without technical support and training – and these tend to be much harder to fund.
• Data is becoming a major issue – its capture, curation and analysis. We are already into petabytes here at Sheffield. The world is into exabytes at least.
Closing remarks
• Industry tends to engage with Academy on problems they need answering.– Needs an expertise offering, even if approached for cycles.
• Trusted relationships take time– Important to maintain channels that work
• HPC is part of a broader offering• Critical to build the right team
– Involve skills/resources in National centers• Must listen to & provide forum for industry
– Shape & maintain a relevant offering to industry– Embed in existing activities: N8 industry innovation forum
• Links to Universities– Appropriate skills emerging from our graduates.
Main Goals we would like to achieve in SP2RC
1. Forward modelling of the the excitation, progression and dissipation of MHD waves in the solar atmosphere based on the single/multi- highly localised magnetic flux tubes (e.g. magnetic bright points/pores, magnetic elements of Active Regions, vortices and inter-granular lanes) by applying realistic sources (even from high-resolution observations) of wave excitation based on one/two fluid MPI/GPU MHD codes.
2. To determine the role of Alfven, kink and longitudinal waves in the wave energy transport from photosphere to the upper layers of the solar atmosphere.
3. Perform multidimensional two fluid MHD numerical
simulations at various spatial and temporal scales within the chromosphere, transition region and low corona by taking into account the crucial solar plasma processes, i.e. heat conduction, ionization, proton-neutral hydrogen collisions, in order to obtain an accurate description of the classic wave dynamics and line synthesis.
4. Apply realistic 3D MHD models of the solar atmosphere investigating the mechanisms governing the transport of energy from the solar photosphere to the solar corona.
Orszag-Tang Test
200x200 Model at t=0.1, t=0.26, t=0.42 and t=0.58s
Timing for Orszag-Tang Using SAC/SMAUG with Different Architetures
0
100
200
300
400
500
600
0 1000 2000 3000 4000 5000
Grid Dimension
Tim
e f
or
10
0 It
era
tio
ns
(s
ec
on
ds
)
NVIDIA M2070
NVIDIA K20
Intel E5 2670 8c
NVIDIA K40
K20(2x2)
K20(4x4)