Exploring the Nature of Cold Frontsin Merging Clusters of Galaxies with the FLASH Code

PI: Maxim Markevitch, Harvard University

Co-I: John ZuHone, University of Chicago

1 Significance of Research

Galaxy groups and clusters are the largest objects in the universe that are gravitationally bound.Groups and clusters may contain anywhere from tens to thousands of galaxies as seen in visiblelight. However, nearly a century of investigation of galaxy clusters has demonstrated that thereis literally more than “meets the eye.” Observations of clusters in the X-ray band have shownthat most of the baryonic component is in the form of plasma [16; 12; 25; 34; 33]. This gas isdiffuse (n ∼ 10−2 − 10−4 cm−3), hot (T ∼ 107 − 108 K), and magnetized (B ∼ 0.1 − 3 µG) [4].This is confirmed by observations in the microwave band, as cosmic microwave background (CMB)photons are inverse-Compton scattered by the electrons in the plasma and cause a correspondingdecrement in the CMB intensity [36]. In addition, measurements of galaxy velocity dispersions anddistortion of background galaxies due to gravitational lensing by clusters indicate that most of thematter in galaxy clusters is in the form of non-baryonic “cold dark matter” (CDM) [40].

Importance of Galaxy Clusters.Due to their size and composition, clusters of galaxies provide a good representation of the

material properties of the universe as a whole. Thus, they are useful for resolving importantquestions of cosmology and fundamental physics as well as being interesting in their own right.Due to their deep gravitational potential wells, clusters are essentially “closed boxes” that retainmost of their gaseous and dark material. Thus, they are excellent cosmic laboratories for studyingvarious important questions in astrophysics. X-ray observations of clusters can be used to studythe turbulence, magnetic fields, and viscous properties of the ICM [19]. Gravitational lensing andobservations of hard X-rays and gamma rays provide observational constraints on the nature ofdark matter (e.g. self-interaction and/or annihilation cross-section) [23; 29].

X-ray Observations of Galaxy Clusters with Chandra.Many of the recent advances in the study of the ICM have been accomplished with the Chandra

X-ray Observatory. In the nearly ten years since launch in 1999 Chandra has made groundbreakingdiscoveries in X-ray astronomy, including observations of supernova remnants, black holes, andclusters of galaxies [38]. The high angular resolution of Chandra (∼ 0.5 seconds of arc per pixel,corresponding to a few kpc for nearby clusters) has allowed astrophysicists to probe the ICM inexceptional detail. This has opened up a host of new scientific questions concerning the ICM forsimulations to address, since we are in a regime where the spatial resolution of the observationsis comparable to or sometimes even better than the resolution of the simulation. This has moti-vated the forging of more direct links between simulation and observation. One technique beingincreasingly adopted is the use of simulated observations of X-ray photons constructed from hy-drodynamic simulations. This allows essentially direct comparisons between the simulations andalready existing observations and serves as a guide for future observations.

Cosmological Structure Formation and Cluster Mergers.Within the CDM paradigm of cosmological structure formation, structures in the universe form

in a “bottom-up” fashion. The smallest structures collapse first and build up over billions of

1

years into larger and larger structures. Today, the largest structures formed in this fashion areclusters and groups of galaxies. Because this process is still ongoing, many clusters of galaxies showcurrent or recent signs of merging with other clusters. Two prominent examples of recent note are1E 0657-56 (the “Bullet Cluster”) [22; 6], and Abell 520 [24; 18].

Cluster mergers make it possible to probe the nature of dark matter and the material propertiesof the ICM. In 1E 0657-56, X-ray observations of the ICM combined with weak lensing analysisdemonstrated a clear separation between the bulk of the cluster mass and the bulk of the baryoniccomponent [6], demonstrating that the bulk of the cluster mass is dark and collisionless. Mergersalso often result in sharp features such as shock fronts and “cold fronts” in the ICM, which can beused to place constraints on the material properties of the gas such as the magnetic field strengthand its viscosity and conductivity [19].

Cold Fronts In Merging Clusters of Galaxies.Observations of the ICM have revealed the existence of “edges” and “fronts” in the extended

X-ray emission. Since the X-ray emission is strongly dependent on the gas density, the implicationis that these features are associated with sharp density gradients. Spectral analysis of these featuresreveal that they come in two categories. The first is shock fronts, where the higher-temperaturegas is on the denser side of the front, corresponding to gas that has been heated by a shock wavepropagating through the ICM. The second category has been dubbed “cold fronts”, with the lowertemperature gas on the denser side of the front.

Cold fronts appear in many clusters, even those that are apparently relaxed and show no recentsigns of merging [20]. In merging clusters, such as 1E 0657-56, cold fronts can form by cool, densecores of galaxy clusters becoming ram-pressure stripped by the surrounding ICM [21]. In relaxedclusters, some authors have shown from simulations that cold fronts can form via the displacement ofthe core of the cluster gas from the minimum of the gravitational potential well due to a disturbanceby a passing subcluster [37; 3]. This results in the “sloshing” of the cluster gas in the potential,creating a pattern of cold fronts.

X-ray observations have shown that the width of the density and temperature jump acrosscold fronts is typically very small, on the order of 2-5 kpc [11]. Also, the fronts are typically verysmooth in appearance and undisturbed by turbulence [19]. This implies that along the front fluidinstabilities (e.g. Kelvin-Helmholtz), thermal conduction, and diffusion are suppressed. Withinclusters of galaxies a stabilizing mechanism does exist: magnetic fields. If a magnetic field issituated parallel to the front, it will suppress movement of particles plasma to the field lines andhence suppress diffusion, conduction, and fluid instabilities. Several authors have shown that sucha magnetic field configuration can naturally arise in a galaxy cluster via “magnetic draping,” wheredense gas moving through the ICM can sweep up enough magnetic field to build a magnetic sheatharound the front [2; 10]. This can occur regardless of the initial field configuration.

Our hydrodynamic simulations of galaxy cluster mergers can provide insights into the followingspecific questions regarding cold fronts in the ICM:

1. Are the cold fronts in clusters from major mergers (such as the 1E 0657-56) or from subclustersdisplacing gas from the central potentials of relaxed clusters?

2. Is the width of the fronts in simulations consistent with those seen in X-ray observations?

3. What constraints can be placed on the viscosity and the strength of the magnetic field fromthe stability of the fronts?

As we describe in the Objectives section, our simulations of controlled mergers between galaxy

2

clusters where we tune initial conditions to match observed clusters will provide these fresh insightsat better resolutions than have been previously achieved.

2 Research Objectives

Structure in the universe, including clusters of galaxies, is thought to form in a “bottom-up” fashionvia gravitational attraction, with the smallest objects forming first and merging into progressivelylarger objects. This merging process is still ongoing, as many clusters of galaxies show signsof recent or current merging. Understanding the many facets of the merging process in galaxyclusters is essential to addressing the major research questions associated with cold fronts in clustersof galaxies. It is believed that cold fronts are more often than not the result of merging clusters,whether major mergers or the accumulation of subclusters by large clusters, and so merging providesopportunities to use cold fronts to probe the physics of the ICM.

Simulations of merging clusters provide insight into the effects of merging on the observableproperties of galaxy clusters. The University of Chicago Center for Astrophysical ThermonuclearFlashes (the “Flash Center”), which is supported by the Department of Energy Advanced Simu-lations and Computing through the Academic Strategic Alliance Program, has developed FLASH,a highly capable, fully modular, extensible, community code that is widely used in astrophysics,cosmology, fluid dynamics, plasma physics, and other fields. The capabilities of the FLASH codeinclude adaptive mesh refinement (AMR), accurate solvers for hydrodynamics and magnetohy-drodynamics, several self-gravity solvers, support for active particles, and cosmological expansion.Therefore, the FLASH code is able to accurately treat the complex interplay between the differentkinds of matter in clusters of galaxies and the forces that cause them.

Advantages of Our Simulation Approach.The FLASH code solves the equations of hydrodynamics using an Eulerian grid approach. Many

previous investigations of galaxy clusters and cosmology have utilized codes that employ Lagrangianmethods that follow the motions of gas particles, dubbed “Smoothed Particle Hydrodynamics”(SPH) methods [26]. Such methods have been popular for their speed and typically low memoryusage when compared with grid-based methods.

Recent comparisons of these two methods have demonstrated that there are significant andfundamental differences in the results obtained from simulations of astrophysical situations relevantto clusters of galaxies [1]. Specifically, grid-based methods are able to resolve important dynamicalinstabilities (e.g. Kelvin-Helmholtz and Rayleigh-Taylor) where existing SPH methods resolve thempoorly or not at all. The identified reason for this discrepancy is that in regions with steep densitygradients spurious pressure forces on the particles are introduced, producing a gap at the fluidinterface where information that would drive instabilities is not transferred.

This is an important consideration when performing simulations of galaxy cluster mergers.Mergers drive shocks into the ICM, heating it and driving some of the gas away from the clusters.Cold fronts are often formed from major mergers or the sloshing of a cluster core due to a mergingsubcluster. Both of these features involve sharp density gradients. In the specific applications wedetail below, we will point out the advantages of the grid-based method that our FLASH simulationsemploy.

Controlled Simulations of Mergers.Since merging occurs in a cosmological context, many investigations of cluster mergers have

been done in the context of cosmological simulations [27; 7]. The advantage of this approach isthat merging can be studied in a realistic context where there may be many clusters merging at

3

Figure 1: Slices through a FLASHsimulation of the galaxy clusterCl 0024+17 undergoing a merger.The variable plotted is gas den-sity. The clusters are initially sep-arated by a distance of ∼3 Mpc,and the smaller, denser clusterdrives a shock into the larger clus-ter, pushing the larger’s gas for-ward and laterally. This gas laterfalls back into the cluster’s darkmatter-dominated potential [39].

once in various dynamical states. However, this is also the disadvantage of this approach, becauseit makes it difficult to tease out the different causal mechanisms and effects that led to the finalmerger product from the initial conditions.

By performing controlled simulations of galaxy cluster mergers, it is possible to separate out thevarious effects from each other and precisely characterize their contribution to the different stagesof the merger, including the final merger product. Two kinds of controlled merger simulations arecommon: (1) simulations which attempt to reproduce real cluster mergers, typically via iteratingover many small resolution simulations until a good fit is achieved [35; 39] (see Figure 1); and(2) explorations of a merger parameter space where initial conditions such as mass ratio, initialvelocity, and impact parameter are varied [31; 32; 28]. It is the first type of simulation that weplan to carry out using the TeraGrid allocation.

Using the FLASH code together with the computational resources of the TeraGrid allocation,we will carry out a rigorous investigation of the nature and origin of cold fronts in merging clustersof galaxies, beginning with simple hydrodynamic models of the ICM building up to models withexplicit viscosity and magnetic fields. We will include models of major mergers between clustersas well as minor mergers between large, relaxed clusters and subclusters. We will also use mockX-ray observations of these clusters to make predictions regarding real observations of cold frontsin clusters.

We now describe in more detail our strategic plan for addressing these questions with oursimulations.

Analysis of Specific Merger Scenarios.Cold fronts are observed in many galaxy clusters, some which show obvious signs of merging

and others which appear relatively relaxed. Much progress has been made over the last decade overunderstanding generally how such fronts could arise in the ICM in either case. Merging is a likelycause of both kinds, whether by cluster cores becoming ram-pressure stripped in major mergers[21] or by infalling substructure initiating gas sloshing in relaxed clusters [37; 3].

However, important unanswered questions remain with regard to the stability of the fronts andthe sharpness of the density and temperature jumps. X-ray observations have shown that thewidth of the density and temperature jump across cold fronts is typically very small, on the orderof 2-5 kpc [11], and the front surfaces are smooth in appearance and undisturbed by turbulence[19]. Since we expect diffusion and conduction to smooth out the fronts and turbulence to breakthem up, a stabilizing mechanism must exist in the ICM.

In order to answer these questions in the context of existing observations of cold fronts in galaxyclusters, we will seek to model specific clusters and merger scenarios. Our ability to construct mock

4

Figure 2: Examples of mock Chandra observations from FLASH simulations. Images are of rawcounts taken in 60 ks exposures. Left: A cold front preceded by a shock front. Center: Two galaxycluster cores sideswiping each other, producing cold fronts. Right: “Sloshing” cold fronts producedby the passage of a smaller cluster, displacing the gas from the primary cluster’s potential well.

X-ray observations of our simulations will enable direct comparisons to these real clusters. Our suiteof simulations will include major mergers as well as simulations of relaxed clusters with infallingsubstructure.

Inclusion of Additional Physics.Since it is expected that features such as cold fronts would be subject to fluid instabilities, we

will seek to understand how adding additional physics to our simulations affects the stability of thefronts. Our base simulations will model inviscid, unmagnetized Eulerian flow. It is expected thatunder these conditions cold fronts should not be stable to Kelvin-Helmholtz and Rayleigh-Taylorinstabilities and disruption by turbulence [19]. However, it is known that the ICM is magnetized,and the viscosity of the ICM is not known. Viscosity would damp fluid motions and preventturbulence from developing at the front surfaces, and magnetic fields could exist in stabilizinglayers at the front surface to suppress instabilities [2; 10].

The FLASH code has support for adding an explicit viscosity to the Euler equations, and thelatest version of FLASH has excellent support for MHD. Therefore we are in an excellent positionto extend our merger simulations into regimes not fully explored as of yet by current simulationsin the study of cold fronts.

Simulation of Observed Quantities Using Simulated Photons.At regular intervals throughout the evolution of the cluster merger, we will output the thermo-

dynamic and kinetic state of the cluster gas, and the positions, velocities, and energies of the darkmatter particles. We will use the former information to construct simulated photon event files.Though such a volume of simulated data would take a very long time to analyze on an individualbasis, we already have in place a pipeline which will make much of this job automatic. Creating suchevent files will help us to make direct comparisons between the cold fronts seen in our simulatedobservations and in real observations of galaxy clusters, particularly with regard to how adding theeffects of different physics in the simulation has on the observations. For some examples of thesesimulated observations where cold fronts have appeared see Figure 2.

Strategic Plan for Simulations.In order to match our simulations to the observed clusters, it will be necessary to adopt an itera-

tive approach. We will carefully choose initial conditions based on the best available measurementsfor a set of low resolution simulations, some of which we may run on resources locally available.After each inexpensive simulation is run we will examine the output and make adjustments to our

5

Figure 3: Cold gas sloshing in the center ofa relaxed cluster simulated with FLASH. Thecolor scale is temperature in units of keV. Thecluster initially possessed a high-gas density,low-temperature core. A passing subclusterseparated the gas from the central dark mat-ter potential. Note the sharpness of the frontat the southward end.

initial conditions to make a closer match to the actual cluster conditions at the time of interest.Once we have confidence in our simulation conditions we will run high-resolution simulations whichwe will analyze to generate our final results.

The clusters we will choose to simulate include some of the most interesting recently studied byChandra. The first, 1E 0657-56, is the prototypical example of a cold front formed in a major merger[22; 6; 35]. Secondly, the cluster Abell 520 is another interesting example of a major merger witha cold front, though at a later state [14]. This is also a cluster merger with an unusual distributionof dark matter, so simulating this system will have application beyond the study of cold fronts [18].Thirdly, we will simulate a merger between a large, relaxed cluster and a subcluster, using relaxedclusters such as Abell 2029 or Abell 1795 as our model. These are large, relaxed clusters with coldfronts [5; 20]. We will then use our simulated X-ray observations of these clusters to measure thesharpness of the fronts under different physical conditions and different times in the simulation.

We propose to run 18 high-fidelity simulations of these different clusters and others with amaximum resolution of ∆x = 5 kpc, varying initial conditions and the physical mechanisms inplay. For example, for 1E 0657-56 we may choose to run three simulations, one with Eulerianhydrodynamics, another with magnetic fields, and another with an explicit viscosity. As we willdetail in the Job Characterization section, the 18 high-fidelity simulations will require a total of 9million CPU-hours of the TeraGrid allocation on the TACC Ranger Sun Constellation System.

Advantages Over Previous Works.It was recently demonstrated that in SPH simulations spurious pressure forces can develop on

particles at jumps in gas density that prohibit communication across the fluid interface [1]. Thishas the effect of suppressing fluid instabilities which would otherwise destroy these features. Assuch, the spurious pressure forces in SPH simulations have the unfortunate tendency of mimickingthe effects that would be expected from the inclusion of additional physics such as magnetic fields.This was also pointed out in a recent study of cold fronts in relaxed galaxy clusters [3]. Using agrid-based approach will alleviate these concerns and will allow us to clearly identify the effects ofincluding different physical mechanisms in our model of the ICM.

6

3 Computational Approach

Programming Languages, Libraries, and Other Software UsedThe FLASH code [13] is a large, modern, component based application code for multiphysics

applications, particularly those relevant to astrophysical simulations. FLASH contains more thathalf a million lines of code in several languages, including FORTRAN90, C and Python. TheFLASH code is used on a wide variety of platforms, and has a fairly large external user base.The code uses MPI for parallel operations, and PARAMESH [17], for managing the discretizedcomputational domain. PARAMESH is a parallel oct-tree based block structured adaptive meshlibrary. FLASH also uses HDF5(NCSA), or PnetCDF(ANL) for optimized parallel IO.

Description of the Underlying Mathematical FormulationThe FLASH code primarily consists of a suite of high-resolution explicit numerical algorithms

for solving partial differential equations of classical and relativistic hydrodynamics and magneto-hydrodynamics. In order to simulate the collisionless dark matter component of galaxy clusters,FLASH implements a particle-mesh method for N -body calculations, which are simple ODE inte-grators for the positions and the velocities of the particles. The code also includes solvers for theelliptic equations of self-gravity. These solvers are used together to simulate self-gravitating flowsas proposed here.

Algorithms and Numerical Techniques EmployedThe hydrodynamic evolution is computed with the Piecewise Parabolic Method (PPM) [8],

a second order finite-volume method for solution of the Euler equations with specialized shock-capturing techniques. The evolution of the dark matter particles is computed using a second-orderleapfrog integrator with forces mapped to the particles from the mesh, and the particle mass isalso mapped to the mesh as the mass density of dark matter particles [15]. For solving the Poissonequation to find the gravitational field of the clusters, we employ a multigrid solver included withFLASH [30].

Parallel Programming System UsedPARAMESH uses MPI for parallel operations, and therefore all parallelism in FLASH is also

MPI based. In addition, for optimized parallel IO operations, either HDF5(NCSA), or PnetCDF(ANL)can be used. We will employ HDF5 parallel IO.

Visualization and AnalysisFor creating movies and plots for visualization of state variables such as density, temperature,

pressure, etc., we will use VisIt, a cross-platform parallel visualization software package developedat Lawrence Livermore National Laboratory. VisIt has been extensively developed to ensure com-patibility with the FLASH code data formats. VisIt is also scriptable, so large visualization jobsmay be run automatically.

However, the bulk of our analysis will be done with mock X-ray observations created from oursimulations. For real galaxy clusters, physical quantities such as density and temperature cannotbe measured directly but must be derived by fitting physically motivated models of the surfacebrightness profiles and spectra of the cluster gas to the distributions of photons detected by X-raytelescopes. We have developed an analysis pipeline that constructs simulated photon event filesbased on a model for X-ray emissivity for the intracluster plasma in our FLASH simulations. Bydoing so we can make a direct link to observations of real galaxy clusters.

7

10−3

0.01

0.1

norm

aliz

ed c

ount

s s−

1 keV

−1

data and folded model

1 2 5

−0.02

0

0.02

0.04

resi

dual

s

Energy (keV)

zuhone 11−Aug−2008 21:37

Figure 4: Products of mock cluster observations from a FLASH simulation of a merger of clustersof galaxies. Left panel: Smoothed X-ray image of clusters viewed along the line of sight of themerger. Right panel: Simulated spectrum and fitted model [39].

To construct our mock observations, we first create a flux map from a FLASH dataset. Wechoose a line of sight, and using a model for the emissivity of the cluster plasma project alongthis line of sight. This flux map is then used as a “distribution function” in sky coordinates andphoton energy for X-ray photons originating from this cluster. This flux map serves as input forsoftware that uses the Chandra response matrices to create simulated photon event files which arethe same format as that of real observations. With these files we may proceed to analyze oursimulated clusters in the same way as real clusters are analyzed, using the same battle-tested toolsas Chandra observers, such as CIAO and XSPEC.

Our process for generating these simulated observations is scriptable and therefore we canautomate and even parallelize the generation of many simulated event files and their subsequentanalysis. Our pipeline has undergone a significant verification study which is detailed in a recentpaper [39].

Other Resources AvailableLocally at the Harvard-Smithsonian Center for Astrophysics we will have workstations available

for visualization and analysis, and for off-site storage of data products. We will also have a 256-processor Linux cluster available for smaller-resolution runs. However, for the large, high-fidelitysimulations the resources of the TeraGrid allocation on the TACC Ranger Sun Constellation Systemwill be required, as detailed in the Job Characterization section. The computational resources ofthe FLASH Center at the University of Chicago are for the simulation of Type Ia supernovaeexplosions and are not available for research involving galaxy cluster mergers.

4 Job Characterization

We determine the total number of CPU hours required to complete a high-resolution simulationof a galaxy cluster merger. First, we note that the total CPU time tCPU in CPU-hours for anindividual run is

tCPU = tadvNstepsNblocks (1)

where Nsteps is the number of time steps taken, tadv is the CPU time required per block cycleadvance, and Nblocks is the total number of blocks in the computational domain. We proceed to

8

estimate tCPU by determining Nsteps, tadv, and Nblocks for this problem setup.The galaxy clusters will be initially established within a three-dimensional cubical domain of

length L on a side and volume L3, where L = 10 Mpc. For resolving shocks and cold fronts ade-quately, we choose a resolution of ∆x = 5 kpc for our simulations. Since we are using an adaptivemesh, this resolution is not constant over the entire grid but is resolved finely only over regionsof interest. These include regions of high mass density and sharp features in the density, temper-ature, and pressure of the gas. However, we restrict this latter refinement criterion to densitiesn <∼10−5 cm−3, well outside the current limits of detectability of the current X-ray observatories.Though the number of blocks will change throughout the simulation, we estimate the average inour simulations to be Nblocks = 50, 000 for our high fidelity simulations.

The time step within our simulations is dictated by two considerations: first, by the Courantcondition,

∆thydro = C ·mingrid

{∆xi

|vx,ijk|+ cs,ijk,

∆yj

|vy,ijk|+ cs,ijk,

∆zk

|vz,ijk|+ cs,ijk

}(2)

where C is the Courant number, cs is the local adiabatic sound speed, and v = {vx, vy, vz} is the fluidvelocity. The minimization is taken over all cells. The second consideration is the correspondingcondition for the active particles:

∆tpart = K · minparticles

{∆xi

|vpx|

,∆yj

|vpy |

,∆zk

|vpz |

}(3)

where K is a constant controlling the fraction of a cell traversed in one step, and vp = {vpx, vp

y , vpz} is

the particle velocity. The minimization here is taken over all particles. For each step, the minimumof these two timesteps will determine the timestep for the simulation. Typically, the minimumof the two timesteps will be the particle timestep, since particles in the high-velocity tail of thedistribution will tend to have speeds larger than the bulk speed and the adiabatic sound speed ofthe gas.

The runtime of the computation will vary depending on the stage of the merger that the observedcluster we are attempting to model is in, but a reasonable maximum value that we estimate willcover the interesting parts of the merger evolution will be tmax = 6 Gyr. We approximate thenumber of steps in the simulation as Nsteps = tmax/∆t.

Based on FLASH simulations on the Hera platform at Lawrence Livermore National Laboratory(a quad-core AMD Opteron Linux cluster with a theoretical peak system performance of 127.2TFLOP/s), we have determined that for this platform the CPU time per block per step requiredfor advance is tadv = 0.457 s. This machine has a similar configuration to the TACC Ranger SunConstellation System. To a first approximation, we may scale the advance time between Hera andRanger by taking the ratio of the clock speeds. The clock speed of the AMD processors on Herais 2.3 GHz, and the clock speed of the AMD processors on Ranger is 2.3 GHz, making this ratiounity.

Combining these results, and scaling to fiducial parameter values, we estimate the CPU time es-timate for a single high-fidelity galaxy cluster merger simulation evolved to the end of the simulationas

tCPU = 4.68 × 105 hrs(

tadv

0.457 s

) (vmax

3000 km/s

) (5 kpc∆x

) (0.5K

) (tmax

6 Gyr

) (Nblocks

50, 000

)(4)

For our high-fidelity simulations, we propose to run on 1024 processors. This implies a total wall-clock time for each simulation of

twall = 45.66 hours(

1024NCPUs

) (tadv

0.457 s

) (vmax

3000 km/s

) (5 kpc∆x

) (0.5K

) (tvir

6 Gyr

) (Nblocks

50, 000

)(5)

9

0

500

1000

1500

2000

100 1000

Aver

age

Tim

e / P

roce

ssor

[sec

onds

]

Number of Processors

Total EvolutionHydrodynamics

GravityParticles Advance

Particles CommunicationRefinement

Particles Refinement

Figure 5: Results of FLASHweak scaling test for thegalaxy cluster merger prob-lem, where the problem sizehas been increased by addingAMR blocks at the same level.

For our study of cold fronts in cluster mergers, we propose to complete a set of 18 high-fidelitysimulations of interesting cluster merger cases at a resolution of 5 kpc, in addition to the low-resolution simulations that will be required to choose the best initial conditions. From Equation 4,it is the case that an increase in resolution by a factor of two involves roughly a factor of sixteenincrease in CPU time (a factor of eight for the increase in block number multiplied by a factor of twofor the decrease in the timestep). Because of this, the total CPU time required is well-approximatedby the size of the 18 high-fidelity simulations. Therefore the total requirement for the study of coldfronts in clusters using the TeraGrid allocation on Ranger will be 9 million CPU-hours.

5 Parallel Performance

The FLASH code has been demonstrated to scale up to thousands of processors on many differentplatforms. Figure 1 demonstrates that as the work is increased by adding more AMR blocks at thesame refinement level, all components of the code for the galaxy cluster merger problem scale wellup to 2,048 processors on the Hera platform at LLNL, with the exception of the multigrid Poissonsolver for gravity, which has an increase in compute time per CPU of ∼30% between 64 and 2048CPUs.

The FLASH multigrid gravity solver begins the Poisson solve by performing an FFT solve on thetop-level block (level 1) which covers the entire domain. The multigrid solution is then performedfrom the top level block all the way up to the blocks at the highest level of refinement (typicallylevel 8 in ”high fidelity” simulations). This is not scalable to large numbers of processors becausethe blocks covering the domain at low levels (levels 1, 2, 3) are owned by only a small number ofprocessors. Therefore, processors which own no blocks or own relatively few blocks at these levelsare starved of work. Detailed analysis of the multigrid gravity unit demonstrates that as morerefinement levels are added (thus increasing the resolution), each level adds the same amount oftime to the computational cost.

In order to overcome the algorithm’s poor load balance, a hybrid multigrid/parallel FFT solveris currently being developed by the FLASH developers. This solver performs a parallel multi-block FFT solve on the highest refinement level that covers the whole computational domain, andbegins the multigrid solve from this coarse level. The parallel FFT algorithm used here has beendemonstrated to scale well to large numbers of processors [9]. There are two main advantages of

10

using this strategy: Firstly, the parallel FFT taking place at the coarse solve level is distributed overa large number of processors. Secondly, the cost of restriction and correction solves in refinementlevels lower than the coarse level are eliminated. Therefore, the resulting workload is more uniformamong the processors and should provide much better scaling.

6 Disk Space Requirements

Restart I/OAt regular intervals we will output a checkpoint file to the parallel scratch filesystem storing the

full state of the simulation in full numerical precision, from which the simulation may be restarted.Our checkpoint files are in HDF5 format, and we will use parallel I/O.

We estimate the size of the grid variables in a single checkpoint file in full precision as NcellNstate

x 8 bytes, where Nstate is the number of quantities stored in the simulation state vector. If thenumber of cells per block equals nxnynz, then Ncell = Nblocknxnynz. For Nstate = 20, Nblock =50, 000, and nxnynz = 4096 we estimate the size of the grid variables in the checkpoint file for ourhigh-fidelity simulations will be approximately 31 GB in size. In addition to these, the checkpointfile contains the active particles of the simulation which have a number of attributes. The totalsize of the stored particle attributes in double precision is NparticlesNattributes x 8 bytes, whereNattributes is the number of particle attributes which are outputted to the checkpoint file. ForNparticles = 2× 107 and Nattributes = 16, we estimate the size of the particle data in each checkpointfile to be 2.4 GB. Combined with the grid data, this makes for a total estimated checkpoint file tobe 33.4 GB in size.

We will keep a cadence of one checkpoint file every 4 hours, so for a ∼46 hour run this will resultin ∼12 checkpoint files. This results in some 400 GB of disk space for each of our 18 high-fidelitysimulations. We will build on the extensive experience of the FLASH Center to intelligently managedisk space. As the simulation is running, we will have in place automated pipeline software whichwill check for the existence of new output files. As time elapses, this software will initiate transfersof checkpoint files to mass storage and delete every other checkpoint file. Therefore, each high-fidelity simulation will require approximately 200 GB of parallel scratch disk space for checkpointfiles.

Thus the total parallel scratch storage required for checkpoint files for the 18 simulations onRanger will be 3.6 TB, and the total mass storage space required will be 7.2 TB.

Analysis I/OAt regular intervals during the course of each simulation we output plotfiles and particle files to

the parallel scratch filesystem from which we analyze the results of the simulation. These analysisfiles are in HDF5 format, and we will use parallel I/O.

The size of a single plotfile in single precision is NcellNplot x 4 bytes, where Nplot is the numberof quantities stored in the simulation state vector which are outputted to the plotfile. If the numberof cells per block equals nxnynz, then Ncell = Nblocknxnynz. For Nplot = 9, Nblock = 50, 000, andnxnynz = 4096 we estimate a single plotfile is 6.87 GB in size.

We also output particle files concurrently with the plotfiles, with a reduced number of parti-cle attributes compared with the checkpoint file, since for analysis we do not need the full set.The total size of the stored particle attributes in double precision is NparticlesNattributes x 8 bytes,where Nattributes is the number of particle attributes which are outputted to the particle file. ForNparticles = 2 × 107 and for Nattributes = 9, our estimate for the size of an average particle file is1.34 GB.

11

We will output 600 plotfiles and particle files per simulation, once every 10 Myr. For ourhigh-fidelity simulations, this is a total of approximately 4.93 TB of parallel scratch space for eachsimulation. To manage space, our plan is to keep no more than the data from three simulationson the parallel scratch space at the same time. Our pipeline software will detect new plotfiles andparticle files and copy them to mass storage, and will fork analysis software which will generatetwo-dimensional flux maps from which our simulated X-ray photons will be generated.

Our X-ray flux maps generated from our simulations are each 2-dimensional, so their total sizeis negligible with comparison to the size of our 3-dimensional simulation outputs.

Thus the total parallel scratch storage required for analysis files for the 18 high-fidelity simula-tions on Ranger will be ∼15.0 TB, and the total mass storage space required will be 90.0 TB.

References

[1] Agertz O, et al., 2007. Monthly Notices of the Royal Astronomical Society, 380:963.[2] Asai N, Fukuda N, and Matsumoto R, 2005. Advances In Space Research, 36:636.[3] Ascasibar Y and Markevitch M, 2006. The Astrophysical Journal, 650:102.[4] Carilli CL and Taylor GB, 2002. Annual Review of Astronomy and Astrophysics, 40:319.[5] Clarke TE, Blanton EL, and Sarazin CL, 2004. The Astrophysical Journal, 616:178.[6] Clowe D, et al., 2006. The Astrophysical Journal Letters, 648:109.[7] Cohn JD and White M, 2005. Astroparticle Physics, 24:316.[8] Colella P and Woodward PR, 1984. Journal of Computational Physics, 54:174.[9] Dubey A and Clune T, 2000. In Proceedings of Frontiers, 99.

[10] Dursi LJ, 2007. The Astrophysical Journal, 670:221.[11] Ettori S and Fabian AC, 2000. Monthly Notices of the Royal Astronomical Society, 317:57.[12] Forman W, et al., 1972. The Astrophysical Journal, 178:309.[13] Fryxell B, et al., 2000. The Astrophysical Journal Supplement Series, 131:273.[14] Govoni F, et al., 2004. The Astrophysical Journal, 605:695.[15] Hockney RW and Eastwood JW, 1988. Computer Simulation Using Particles. Taylor &

Francis, New York, NY.[16] Kellogg E, et al., 1972. The Astrophysical Journal, 174:65.[17] MacNeice P, et al., 2000. Computer Physics Communications, 126(3):330.[18] Mahdavi A, et al., 2007. The Astrophysical Journal, 668:806.[19] Markevitch M and Vikhlinin A, 2007. Physics Reports, 443:1.[20] Markevitch M, Vikhlinin A, and Mazzotta P, 2001. The Astrophysical Journal Letters,

562:153.[21] Markevitch M, et al., 2000. The Astrophysical Journal, 541:542.[22] Markevitch M, et al., 2002. The Astrophysical Journal Letters, 567:27.[23] Markevitch M, et al., 2004. The Astrophysical Journal, 606:819.[24] Markevitch M, et al., 2005. The Astrophysical Journal, 627:733.[25] Mitchell RJ, et al., 1976. Monthly Notices of the Royal Astronomical Society, 175:29.[26] Monaghan JJ, 1992. Annual Review of Astronomy and Astrophysics, 30:543.[27] Nagai D and Kravtsov AV, 2003. The Astrophysical Journal, 587:514.[28] Poole GB, et al., 2006. Monthly Notices of the Royal Astronomical Society, 373:881.[29] Randall SW, et al., 2008. The Astrophysical Journal, 679:1173.

12

[30] Ricker PM, 2008. The Astrophysical Journal Supplement Series, 176:293.[31] Ricker PM and Sarazin CL, 2001. The Astrophysical Journal, 561:621.[32] Ritchie BW and Thomas PA, 2002. Monthly Notices of the Royal Astronomical Society,

329:675.[33] Sarazin CL, 1988. X-ray emission from clusters of galaxies. Cambridge University Press,

Cambridge, MA.[34] Serlemitsos PJ, et al., 1977. The Astrophysical Journal, 211:63.[35] Springel V and Farrar GR, 2007. Monthly Notices of the Royal Astronomical Society, 380:911.[36] Sunyaev RA and Zeldovich YB, 1972. Comments on Astrophysics and Space Physics, 4:173.[37] Tittley ER and Henriksen M, 2005. The Astrophysical Journal, 618:227.[38] Weisskopf MC, et al., 2002. Publications of the Astronomical Society of the Pacific, 114:1.[39] ZuHone JA, et al., 2008. arXiv:08080930.[40] Zwicky F, 1937. The Astrophysical Journal, 86:217.

13