Proceedings of the 7th Python in Science Conference (SciPy 2008)

Analysis and Visualization of Multi-Scale Astrophysical Simulations UsingPython and NumPyMatthew Turk (mturk@slac.stanford.edu) – KIPAC / SLAC / Stanford, USA

The study the origins of cosmic structure requireslarge-scale computer simulations beginning withwell-constrained, observationally-determined, initialconditions. We use Adaptive Mesh Refinementto conduct multi-resolution simulations spanningtwelve orders of magnitude in spatial dimensions andover twenty orders of magnitude in density. Thesesimulations must be analyzed and visualized in amanner that is fast, accurate, and reproducible. Ipresent "yt," a cross-platform analysis toolkit writ-ten in Python. "yt" consists of a data-managementlayer for transporting and tracking simulation out-puts, a plotting layer, a parallel analysis layer forhandling mesh-based and particle-based data, aswell as several interfaces. I demonstrate how theorigins of cosmic structure – from the scale of clus-ters of galaxies down to the formation of individ-ual stars – can be analyzed and visualized using aNumPy-based toolkit. Additionally, I discuss effortsto port this analysis code to other adaptive meshrefinement data formats, enabling direct compari-son of data between research groups using differentmethods to simulate the same objects.

Analysis of Adaptive Mesh RefinementData

I am a graduate student in astrophysics, studying theformation of primordial stars. These stars form fromthe collapse of large gas clouds, collapsing to higherdensities in the core of extended star-forming regions.Astrophysical systems are inherently multi-scale, andthe formation of primordial stars is the best exam-ple. Beginning with cosmological-scale perturbationsin the background density of the universe, one mustfollow the evolution of gas parcels down to the massscale of the moon to have any hope of resolving theinner structure and thus constrain the mass scale ofthese stars.In order to do this, I utilize a code designed to in-sert higher-resolution elements within a fixed mesh, viaa technique called adaptive mesh refinement (AMR).Enzo [ENZ] is a freely-available, open source AMRcode originally written by Greg Bryan and now devel-oped through the Laboratory for Computational As-trophysics by a multi-institution team of developers.Enzo is a patch-based multi-physics AMR/N-body hy-brid code with support for radiative cooling, multi-species chemistry, radiation transport, and magneto-hydrodynamics. Enzo has been used to simulate awide range of astrophysical phenomena, such as pri-mordial star formation, galaxy clusters, galaxy forma-tion, galactic star formation, black hole accretion and

jets from gamma ray bursts. Enzo is able to insert upto 42 levels of refinement (by factors of two) allowingfor a dynamic range between cells of up to 242. Onthe typical length scale of primordial star formation,this allows us to resolve gas parcels on the order of ahundred miles, thus ensuring the simulations fully re-solve at all times the important hydrodynamics of thecollapse.A fundamental but missing aspect of our analysispipeline was an integrated tool that was transparentlyparallelizable, easily extensible, freely distributable,and built on open source components, allowing for fullinspection of the entire pipeline. My research advisor,Prof. Tom Abel of Stanford University, suggested Iundertake the project of writing such a tool and ap-proach it from the standpoint of attacking the problemof extremely deep hierarchies of grid patches.Initially, yt was written to be a simple interface be-tween AMR data and the plotting package “Hippo-Draw,” which was written by Paul Kunz at the Stan-ford Linear Accelerator Center [HIP]. As time passed,however, it moved more toward a different mode of in-teraction, and it grew into a more fully-featured pack-age, with limited data management, more abstract ob-jects, and a full GUI and display layer built on wx-Python [WX] and Matplotlib [MPL], respectively. Uti-lizing commodity Python-based packages, I present afully-featured, adaptable and versatile means of ana-lyzing large-scale astrophysical data. It is based pri-marily on the library NumPy [NPY], it is mostly writ-ten in Python, and it uses Matplotlib, and optionallyPyTables and wxPython for various sub-tasks. Addi-tionally, several pieces of core functionality have beenmoved out to C for fast numerical computation, and aTVTK-based [TVTK] visualization component is be-ing developed.

Development Philosophy

From its beginning, yt has been exclusively free andopen source software, and I have made the decisionthat it will never require components that are not opensource and freely available. This enables it to be dis-tributed, not be dependent on licensing servers, andto make available to the broader community the workput forth by me, the other developers, and the broadercontributing community toward approachable analysisof the data. The development has been driven, andwill continue to be driven, by my needs, and the needsof other developers.Furthermore, no feature that I, or any other memberof the now-budding development team, implement willbe hidden from the community at large. This philoso-phy has served the toolkit well already; it has already

M. Turk: Proc. SciPy 2008, G. Varoquaux, T. Vaught, J. Millman (Eds), pp. 46–51 46

Proceedings of the 7th Python in Science Conference (SciPy 2008)

been examined and minor bugs have been found andcorrected.In addition to these commitments, I also sought theability to produce publication-quality plots and to re-duce the difficulty of multi-step operations. The usershould be presented with a consistent, high-level, inter-face to the data, which will have the side-effect of en-abling different entry points to the toolkit as a whole.The development of yt takes place in a publicly acces-sible subversion repository with a Trac frontend [YT].Sphinx-based documentation is available, and auto-matically updated from the subversion repository asit is checked in. In order to ease the process of in-stallation, a script is included to install the entire setof dependencies along with the toolkit; furthermore,installations of the toolkit are maintained at severaldifferent supercomputing centers, and a binary versionfor Mac OS X is provided.

Organization

To provide maximum flexibility, as well as a concep-tual separation of the different components and tasksto which components can be directed, yt is packagedinto several sub-packages.The analysis layer, lagos, provides several features be-yond mere data access, including extensive analyticalcapabilities. At its simplest level, lagos is used toaccess the parameters and data in a given time-basedoutput from an AMR simulation. However, on topof that, different means of addressing collections ofdata are provided, including from an intuitive object-oriented perspective, where objects are described byphysical shapes and orientations.The plotting layer, raven, has capabilities for plottingone-, two- and three-dimensional histograms of quan-tities, allowing for weighting and binning of those re-sults. A set of pixelization routines have been writtenin C to provide a means of taking a set of variable-sizepixels and constructing a uniform grid of values, suit-able for fast plotting in Matplotlib - including caseswhere the plane is not axially perpendicular, allow-ing for oblique slices to be plotted and displayed withpublication-quality rendering. Callbacks are availablefor overlaying analytic solutions, grid-patch bound-aries, vectors, contours and arbitrary annotation.Additionally, several other sub-packages exist that ex-tend the functionality in various different ways. Thedeliverator package is a Turbogears-based [TG] im-age gallery that listens for SOAP-encoded informationabout images on the web, fido stores and retrievesdata outputs, and reason is the wxPython-based [WX]GUI.

Object Design and Protocol

One of the difficulties in dealing with rectilinear adap-tive mesh refinement data is the fundamental discon-nect between the geometries of the grid structure and

the objects described by the simulation. One does notexpect galaxies to form and be shaped as rectangularprisms; as such, access to physically-meaningful struc-tures must be provided. To that end, yt provides thefollowing:

• Sphere

• Rectangular prism

• Cylinder / disk

• “Extracted” regions based on logical operations

• Topologically-connected sets of cells

Each of these regional descriptors is presented to theuser as a single object, and when accessed the data isreturned at the finest resolution available; all overlap-ping coarse grid cells are removed transparently. Thiswas first implemented as physical structures resem-bling spheres were to be analyzed, followed by disk-likestructures, each of which needed to be characterizedand studied as a whole. By making available these in-tuitive and geometrically meaningful data selections,the underlying physical structures that they trace be-come more accessible to analysis and study.The objects are designed so that code snippets such asthe following are possible:

>>> sp = amr_hierarchy.sphere(... center, radius)>>> print sp["Density"].min()>>> L_vec = sp.quantities["AngularMomentumVector"]()>>> my_disk = amr_hierarchy.disk(center,... L_vec, radius, radius/100.0)>>> print my_disk["Density"].min()

The abstraction layer is such that there are severalmeans of interacting with these three-dimensional ob-jects, each of which is conceptually unified, and whichrespects a given set of data protocols. Due to the flex-ibility of Python, as well as the versatility of NumPy,this functionality has been easily exposed in the formof multiple returned arrays of data, which are fast andeasily manipulated. Above can be seen the calcula-tion of the angular momentum vector of a sphere, andthen the usage of that vector to construct a disk witha height relative to the radius.These objects handle cell-based data fields natively,but are also able to appropriately select and returnparticles contained within them. This has facilitatedthe inclusion of an off-the-shelf halo finder, which al-lows users to quantify the clustering of particles withina region.In addition to the object model, a flexible interface toderived data fields has been implemented. All fields,including derived fields, are allowed to be defined byeither a component of a data file, or a function thattransforms one or more other fields, thus allowing mul-tiple layers of definition to exist, and allowing the userto extend the existing field set as needed. Furthermore,these fields can rely on the cells from neighboring gridpatches - which will be generated automatically by ytas needed - which enables the creation of fields thatrely on finite-difference stencils.

47 http://conference.scipy.org/proceedings/SciPy2008/paper_11

Analysis and Visualization of Multi-Scale Astrophysical Simulations Using Python and NumPy

A two-dimensional phase diagram of the distributionof mass in the Density-Temperature plane for a col-lapsing gas cloud

The combination of derived fields, physically-meaningful data objects and a unique data-access pro-tocol enables yt to construct essentially arbitrary rep-resentations of arbitrary collections of data. For in-stance, the user is able to take arbitrary profiles of dataobjects (radial profiles, probability distribution func-tions, etc) in one, two and three dimensions. Thesecan be plotted from within the primary interface, andthen output in a publication-ready format.

Two-Dimensional Data Representations

In order to make images and plots, yt has severaldifferent classes of two-dimensional data representa-tions, all of which can be turned into images. Eachof these objects generates a list of variable-resolutionpoints, which are then passed into a C-based pix-elization routine that transforms them into a fixed-resolution buffer, defined by a width, a height, andphysical boundaries of the source data.

The simplest means of examining data is through theusage of axially-parallel slices through the dataset.This has several benefits - it is easy to calculate whichgrids and which cells are required to be read off disk(and most data formats allow for easy “striding” ofdata off disk, which reduces this operation’s IO over-head) and it is easy to automate the process to stepthrough a given dataset.

However, at some length scales in star formation prob-lems, gas is likely to collapse into a disk, which is oftennot aligned with the axes of the simulation. By slicingalong the axes, patterns such as spiral density wavescould be missed, and ultimately go unexamined. Inorder to better visualize off-axis phenomena, I imple-mented a means of creating an image that is misalignedwith the axes.

An oblique slice through the center of a star forma-tion simulation. The image plane is normal to theangular momentum vector.

This “cutting plane” is an arbitrarily-aligned planethat transforms the intersected points into a new co-ordinate system such that they can be pixelized andmade into a publication-quality plot. This techniquerequired a new pixelization routine, in order to ensurethat the correct voxels were taken and placed on theplot, which required an additional set of checks to de-termine if the voxel intersected with the image plane.The nature of adaptive mesh refinement is such thatone often wishes to examine either the sum of valuesalong a given sight-line or a weighted-average along agiven sight-line. yt provides an algorithm for gener-ating line integrals in an adaptive fashion, such thatevery returned (x, y, v, dx, dy) point does not containdata from any points where dx < dxp or dy < dyp.We do this in a multi-step process, operating on eachlevel of refinement in turn. Overlap between gridsis calculated, such that, along the axis of projection,each grid is associated with a list of grids that it over-laps with on at least one cell. We then iterate overeach level of refinement, starting with the coarsest,constructing lists of both “further-refinable” cells and“fully-refined” cells. A combination step is conducted,to combine overlapping cells from different grids; all“further-refinable” cells are passed to the next level asinput to the overlap algorithm, and we continue re-cursing down the level hierarchy. The final projectionobject, with its variable-resolution cells, is returned tothe user.Once this process is completed, the projection objectrespects the same data protocol, and can be plotted inthe same way, as an ordinary slice.

Contour Finding

Ofttimes, one needs to identify collapsing objects byfinding topologically-connected sets of cells. The na-ture of adaptive mesh refinement, where in a givenset cells may be connected across grid and refinementboundaries, requires sophisticated means for such iden-tification.

http://conference.scipy.org/proceedings/SciPy2008/paper_11 48

Proceedings of the 7th Python in Science Conference (SciPy 2008)

Unfortunately, while locating topologically-connectedsets inside a single-resolution grid is a straightforwardbut non-trivial problem in recursive programming, ex-tending this in an efficient way to hierarchical datasetscan be problematic. To that end, the algorithm imple-mented in yt checks on a grid-by-grid basis, retrievingan additional set of cells at the grid boundary. Anycontour that crosses into these ’ghost zones’ mandatesa reconsideration of all grids that intersect with thecurrently considered grid. This process is expensive,as it operates recursively, but ensures that all contoursare automatically joined.Once contours are identified, they are split into indi-vidual derived objects that are returned to the user.This presents an integrated interface for generatingand analyzing topologically-connected sets of relatedcells. In the past, yt has been used to conduct thisform of analysis and to study fragmentation of col-lapsing gas clouds, specifically to examine the gravi-tational boundedness of these clouds and the scales atwhich fragmentation occurs.

Parallel Analysis

As the capabilities of supercomputers grow, the size ofdatasets grows as well. In order to meet these chang-ing needs, I have been undertaking an effort to par-allelize yt to run on multiple independent processingunits. Specifically, I have been utilizing the MessagePassing Interface (MPI) via the MPI4Py [MPI] mod-ule, a lightweight, NumPy-native wrapper that enablesnatural access to the C-based routines for interprocesscommunication. My goal has been to preserve at alltimes the API, such that the user can submit an un-changed serial script to a batch processing queue, andthe toolkit will recognize it is being run in parallel anddistribute tasks appropriately.The tasks in yt that require parallel analysis can be di-vided into two different broad categories: those tasksthat can act on data in an unordered, uncorrelatedfashion, and those tasks that act on a decomposed im-age plane.To parallelize the unordered analysis, a set of iteratorshave been implemented utilizing an initialize/finalizestructure. Upon initialization of the iterator, it callsa method that determines which sets of data will beprocessed by which processors in the MPI group. Theiteration proceeds as normal, and then, before the Sto-pIteration exception is raised, it finalizes by broad-casting the final result to every processor. The un-ordered nature of the analysis allows the grids to beordered such that disk access is minimized; on high-performance file systems, this results in close-to-idealscaling of the analysis step.

Constraints of Scale

In order to manage simulations consisting of multiplehundreds of thousands of discrete grid patches - as well

as their attendant grid cell values - I have undertakenoptimization using the cProfile module to locate andeliminate as many bottlenecks as possible. To thatend, I am currently in the process of reworking the ob-ject instantiation to rely on the Python feature ’slots,’which should speed the process of generating hundredsof thousands of objects. Additionally, the practice ofstoring data about simulation outputs between instan-tiation of the Python objects has been extended; thisspeeds subsequent startups, and enables a more rapidresponse time.

Enzo data is written in one of three ways, the mostefficient -and prevalent- way being via the Hierarchi-cal Data Format (HDF5) [HDF] with a single file perprocessor that the simulation was run on. To limit theeffect that disk access has on the process of loadingdata, hand-written wrappers to the HDF5 have beeninserted into the code. These wrappers are lightweight,and operate on a single file at a time, loading data inthe order it has been written to the disk. The packagePyTables was used for some time, but the instantia-tion of the object hierarchy was found to be too muchoverhead for the brief and well-directed access desired.

Frontends and Interfaces

yt was originally intended to be used from the com-mand line, and images to be viewed either in a webbrowser or via an X11 connection that forwarded theoutput of an image viewer. However, a happy side-effect of this attitude - as well as the extraordinarilyversatile Matplotlib “Canvas” interface - is that the ytAPI, designed to have an a single interface to analysistasks, is easily accessed and utilized by different inter-faces. By ensuring that this API is stable and flexible,GUIs, web-interfaces, and command-line scripts can beconstructed to perform common tasks.

A typical session inside the GUI

For scientific computing as a whole, such flexibility isinvaluable. Not all environments have access to thesame level of interactivity; for large-scale datasets, be-ing able to interact with the data through a script-ing interface enables submission to a batch process-ing queue, which enables appropriate allocation of re-sources. For smaller datasets, the process of interac-tively exploring datasets via graphical user interfaces,

49 http://conference.scipy.org/proceedings/SciPy2008/paper_11

Analysis and Visualization of Multi-Scale Astrophysical Simulations Using Python and NumPy

exposing analytical techniques not available to an of-fline interface, is extremely worthwhile, as it can behighly immersive.The canonical graphical user interface is written in wx-Python, and presents to the user a hierarchical listingof data objects: static outputs from the simulation, aswell as spatially-oriented objects derived from thoseoutputs. The tabbed display pane shows visual rep-resentations of these objects in the form of embeddedMatplotlib figures.Recently an interface to the Matplotlib ’pylab’ inter-face has been prepared, which enables the user to in-teractively generate plots that are thematically linked,and thus display an uniform spatial extent. Furtherenhancements to this IPython interface, via the pro-file system, have been targeted for the next release.Knoboo [KBO] has been identified as a potential web-based interface, in the same style as Sage. It is alightweight software package designed to display ex-ecuted Python code in the browser but to conduct theexecution on the backend. With a disjoint web-serverand execution kernel model, it enables the frontend tocommunicate with a remote kernel server where thedata and analysis packages would reside. Because ofits flexibility in execution model, I have already beenable to conduct analysis remotely using Knoboo as myuser interface. I intend to continue working with theKnoboo developers to enhance compatibility betweenyt and Knoboo, as web-based interfaces are a powerfulway to publish analysis as well as to enable collabora-tion on analysis tasks.

Generalization

As mentioned above, yt was designed to handle andanalyze data output from the AMR code Enzo. Dr.Jeff Oishi of Berkeley is leading the process of convert-ing the toolkit to work equally well with data fromother AMR codes; however, different codes make sep-arate sets of assumptions about outputted data, andthis must be generalized to be non-Enzo specific.In this process, we are having to balance a desire forgeneralization with a desire for both simplicity andspeed. To that extent, we are attempting to make min-imally invasive changes where possible, and rethinkingaspects of the code that were not created in the mostgeneral fashion.By providing a unified interface to multiple, often com-peting, AMR codes, we will be able to utilize similar -if not identical - analysis scripts and algorithms, whichwill enable direct comparison of results between groupsand across methods.

Future Directions

As the capabilities of yt expand, the ability to ex-tend it to perform new tasks extend as well. Recently,the beginnings of a TVTK-based frontend were im-plemented, allowing for interactive, physically-oriented

3D visualization. This relies on the vtkCompositeDat-aPipeline object, which is currently weakly supportedacross the VTK codebase. However, the power ofTVTK as an interface to VTK has significant promise,and it is a direction we are targeting.

A visualization within yt, using the TVTK toolkit tocreate 3D isocontours and cutting planes.

Work has begun on simulated observations from large-scale simulations. The first step toward this is simu-lating optically thin emissions, and then utilizing ananalysis layer that operates on 2D image buffers.By publishing yt, and generalizing it to work on mul-tiple AMR codebases, I hope it will foster collabora-tion and community efforts toward understanding as-trophysical problems and physical processes, while fur-thermore enabling reproducible research.

Acknowledgments

I’d like to acknowledge the guidance of, first and fore-most, my PhD advisor Prof. Tom Abel, who inspiredme to undertake this project in the first place. Addi-tionally, I’d like to thank Jeff Oishi, a crucial memberof the development team, and Britton Smith and BrianO’Shea, both of whom have been fierce advocates forthe adoption of yt as a standard analysis toolkit.Much of this work was conducted at Stanford Univer-sity and the Kavli Institute for Particle Astrophysicsand Cosmology, and was supported (in part) by U.S.Department of Energy contract DE-AC02-76SF00515.

References

[ENZ] http://lca.ucsd.edu/projects/enzo[HIP] http://www.slac.stanford.edu/grp/ek/

hippodraw/[WX] http://wxpython.org/[MPL] http://matplotlib.sf.net/[NPY] http://numpy.scipy.org/[TVTK] http://svn.enthought.com/enthought/wiki/

TVTK[YT] http://yt.enzotools.org/[TG] http://turbogears.org/[MPI] http://mpi4py.scipy.org/[HDF] http://hdfgroup.org/[KBO] http://knoboo.com/

http://conference.scipy.org/proceedings/SciPy2008/paper_11 50