Cornell Theory Center Aug 15 2000

CCTKThe Cactus Computational

ToolkitWerner Benger

Max-PIanck-Institut für Gravitationsphysik(Albert-Einstein-Institute at Golm/Potsdam – AEI)

andKonrad-Zuse-Center for

Information Technology Berlin (ZIB)Werner.Benger@aei-potsdam.mpg.de

Introduction

Cactus: Original Motivation Numerical Relativity

Ongoing Research

What is the Cactus Computational Toolkit? Cactus Design - Flesh and Thorns

What does Cactus provide?

The Cactus Framework How to use Cactus

Interfacing Cactus

Future Developments Metacomputing for Numerical Relativity

Plans...

Cactus - Original Motivation: Numerical Relativity

Kepler: stable ellipsesEinstein: no stable orbits because of gravitational radiation(GR required for extreme gravity)

Astronomy with gravitational wave detectors (taken into operation by this year 2000!)

The Two-Body problem in General Relativity is still unsolved.

Axial symmetric Collisions of Black Holes (NCSA 1993-96)

Pre-Cactus (H-Code, G-Code), Cactus 1.0, 2.x, Cactus 3.xUS - NSF Grand Challenge Projects

Requirement for Collaborative Code

A distorted single black hole

Two Black Holes

Focus of 3D Numerical Evolutions during the last couple years

Black Holes (prime source for GW) – from Misner BH collisions to grazing BH collisions with initial spin and momentum (Brandt-Bruegmann initial data)

Gravitational Waves (Evolution of Brill Waves, collapse of pure GW, investigation of critical amplitude, i.e. when do black holes form)

Neutron Stars (NASA Neutron Star Grand Challenge, GR hydrodynamics, neutron stars colliding to black holes,...)

Visualization of 3D data with ZIB's Amira

Efforts for Remote Visualization, Remote Monitoring, Remote Steering German Gigabit Project (TIKSL), KDI Portal

German Gigabit Project: TIKSL

Remote Visualization

Remote Steering

Online Monitoring

Globus Services

HDF5 Interface

Data Grid Access (DPSS)

What is the Cactus Computational Toolkit?

Portable Application Framework

"Flesh" provides registration and scheduling facilities

Memory management ("gridarrays","gridfunctions")

"Thorns" are exchangeable code segments

Runtime activation/deactivation of thorns

I/O layers (parallel I/O)

Parameter handling

Exchangeable Parallelization layers (MPI, Globus-MPI, Shared Memory,...)

Data Types (Portability)

Cactus data types to provide portability across platforms

CCTK_REAL CCTK_REAL4, CCTK_REAL8, CCTK_REAL16

CCTK_INT CCTK_INT2, CCTK_INT4, CCTK_INT8

CCTK_CHAR

CCTK_COMPLEX CCTK_COMPLEX8, CCTK_COMPLEX16, CCTK_COMPLEX32

Cactus Flesh

Make System Organizes builds as configurations which hold everything needed to build

with a particular set of options on a particular architecture.

API Functions which must be there for thorns to operate.

Scheduler Sophisticated scheduler which calls thorn-provided functions as and when

needed.

CCL Configuration language which tells the flesh all it needs to know about the

thorns.

Data Structures (Memory Management)

Grid Arrays A multidimensional and arbitrarily sized array distributed among

processors

Grid Functions A field distributed on the multidimensional computational grid (a Grid

Array sized to the grid)

– Every point in a grid may hold a different value “f(x,y,z)”

Grid Scalars Values common to all the grid points

Parameters Values/Keywords that affect the behavior of the code (initialization,

evolution, output, etc..)

– parameter checking, steerable parameters

Cactus Thorns

Flesh (core) written in C Thorns (modules) grouped in packages written in F77, F90, C, C++ Thorn-Flesh interface fixed in 3 files written in CCL (Cactus

Configuration Language): interface.ccl: Grid Functions, Arrays, Scalars (integer, real, logical, complex) param.ccl: Parameters and their allowed values schedule.ccl: Entry point of routines, dynamic memory and communication

allocations

Object oriented features for thorns (public, private, protected variables, implementations, inheritance) for clearer interfaces

Compilation: PERL parses the CCL files and creates the flesh-thorn interface code at

compile time Particularly important for the FORTRAN-C interface. FORTRAN arg. lists

must be known at compile time, but depend on the thorn list

Interface

The concept: contract with the rest of the code Now it is only for the data structures: variables and parameters adding routines and arguments

Private The variables that you want the flesh to allocate/communicate but

no other thorn to see.

Public The variables that you want everybody to see (that means that

everybody can modify them too!) Inheritance

Protected Variables that you want only your friends to see!

Implementation

Why Two or more thorns that provide the same functionality but different

internal implementation– Interchangeable pieces that allow easy comparison and evolution in

the development process

– They are compiled together and only one is activated at runtime

How If all the other thorns need to see the same contract, then thorns

implementing a certain functionality must– Have the same public and protected variables

– The same concept applies to parameters and scheduling

Example Wildly different evolution approaches for the same equations, so all

the analysis and initial data thorns remain the same.

Scheduling

Thorns schedule when their routines should be executed

Basic evolution skeleton idea standard scheduling points INITIAL, EVOL, ANALYSIS fine control: run this routine BEFORE/AFTER that routine

Extend/customize with scheduling groups Add my routine to this group of routines Run the group WHILE some condition is met

Future redesign The scheduler is really a runtime selector of the computation flow. Much more power can be added to this concept

Parallelizing an Application Thorn

CCTK_SyncGroup– synchronize ghostzones for a group of grid variables

– just add Synchronization to Scheduler configuration file as well

CCTK_Reduce– call any registered reduction operator, e.g. maximum value over the

grid

CCTK_Interpolate– call any registered interpolation operator

CCTK_MyProc– unique processor number within the computation

CCTK_nProcs– total number of processors

CCTK_Barrier– waits for all processors to reach this point

PUGH

The standard parallel driver supplied with Cactus is supplied by thorn PUGH

Driver thorn: Sets up grid variables, handles processor decomposition, deals with processor communications

1,2,3D Grid Arrays and Grid Functions (beta6)

Uses MPI

Custom processor decomposition

Otherwise decomposes in z, then y, then x directions

How to use Cactus

[Optional: Develop thorns, according to some rules e.g. specify variables through interface.ccl

Specify calling sequence of the thorn subroutines for given problem and algorithm (schedule.ccl)]

Specify which thorns are desired for simulation (e.g. Einstein equations + special method 1 +HRSC hydro+wave finder + AMR + live visualization module + remote steering tool…)

Specified code is then created, with only those modules, those variables, those I/O routines, this MPI layer, that AMR system,…, needed (minimal binary)

Subroutine calling lists generated automatically

Automatically created for desired computer architecture

Run it…(local, remote, on the Grid using Globus environment)

How does Cactus help ?

Collaborative working

Different people are experts on different parts of the physical problem

Each person can write a thorn which solves their part, e.g. the metric evolution, hydrodynamics, apparent horizon finding, ...

Thorns are encapsulated, and different thorns with the same functionality are interchangeable

Parallelism

Cactus provides a parallel layer

Layer is independent of underlying machine architecture

Researchers don't need to think deeply about parallelism Choice of parallel library layers (Native MPI, MPICH, MPICH-G(2), LAM,

WMPI, PACX, HPVM, MPIPro)

...

IO Cactus provides optimized IO layers for the various machines

Possible to output very large datasets in a short time

Parallel I/O (Parallel HDF5, Panda, John Shalf's FlexIO - various interfaces to MPI-I/O)

Checkpointing A lot of runs take more time than queing systems allow

Cactus provides mechanisms to dump out the entire state of the simulation and then to read it in again either on the same machine or another

Platform independence

Cactus provides a platform independent environment

Various "strange compilation" issues on different machines are already handled by the CCTK environment

CACTUS on NT

The NT Supercluster was used in three demos and two of the HPC challenges at SC ’98 , held in Orlando, Florida,

November 9-13, 1998.http://www.ncsa.uiuc.edu/General/CC/ntcluster/sc98/

Who Uses Cactus

Numerical Relativity AEI

WashU

NCSA

Penn State

UIB

Southampton

PRL

...

Computer Science Panda IO Project (UIUC)

Globus (Argonne)

Cluster evaluation

– Roadrunner (UNM)

– NT cluster (NCSA)

– ...

Autopilot (UIUC)

Gigabit Project (DFN)

Horizons (A Metacomputing Application)

Singularity at center hidden by event horizon Surface through which nothing in the interior can escape

Event horizon only really detectable if we have the whole spacetime

Not possible while running the simulation

Apparent horizon always within event horizon Many methods to detect this

If a horizon is found Definitely have a black hole Can compute Gaussian Curvature to inspect oscillations and correlation to

emitted gravitational waves

Distributed Computation of AH

SpaceTime Evolution

gmn t, K mn t

Apparent Horizon Computation

AH(t=9.0)

AH(t=11.0)

AH(t=16.0)

RZG CRAY T3E, 512 Nodes(Garching/Munich)

ZIB T3E, 16 Nodes

AEI Origin 2000, 16 Nodes

Cornell NT Cluster, 16 Nodes

GlobusServices

All the required technique (TIKSL NetHDF5, Globus MDS Queries, ...) is already there!

Cactus communication layer Parallel driver thorn (e.g. PUGH) currently provides both variable

management and communication … abstract send and receives etc

Abstract communication from driver thorn easily implement different parallel paradigms shared memory, threads, Corba, OpenMP, PVM, ...

Compact groups (different layout in memory for improved Cache performance)

Unstructured Meshes/Finite Elements/Spectral Methods Unstructured Multigrid Solver Convergence/Multiple Coordinate Patches Capability browsing mechanism Command line interface … connect directly to Cactus,

scheduling GUIs, Documentation, GUIs, Documentation ….

Coming up (Cactus 4.2)

What physical systems are there now ?

Initial Data Schwarzschild

Misner

Brandt-Bruegmann puncture data

Brill waves

Teukolsky waves

Distorted Brill wave and black hole

TOV

Colliding neutron stars

Orbiting neutron stars

Boson star

Dust

...

Analysis Apparent horizon finders

fast flow

minimisation

Wave extraction

Riemann Invariants

Newman-Penrose quantities

Constraint evaluation

Geodesic tracking

Overview

Introduction Cactus - Original Motivation: Numerical Relativity Axial symmetric Collisions of Black Holes

(NCSA 1993-96) Focus of 3D Numerical Evolutions

during the last couple years Visualization of 3D data with ZIB's Amira German Gigabit Project: TIKSL

Overview, II

What is the Cactus Computational Toolkit? Data Types (Portability) Cactus Flesh Data Structures (Memory Management) Cactus Thorns Interface Implementation Scheduling Parallelizing an Application Thorn PUGH How to use Cactus How does Cactus help ? CACTUS on NT Who Uses Cactus Horizons (A Metacomputing Application) Distributed Computation of AH