PRO

OF

Tech Science Press Paper CMES Galley Proof Only Please Return in 48 Hours.

Parallel Octree-Based Finite Element Method for Large-Scale EarthquakeGround Motion Simulation

J. Bielak 1, O. Ghattas 2, and E.-J. Kim 3

Abstract: We present a parallel octree-based finite el-ement method for large-scale earthquake ground motionsimulation in realistic basins. The octree representa-tion combines the low memory per node and good cacheperformance of finite difference methods with the spa-tial adaptivity to local seismic wavelengths characteris-tic of unstructured finite element methods. Several testsare provided to verify the numerical performance of themethod against Green’s function solutions for homoge-neous and piecewise homogeneous media, both with andwithout anelastic attenuation. A comparison is also pro-vided against a finite difference code and an unstruc-tured tetrahedral finite element code for a simulation ofthe 1994 Northridge Earthquake. The numerical tests allshow very good agreement with analytical solutions andother codes. Finally, performance evaluation indicatesexcellent single-processor performance and parallel scal-ability over a range of 1 to 2048 processors for North-ridge simulations with up to 300 million degrees of free-dom.

keyword: Earthquake ground motion modeling, oc-tree, parallel computing, finite element method, elasticwave propagation

1 Introduction

Wave propagation simulations for earthquake-inducedground motion have been performed for over 30 yearsto gain a better understanding of the distribution of theearthquake motion in urban regions in space and time.Such insight has contributed to the development of build-ing codes in which a seismic-prone region is divided intodifferent zones of comparable seismic hazard, with thegoal of reducing seismic risk. The dramatic improve-ment in supercomputing performance has more recentlyenabled seismologists and earthquake engineers to more

1 Carnegie Mellon University, Pittsburgh, PA, USA.2 University of Texas at Austin, Austin, TX, USA.3 Duke University, Durham, NC, USA.

accurately understand the effects of source, wave prop-agation, and local site conditions on the ground motion.In particular, using parallel computers with several thou-sand processors, it has now become possible to modelground motion in large, highly heterogeneous basins,such as the Los Angeles (LA) basin, with sufficient reso-lution to capture frequencies of interest, for realistic ge-ological models.

An earthquake ground motion simulation entails solvingnumerically the elastodynamic wave equations. Thereare several numerical methods available for ground mo-tion simulations. The finite difference method (FDM),the boundary element method (BEM) and the finite ele-ment method (FEM) are commonly used. In seismologyand earthquake engineering, the FDM has been the mostpopular technique due to its satisfactory accuracy, ease ofimplementation, and low memory needed per grid point[e.g., Virieux (1984); Levander (1988); Graves (1996);Pitarka, Irikura, Iwata, and Sekiguchi (1998)]. A numberof earthquake ground motion simulations in the greaterLA basin have been computed using the FDM [e.g., Vi-dale and Helmberger (1988); Frankel and Vidale (1992);Yomogida and Etgen (1993); Schrivner and Helmberger(1994); Olsen and Archuleta (1996); Graves (1998)].However, for heterogeneous media with large contrastsin material stiffness, the (conventional form of the) FDMsuffers due to its reliance on a regular grid, as illustratedin Figure 1a. The regular grid is not capable of adaptingto the local wavelengths of propagating waves, which areshorter for softer materials. Thus, grid resolution is dic-tated by the shortest wavelengths, which results in over-refined grids in stiffer regions. In addition, material inter-faces are resolved with O(h) geometric error, unless theinterfaces are axis-aligned. Finally, the time step in anexplicit time integrator (which is most commonly usedin wave propagation) is artificially small to accommo-date the CFL stability condition in the over-refined stiffregions.

The main advantage of the BEM is its unique ability

PRO

OF

12 Tech Science Press CMES Galley Proof Only Please Return in 48 Hours.

(a) Regular grid (b) FEM

(c) Octree

Figure 1 : Examples of spatial discretization of abasin for different numerical methods. (a) Regular grid[Pitarka, Irikura, Iwata, and Sekiguchi (1998)]. (b) Un-structured tetrahedral mesh [Bao, Bielak, Ghattas, Kalli-vokas, O’Hallaron, Schewchuk, and Xu (1998)]. (c) Oc-tree mesh [Kim, Bielak, and Ghattas (2003)].

to provide a complete solution in terms of boundaryvalues only, with substantial savings in modeling ef-fort. Zang and Chopra (1991) used a BEM approachto study the 3D response of a canyon in an elastichalf-space. Others have investigated different types ofintegral equations and their numerical treatment [e.g.,Sanchez-Sesma (1978); Wong (1979), Sanchez-Sesmaand Esquivel (1979); Wong (1982); Dravinski (1982);Sanchez-Sesma, Bravo, and Herrera (1985); Kawase(1988)]. But the BEM is not appropriate for groundmotion simulations in heterogeneous basins, since it re-quires piecewise-homogeneity of the domain.

On the other hand, the FEM is designed to be effectivefor heterogeneous media. Unlike the FDM, the FEM iscapable of adapting the mesh to local features of the so-lution, as shown in Figure 1b. Thus over-refinement isavoided and longer time steps can be taken, since theCFL stability limit is on the order of that required foraccuracy. These advantages become more dramatic withincreasing contrast in shear wave velocity in the hetero-geneous medium. However, the FEM comes with severaldisadvantages relative to FDM: generating an appropri-

ate wavelength-adaptive mesh can require considerableeffort; storing the system stiffness matrix requires con-siderable memory, since the “stencil” varies from nodeto node; and indirect addressing associated with unstruc-tured meshes results in generally poorer cache perfor-mance. In fact, these disadvantages are not intrinsic tothe FEM; they are associated with unstructured meshmethods in general. For regular grids, the FEM can beimplemented with all of the advantages of the FDM. Be-cause of the difficulties in meshing complex heteroge-neous basins, there are fewer examples of FEM-basedearthquake ground motion simulations [e.g., Toshinawaand Ohmachi (1988); Li, Bielak, and Ghattas (1992);Bao, Bielak, Ghattas, Kallivokas, O’Hallaron, Shew-chuk, and Xu (1996); Bao, Bielak, Ghattas, Kallivokas,O’Hallaron, Schewchuk, and Xu (1998); Komatitsch andTromp (1999); Yoshimura, Bielak, Hisada, and Fernan-dez (2003); Akcelik, Bielak, Biros, Epanomeritakis, Fer-nandez, Ghattas, Kim, Lopez, O’Hallaron, Tu, and Ur-banic (2003)].

In this article, we present, verify, and demonstrate anoctree-based finite element method for seismic wavepropagation that overcomes the difficulties of the un-structured mesh-based FEM cited above [Kim (2003)].The octree method combines the adaptivity of the FEMto local wavelengths with the low memory requirementsand good cache performance associated with the FDM.The keys are an octree data structure for generating, rep-resenting, and refining the mesh (as illustrated in Fig-ure 1c); a matrix-free implementation enabled by theregular shape of the elements; and enforcement of al-gebraic constraints at hanging nodes to maintain a con-forming approximation across element interfaces. Sec-tion 2 describes the ingredients of the octree method,while Section 3 discusses the discretization and solutionof the elastic wave propagation equation. Incorporationof anelastic attenuation is described in Section 4. Sec-tion 5 presents numerical verification tests of the octreemethod, while Section 6 provides performance and scal-ability results to 2048 processors.

2 Octree-based finite element method

Octrees have been used as a basis for finite element ap-proximation since at least the early 90s [Young, Melvin,Bieterman, Johnson, Samant, and Bussoletti (1991)].Our interest in octrees stems from their ability to adaptto the wavelengths of propagating seismic waves while

PRO

OF

CMES Galley Proof Only Please Return in 48 Hours. 13

maintaining a regular shape of finite elements. here,leaves associated with the lowest level of the octree areidentified with trilinear hexahedral finite elements andused for a Galerkin approximation of a suitable weakform of the elastic wave propagation equation. The hexa-hedra are recursively subdivided into 8 elements until alocal refinement criterion is satisfied. For seismic wavepropagation in heterogeneous media, the criterion is thatthe longest element edge should be such that there re-sult at least p nodes per local shear wavelength, as de-termined by the local shear wave velocity β and themaximum frequency of interest fmax. In other words,hmax < β

p fmax. For trilinear hexahedra and taking into ac-

count the accuracy with which we know typical basinproperties, we take typically p = 10. An additional con-dition that drives mesh refinement is that the element sizenot differ by more than a factor of two across neighboringelements (the octree is then said to be balanced). Notethat as in finite difference methods, the octree does notexplicitly represent material interfaces within the earth,and instead accepts O(h) error in representing them im-plicitly. This is usually justified for earthquake model-ing since the location of interfaces is known at best tothe order of the seismic wavelength. If needed, higher-order accuracy in representing arbitrary interfaces can beachieved by local adjustment of the finite element basis[e.g., Young, Melvin, Bieterman, Johnson, Samant, andBussoletti (1991)].

Figure 2 depicts the octree mesh (and its 2D counter-part, a quadtree). The left drawing illustrates a factor-of-two edge length difference (a “legal” refinement) anda factor-of-four difference (an “illegal” refinement). Un-less additional measures are taken, so-called hangingnodes that separate different levels of refinement (in-dicated by solid circles and the subscript h in the fig-ure) result in a possibly discontinuous field approxima-tion, which can destroy the convergence properties of theGalerkin method. Several possibilities exist to remedythis situation by enforcing continuity of displacementfield across the interface either strongly (e.g., by con-struction of special transition elements) or weakly (e.g.,via mortar elements or discontinuous Galerkin approxi-mation). The simplest technique is to enforce continuityby algebraic constraints that require the displacement ofthe hanging node be the average of the displacement ofits anchored neighbors (indicated by open circles and thesubscript a). As illustrated in Figure 2, the displacement

uuui

a uuuh uuu ja

legal

illegal

uuuhuuuh

uuuia uuuk

a

uuu ja uuul

a

hanging node

anchored node

Figure 2 : Quadtree and Octree-Based Mesh

of an edge hanging node, uuuh, should be the average of itstwo edge neighbors uuui

a and uuu ja, and the displacement of a

face hanging node, uuuh, should be the average of its fourface neighbors uuui

a, uuu ja, uuuk

a, and uuula. Efficient implementa-

tion of these algebraic constraints will be discussed in thenext section. As evident from the figure, when the octreeis balanced, an anchored node cannot also be a hangingnode.

Our earlier earthquake modeling code was based onan unstructured mesh data structure and linear tetrahe-dral finite elements [Bao, Bielak, Ghattas, Kallivokas,O’Hallaron, Shewchuk, and Xu (1996); Bao, Bielak,Ghattas, Kallivokas, O’Hallaron, Schewchuk, and Xu(1998)]. Compared to that approach, the present octree-based method has several important advantages:

• The octree meshes are much more easily generatedthan general unstructured tetrahedral meshes, par-ticularly when the number of elements increasesabove 50 million. We use an efficient out-of-core octree-based hexahedral mesh generator [Tu,O’Hallaron, and Lopez (2002); Tu and O’Hallaron(2004)] that can generate meshes of sizes thatare limited only by available disk space. (Sinceeach basin is meshed just once for a given resolu-tion of interest—but subjected to many earthquakescenarios—mesh generation can be done off-line.)

• The hexahedra provide somewhat greater accuracyper node (the asymptotic convergence rate is un-changed, but the constant is typically improved overtetrahedral approximation).

• The hexahedra all have the same form of the ele-ment stiffness matrices, scaled simply by element

PRO

OF

14 Tech Science Press CMES Galley Proof Only Please Return in 48 Hours.

size and material properties (which are stored asvectors), and thus no matrix storage is required atall. This results in a substantial decrease in requiredmemory—about an order of magnitude, comparedto our node-based tetrahedral code.

• Because of the matrix-free implementation, (stiff-ness) matrix-vector products are carried out at theelement level. This produces much better cache uti-lization by relegating the work that requires indirectaddressing (and is memory bandwidth-limited) tovector operations, and recasting the majority of thework of the matrix-vector product as local element-wise dense matrix computations. The result is a sig-nificant boost in performance.

These features permit earthquake simulations to substan-tially higher frequencies and lower resolved shear wavevelocities than heretofore possible. In the next section,we describe the octree-based discretization and solutionof the elastic wave equation.

3 Wave propagation model and octree discretization

We model seismic wave propagation in the earth viaNavier’s equations of linear elastodynamics. Let uuu rep-resent the vector field of the three displacement compo-nents, λ and µ the Lame moduli and ρ the density distri-bution, bbb a time-dependent body force representing theseismic source, and ttt =

[µ(∇uuu+∇uuuT

)+λ(∇ ·uuu)I

]nnn the

surface traction vector. Let Ω be an open bounded do-main in R

3 with free surface ΓFS, truncation boundaryΓAB, and outward unit normal to the boundary nnn. Theinitial–boundary value problem is then written as:

ρuuu−∇ ·[µ(

∇uuu+∇uuuT)

+λ(∇ ·uuu)I]

= bbb

in Ω× (0,T ) ,nnn×nnn×ttt = nnn×nnn× uuu

√ρµ on ΓAB × (0,T) ,

nnn ·ttt = nnn · uuu√

ρ(λ+2µ) on ΓAB × (0,T) ,

ttt = 000 on ΓFS × (0,T ) ,uuu = 000 in Ω×t = 0 ,

uuu = 000 in Ω×t = 0 , (1)

With this model, p waves propagate with velocity α =√(λ+2µ)/ρ, and s waves with velocity β =

√µ/ρ. The

continuous form above does not include material atten-uation, which we introduce at the discrete level via a

Rayleigh damping model, as discussed below. The vec-tor bbb comprises a set of body forces that equilibrate aninduced displacement dislocation on a fault plane, pro-viding an effective representation of earthquake ruptureon the plane. For example, for a seismic excitation ide-alized as a point source, bbb = −µvAMMM f (t)∇∇∇δ(xxx−ξξξ) [Akiand Richards (1980)]. In this expression, v is the averageearthquake dislocation; A the rupture area; MMM the (nor-malized) seismic moment tensor, which depends on theorientation of the fault; f (t) the (normalized) time evolu-tion of the rupture; and ξξξ the source location.

Since we model earthquakes within a portion of the earth,we require appropriately positioned absorbing bound-aries to account for the truncated exterior. For simplicity,in (1) the absorbing boundaries are given as dashpots onΓAB, which approximate the tangential and normal com-ponents of the surface traction vector ttt with time deriv-atives of corresponding components of the displacementvector. Even though this absorbing boundary is approx-imate, it is local in both space and time, which is partic-ularly important for large-scale parallel implementation.Finally, we enforce traction-free conditions on the freesurface of the earth.

We apply standard Galerkin finite element approxima-tion in space to the appropriate weak form of the initial-boundary value problem (1). Let U be the space of ad-missible solutions (which depends on the regularity ofbbb), Uh be a finite element subspace of U, and vvvh be a testfunction from that subspace. Then the weak form is writ-ten as follows.Find uuuh ∈ Uh such that

Ω

[ρuuuh ·vvvh +

µ2

(∇uuuh +∇uuuT

h

)·(

∇vvvh +∇vvvTh

)]dxxx

+

Ω[λ(∇ ·uuuh)(∇ ·vvvh)−bbb ·vvvh] dxxx

=

ΓAB

√ρµ(nnn×nnn× uuuh) · (nnn×nnn×vvvh) dsss

+

ΓAB

√ρ(λ+2µ)(nnn · uuuh) (nnn ·vvvh) dsss, ∀vvvh ∈ Uh. (2)

Finite element approximation is effected via piecewisetrilinear basis functions and associated trilinear hexahe-dral elements on an octree mesh. This strikes a balancebetween simplicity, low memory (since all element stiff-ness matrices are the same modulo scale factors), andreasonable accuracy.4

4 The output quantities of greatest interest are displacements and

PRO

OF

CMES Galley Proof Only Please Return in 48 Hours. 15

Upon spatial discretization, we obtain a system of ordi-nary differential equations of the form

MMMuuu+(CCCAB +CCCatt)uuu+KKKuuu = bbb,

uuu(000) = 000,

uuu(000) = 000, (3)

where MMM and KKK are mass and stiffness matrices, arisingfrom the terms involving ρ and (µ,λ) in (2), respectively;bbb is a body force vector resulting from a discretizationof the seismic source model; and damping matrix CCCAB

reflects contributions of the absorbing boundaries. Wehave also introduced the damping matrix CCCatt to simulatethe effect of energy dissipation due to anelastic materialbehavior; it consists of a linear combination of mass andstiffness matrices and its form will be discussed in thenext section.

The time dimension is discretized using central differ-ences. The algorithm is made explicit using a diagonal-ization scheme that (1) lumps the mass matrix MMM, theabsorbing boundary matrix CCCAB, and the mass compo-nent of the the material attenuation matrix CCCatt (all ofwhich have Gram structure and are therefore spectrallyequivalent to the identity), and (2) splits the diagonal andoff-diagonal portions of the stiffness component of CCCatt ,time-lagging the latter. The resulting update for the dis-placement field at time step k +1 is given by[MMM +

∆t2

CCCABdiag +

∆t2

CCCattdiag

]uuuk+1

=[2MMM−∆t2KKK − ∆t

2CCCAB

off −∆t2

CCCattoff

]uuuk

+[

∆t2

CCCAB +∆t2

CCCatt −MMM

]uuuk−1 +∆t2bbbk. (4)

The time increment ∆t must satisfy a local CFL conditionfor stability. Space is discretized over the octree mesh(each leaf corresponds to a hexahedral element) that re-solves local seismic wavelengths as discussed above.This insures that the CFL-limited time step is of the orderof that needed for accuracy, and that excessive dispersionerrors do not arise due to over-refined meshes.

Spatial discretization via refinement of an octree pro-duces a non-conforming mesh, resulting in a discontin-uous displacement approximation. We restore continuityof the displacement field across refinement interfaces by

velocities, as opposed to stresses.

imposing algebraic constraints that require the displace-ment at a hanging node to be consistent with the approx-imation along the neighboring element face or edge, asdiscussed above. We can express these algebraic conti-nuity constraints in the form

uuu = BBBuuu,

where uuu denotes the displacements at the non-hanging(i.e., independent) nodes, and BBB is a sparse constraint ma-trix. In particular, BBBi j = 1

4 if hanging node i is a faceneighbor of anchored node j and 1

2 if it is an edge neigh-bor; BBBi j = 1 simply identifies a non-hanging node; andBBBi j = 0 otherwise. Rewriting the linear system (4) as

AAAuuuk+1 = ccc(uuuk,uuuk−1)

we can impose the continuity constraints via the projec-tion

BBBTAAABBBuuuk+1 = BBBTccc(uuuk,uuuk−1). (5)

The reduced matrix BBBTAAABBB is then further lumped to ren-der it diagonal, so that the constrained update (5) is ex-plicit. In fact the resulting reduced matrix can be con-structed simply by dividing the contributions of the hang-ing nodes and adding them in equal portions to diagonalcomponents of the corresponding anchored nodes. Therighthand side of (5) is determined at each time step bycomputing ccc(uuuk,uuuk−1), i.e. the righthand side of (4), inan element-by-element fashion, and then applying the re-duction BBBTccc. This amounts again to dividing the contri-butions of the hanging components of ccc equally amongthe corresponding anchored nodes. The work involvedin enforcing the constraints is proportional to the num-ber of hanging nodes, which can be a sizable fraction ofthe overall number of nodes for a highly irregular octree,but is at most of O(N). Therefore, the per-iteration com-plexity of the update (5) remains linear in the number ofnodes.

The combination of an octree-based wavelength-adaptivemesh, piecewise trilinear Galerkin finite elements inspace, explicit central differences in time, constraints thatenforce continuity of the displacement approximation,and local-in-space-and-time absorbing boundaries yieldsa second-order-accurate in time and space method that iscapable of scaling up to the very large problem sizes thatare required for high resolution earthquake modeling. Inthe next section we discuss incorporation of anelastic at-tenuation.

PRO

OF

16 Tech Science Press CMES Galley Proof Only Please Return in 48 Hours.

4 Anelastic Attenuation

Many studies have found that anelastic attenuation playsan important role in earthquake-induced ground mo-tion modeling in sedimentary basins such as Los An-geles because elastic waves may be overamplified bytrapped waves that are reflected and refracted withinbasins [Frankel and Vidale (1992); Yomogida and Et-gen (1993); Olsen and Archuleta (1996); Olsen, Nig-bor, and Konno (2000); Wald and Graves (1998); Pitarka,Irikura, Iwata, and Sekiguchi (1998); Sato, Graves, andSomerville (1999)]. There are several difficulties in in-corporating attenuation within time domain seismic wavepropagation simulations, including the large memorynecessary to store additional variables that are usuallyrequired to represent anelastic attenuation, and the dif-ficulty in estimating the quality factor (Q) for highly het-erogeneous media. Day and Bradley (2001) have sug-gested a coarse-grained implementation of the memoryvariables that improves performance in terms of accu-racy of solution and efficiency of memory use. More-over, Olsen, Day, and Bradley (2003) have demonstratedan effective Q model for the LA basin in the context ofsimulations of the 1994 Northridge earthquake.

In this study, we instead use a Rayleigh damping modelbecause of its convenience, simplicity, and consistencywith our existing seismic wave propagation code. Here,the element damping matrix is taken proportional to ele-ment mass and stiffness matrices,

CCCe = κ1MMMe +κ2KKKe, (6)

where CCCe is the element material damping matrix, MMMe isthe element mass matrix, KKKe is the element stiffness ma-trix, and κ1 and κ2 are scalar coefficients that are to bedetermined element-wise with the goal that the dampingratio for each mode is independent of the frequency overa given frequency range. Pre- and post-multiplying (6)by the nth eigenvector,

φTnCCCeφn = κ1φT

n MMMeφn +κ2φTn KKKeφn, (7)

and making use of orthonormality and the modal damp-ing assumption, we obtain

2ωnζn = κ1 +κ2ω2n (8)

or

ζn =1

2ωnκ1 +

ωn

2κ2, (9)

0 500 1000 1500 2000 2500 3000 3500 40000

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

shear wave velocity (m/s)

dam

ping

rat

io (

%)

Figure 3 : Relation between damping ratio and shearwave velocity

where the damping ratio ζn is equivalent to 12Q and ωn

is the natural frequency associated with the nth mode.Assuming that the damping ratio ζ varies with the inverseof the shear wave velocity, β [as in Bao, Bielak, Ghattas,Kallivokas, O’Hallaron, Schewchuk, and Xu (1998)],

ζ =γ1

β+ γ2, (10)

where γ1 and γ2 are scalars determined by assuming thatζ is 2% at the shear velocity β of 200 m/s while ζ is 0.5%at β of 1000 m/s. This yields γ1 = 5.333 and γ2 = 66.67,for which the relation between the damping ratio and theshear wave velocity is shown in Figure 3. For any givenshear velocity, then, (10) determines the desired dampingratio. Our goal is to find the Rayleigh constants κ 1 and κ2

such that (9) yields as close to this desired damping ra-tio as possible over a realistic frequency range (ω1,ω2).This can be formulated as the linear least squares prob-lem:

min(κ1,κ2)

ω2

ω1

(ζ− κ1

2ω− κ2ω

2

)2dω (11)

For example, solving this least squares problem for ζ =0.5% over the frequency range (0.1,1.1) Hz gives the plotin Figure 4 of ζ as determined by (9). The near-constantζ, and thus Q, over the frequency range of interest, is ev-ident in the figure. The Rayleigh damping model, which

PRO

OF

CMES Galley Proof Only Please Return in 48 Hours. 17

0 0.2 0.4 0.6 0.8 1 1.20

0.5

1

1.5

2

2.5

3

3.5

4

frequency (Hz)

ζ, d

ampi

ng r

atio

(%

)

effective frequency range: 0.1 Hz − 1.1 Hz

target damping ratio: 0.5 %

Figure 4 : Relation between damping ratio and fre-quency for an optimized Rayleigh damping model

increases both linearly and inversely with frequency, pro-vides a reasonable damping model for many soils, al-though very low and very high frequencies are over-damped.

5 Numerical Tests for Validation

To verify our octree-based seismic wave propagationcode, we have conducted a number of numerical testsranging from idealized models to realistic basins withvarious properties and excitations Kim (2003). Here, wepresent four representative examples. In all cases, themesh is generated with the out-of-core octree/hexahedralmesh generator Euclid [Tu, O’Hallaron, and Lopez(2002); Tu and O’Hallaron (2004)], and partitionedwith the parallel mesh partitioner ParMETIS [Karypis,Schloegel, and Kumar (2002)].

First, we solve a layered half space problem with pointsource excitation. The problem description is given inTable 1. Figure 5a illustrates the location of the pointsource excitation.

The medium is piecewise-homogeneous, consisting of ahomogeneous elastic layer over an elastic half space. Theexcitation is defined as a double-couple, located in thehalf space as a point source. The mesh size is chosen sothat shear wavelengths corresponding to a frequency of1 Hz are resolved with 10 nodes per wavelength. Thisresults in an element size in the half space of 450 m

Table 1 : Layered half space problem descriptiondomain size: 36×36×18 km3

depth of layer: 1.8 kmlayer: ρ = 2500 kg/m3

α = 3900 m/sβ = 2250 m/s

half space: ρ = 3048 kg/m3

α = 7800 m/sβ = 4500 m/s

excitation: M0[ tT0− 1

2πsin(2π tT0

)] 0 ≤ t ≤ T0

M0 t > T0

strike, dip, rake: 30, 40, 60

magnitude (M0): 1.4×1013 Nmrise time (T0): 1 s

source location: (−225 m, 225m, −6075 m)

and in the top layer of 225 m, resulting in a mesh with435,200 elements, 469,485 total nodes, and 19,360 hang-ing nodes. The time step size is 0.02 s for the 40 s sim-ulation. A Cartesian coordinate system is chosen with xpositive in the East (E) direction, y positive in the North(N) direction, and z positive in the Upward (U) direction.This axis convention applies to the remaining examples.Figures 5b, 5c, and 5d provide a comparison of the oc-tree FEM solution with that from an analytical Green’sfunction code.5 We compare at 8 observation points nearthe top surface along a diagonal from the epicenter. Bothnumerical and analytical results are low-pass filtered to 1Hz, the target resolution. These results show good graph-ical agreement between the two solutions.

To investigate the influence of mesh irregularity due tooctree refinement, we construct a mesh that is adaptedto the shear wave velocities of the Southern CaliforniaEarthquake Center (SCEC) model version 2.2 [Magis-trale, Day, Clayton, and Graves (2000)] for a frequencyof 0.2 Hz. We replace the spatially-variable SCEC ve-locity model with a homogeneous material model in or-der to compare with the Green’s function solution. Table2 lists the problem data. Figure 6 depicts the irregularmesh for this model. The ratio of the largest element size(h = 1250 m) and smallest element size (h = 156.25 m) is8, corresponding to an octree with four levels. Compar-isons with Green’s function solutions are shown in Figure7a, 7b, and 7c for a number of observation points that are

5 The Green’s function code was written by Y. Hisada.

PRO

OF

18 Tech Science Press CMES Galley Proof Only Please Return in 48 Hours.

Hypocenter

18 k

m18

km

18 km 18 km

12 k

m6

km

12 km6 km

x

z

x

y y

z

Isometric view inthe simulation model

front view

top view side view

x

y

z

Observation Points

1. ( 0, 0, -50)2. ( 2000, 2000, -50)3. ( 4000, 4000, -50)4. ( 6000, 6000, -50)5. ( 8000, 8000, -50)6. (10000, 10000, -50)7. (12000, 12000, -50)8. (14000, 14000, -50)

12

34

56

78

Hypocenter(-225, 225 -6075)

Strike = 30 (deg)Dip = 40 (deg)

Rake = 60 (deg)

(a) Layered half space model 0 5 10 15 20 25 30 35 40

1

2

3

4

5

6

7

8

E−W velocity (m/s): point source

loca

tion

time(sec)

10−5 m/s

GREEN−elasticQUAKE−elastic

(b) E-W component of response

0 5 10 15 20 25 30 35 40

1

2

3

4

5

6

7

8

N−S velocity (m/s): point source

loca

tion

time(sec)

10−5 m/s

GREEN−elasticQUAKE−elastic

(c) N-S component of response

0 5 10 15 20 25 30 35 40

1

2

3

4

5

6

7

8

U−D velocity (m/s): point source

loca

tion

time(sec)

10−5 m/s

GREEN−elasticQUAKE−elastic

(d) U-D component of response

Figure 5 : Comparison of octree (dotted) with Green’s function (solid) solution for layered half space problem.(a) Problem description. (b) E-W component of velocity. (c) N-S component of velocity. (d) U-D component ofvelocity.

located in a similar pattern as in the half space problem.Results are filtered to 0.2 Hz. The comparison with theanalytical solution is again very good, suggesting correcttreatment of the mesh size transitions.

To examine the implementation of anelastic attenuation,we use the same model that was used for the layered halfspace problem (Figure 5a). We choose damping ratiosζ1 = 15% (Qs = Qp = 3.33) for the layer and ζ2 = 3%(Qs = Qp = 16.67) for the half space. The damping ra-

tios are high for the given properties; in fact our goal isto verify the anelastic implementation with the Green’sfunction solution for large attenuation. This comparisonis shown in Figure 8a, 8b, and 8c and again the agreementis very good.

Finally, we perform a simulation of the 1994 Northridgeearthquake in the LA Basin and compare our octree-based FEM results with synthetic seismograms from theFDM code of Graves (1998) and our earlier tetrahe-

PRO

OF

CMES Galley Proof Only Please Return in 48 Hours. 19

Table 2 : Half space with irregular octree meshdomain size: 80×80×30 km3

half space: ρ = 2000 kg/m3

α = 5000 m/sβ = 2500 m/s

excitation: M0[ tT0− 1

2πsin(2π tT0

)] 0 ≤ t ≤ T0

M0 t > T0

strike, dip, rake: 30, 40, 60

magnitude (M0): 1.4×1013 Nmrise time (T0): 1 s

source location: (40.625 km, 40.625 km, −14.375 km)

0 10 20 30 40 50 60

1

2

3

4

5

6

7

8

9

E−W component

Time (sec)

Loca

tion

0.1 × 10−3 cm/s

GREENQUAKE

(a) E-W component of response

0 10 20 30 40 50 60

1

2

3

4

5

6

7

8

9

N−S component

Time (sec)

Loca

tion

0.1 × 10−3 cm/s

GREENQUAKE

(b) N-S component of response

0 10 20 30 40 50 60

1

2

3

4

5

6

7

8

9

U−D component

Time (sec)

Loca

tion

0.2 × 10−3 cm/s

GREENQUAKE

(c) U-D component of response

Figure 7 : Comparison of octree (dotted) with Green’s function (solid) solution for irregular mesh of LA Basin, withhomogeneous properties. (a) E-W component of velocity. (b) N-S component of velocity. (c) U-D component ofvelocity.

0 5 10 15 20 25 30 35 40

1

2

3

4

5

6

7

8

9

E−W velocity (m/s): ζ1 = 0.15, ζ

2 = 0.03

loca

tion

time(sec)

10−5 m/s

QUAKE−anelasticGREEN−anelastic

(a) E-W component of response

0 5 10 15 20 25 30 35 40

1

2

3

4

5

6

7

8

9

N−S velocity (m/s): ζ1 = 0.15, ζ

2 = 0.03

loca

tion

time(sec)

10−5 m/s

QUAKE−anelasticGREEN−anelastic

(b) N-S component of response

0 5 10 15 20 25 30 35 40

1

2

3

4

5

6

7

8

9

U−D velocity (m/s): ζ1 = 0.15, ζ

2 = 0.03

loca

tion

time(sec)

10−5 m/s

QUAKE−anelasticGREEN−anelastic

(c) U-D component of response

Figure 8 : Comparison of octree (dotted) with Green’s function (solid) solution for anelastic model of layered halfspace problem. (a) E-W component of velocity. (b) N-S component of velocity. (c) U-D component of velocity.

PRO

OF

20 Tech Science Press CMES Galley Proof Only Please Return in 48 Hours.

Table 3 : Performance analysis of octree code for 1994 Northridge earthquake simulation

Model 0.1 Hz 0.2 Hz 0.5 Hz 1.0 Hz 1.0 Hz 1.0 Hz

Procs 1 16 128 512 1024 2048

Nodes (×106) 0.134 0.618 14.8 47.6 102 102

MFLOPS/s 505 491 469 451 450 443

Par Effcny 100% 97% 93% 89% 89% 88%

CPU Effcny 25.3% 24.6% 23.5% 22.6% 22.5% 22.2%

Figure 6 : Irregular mesh for LA basin model

dral code [Bao, Bielak, Ghattas, Kallivokas, O’Hallaron,Shewchuk, and Xu (1996); Bao, Bielak, Ghattas, Kalli-vokas, O’Hallaron, Schewchuk, and Xu (1998)]. The di-mensions of the basin model are 80 × 80 × 30 km3. Ma-terial properties are taken from the SCEC velocity modeland the fault rupture model is based on Wald, Heaton,and Hudnut (1996). The maximum resolved frequencyis 0.5 Hz. Anelastic attenuation is based on Q factorsgiven by Olsen, Day, and Bradley (2003). The resultingmesh has approximately 12 million elements and 15 mil-lion nodes. Figure 9 compares synthetic results from thethree codes at a number of observation stations. The syn-thetic seismograms have been low-pass filtered using aButterworth filter with 4 poles and 2 passes. The overallagreement once again is very good.

6 Performance and scalability

Our octree-based finite element code has achieved excel-lent performance and isogranular scalability over a rangeof 1 to 2048 processors on the HP AlphaServer Cluster

at the Pittsburgh Supercomputing Center. Table 3 pro-vides details on the performance of the code for a seriesof simulations of the 1994 Northridge earthquake in theLA basin corresponding to highest resolved frequencyranging from 0.1 Hz (run on 1 processor) to 1 Hz (runon 2048 processors). Problem size varies from 134,500to over 100 million nodes. Scalability is excellent: 88%parallel efficiency is maintained in scaling from 1 to 2048processors. The scalar efficiency is also very good, de-grading to just 22.2% of peak on each Alpha node (basedon 2 GFLOPS/s peak). This latter figure is very good fora modern microprocessor, considering the irregular na-ture of the computations. The 2048 processor simulationsustains nearly a TFLOPS/s.

7 Concluding Remarks

We have presented a parallel octree-based finite elementmethod for large-scale earthquake ground motion simu-lation in realistic basins. The octree representation com-bines the low memory per node and good cache perfor-mance of finite difference methods with the spatial adap-tivity to local seismic wavelengths characteristic of un-structured finite element methods. Several tests are pro-vided to verify the numerical performance of the methodagainst Green’s function solutions for homogeneous andpiecewise homogeneous media, both with and withoutanelastic attenuation. A comparison is also providedagainst a finite difference code and an unstructured tetra-hedral finite element code for a simulation of the 1994Northridge Earthquake. The numerical tests all showvery good agreement with analytical solutions and othercodes. Finally, performance evaluation indicates excel-lent single-processor performance and parallel scalabil-ity over a range of 1 to 2048 processors for Northridgesimulations with up to 300 million degrees of freedom.

Acknowledgement: Two words stand out in our

PRO

OF

CMES Galley Proof Only Please Return in 48 Hours. 21

SSA

N−S URSCorp

ARCHM

QUAKE17 cm/s

E−W

14 cm/s

U−D

8 cm/s

60 sec

JFP N−S URSCorp

ARCHM

QUAKE255 cm/s

E−W

86 cm/s

U−D

96 cm/s

60 sec

PKC

N−S URSCorp

ARCHM

QUAKE33 cm/s

E−W

28 cm/s

U−D

13 cm/s

60 sec

MCN

N−S

8 cm/s

E−W

3 cm/s

U−D

5 cm/s

TAR

N−S

15 cm/s

E−W

6 cm/s

U−D

8 cm/s

VLA N−S

12 cm/s

E−W

11 cm/s

U−D

6 cm/s

HSL N−S

16 cm/s

E−W

29 cm/s

U−D

11 cm/s

OBG

N−S

12 cm/s

E−W

14 cm/s

U−D

6 cm/s

IGU

N−S

9 cm/s

E−W

9 cm/s

U−D

6 cm/s

LBL

N−S

4 cm/s

E−W

6 cm/s

U−D

3 cm/s

DWY

N−S

9 cm/s

E−W

10 cm/s

U−D

7 cm/s

Figure 9 : Comparison of synthetic seismograms at 11 stations for 1994 Northridge Earthquake simulation. Topseismogram corresponds to FDM simulation; middle to tetrahedral FEM; bottom to octree FEM.

PRO

OF

22 Tech Science Press CMES Galley Proof Only Please Return in 48 Hours.

minds when we think of Cliff Astill: vision and integrity.

Soon after moving to his new position as Program Di-rector of Ground Motion and Geotechnical EarthquakeEngineering at NSF in the late 1980s, Cliff came to therealization that, along with experimentation and directfield observation, model-based simulation would have toplay a leading role in research, along with experimenta-tion and direct field observation, if real advances were tobe made in the understanding and prediction of the earth-quake behavior of complex geotechnical systems. In hisunassuming, yet highly effective style, he was instrumen-tal in creating VELACS, a multi-investigator project forthe validation of liquefaction analysis using centrifugestudies. This was the first large-scale concerted effort toevaluate the utility of numerical codes that included porepressure development and liquefaction modeling abilitiesto function as real-life design codes. Through this andother activities, Cliff became one of the primary forces insetting the course of geotechnical earthquake engineeringfor the next generation.

In our own work, Cliff provided a willing ear, and ini-tially his own discretionary funds, to support our ex-ploratory work on what would later become a major ef-fort on earthquake ground motion simulation in largebasins. Even though we were neophytes at the time, withhis wisdom and encouragement he made us feel from thevery beginning as if we knew more than we did, therebygiving us the confidence to actually achieve somethinguseful. Most appropriately, Cliff later served as NSFcognizant officer for both the Grand Challenges (CMS-9318163) and KDI (CMS-9980063) projects. The lattersupported the bulk of the underlying work for the presentpaper.

While our initial relationship was purely professional,over the years it grew into a real friendship. Throughit, we came to know Cliff as a humanist, lover of litera-ture, and great athlete. Never would we have imaginedthat he was a whitewater canoeist of the highest caliberor that he would regularly participate with his wife inwalkathons that lasted more than 20 hours.

The illness that would ultimately take his life made ap-parent the strength of character that we already knewwell. Always having had a keen intellectual curiosity,Cliff became an expert on his own disease, and was ableto participate in his own treatment, and sometimes guidehis physicians in the appropriate course of action. Evenat the stage when many might have despaired, Cliff lived

life at its fullest to the very end.

For us, it is a privilege to have had the opportunityof knowing and working with Cliff Astill, a first-ratescholar, kind and generous person, gentleman, and awonderful human being. We dedicate this paper to hismemory.

We would also like to express our appreciation to GeorgeK. Lea, who himself has provided guidance and encour-agement to us through the years, for his kind invitation tosubmit a contribution to this special issue of the journal.

Partial support for this work was also provided throughgrants from the NSF-ITR program (EAR-0326449) andthe Southern California Earthquake Center. SCEC isfunded by NSF Cooperative Agreement EAR-0106924and USGS Cooperative Agreement 02HQAG0008. TheSCEC contribution number for this paper is 947. We alsothank the Pittsburgh Supercomputing Center for its con-tinued support of our large scale earthquake modeling re-search. In particular, the simulations in this work wereconducted on the HP AlphaServer parallel system at PSCunder awards BCS020001P and MCA01S002P.

References

Akcelik, V.; Bielak, J.; Biros, G.; Epanomeritakis, I.;Fernandez, A.; Ghattas, O.; Kim, E. J.; Lopez, J.;O’Hallaron, D. R.; Tu, T.; Urbanic, J. (2003): High-resolution forward and inverse earthquake modeling onterascale computers. In Proceedings of ACM/IEEESC2003, Phoenix, AZ.

Aki, K.; Richards, P. G. (1980): Quantitative Seis-mology: Theory and Methods, volume I. W. H. Freemanand Co.

Bao, H.; Bielak, J.; Ghattas, O.; Kallivokas, L.;O’Hallaron, D. R.; Schewchuk, J. R.; Xu, J. (1998):Large-scale simulation of elastic wave propagation inheterogeneous media on parallel computers. ComputerMethods in Applied Mechanics and Engineering, vol.152, pp. 85–102.

Bao, H.; Bielak, J.; Ghattas, O.; Kallivokas, L. F.;O’Hallaron, D. R.; Shewchuk, J. R.; Xu, J. (1996):Earthquake ground motion modeling on parallel comput-ers. In Supercomputing ’96, Pittsburgh, Pennsylvania.

PRO

OF

CMES Galley Proof Only Please Return in 48 Hours. 23

Day, S. M.; Bradley, C. R. (2001): Memory-efficientsimulation of anelastic wave propagation. Bulletin of theSeismological Society of America, vol. 91, pp. 520–531.

Dravinski, M. (1982): Influence of interface depth uponstrong ground motion. Bulletin of the Seismological So-ciety of America, vol. 72, pp. 596–614.

Frankel, A.; Vidale, J. (1992): A three-dimensionalsimulation of seismic waves in the Santa Clara Valley,California from a Loma Prieta aftershock. Bulletin ofthe Seismological Society of America, vol. 82, pp. 2045–2074.

Graves, R. W. (1996): Simulating seismic wave propa-gation in 3D elastic media using staggered-grid finite dif-ferences. Bulletin of the Seismological Society of Amer-ica, vol. 86, pp. 1091–1106.

Graves, R. W. (1998): Three-dimensional finite-difference modeling of the San Andreas fault: Source pa-rameterization and ground motion levels. Bulletin of theSeismological Society of America, vol. 88, pp. 891 – 897.

Karypis, G.; Schloegel, K.; Kumar, V. (2002):ParMETIS 3.0. http://www-users.cs.umn.edu/ ˜karypis/metis/parmetis/, 2002.

Kawase, H. (1988): Time domain response of a semi-circular canyon for incident SV, P, and Rayleigh wavescalculated by the discrete wavenumber boundary elementmethod. Bulletin of the Seismological Society of Amer-ica, vol. 78, pp. 1415–1437.

Kim, E. J. (2003): Parallel octree-based multireso-lution mesh method for large scale earthquake groundmotion simulation. PhD thesis, Deptartment of Civiland Environmental Engineering, Carnegie Mellon Uni-versity, 2003.

Kim, E. J.; Bielak, J.; Ghattas, O. (2003): Large-scale Northridge Earthquake simulation using octree-based multiresolution mesh method. In 16th ASCE En-gineering Mechanics Conference.

Komatitsch, D.; Tromp, J. (1999): Introduction to thespectral-element method for 3-D seismic wave propaga-tion. Geophysics Journal International, vol. 139, pp.806–822.

Levander, A. R. (1988): Fourth-order finite-differenceP-SV seismograms. Geophysics, vol. 53, pp. 1425 –1436.

Li, X.; Bielak, J.; Ghattas, O. (1992): Three-dimensional earthquake site response on a CM-2. In Pro-ceedings of the 10th World Conference on EarthquakeEngineering.

Magistrale, H.; Day, S.; Clayton, R. W.; Graves, R.(2000): The SCEC Southern California reference three-dimensional seismic velocity model version 2. Bul-letin of the Seismological Society of America, vol. 90,pp. S65–S76.

Olsen, K. B.; Archuleta, R. J. (1996): Three-Dimensional simulation of earthquakes on the Los An-geles fault system. Bulletin of the Seismological Societyof America, vol. 86, pp. 575–586.

Olsen, K. B.; Day, S. M.; Bradley, C. R. (2003): Es-timation of Q for long-period (> 2 s) waves in the LosAngeles Basin. Bulletin of the Seismological Society ofAmerica. In press.

Olsen, K. B.; Nigbor, R.; Konno, T. (2000): 3-Dviscoelastic wave propagation in the upper Borrego Val-ley, California constrained by borehole and surface data.Bulletin of the Seismological Society of America, vol. 90,pp. 134–150.

Pitarka, A.; Irikura, K.; Iwata, T.; Sekiguchi, H.(1998): Three-dimensional simulation of the near faultground motion for the 1995 Kyogo-ken Nanbu (Kobe),Japan, earthquake. Bulletin of the Seismological Societyof America, vol. 88, pp. 428 – 440.

Sanchez-Sesma, F. J. (1978): Ground motion amplifi-cation due to canyons of arbitrary shape. In Proceedingsof 2nd International Conference on Microzonation, pp.729–738, San Francisco, California.

Sanchez-Sesma, F. J.; Bravo, M. A.; Herrera, I.(1985): Surface motion of topographical irregularitiesfor incident P, SV, and Rayleigh waves. Bulletin of theSeismological Society of America, vol. 75, pp. 263–269.

Sanchez-Sesma, F. J.; Esquivel, J. A. (1979): Groundmotion on alluvial valley under the incident plane SHwaves. Bulletin of the Seismological Society of Amer-ica, vol. 69, pp. 1107–1120.

PRO

OF

24 Tech Science Press CMES Galley Proof Only Please Return in 48 Hours.

Sato, T.; Graves, R. W.; Somerville, P. G. (1999):Three-dimensional finite-different simulations of long-period strong motions in the Tokyo metropolitan areaduring the 1990 Odawara Earthquake (Mj 5.1) and theGreat 1923 Kanto earthquake (Ms 8.2) in Japan. Bul-letin of the Seismological Society of America, vol. 89,pp. 579–607.

Schrivner, C. W.; Helmberger, D. V. (1994): Seis-mic waveform modeling in the Los Angeles Basin. Bul-letin of the Seismological Society of America, vol. 84, pp.1310–1326.

Toshinawa, T.; Ohmachi, T. (1988): Love-wave propa-gation in a three-dimensional basin. Bulletin of the Seis-mological Society of America, vol. 82, pp. 1661–1677.

Tu, T.; O’Hallaron, D.; Lopez, J. (2002): Etree:A database-oriented method for generating large octreemeshes. In Proceedings of the Eleventh InternationalMeshing Roundtable.

Tu, T.; O’Hallaron, D. R. (2004): A computationaldatabase system for generating unstructured hexahedralmeshes with billions of elements. In Proceedings ofACM/IEEE SC2004, Pittsburgh, PA.

Vidale, J. E.; Helmberger, D. V. (1988): Elastic finite-difference modeling of the 1971 San Fernando, Califor-nia, earthquake. Bulletin of the Seismological Society ofAmerica, vol. 78, pp. 122–141.

Virieux, J. (1984): SH-wave propagation in hetero-geneous media: Velocity-stress finite-difference method.Geophysics, vol. 49, pp. 1933 – 1957.

Wald, D. J.; Graves, R. W. (1998): The seismic re-sponse of the Los Angeles Basin, California. Bulletin ofthe Seismological Society of America, vol. 88, pp. 337–356.

Wald, D. J.; Heaton, H.; Hudnut, K. W. (1996): Theslip history of the 1994 Northridge, California, Earth-quake determined from strong-motion, teleseismic, GPS,and leveling data. Bulletin of the Seismological Societyof America, vol. 86, pp. S49–S70.

Wong, H. L. (1979): Diffraction of P, SV, and Rayleighwaves by surface topographies. Technical Report ReportCE79-05, Department of Civil Engineering, Universit ofSouthern California, 1979.

Wong, H. L. (1982): Effect of surface topography onthe diffraction of P, SV, and Rayleigh waves. Bulletin ofthe Seismological Society of America, vol. 72, pp. 1167–1183.

Yomogida, K.; Etgen, J. T. (1993): 3-D wave propaga-tion in the Los Angeles basin for the Whittier-Narrowsearthquake. Bulletin of the Seismological Society ofAmerica, vol. 83, pp. 1325–1344.

Yoshimura, C.; Bielak, J.; Hisada, Y.; Fernandez,A. (2003): Domain reduction method for three-dimensional earthquake modeling in localized regions,Part II: Verification and application. Bulletin of the Seis-mological Society of America, vol. 93, pp. 825 – 840.

Young, D.; Melvin, R.; Bieterman, M.; Johnson, F.;Samant, S.; Bussoletti, J. (1991): A locally re-fined rectangular grid finite element method: applica-tion to computational fluid dynamics and computationalphysics. Journal of Computational Physics, vol. 92, pp.1–66.

Zang, L.; Chopra, A. (1991): Three dimensionalanalysis of spatially varying ground motion around uni-form canyon in a homogeneous half space. EarthquakeEngineering and Structural Dynamics, vol. 20, pp. 911–926.