PARALLEL COMPUTING IN MULTIBODY SYSTEM DYNAMICS:...

PARALLEL COMPUTING IN MULTIBODY SYSTEM DYNAMICS:WHY, WHEN AND HOW

Dan Negrut∗, Radu Serban, Hammad Mazhar, Toby HeynSimulation-Based Engineering Lab

Department of Mechanical EngineeringUniversity of Wisconsin–Madison

Madison, WI 53706Email: {negrut,serban,mazhar,heyn}@wisc.edu

ABSTRACTThis paper addresses three questions related to the use of

parallel computing in Multibody Dynamics (MBD) simulation.The “why parallel computing?” question is answered based onthe argument that in the upcoming decade parallel computingrepresents the main source of speed improvement in MBD simu-lation. The answer to “when is it relevant?” is built around theobservation that MBD software users are increasingly interestedin multi-physics problems that cross disciplinary boundaries andlead to large sets of equations. The “how?” question is ad-dressed by providing an overview of the state of the art in paral-lel computing. Emphasis is placed on parallelization approachesand support tools specific to MBD simulation. Three MBD ap-plications are presented where parallel computing has been usedto increase problem size and/or reduce time to solution. The pa-per concludes with a summary of best practices relevant whenmapping MBD solutions onto parallel computing hardware.

1 Introduction and Purpose.Multibody dynamics (MBD) is an area of Computational

Dynamics concerned with the dynamics of mechanical systemsconsisting of collections of bodies experiencing large displace-ments and possibly large deformations. The three dimensional(3D) bodies are subject to external forces and their time evolu-tion can be constrained through kinematic constraints that couplethe dynamics of pairs of bodies. The bodies can interact mutu-ally through friction and contact, or alternatively, in Fluid-SolidInteraction (FSI) problems, their dynamics interplays with the

∗Address all correspondence to this author.

dynamics of a fluid through a two way coupling mechanism.Historically, MBD problems have relatively been small. A

full vehicle dynamics analysis or an engine dynamics simulationin a commercial code such as ADAMS [50] typically leads tofewer than 50,000 equations. By Finite Element Analysis (FEA)or Computational Fluid Dynamics (CFD) standards, these aresmall problems. While nonetheless complex, MBD simulationhas mostly led to computational problems that by and large havenot justified heavy reliance on parallel computing. In fact, com-pared to solution providers in the FEA and CFD markets, all ma-jor players in the MBD arena: ADAMS, RecurDyn, LMS Vir-tual.Lab, or Simpack, have, until recently, been slow in adopt-ing parallel computing in the equation formulation and numeri-cal solution stages. There are still numerous studies in the MBDliterature regarding how parallelism can reduce time to solution,see for instance [2, 12, 18, 49]. Parallel computing has alreadybeen applied in our community to address domain specific prob-lems, see [8, 22, 67]. Finally, software for general purpose paral-lel MBD is discussed in [31, 42, 55].

Two forces are gradually changing the MBD landscape in re-lation to the simulation of very large systems. First, users of thesesimulation tools are requesting support for modeling features thatcross the boundary of traditional MBD into multi-physics prob-lems. These problems might include a fluid component, a gran-ular component, a terrain component, etc., to yield models char-acterized by large sets of equations that stand to benefit fromparallel computing. Second, once the privilege of a select groupof research institutions, hardware that supports effective parallelcomputing is now ubiquitous. Due to low costs, less than $4000for a 60 core Intel Xeon Phi chip or $3000 for a 2600 scalar pro-cessor Nvidia Kepler graphics processing unit (GPU) card, there

1 Copyright c© by ASME

is a manifest democratization of the access to parallel computinghardware.

Given these ongoing changes in the MBD simulation arena,the purpose of this paper is fourfold: (i) briefly present actualMBD problems whose modeling results in large sets of equationshence confirming the aforementioned trend; (ii) summarize thestate of the art in parallel computing and comment on how it canbe leveraged in MBD simulation; (iii) present several applica-tions in which parallel computing was used to reduce the timeto solution and/or increase the size of the problem investigated;and (iv) summarize good practices relevant when mapping MBDsolutions onto parallel computing hardware.

The paper is structured around these four goals as follows.Section 2 summarizes how MBD problems in granular dynamicsand fluid-solid interaction lead to large and challenging numeri-cal problems that stand to benefit from parallel computing. Sec-tion 3 discusses parallel computing issues: Is parallel computinga fad?, What parallel computing avenues should one adopt in thecontext of MBD modeling and simulation? and finally, What arethe tools of the trade (libraries, debuggers, profilers, integrateddevelopment environments) that one can use to leverage parallelcomputing in MBD?. Section 4 concentrates on issues that caninfluence the choice of (a) modeling formulation; (b) numericalmethods; and (c) the parallel computing platform, three factorsthat combine to provide the foundation of a parallel MBD simu-lation tool. Section 5 describes three applications that relied ondifferent parallel computing platforms to increase problem sizeand/or reduce time to solution. Section 6 rounds off the paperwith closing remarks and several recommendations.

Note that the discussion herein was predicated on the as-sumption that the MBD problem size is large to the extent thatthe overhead associated with using high level parallel program-ming approaches is negligible. Approaches suitable for real-timesimulation, when the interest is in characterizing on a tight andpredefined schedule the dynamics of small problems, would re-volve around programming techniques drawing on low level par-allel libraries: e.g., pthreads. These libraries often work withinthe context of a dedicated and stripped down version of an oper-ating system running on a multicore CPU chip. This discussion;i.e., parallel computing for real-time simulation, falls outside thescope of this contribution.

2 MBD: The Need for Parallel Computing.MBD simulation plays an important role in the overall prod-

uct life cycle by providing early insights in engineering designsthrough virtual prototyping. The complexity of the models inMBD simulation has evolved from component to subsystem tosystem. Recent advances in processor performance and memoryspeed/size, combined with the widespread availability of parallelcomputing platforms, provide renewed interest in what might becalled many-body dynamics: understanding through direct nu-

merical simulation the emergent behavior of large systems of 3Dbodies interacting through friction, contact, and potentially longrange electrostatic forces. Similar to Molecular Dynamics, themacroscale dynamics represents the emergent behavior of mil-lions of 3D bodies whose motion is monitored in time. A list ofapplications that rely on many-body dynamics includes: packingof polyhedra for crystalline pattern formation studies; packing ofcompounds in pharmaceutical pills; drug delivery for powdersand suspensions; ground vehicle mobility on granular terrain;additive manufacturing, in which layer upon layer of granularmaterial is moved, shaped, and melted by a 3D printer; granularimpact cratering of relevance in astrophysics, etc.

At its core, the many-body dynamics problem is posedlike the typical MBD problem. The equations of motion areformulated herein using an absolute, or Cartesian, representa-tion of the attitude of each rigid body in the system. Thestate of the system is represented by the generalized positionsq =

[rT1 , ε

T1 , . . . , r

Tnb, εTnb

]T ∈ R7nb and their time derivativesq, where nb is the number of bodies, rj ∈ R3 is the absolutelocation of the center of mass of body j and the quaternions(Euler parameters) εj ∈ R4 are used to represent body orien-tation. Instead of using quaternion derivatives, ε, it is more ef-fective to work with the angular velocities ωj ∈ R3, expressedin a local reference frame attached to body j; in other words,the formulation is derived in terms of the generalized velocitiesv =

[rT1 , ω

T1 , . . . , r

Tnb, ωTnb

]T ∈ R6nb . Given that εj = 12G

Tj ωj ,

with the 3 × 4 matrix Gj ≡ G(εj) defined as in [26], the rela-tionship between the time derivatives of the generalized positionsand the generalized velocities is the linear mapping q = L(q)v.

Bilateral constraints representing kinematic joints (spheri-cal, prismatic, revolute, etc.) are included as algebraic equationsrestricting the relative position of two rigid bodies. Each con-straint i ∈ B, the set of bilateral constraints, is represented bythe scalar algebraic equation Ψi(q, t) = 0 and transmits reac-tion forces to the connected bodies by means of a multiplier γi,b.Assuming smoothness of the constraint manifold, Ψi(q, t) canbe differentiated to obtain the Jacobian ∇qΨi = [∂Ψi/∂q]

T . Inwhat follows we will also use the notation∇Ψi ≡ LT (q)·∇qΨi.

Contact and friction forces are included in the systemthrough complementarity constraints. Given a large number ofrigid bodies with different shapes, modern collision detection al-gorithms are able to efficiently find a set of contact points, thatis, points where a gap function Φ(q) can be defined for each pairof near-enough geometry features. Where defined, such a gapfunction must satisfy the nonpenetration condition Φ(q) ≥ 0for all contact points. More precisely, when a contact i is ac-tive (Φi(q) = 0), a normal force and a tangential friction forceact on each of the two bodies at the contact point; if the con-tact is not active (Φi(q) > 0), no contact or friction forces exist.This results in a mathematical description of the model knownas a complementarity problem [75]. In what follows, A(q(t))


Body

A Body

B

0 x

y

z

ni

ui

wi

rB

rA

si,A

i-th contact

si,B

x

y

z

x y

z

FIGURE 1. Contact i between two bodies A,B ∈ {1, 2, . . . , nb}

denotes the set of all active contacts for a given configuration qof the system at time t.

The contact model can be augmented with the classicalCoulomb model to define the friction forces acting at the inter-face between two bodies in contact. Consider the contact i ∈ Abetween two bodies A and B as shown in Fig. 1. Let ni be thenormal at the contact pointing toward the exterior of the body oflower index, which by convention is considered to be body A.Let ui and wi be two unit vectors in the contact plane such thatni,ui,wi ∈ R3 are mutually orthogonal. The frictional contactforce is applied on the system by means of multipliers γi,n ≥ 0,γi,u, and γi,w, leading to the normal and tangential componentsFi,N = γi,nni and Fi,T = γi,uui + γi,wwi, respectively. TheCoulomb model is expressed as the maximum dissipation princi-ple:

(γi,u, γi,w) = argmin√γ2i,u+γ

2i,w≤µiγi,n

vTi,T (γi,uui + γi,wwi) (1)

where µi is the friction coefficient associated with contact i.The time evolution of the dynamical system is governed by

the following differential problem with set-valued functions andcomplementarity constraints, which is equivalent to a differential

variational inequality (DVI) [61]:

q = L(q)v

Mv = f (t,q,v) +∑i∈B

γi,b∇Ψi

+∑i∈A

(γi,nDi,n + γi,uDi,u + γi,wDi,w)

i ∈ B : Ψi(q, t) = 0

i ∈ A(q(t)) : 0 ≤ γi,n ⊥ Φi(q) ≥ 0, and

(γi,u, γi,w) = argmin√(γi,u)2+(γi,w)2≤µiγi,n

vT (γi,uDi,u + γi,wDi,w)

(2)where f (t,q,v) denotes the external (applied) generalizedforces and the notation ⊥ is used to indicate orthogonality, i.e.,γTi,nΦi(q) = 0.

The tangent space generators Di = [Di,n, Di,u, Di,w] ∈R6nb×3 are defined as

DTi = [0 · · · −AT

i ATi AA˜si,A · · · AT

i −ATi AB˜si,B · · · 0]

(3)where Ai = [ni,ui,wi] ∈ R3×3 is the orientation matrix asso-ciated with contact i, AA = A (εA) and AB = A (εB) are therotation matrices of bodies A and B, respectively, and the vec-tors si,A and si,B represent the contact point positions in body-relative coordinates as illustrated in Fig. 1. The mass matrix M isconstant and diagonal due to the choice of body-fixed centroidalreference frames.

Under numerical discretization, see for instance [3, 27, 28,76], the numerical solution calls at each time step for solving aquadratic optimization problem with conic constraints:

min q(γ) = 12γ

TNγ + rT γ

subject to√γi,u2 + γi,w2 ≤ µiγi,n , ∀i ∈ A ,

(4)

where N and r are a constant symmetric positive semidefi-nite matrix and a constant vector that depend on system stateat the beginning of the time step; γ =

[γT1 , γ

T2 , . . . , γ

Tnc

]Tand

γi = [γi,n, γi,u, γi,w]T

= h [γi,n, γi,u, γi,w]T is the triplet ex-

pressing the magnitude of the contact impulses for contact i, withh denoting the integration step-size. This constrained optimiza-tion problem is solved by an iterative method or interior pointmethod [27]. In either case, the dimension of the problem islarge. As an example, the penetration study illustrated in Fig. 2used 0.6 million bodies [27]. There are approximately two mil-lion contacts at any time step, each introducing a set of threeunknowns γi, resulting in an optimization problem of dimensionaround six million. Parallel computing is essential in solving thissub-problem and the collision detection that determines the setof contacts in the first place. For a typical integration step-size


FIGURE 2. Cross-section view of 3D normal contact forces in gran-ular material (modeled using 0.6 million rigid bodies), during impact byspherical object.

h = 10−4s, a number of 10, 000 solutions of Eq. 4 and 10, 000collision detections are carried out for each second of simulation.

There are several alternative approaches to modeling fric-tional contact beyond the one presented above. The most widelyused relies on a a penalty/regularization approach: a small pene-tration is allowed between two bodies and the interface frictional-contact force is proportional to the amount of penetration andrelative velocity of the bodies at the contact point. A recent re-view of literature on penalty methods is provided in [35]. Whilepenalty methods do not call for the costly solution of the opti-mization problem in Eq. 4, they are hampered by much smallerintegration time steps (h ≈ 10−6s). Like the DVI approachdescribed above, penalty methods also have to deal with verylarge sets of differential equations resulting from the use of sevengeneralized positions and six generalized velocities for each 3Dbody in the system (almost 8 million equations for the problemin Fig. 2).

The Fluid-Solid Interaction (FSI) problem is very brieflydiscussed herein as another MBD application that leads to largesets of equations and whose solution requires complex numericalmethods. The fluid flow can be represented in either an Eulerianor a Lagrangian framework. Provided that the interfacial forcesare captured thoroughly, the Lagrangian framework is capableof tracking the domain deformation introduced by the motion ofthe solid phase at almost no extra cost. Smoothed Particle Hy-drodynamics (SPH) [21, 40, 46], its modifications [15, 45], andvariations [37] have been widely used for the simulation of thefluid domain in a Lagrangian framework. The main evolution

equations of the fluid flow using SPH are expressed as

dρadt

= ρa∑b

mb

ρb(va − vb) · ∇aWab (5)

dvadt

= −∑b

mb

[(pbρ2a

+paρ2b

)∇aWab (6)

− (µa + µb)rab · ∇aWab

ρ2ab(r2ab + εh2ab)

vab

],

which are solved in conjunction with dra/dt = va to update thefluid properties at each SPH marker a. Above, ρ, µ, v, and p arelocal fluid density, viscosity, velocity, and pressure, respectively,m is the representative fluid mass assigned to the SPH marker,and rab is the distance between two fluid markers a and b. Thekernel functionWab ≡W (rab, h) is used to smooth out the localfluid properties within a resolution length h. The fluid flow evo-lution equations are solved explicitly, where pressure is relatedto density via an appropriate state equation to maintain the com-pressibility below 1%. Accuracy and stability can be improvedthrough an XSPH modification [45] and Shephard filtering [13].

FIGURE 3. Coupling of the fluid and solid phases. BCE and fluidmarkers are represented by black and white circles, respectively.

The fluid-solid coupling should satisfy the no-slip and im-penetrability conditions on the surface of the solid obstacles. Byattaching Boundary Condition Enforcing (BCE) markers on thesurface of the solid objects (see Fig. 3), the local relative veloc-ity between the two phases is enforced to be zero at the markerlocations. The position and velocity of the BCE markers are up-dated according to the motion of the solid phase, resulting in thepropagation of the solid motion to the fluid domain. In turn, theinteraction forces evaluated at the BCE markers are used to cal-culate the total force and torque exerted by the fluid on the solid.


FSI applications lead to large problems owing to the num-ber of SPH markers that need to be considered in the solution.The results reported in [64,65] drew on models with close to onemillion degrees of freedom and thus provide ample opportunitiesfor parallel computing. For the specifics of a parallel implemen-tation of the FSI solution, the interested reader is referred to [42].

3 Parallel Computing: the Long Term View, Alterna-tives, and Solution Development SupportFor the purpose of this discussion, parallel computing refers

to the process of enlisting multiple hardware assets (cores or pro-cessors) to simultaneously execute, in a fashion at least partiallycontrolled by the application developer, either (a) multiple se-quences of instructions of a program at the same time; or (b) thesame sequence of instructions acting at the same time on multipleinstances of data. Following Flynn’s taxonomy of parallel com-puters [20], the former covers multiple instruction multiple data(MIMD) architectures, while the latter refers to single instruc-tion multiple data (SIMD) architectures. Herein, we concentrateon (i) distributed computing using the Message Passing Inter-face (MPI) [24], which falls into (a) above; (ii) general-purposeGPU computing (GPGPU), which is an example of (b) above;and (iii) symmetric multiprocessing (SMP) with OpenMP [59],which can fall under (a) or (b). Details about MPI, GPGPU,and OpenMP will be provided after discussing the grounds fora migration to parallel computing in CAE in general and MBDin particular. The section will conclude with a brief summaryof productivity tools: compilers, debuggers, profilers, develop-ment environments, and libraries that support the three main par-allel computing platform considered in this paper; i.e., GPGPU,OpenMP, and MPI.

3.1 Is Parallel Computing a Fad?There were three factors that precipitated the rise of paral-

lel computing in the mid 2000s. Also called the three walls tosequential computing, these factors marked a shift in the chipindustry towards multi-core designs.

The memory wall is a manifestation of the fact that, fordecades, memory speed increased at a pace significantly slowerthan that of data processing. For instance, from 1986 to 2000, theCPU speed improved at an annual rate of 55%, while the mainmemory access speed improved only at 10% annually [63]. Itbecame apparent that making a faster and faster processor willonly lead to scenarios where this fast processor would be idlewaiting on data to be moved back and forth. Indeed, a mem-ory request that leads to a cache miss and thus triggers a mainmemory (RAM) transfer typically requires hundreds of cycles. Ifthe cache miss is combined with a translation look-aside buffer(TLB) miss, this cost can be orders of magnitude larger. Com-paring this to the one cycle required nowadays to carry out an

add-multiply operation, led to the assertion that computation isnot about number crunching, which basically takes no time, butrather data transfer.

There are two aspects to memory speed: latency and band-width. Latency is essentially the amount of time it takes for amemory transaction to begin; once the transaction has begun,bandwidth is a measure of the rate at which data can be trans-ferred from the provider of data to the consumer of data. Reduc-ing latency is difficult; increasing bandwidth is relatively easyby adding more lanes in the memory bus, increasing the mem-ory clock frequency, or using techniques that allow moving bitsover several phases of the signal and/or wavelengths. In otherwords, there is little difference between moving 4 bytes or 512bytes since the cost is not dictated by the bandwidth constraint asmuch as it is by the latency associated with the data movementrequest. As such, it became apparent that if data must be movedthen it is advantageous to move it in big chunks. This favoredparallel execution, especially in its SIMD flavor.

The power wall has manifested as a constraint placed on theclock frequency at which a chip could operate. It was postu-lated that the transistors used in computers ”continue to functionas voltage-controlled switches - while all key figures of meritsuch as layout density, operating speed, and energy efficiencyimprove, provided geometric dimensions, voltages, and dopingconcentrations are consistently scaled to maintain the same elec-tric field” [14]. It thus makes sense to decrease both the typi-cal feature size and voltage in a chip, since when these changesare combined with appropriate adjustments in doping concentra-tions the end product is superior: more transistors per unit area,which is essentially what fueled Moore’s law for the last fourdecades. However, there are physical limits that challenge thisquest for smaller operation voltage and feature length. Presently,the voltage can only be marginally decreased – aiming for volt-ages below 0.3 V introduces risk because controlling the transis-tors becomes very hard in an electrically noisy environment. De-creasing the feature length is challenging too since the insulatorthat separates the gate from the rest of the MOS transistor be-comes narrower as the feature length decreases. This translatesinto power leaks, which are important since power dissipationon a chip changes as follows: it is increased by the leaks, it in-creases quadratically with the voltage and linearly with the clockfrequency at which the transistors are operated. As seen, lower-ing the voltage is not an option anymore and as such, as the fea-ture length goes down, the frequency must be reduced or at bestcan stay constant. Otherwise, the amount of heat dissipated goesup, which can trigger a thermal runaway; i.e., a situation wherean increase in temperature raises the resistance of the transistorsleading to even more power being dissipated. The device eitherfails sooner or functions unreliably.

Finally, the instruction level parallelism (ILP) wall reflectsthe inability of parallel execution at the chip level to yield fur-ther substantial efficiency gains in sequential programming. In


other words, techniques that have benefited sequential comput-ing in the past – instruction pipelining, superscalar execution,out-of-order execution, register renaming, branch prediction, andspeculative execution – have fulfilled to a very large extent theirpotential. Interestingly, despite the use of the word parallelism,the above ILP techniques were all meant to speed up sequen-tial computing. For instance, pipelining allows the simultaneousexecution of several instructions at the same time; however, allthese instructions belonged to one sequential stream of instruc-tion, which now was executed faster since at each clock tick oneinstruction would be finished. The obvious mechanism of obtain-ing higher speedups by making the pipeline deeper is counter-balanced by the consequent rise in instruction dependency. Reg-ister forwarding, register renaming, and several other techniqueseliminate some of these conflicts but pipeline stalls are inher-ent in deep pipelines. Superscalar solutions employ two parallelpipelines that nonetheless work on the same stream of instruc-tions and support speculative execution, out-of-order execution,branch prediction, etc. All these tricks eventually reached a levelwhere the complexity of managing them did not justify the returnon investment in terms of power and/or die area costs.

In the early to mid 2000s it became apparent that the mo-mentum behind the sequential computing in the server and con-sumer market was waning. Memory access latencies could notbe hidden behind one stream of instructions and the gap betweendata processing and data movement was getting wider; the fre-quency couldn’t be increased any further without resorting to so-lutions that were more expensive than the market was accepting;and the ILP had for most part exhausted its set of tricks. Moore’slaw remained one bright spot in the picture and provided a safebet for continued performance improvement. Two of the leadingchip manufacturers, Intel and IBM, had roadmaps in place thatsuggested that the feature length was bound to keep decreasingfor close to two decades. For instance, Intel released Ivy Bridgein 2012, which is based on 22 nm technology. It will release achip based on 14 nm technology next year and 10 nm technol-ogy in 2016, with 7 nm and 5 nm estimated for 2020 and 2022,respectively. This ongoing miniaturization trend sends a clearmessage that more and more cores will be available on a chip,making the parallel computing more and more prevalent.

To put things in perspective and understand the pace atwhich the technology advanced, less than one decade ago, a 1TB distributed memory IBM Blue Gene/L with 1024 PowerPCcores used to deliver 5.7 TFlops. It came at a price of approx-imately $1.4 million, required a dedicated power source, andneeded hardware and software support staff. Today, a powerfuldesktop with 1 TB of RAM, a motherboard with two Intel Xeonchips for 32 cores (64 virtual cores), and four Kepler GPU cardsdelivers more than 6 Tflops at a price of $70,000. It does nothave steep power requirements, it does not require a system ad-ministrator to maintain it, can run Linux or Windows, and offersa comprehensive set of productivity tools. This trend towards

more performance for lower costs is expected to continue in thesame directions, namely more cores packaged on a chip and anincrease in memory bandwidth allowing to hide memory latencywith useful execution. In other words, the new parallel comput-ing paradigm is anchored into technology realities.

3.2 Relevant Flavors of Parallel ComputingWhile the hardware transition to multi- and many-core chip

layouts happened quickly, the software shift to using parallelcomputing languages and libraries to produce programs thatleverage this hardware has been and continues to be arduous.

For fine gain parallelism that suits the SIMD model, GPUcomputing is the best match in terms of hardware and soft-ware support. The most recent GPU card from Nvidia belongsto the Kepler architecture. It has 250-320 GB/s bandwidth1 tofeed about 2500 scalar processors, which can deliver up to 1.31TFlops. This is approximately 25% of an IBM Blue Gene/L ofone decade ago. The GPU’s weak spot is associated with itsmemory size. At 6-8 GB is in many cases sufficient but verylarge flexible body dynamics problems or fluid-solid interactionproblems might run into memory space issues. These are usu-ally resolved by shuttling data back and forth between the GPUand the host CPU. Yet this happens over a PCI Express bus thatusually sustains 8 GB/s bandwidths and thus becomes a sim-ulation bottleneck. In terms of software support, these GPUsare programmed using either OpenCL, which is an open stan-dard [52], or CUDA [56]. The latter currently has wider adoptionand counts on a more mature set of productivity tools.

In our experience, GPU computing turned out to be ex-tremely suitable for simulations of granular material, as furtherdiscussed in section 5.3. Another application suitable for GPUcomputing is large scale flexible body dynamics where one hasto handle tens to hundreds of thousands of elements. Such aproblem was approached in the framework of Absolute NodalCoordinate Formulation (ANCF) in [34]. Likewise, any analysisthat involves a large set of data that needs to be processed is verylikely amenable to GPU computing. For instance, FEA wouldfall into this category, since each node and/or element in the meshshould be visited in order to evaluate a stiffness matrix and, forimplicit integration-based nonlinear FEA, to compute the ingre-dients used in assembling the numerical discretization Jacobian.Likewise, N-body problems, where one has to loop through allbodies to assess the influence of the neighboring bodies, providea good setup for effective use of GPU computing.

GPU computing falters when dealing with problems withrelatively small data sets or when the tasks performed are differ-ent for each member of the data set. Unfortunately, this turnsout to be the case of classical MBD. Most systems include lessthan 500 rigid bodies. Even when one brings into the picture afloating frame of reference formulation for handling flexibility,

1We use GB to denote gigabytes and Gb for gigabits. Eight Gb make one GB.6 Copyright c© by ASME

typically the size of the problem does not justify the use of GPUcomputing. Providing a size after which GPU makes sense de-pends on the type of GPU being considered. Today’s high endGPUs have 2,500 scalar processors or more. To get a good oc-cupancy of the hardware, one would have to use in CUDA closeto 30,000 threads to keep these 2,500 busy. In other words, onewould need to deal with an MBD problem where there are at least30,000 quantities that need to be computed. Moreover, since theGPU is a SIMD-type device, it is desirable to have these threadsto work on very similar problems e.g., compute the stiffness ma-trix of 30,000 elements, or evaluate the normal force at 30,000contact points, etc.

Parallel computing with OpenMP [59] is more forgivingin handling scenarios that deviate from the SIMD model. Un-like GPU computing, which for meaningful speedups requiresproblems that call upon the use of at least tens of thousands ofparallel threads, OpenMP can yield good speed gains by usinga relatively small number of threads. While GPU threads arelightweight and switching the hardware from the execution ofone thread to a different thread incurs little overhead, OpenMPthreads are heavy; starting or stopping a thread is costly. In fact,the operating system typically fires up a number of threads equalto the number of cores present on the system and manages thesethreads for the duration of the simulation by allocating jobs tothem. The term system here refers to workstations that typicallyhave between 4 and 64 cores that operate in a symmetric mul-tiprocessing (SMP) framework. In this context, symmetric indi-cates that the main memory is shared and each of the identicalcores has similar access to this memory albeit at slightly differ-ent latencies based on the physical proximity of a core to thememory bank accessed.

The OpenMP standard is supported very well by Intel andAMD chips. OpenMP can be presently used on a power work-station that might have 0.5 TB of RAM and 64 cores. Additionalcompute power can be accessed if the system has Intel Xeon Phiaccelerators, each of which brings to the table more than 250parallel threads. Unlike GPU computing, OpenMP (a) supportsvery well coarse grain parallelism, and (b) allows one to workin a more convenient fashion with very large data sets. Existingworkstations routinely have hundreds of GB of main memory,which is two orders of magnitude more memory than one canaccess on the GPU. The latter can directly access the host mainmemory, but this comes at a stiff price as the transaction is car-ried out over the PCI express bus. The GPU wins when it comesto sheer memory bandwidth and flop rate. Number crunching isbasically free as long as data is available; the problems crop upwhen cache misses require costly trips to main memory. GPUcan do a better job here since (i) it operates at higher bandwidth(roughly 300 GB/s as opposed to roughly 50 GB/s for multi-core CPUs); and (ii) it is better at hiding memory latency withuseful work. The latter is accomplished by overcommitting the2,500 scalar processors on the GPU to handle tens of thousands

of threads to increases the chance that the scheduler will find awarp of threads that are ready for execution and thus avoid stallsof the scalar processors. This is very similar to the idea of hyper-threading except that while on the Intel architecture one can havetwo candidates for occupying the hardware, in GPU computingone warp scheduler has, on average, up to 16 choices from whichit can select the next warp of threads that will be executed. Thisis because each stream multiprocessor on a GPU card can be as-signed for execution up to 64 warps (groups of 32 threads) and ituses four warp schedulers.

In terms of source code modification, OpenMP provides avery low entry point into parallel computing. For instance, a for-loop that might evaluate the generalized forces acting on eachbody in the system can be parallelized by simply adding oneline of code. Unfortunately, OpenMP does not scale well; i.e.,it works on a powerful workstation but cannot be implementedfor distributed memory architectures. Specifically, one cannotspread parallelism on two workstations solely using OpenMP.

Going beyond one workstation is typically achieved by us-ing implementations of the Message Passing Interface (MPI)standard [24], such as MPICH2 [25] or Open MPI [23]. MPIwill become indispensable when large amounts of memory areneeded, such as for large granular dynamics or fluid-solid inter-action problems. The MPI nodes are linked together through afast interconnect for passing data back and forth. Typical band-widths are of the order of 40 Gb/s; i.e., close to what one wouldget over a PCI connection. However, at 1 microsecond [53], thelatency is significantly larger, which is likely to impact MBDsimulation since dynamics simulation is communication bound.Specifically, at each time step, the nodes need to exchange datato advance the dynamics simulation. If, for instance, the step sizein a penalty approach to handling frictional contact is 10−6 s, thismeans that each second of simulation requires 1,000,000 data ex-changes. This places a significant bottleneck on the simulationessentially overshadowing any other aspect of the computation tothe point where the execution time will be mostly spent in send-ing messages back and forth between the collaborating nodes andwaiting for such messages to arrive.

3.3 Productivity Tools and Library SupportToday, a developer can write parallel software without writ-

ing any line of code that will execute directly on the GPU. Thereare several relatively mature libraries that provide a low entrypoint for GPU computing; the user only needs to make a callinto such a library to obtain, for instance, the solution of a sparselinear system using an iterative method that will be executed onthe GPU. CUSP [9] is an open source library of generic parallelalgorithms for sparse matrix and graph computations. It pro-vides several solvers for linear systems (conjugate-gradient, bi-conjugate gradient, biconjugate gradient stabilized, generalizedminimum residual, multi-mass conjugate-gradient, and multi-


mass biconjugate gradient stabilized), preconditioners (algebraicmultigrid based on smoothed aggregation, approximate inverse,diagonal), and several utilities related to manipulating dense andsparse matrices. Basic operations such as prefix scans, gather,scatter, sort, etc. are supported by the open source template li-brary Thrust [30]. Operations that combine several arrays toproduce an output array can be implemented by writing only afew lines of C++ code that are subsequently executed in an opti-mized fashion on the GPU.

As an alternative to using GPGPU libraries, a developercan implement functions (kernels) that are directly executed onthe GPU. This is not a low entry point to GPGPU comput-ing since writing efficient kernels requires a good understandingof the GPU’s deep memory hierarchy (registers, local memory,shared memory, constant memory, texture memory, global mem-ory, cache levels) and how to set up an optimal kernel executionconfiguration, namely defining (i) the number of parallel threadsin a block and (ii) the number of such blocks of threads launchedin a grid. For instance, if a number of 0.5 million force compu-tations are necessary, the user should specify how a number of atleast 0.5 million threads will be launched to compute the forces inparallel. One can choose a large number of threads per block anda small number of blocks; alternatively, one can decide on a smallnumber of threads but large number of blocks. These decisionsare based on several factors, including the number of registers(per thread) used by the kernel and the amount of required sharedmemory (per block). Moreover, these execution configurationsmay have to change from GPU to GPU, based on hardware ca-pabilities. The implementation effort for developing GPU ker-nels can be significantly reduced by available development tools.Nvidia’s Nsight development platform offers an integrated envi-ronment for building, debugging, profiling and tracing parallelapplications using CUDA C/C++. Available both under Win-dows and Linux and integrated with Visual Studio and Eclipse,respectively, the Nsight debugger supports all common debug-ging tasks (setting breakpoints in CUDA source, viewing localvariables and inspecting memory, navigate the current call stack,etc.) as well as switching the debugging context to any thread orwarp of threads. The latter is critical for effective debugging ofGPU parallel software which routinely involves launching tensof thousands of parallel threads. The analysis and profiling toolsprovided with Nsight help the developer understand overall per-formance and identify hot spots and bottlenecks. The user cansee the combined CPU/GPU activity along a timeline or queryperformance information at a very fine level of granularity (e.g.,level 1 and 2 cache hits, effective memory bandwidths and deviceutilization, register usage).

OpenMP enjoys a similar level of library and productivitytools support. The philosophy behind OpenMP, shared also bythe language cilk and the Intel Threading Building Blocks TBBlibrary is that the developer need only expose parallelism afterwhich the runtime environment, through an internal scheduler,

will manage the distribution of the work to parallel executionthreads. Exposing parallelism can be done with compile-time di-rectives (OpenMP), simple language extensions (cilk), or librarycalls (TBB). In terms of library support, the Intel Math Kernel Li-brary (MKL) is a mature solution that leverages multi-core chipsfor fast math operations, including the solution of sparse, dense,and banded linear systems. Preconditioners are available, as areiterative solvers. Finally, debugging and profiling OpenMP inparticular and any multi-core code in general is facilitated by aseries of productivity tools released by Intel: Intel VTune Am-plifier, Intel Inspector, and Intel Advisor. The tools can be usedfor debugging, code profiling, and code analysis.

Distributed (message passing) computing remains the priv-ilege of a relatively small number of organizations. This is re-flected in the relatively low number of options available for MPIdistributed computing, of which two solutions stand out: Open-MPI [58] and MPICH2 [48]. The latter implements MPI-3, themost recent version of the standard, and is the origin of severalother vendor implementations tuned to specific hardware; Intel,IBM, and Cray each have their own MPI derivative based onMPICH2.

Several open source libraries are available for large scaledistributed simulation using MPI. LAMMPS [38, 66] providesfunctionality originally supporting molecular dynamics simula-tions. Over time, LAMMPS or derivatives have been used inperidynamics [62] or more recently in granular dynamics appli-cations [39]. Unlike LAMMPS, which focuses on many-bodydynamics, PETSc [7] is an infrastructure that leverages MPIparallel computing for scientific applications modeled with par-tial differential equations. It provides extensive support for lin-ear algebra, such as algebraic multigrid, direct solvers, iterativesolvers, and preconditioners. It offers explicit and implicit inte-grators, including an interface to the multistep implicit integratorCVODE from the Sundials suite [29].

4 Parallel Computing in MBD4.1 Equation Formulation Aspects

Broadly speaking, mutibody formulations based on theNewton-Euler equations can be categorized into Cartesian (ordescriptor form) or recursive formulations. The Cartesian for-mulation [26,60,73,74] gained broader acceptance early on as itwas used in commercial packages such as ADAMS and DADS.In such an approach, the equations of motion can be formed ina straightforward manner, as illustrated in Section 2, simply bywriting the free body differential equations for all bodies in thesystem and append the algebraic (nonlinear) equations associatedwith each joint and constraint in the system.

A variety of recursive formulations have been proposed be-ginning with the early 1980s, first in the robotics community forchain manipulators and open-loop systems [4, 17] and later onextended to more general multibody systems, including systems


with closed-loop kinematic chains. Recursive formulations areinherently O(n) for the inverse dynamics problem, where n rep-resents the number of degrees of freedom. For the more com-plicated forward dynamics problem, various formulations havebeen proposed which, based on how they deal with the factoriza-tion of the resulting generalized mass matrix, can be classifiedas O(n3), O(n2), or O(n) and arise successively as a ”naturalconsequence of a progressive exploitation of the structure of themass matrix” as noted by Jain in [33]. Jain also showed that, ex-cept for minor differences, all linear complexity recursive algo-rithms, i.e., those that result in explicit derivations of the inverseof the mass matrix [5, 6, 11, 17, 68–70], are essentially the samealgorithm. The equations of motion, expressed in terms of a min-imal set of generalized states, are formed implicitly through suc-cessive outward (base-to-tips) and inward (tips-to-base) sweepsof the underlying multibody tree to recursively propagate kine-matics and articulated inertias, respectively.

More recent divide and conquer formulations [18, 19, 51]achieve O(n) complexity for serial implementation but, due totheir underlying binary tree structure, have logarithmic complex-ity when implemented in parallel.

Our goal here is neither to provide an exhaustive review ofprevious work in this area nor to advocate one choice over an-other. Instead, we will briefly comment on how they relate tothe use of parallel computing in MBD. The differences in termsof type, structure, and size of the resulting equations in the twoformulations can be summarized as follows:

Type For any non-trivial multibody system, the Cartesian for-mulation always results in a set of differential algebraicequations (DAEs).On the other hand, recursive formulations result in systemsof ordinary differential equations (ODEs) for open-loop sys-tems and minimal-size DAEs for closed-loop system (usingfor example the cut joint approach [78]).

Structure The Cartesian formulation does not take into accountthe structure (topology) of the MBD, at least not at the equa-tion formulation phase, but results in a set of equations witha very well defined and, most importantly, simple structure.Addition/removal of bodies or constraints is a trivial task.On the other hand, recursive formulations effectively im-pose certain computational paths (namely outward/inwardrecursive traversals of the topological tree) which result inmuch smaller but more complex, especially for O(n) for-mulations, system of equations. Addition or removal of bod-ies or constraints is a more complex task as it requires a re-evaluation of the system’s topology.

Size There is a significant disparity in the size of the resultingDAE or ODE systems for the two main classes of formula-tions considered here. For a system consisting of nb bodiesand having n degrees of freedom, a Cartesian formulationresults in a DAE with 6nb and 6nb−n differential and alge-

braic equations, respectively. If the system does not containany kinematic closed loops, a recursive formulation leadsto a system of n ODEs. However, in the presence of closedloops, them constraints introduced in the process of extract-ing the topological spanning tree lead to larger-size (namely2n + m) DAE systems. Note that if the multibody systemcontains many loops, such thatm = O(n), in addition to theincrease in problem size, one must also re-evaluate the al-gorithmic complexity of recursive formulations since O(n)algorithms actually perform as O(n+ nm2).

From the above observations it becomes apparent that thereis significant potential for fine grain parallelism within the Carte-sian framework. The fashion in which the equations of motionare generated and assembled, particularly the fact that constraintsinduced by a joint only involve states associated with the bod-ies connected by that joint, exposes a large number of indepen-dent/decoupled calculations. Hence, large problems posed inthe Cartesian framework fit well the SIMD model supported byGPGPU computing. On the other hand, the problem size andthe computational flow imposed by recursive formulations favortask parallelism that mirrors the structure of the spanning tree.OpenMP represents a good parallel computing platform with thecaveat that attention needs to be paid to load balancing, whichis influenced by the spanning tree associated with the system:vastly different branch sizes hurt load balancing as they’ll requiredifferent levels of traversing effort.

4.2 Numerical Solution AspectsThe numerical solution of dynamics problems involving

frictional contact in a DVI formulation (such as that in Eq. 2)require new numerical approaches as one moves towards verylarge-scale many-body problems involving hundreds of thou-sands to millions of bodies and contacts. Classical approachesbased on posing and solving linear complementarity problems(LCP) obtained as a result of polyhedral approximations of thefriction cone not only suffer from accuracy issues stemming fromthe induced anisotropies but also lead to an increase in problemsize that quickly becomes prohibitive. These limitations can beavoided by transforming the original nonlinear complementar-ity problem into a cone complementarity problem (CCP) whichcan be shown [3, 76] to represent the first-order optimality con-ditions of the constrained optimization problem of Eq. 4. Usualapproaches to solving this problem rely on fixed-point iterations(Jacobi or Gauss-Seidel) with projection on a convex set; how-ever, since these methods require a large number of iterationswhen handling large systems of engineering relevance, the CCPsolution, which can account for up to two thirds of the total so-lution time, leads to prohibitively long simulations. Moreover,some of these methods (e.g. Gauss-Seidel) are not amenable toparallel computing. Alternative solvers, such as gradient pro-jected minimum residual (for frictionless problems), precondi-


tioned spectral projected gradient with fallback, and primal-dualinterior point methods, have been shown to have improved con-vergence properties leading to significant speedups [27, 28].

The efficiency of MBD simulations is directly influenced bythe size of the integration step-size. Large step sizes are desir-able especially in many-body problems with frictional contact,regardless of the modeling approach (DVI or DEM), since eachtime step requires a costly collision detection stage that can in-volve millions of contact events. As discussed in §2, the exist-ing integration schemes require typical step sizes of h = 10−4

for a DVI approach and as low as h = 10−6 for DEM. In ourimplementation, each step of the DVI uses half-implicit numer-ical integration that calls for the solution of a large optimizationproblem. It quickly becomes apparent that the promise of largestep-sizes associated with fully implicit methods is very attrac-tive, since in this case the step size is unrestricted by stabilityconsiderations. However, fully implicit integration schemes leadto nonlinear systems typically solved using Newton-like meth-ods and hence they require (i) the ability to evaluate Jacobianinformation and (ii) efficient linear solvers. Note that linearsolvers are important not only for implicit integration, but alsofor interior point methods used to solve the CCP associated withthe problem in Eq. 4. Even though very sparse, the sheer sizeof the resulting matrices in problems such as granular dynam-ics or Lagrangian formulations for fluid-solid interaction (e.g.,SPH) make direct methods for linear systems less attractive.The alternative is to use matrix-free iterative techniques, suchas the Krylov subspace iterative methods, which do not requireexplicitly forming the system’s matrix but only matrix-vectorproducts [34]. Unfortunately, for fast convergence these itera-tive methods still require the use of adequate preconditioners.While domain-specific preconditioners for MBD problems arestill lacking, general-purpose preconditioners are available forall parallel computing platforms discussed herein: PETSc [7]and hypre [16] libraries for MPI computing, Intel’s MKL forOpenMP, and CUSP [9] for GPGPU.

4.3 Software Development AspectsThe parallel computing of mid 1990s relied extensively on

posix threads [57]. A very low level approach to parallel com-puting, posix threads are currently used only when performancearguments trump the development and maintenance effort associ-ated with this parallel programming approach. Today, alternativesolutions provide a parallel programming entry point in whichthe technical effort is focused more on modeling and numericalsolution aspects and less on the implementation constructs re-quired by these programming platforms/models.

While it is clear that when measured by the amount of datathat needs to be processed during simulation very large modelswill always fall back on a distributed computing model supportedby the MPI standard, simulations that fit on one workstation can

be run either with OpenMP or relying on GPGPU computing.Debating which of the two platforms is superior falls outside thescope of this paper. However, for task parallelism, where severalindependent and sizable tasks need to be performed, OpenMPhas intrinsic support that will give it an edge. For instance, for atypical multibody dynamics problem with 300 bodies, 100 joints,and 200 forces, OpenMP parallelizes the evaluation of the Jaco-bian associated with implicit integration very naturally: one taskwill be associated with the computation of the contribution of the300 bodies, another task will be associated with the evaluation ofthe contribution of the 100 joints, and yet a third task will handlethe contribution of the 200 forces. Three OpenMP tasks can belaunched, each one of them with an inner loop that can be par-allelized, and native dynamic load balancing will distribute theload and keep all cores busy. Here GPU computing is less attrac-tive since the problem and associated coarse grain parallelismdoes not fit the SIMD model.

For many-body dynamics though, such as granular dynam-ics simulation, mobility on discrete terrain, fluid-solid interac-tion using SPH, etc., where there are tens of thousands of datainstances to be processed, GPGPU is very attractive. Collisiondetection, evaluating the force at each contact point, or advanc-ing the state of the system through explicit integration are com-putations that map well on GPUs owing to the SIMD nature ofthe job at hand. It is this type of fine grain parallelism; i.e., verylarge number of identical operations that need to be performed ona large number of items, that is the forte of GPGPU computing.

There is a gray area with no clear answer to whetherOpenMP or GPU computing has the upper hand. In this casewe implemented a coding style that allowed us to experimentwith either platform by setting an environment variable at com-pile time. To support this, the code will have to be modified ina very limited and localized fashion. Specifically, both for datastructures and function definitions a different syntax is neededdepending on whether GPU or CPU computing is used. As anexample, consider the code in the following listings. The pur-pose of this code is to update the velocity of each body giventhe force acting on it. If GPU computing is used, this should beperformed with a kernel with one thread per body. If CPU com-puting is used, the velocity update would be achieved by loopingover all bodies in the system using OpenMP through the inclu-sion of a simple #pragma omp parallel for.

First, a function is written to perform the desired operation.In this case, the function should increment the velocity of a givenbody based on the mass, inertia, force, torque, and time step.Note that this function is defined with host device ,which means it is compiled to be executed on both the CPU(host) and GPU (device).


Listing 1. Function to update the velocity of a bodyh o s t d e v i c e

void a d d F o r c e s ( u i n t& i , r e a l ∗ inv mass , r e a l 3 ∗ f o r c e s ,r e a l 3 ∗ v e l ) {

v e l [ i ] += f o r c e s [ i ] ∗ i n v m a s s [ i ] ;}

The above function, in which the value of the step sizeh has already been factored in the inverse of the mass ma-trix, should be called in different ways if CPU or GPU isused. If the GPU is used, a GPU kernel (shown in thelisting below) should call the function. Note the inclusionof global , which signifies a kernel function, and themacro INIT CHECK THREAD BOUNDED, which ensures thatthe thread IDs executing the kernel are appropriate.Listing 2. GPU kernel to update the velocity of all bodies in the system

g l o b a lvoid d e v i c e a d d F o r c e s ( r e a l ∗ mass , r e a l 3 ∗ f o r c e s , r e a l 3 ∗

v e l ) {INIT CHECK THREAD BOUNDED(INDEX1D , n u m o b j c o n s t ) ;a d d F o r c e s ( index , mass , f o r c e s , v e l ) ;

}

For parallel execution on the CPU, the code should loop overall bodies in the system using an OpenMP for-loop, invoking thesingle-body update function as in the following listing.

Listing 3. Host loop to update the velocity of all bodies in the systemvoid h o s t a d d F o r c e s ( r e a l ∗ mass , r e a l 3 ∗ f o r c e s ,

r e a l 3 ∗ v e l ) {#pragma omp p a r a l l e l f o r

f o r ( u i n t i n d e x = 0 ; i n d e x < num obj ; i n d e x ++) {a d d F o r c e s ( index , mass , f o r c e s , v e l ) ;

}}

Finally, a compile-time flag, ENABLE GPU, is used to con-trol which parallel computing model is used. If the flag is de-fined, the GPU kernel is called; otherwise, the host function iscalled.Listing 4. Velocity update, selecting between CPU and GPU execution

# i f d e f ENABLE GPUCOPY TO CONST MEM( num obj ) ;d e v i c e a d d F o r c e s <<<BLOCKS( num obj ) , THREADS>>>(

d e v i c e m a s s d a t a ,d e v i c e f r c d a t a ,d e v i c e v e l d a t a ) ;

# e l s eh o s t a d d F o r c e s (

d e v i c e m a s s d a t a . d a t a ( ) ,d e v i c e f r c d a t a . d a t a ( ) ,d e v i c e v e l d a t a . d a t a ( ) ) ;

# e n d i f

This strategy is used for all code sections in which parallelcomputing can be leveraged. A single function is written to per-form the necessary operations, capable of executing on the CPUand GPU (Listing 1). One wrapper function is written for theGPU, where a kernel calls the function within each thread (List-

ing 2), while a second wrapper is written for the CPU, where thefunction is called from within an OpenMP for-loop (Listing 3).Finally, the compile-time flag is used to selectively compile theappropriate wrapper function in the code (Listing 4).

Note that primitive functions are handled by Thrust directly.For example, sorting a vector can be accomplished by callingthrust::sort(myVec.begin(), myVec.end()); IfmyVec is a host vector, without any user intervention, Thrustwill use OpenMP to sort it. If myVec is a device vector, Thrustwill use the GPU.

5 Examples: GPGPU, OpenMP, and MPI Parallel Com-puting in MBDThree applications are discussed next to showcase the use of

the parallel computing techniques discussed above. GPU com-puting is used in conjunction with the dynamics of a tracked ve-hicle operating on granular terrain. An OpenMP-enabled granu-lar dynamics simulation is used to identify pattern formation ingranular material that is shaken at a critical frequency. Finally,MPI-enabled distributed computing is used to investigate the dy-namics of a wheeled vehicle operating on discrete deformableterrain. These models were run in Chrono, an open source par-allel simulation framework for computational dynamics [42,72].

5.1 GPU Computing Example: Light Tracked VehicleMobility Simulation

A detailed account of how GPU computing is used to solvemultibody dynamics problems can be found in [54]. The keykernels of the GPU implementation are listed in the followingpseudocode:

Parallel Kernels for Solution of Dynamics Problem

1. Parallel Collision Detection2. (Body parallel) Force kernel3. (Contact parallel) Contact pre-processing kernel4. Inner Iteration Loop:

(a) (Contact parallel) CCP contact kernel(b) (Bilateral-Constraint parallel) CCP constraint kernel(c) (Body parallel) Body velocity update kernel

5. (Body parallel) Time integration kernel

Using a combination of joints and linear actuators, a trackedvehicle model was simulated navigating over deformable terrainmade up of gravel-type granular material. The vehicle is modeledto represent a small, lightweight tracked vehicle much like anautonomous robot.

There are two tracks, each with 61 track shoes (see Fig. 4).Each track shoe is made up of two cylinders and three rectangu-lar plates and has a mass of 0.34 kg. Each shoe is connected toits neighbors using one revolute joint on each side. Within each


FIGURE 4. Light tracked vehicle operating on gravel.

FIGURE 5. Magnitude of force experienced by one revolute joint ongranular terrain.

track there are five rollers, each with a mass of 15 kg, one idlerand one sprocket both with a mass of 15 kg. The chassis has amass of 200 kg and moments of inertia were computed for allparts using a CAD package. The purpose of the rollers is to keepthe tracks separated and support the weight of the vehicle as itmoves forward. The idler keeps the track tensioned. It is usu-ally modeled with a linear spring/actuator but for the purposesof demonstration it was attached, using a revolute joint, to thevehicle chassis. The sprocket is used to drive the vehicle and isattached, using another revolute joint, to the chassis. Torque isapplied to drive the track, with each track driven independentlyof the other.

The track for the vehicle was created by first generating aring of connected track shoes. This ring was dropped onto asprocket, five rollers, and an idler which was connected to thechassis using a linear spring. The idler was pushed with 2000 Nof force until the track was tensioned and the idler had stoppedmoving. This pre-tensioned track was then saved to a data fileand loaded for the simulation of the complete vehicle.

Figure 5 shows the joint forces for one revolute joint when

FIGURE 6. Top view of simulation results obtained for Γ = 0.4 andf? = 0.38

the tracked vehicle was simulated as it moved over a bed of84,000 granular particles. The particles were modeled as largepieces of gravel with a radius of 0.075 m, and a density of1900 kg/m3. A torque of 100 Nm was applied to both sets oftracks to move the vehicle. Unlike the case where the vehiclemoves on a flat section of ground, the forces experienced by therevolute joints are much noisier. Individual grains slip and stickunder the tracks as the vehicle moves causing large vibrationsto travel through the shoes. These vibrations would be reducedwhen modeling a more compliant terrain material that can dissi-pate energy on contact.

5.2 OpenMP Example: Pattern Formation in GranularDynamics

This simulation was used to study the pattern formation ingranular material shaken vertically. These patterns emerge inlayers of material that is driven only by floor vibrations and havebeen studied extensively both in 2D and in 3D [10, 41, 44, 77].This phenomenon is important in a variety of fields, from study-ing the formation of crystalline structures to processing and han-dling different types of granular material [32]. Without frictionthese patterns cannot emerge [47]. The simulation is challeng-ing due to the number of particles involved and the velocity andamplitude of the vibrations of the moving plate.

The simulation was performed on a Intel Xeon E5-2630with 6 physical cores and 12 virtual cores. The simulation in-


cluded 60,000 rigid particles, was advanced with a time step of5× 10−4 s, and was used to investigate the system dynamics for30 s. An OpenMP parallel version of the Accelerated ProjectedGradient Descent solver was used with 50 iterations performedat each time step [27]. The parameters for friction, packing ratio,and vibration amplitude and frequency used were as providedin [10]. A friction value of µ = 0.5 was chosen. The parti-cle size, gravity and the total number of particles were used tocompute the non-dimensional frequency f? and amplitude Γ ofthe driving plate. Figure 6 shows a simulation with parametersΓ = 0.4 and f? = 0.38. The results obtained are in line with theexperimental data, which confirms that this choice of parametersproduces a hexagonal pattern in the granular material [10].

5.3 MPI Example: Vehicle MobilityWhen relying on MPI parallel computing, Chrono divides a

simulation domain into a number of sub-domains that can eachbe simulated in parallel on discrete compute nodes. The solvertransparently sets up the necessary communication channels be-tween participating compute nodes and carries out the simulationincluding collision detection in a distributed and parallel fash-ion [43].

The general algorithmic flow of the domain decompositionapproach for DEM simulations is given in the following pseu-docode, which specifies the operations performed by a givennode simulating a single sub-domain. All nodes operate indepen-dently, in parallel, subject to communication and synchronizationbarriers.

PSEUDOCODE DEM MPI()(1) Identify boundaries of sub-domain Si(2) Initialize communication pathways to

each sub-domain Sj(3) Initialize data structures(4) for t = 0 to tend(5) Perform collision detection(6) Compute collision forces(7) Barrier: Intermediate communica-

tion of net forces on shared objects(8) Compute net forces(9) Perform integration to update state(10) Barrier: Final synchronization(11) Update set of bodies in sub-

domain(12) t← t+ h(13) endfor

The example considered represents a replica of the Marsrover Opportunity. Details on the rover model, including inertiaand geometry properties, can be found in [27]. The rover chassiswas 100 kg; the mass of each wheel was 3 kg. The wheels wererepresented as cylinders with radius 10 cm and width 12.5 cm. A

FIGURE 7. Snapshot from Mars Rover simulation at t = 4.41s, withtwo million terrain bodies using MPI-enabled parallelism on 64 cores.Animation available at [71].

set of 30 grousers distributed evenly around the circumference ofeach wheel extend across the wheel’s width. Each grouser has athickness of 0.5 cm and extends away from the wheel by 0.5 cm.The terrain was composed of 2,016,000 rigid spherical bodieswith radius 5 mm and mass 4.5 g. The elements in the terrainwere initialized in random positions above the simulation domainand allowed to settle under gravity. The simulation domain wasrectangular, with length of 3 m in the driving direction and widthof 1.5m. All six wheels of the rover were driven with constantangular velocity of π rad/s, or 30 rpm. The integration time stepwas 10−5 s and 6 s of dynamics were simulated, requiring about298 hours of computation time. The DEM stiffness and damp-ing parameters had the values kn = kt = 2 × 105, γn = 750,and γt = 0. The simulation domain was partitioned into 64 sub-domains by dividing both the length and the width evenly into 8sub-domains in each direction. The resulting sub-domains wererectangular, with length 0.375 m and width 0.1875 m. The depthdirection was not divided.

A snapshot from the simulation can be seen in Fig. 7. Theterrain particles are colored by sub-domain and shared bodiesare colored white. Further, the wheels of the rover are checkeredwith the color of the master sub-domain. In Fig. 7 the wheels arepink, which indicates that the pink sub-domain has the mastercopy of the entire rover vehicle. Figure 8 shows the “footprint”of the rover. The terrain particles are colored uniformly, hidingthe sub-domain division, and the rover and its wheels are notshown. The amount of simulation time is still prohibitively largefor this problem, which aims at understanding how different ve-hicle topologies influence the dynamics of the rover. By orders ofmagnitude, the simulation bottleneck is associated with the data


FIGURE 8. Snapshot from Mars Rover simulation at t = 4.41s,showing the “footprint” of the rover wheels with grousers in granularterrain composed of two million rigid bodies.

communication required by an exchange protocol that handledthe cross-boundary motion and interaction of bodies.

6 Closing RemarksOnce the privilege of national research labs or supercom-

puting centers, parallel computing is now ubiquitous. In fact, itis virtually impossible today to purchase a workstation or laptopthat doesn’t have at least four cores that can support parallel com-puting. These two observations frame the answer to the why inthe title of this contribution: sequential computing promises onlymodest gains in efficiency, insufficient for fueling the evolutionof the MBD field in the next decade. The continuous drive inMBD towards multi-physics problems yielding ever larger sys-tems of equations provides the backdrop for the when in the ti-tle of the paper. Fluid-structure interaction, granular dynamics,and vehicle mobility on deformable discrete terrain represent ex-ample applications that will stand to benefit from the interplaybetween MBD and parallel computing. Finally, using as a justi-fication several applications in which we have relied on differentparallel computing platforms, we discussed aspects related to themodeling, numerical solution, and software implementation as-pects of MBD simulation; i.e., we addressed the how aspect al-luded to in the title of the contribution. Several comments madeat various points in this paper are summarized as follows:Parallel computing platform. Large models involving anamount of data that exceeds the memory available on a work-station require a distributed computing model facilitated by MPIparallelism. As discussed in §5.3, MPI incurs high communica-tions cost that presently can limit duration or size of the prob-

lem being analyzed. This is particularly the case if the simula-tion proceeds at small step-sizes and data communication dwarfsany other component of the numerical solution. If the problemis small enough – tens to hundreds of GB on the CPU and upto 12 GB on the GPU – the approach described in §4.3 allowsone to experiment with OpenMP or GPU computing and makea choice on a case-by-case basis. GPGPU computing is attrac-tive for fine level granularity when the computational task fitsthe SIMD model. Problems characterized by task parallelism areideally suited for OpenMP or related approaches (cilk, TBB).Equation formulation. Recursive formulations do not have alevel of granularity that fits well the SIMD model but are wellsuited for task-level parallelization. The finer degree of granular-ity associated with the Cartesian formulation makes it ideal forGPGPU computing.Numerical solution. Minimizing time to solution may requireadoption of numerical solution techniques different from thoseused for sequential calculations. This might require embracingstrategies that are manifestly suboptimal in a sequential comput-ing framework but which expose a high degree of parallelismand come with low synchronization demands. Moreover, effec-tive parallel computing requires a shift in how we think aboutwriting software. Rather than organizing data structures for theconvenience of the programmer by reflecting the domain specificorganization of a problem (bodies, joints, forces, etc.), the datastructures should be organized to reflect the hardware constraintsof the underlying architecture, which most often prefers data tobe organized in long contiguous arrays that are suitable for par-allel processing and ideal for vectorization. The latter approachthough renders the software hard to design, debug, and maintainas it loses the connection to the domain problem that it solves.

We conclude with a few recommendations that coalescedduring several years of experience with using parallel comput-ing in MBD simulation.

1. The most important observation is that number crunchingis basically free while data movement/memory transactionsare one to two orders of magnitude more expensive. Formemory transactions the most important aspect is memoryaccess patterns. For GPGPU computing uncoalesced mem-ory accesses reduce bandwidth, while for OpenMP violatingspatial and temporal locality in data accesses increase thelikelihood of cache misses. The recommendation is at thesoftware design stage to organize model data in structuresthat are poised to minimize the amount of traffic to mainmemory or inter-core communication (see the ”‘Numericalsolution”’ remark above).

2. If available, use domain libraries: the solution is alreadythere, bug-free and optimized. Reinventing the wheel maybe useful for learning purposes but is unlikely to lead to per-formance improvements.

3. Debugging parallel code is challenging. Although involving


a steep learning curve, using debuggers is the only way toaccess program state information that cannot be replicatedby rudimentary debugging techniques (such as using printstatements). NVIDIA’s cuda-gdb for GPU computing, In-tel’s Parallel Studio Suite for OpenMP computing, and In-tel’s Cluster Studio for MPI computing represent debuggingsolutions widely used in the developer community.

4. Use dedicated profiling tools as the only way to conclusivelyfind the hot spots and bottlenecks in a parallel program andthen direct the optimization effort where it matter most. Inthis context, the recommendation is to always keep in mindAmdahl’s law [1] and parallelize the portions of the codethat are the actual computational bottlenecks. Finally, recallthat premature optimization is the root of all evil [36].

AcknowledgementsThe hardware used to produce the results reported in this paper

was procured through the Army Research Office instrumentation grantW911NF-11-1-0327. We thank NVIDIA for sponsoring our researchprograms in the area of high performance computing. We are gratefulto Alex Dirr, Prateek Gupta, and Arindam Sinha for their feedback on adraft of this document.

REFERENCES[1] G. Amdahl. Validity of the single processor approach to

achieving large-scale computing capabilities. AFIPS Con-ference Proceedings, 30:483485, 1967.

[2] K. S. Anderson and S. Duan. Highly parallelizable low or-der dynamics algorithm for complex multi-rigid-body sys-tems. AIAA Journal of Guidance, Control and Dynamics,23(2):355–364, 2000.

[3] M. Anitescu and G. D. Hart. A constraint-stabilizedtime-stepping approach for rigid multibody dynamics withjoints, contact and friction. International Journal for Nu-merical Methods in Engineering, 60(14):2335–2371, 2004.

[4] W. W. Armstrong. Recursive solution to the equationsof motion of an N-link manipulator. In Proc. 5th WorldCongress on Theory of Machines and Mechanism, vol-ume 2, pages 1343–1346, Montreal, 1979.

[5] D.S. Bae and E.J. Haug. A recursive formulation for con-strained mechanical system dynamics: Part I. open loopsystems. Mechanical Structures and Machines, 15(3):359–382, 1987.

[6] D.S. Bae and E.J. Haug. A recursive formulation for con-strained mechanical system dynamics: Part II. closed loopsystems. Mechanical Structures and Machines, 15(4):481–506, 1987.

[7] S. Balay, J. Brown, K. Buschelman, V. Eijkhout,W. Gropp, D. Kaushik, M. Knepley, L. CurfmanMcInnes, B. Smith, and H. Zhang. PETSc users man-

ual revision 3.3. http://www.mcs.anl.gov/petsc/petsc-3.3/docs/manual.pdf, 2012.

[8] O. A. Bauchau. Parallel computation approaches forflexible multibody dynamics simulations. Journal of theFranklin Institute, 347(1):53 – 68, 2010.

[9] Nathan Bell and Michael Garland. CUSP: Generic parallelalgorithms for sparse matrix and graph computations, 2012.Version 0.3.0.

[10] C. Bizon, M. D. Shattuck, J. B. Swift, W. D. McCormick,and H. L. Swinney. Patterns in 3D Vertically OscillatedGranular Layers: Simulation and Experiment. Phys. Rev.Lett., 80(1):57–60, 1998.

[11] H. Brandl, R. Johanni, and M. Otter. A very efficient al-gorithm for the simulation of robots and similar multibodysystems without inversion of the mass matrix. In IFAC/I-FIP/IMACS Symposium (lst), Vienna, Austria, pages 95–100, 1986.

[12] J. Cuadrado, J. Cardenal, P. Morer, and E. Bayo. Intelligentsimulation of multibody dynamics: space-state and descrip-tor methods in sequential and parallel computing environ-ments. Multibody System Dynamics, 4(1):55–73, 2000.

[13] R. A. Dalrymple and B. D. Rogers. Numerical modeling ofwater waves with the SPH method. Coastal Engineering,53(2):141–147, 2006.

[14] R. Dennard, F. Gaensslen, L. Rideout, E. Bassous, andA. LeBlanc. Design of ion-implanted MOSFET’s with verysmall physical dimensions. IEEE Journal of Solid-StateCircuits, 9(5):256–268, 1974.

[15] G.A. Dilts. Moving-least-squares-particle hydrodynamics–I. consistency and stability. International Journal for Nu-merical Methods in Engineering, 44(8):1115–1155, 1999.

[16] R.D. Falgout and U. Meier Yang. hypre: a library of highperformance preconditioners. In Preconditioners, LectureNotes in Computer Science, pages 632–641, 2002.

[17] R. Featherstone. The calculation of robot dynamics usingarticulated-body inertias. International Journal of RoboticsResearch, 2(1):13–30, 1983.

[18] R. Featherstone. A divide-and-conquer articulated-body al-gorithm for parallel O(log(n)) calculation of rigid body dy-namics. Part 2: Trees, loops and accuracy. InternationalJournal of Robotic Research, 18(3):876892, 1999.

[19] A. Fijany, I. Shraf, and G.M.T. DEleuterio. Parallel O(logn) computation of manipulator forward dynamics. IEEETrans. Robotics and Automation, 11(3):389400, 1995.

[20] Michael J Flynn. Some computer organizations andtheir effectiveness. Computers, IEEE Transactions on,100(9):948–960, 1972.

[21] R. A. Gingold and J. J. Monaghan. Smoothed parti-cle hydrodynamics-theory and application to non-sphericalstars. Monthly Notices of the Royal Astronomical Society,181(1):375–389, 1977.

[22] F. Gonzalez, A. Luaces, U. Lugrıs, and M. Gonzalez. Non-


intrusive parallelization of multibody system dynamic sim-ulations. Computational Mechanics, 44(4):493–504, 2009.

[23] R. Graham, T. Woodall, and J. Squyres. Open MPI: A flex-ible high performance MPI. In Parallel Processing and Ap-plied Mathematics, pages 228–239. Springer, 2006.

[24] W. Gropp, E. Lusk, and A. Skjellum. Using MPI: PortableParallel Programming with the Message-Passing Interface,Second Edition. MIT Press, 1999.

[25] William Gropp. MPICH2: A new start for MPI implemen-tations. In Recent Advances in Parallel Virtual Machineand Message Passing Interface, page 7. Springer, 2002.

[26] E. J. Haug. Computer-Aided Kinematics and Dynamics ofMechanical Systems Volume-I. Prentice-Hall, EnglewoodCliffs, New Jersey, 1989.

[27] T. Heyn. On the Modeling, Simulation, and Visualiza-tion of Many-Body Dynamics Problems with Frictionand Contact. PhD thesis, Department of Mechani-cal Engineering, University of Wisconsin–Madison,http://sbel.wisc.edu/documents/TobyHeynThesis PhDfinal.pdf,2013.

[28] T. Heyn, M. Anitescu, A. Tasora, and D. Negrut. UsingKrylov subspace and spectral methods for solving comple-mentarity problems in many-body contact dynamics sim-ulation. International Journal for Numerical Methods inEngineering, 95(7):541–561, 2013.

[29] A.C. Hindmarsh, P.N. Brown, K.E. Grant, S.L. Lee, R. Ser-ban, D.E. Shumaker, and C.S. Woodward. SUNDI-ALS: Suite of nonlinear and differential/algebraic equa-tion solvers. ACM Transactions on Mathematical Software(TOMS), 31(3):363–396, 2005.

[30] Jared Hoberock and Nathan Bell. Thrust: A parallel tem-plate library, 2010. Version 1.7.0.

[31] K. Iglberger and U. Rude. Massively parallel rigid body dy-namics simulations. Computer Science-Research and De-velopment, 23(3-4):159–167, 2009.

[32] Heinrich M. Jaeger, Sidney R. Nagel, and Robert P.Behringer. Granular solids, liquids, and gases. Rev. Mod.Phys., 68:1259–1273, Oct 1996.

[33] A. Jain. Unified formulation of dynamics for serial rigidmultibody systems. J. Guidance, 14(3):531–542, 1991.

[34] N. Khude, I. Stanciulescu, D. Melanz, and D. Negrut. Effi-cient parallel simulation of large flexible body systems withmultiple contacts. ASME Journal of Computational andNonlinear Dynamics, 8(4):041003–041003, 2013.

[35] Y. A. Khulief. Modeling of impact in multibody systems:An overview. Journal of Computational and Nonlinear Dy-namics, 8:021012, 2013.

[36] D. Knuth. Structured programming with go to statements.ACM Computing Surveys (CSUR), 6(4):261–301, 1974.

[37] S. Koshizuka, A. Nobe, and Y. Oka. Numerical analysisof breaking waves using the moving particle semi-implicitmethod. International Journal for Numerical Methods in

Fluids, 26(7):751–769, 1998.[38] LAMMPS. A Molecular Dynamics Simulator.

http://lammps.sandia.gov/, 2013.[39] LIGGGHTS. Open Source Discrete Element

Method Particle Simulation Code. http://cfdem.dcs-computing.com/?q=OpenSourceDEM, 2013.

[40] L. B. Lucy. A numerical approach to the testing of the fis-sion hypothesis. The Astronomical Journal, 82:1013–1024,1977.

[41] S. Luding, E. Clment, J. Rajchenbach, and J. Duran. Sim-ulations of pattern formation in vibrated granular media.EPL (Europhysics Letters), 36(4):247, 1996.

[42] H. Mazhar, T. Heyn, A. Pazouki, D. Melanz, A. Seidl,A. Bartholomew, A. Tasora, and D. Negrut. Chrono: a par-allel multi-physics library for rigid-body, flexible-body, andfluid dynamics. Mechanical Sciences, 4(1):49–64, 2013.

[43] Hammad Mazhar, Toby Heyn, and Dan Negrut. A scal-able parallel method for large collision detection prob-lems. Multibody System Dynamics, 26:37–55, 2011.10.1007/s11044-011-9246-y.

[44] Francisco Melo, Paul Umbanhowar, and Harry L. Swinney.Transition to parametric wave patterns in a vertically os-cillated granular layer. Phys. Rev. Lett., 72:172–175, Jan1994.

[45] J. J. Monaghan. On the problem of penetration in particlemethods. Journal of Computational Physics, 82(1):1–15,1989.

[46] J. J. Monaghan. Smoothed particle hydrodynamics. Re-ports on Progress in Physics, 68(1):1703–1759, 2005.

[47] S. J. Moon, J. B. Swift, and H. L. Swinney. Role of frictionin pattern formation in oscillated granular layers. Phys. Rev.E, 69:031301, 2004.

[48] MPICH2. High Performance Portable MPI.http://http://www.mpich.org/, 2013.

[49] L. Mraz and M. Valasek. Solution of three key problems formassive parallelization of multibody dynamics. MultibodySystems Dynamics, 29(1):21–39, JAN 2013.

[50] MSC.Software. ADAMS User Manual. Available online athttp://www.mscsoftware.com, 2014.

[51] R.M. Mukherjee and K.S. Anderson. Orthogonal comple-ment based divide-and-conquer algorithm for constrainedmultibody systems. Nonlinear Dyn, 48:199–215, 2007.

[52] A. Munshi, B. Gaster, T. Mattson, and D. Ginsburg.OpenCL programming guide. Addison-Wesley Profes-sional, 2011.

[53] MVAPICH2. MVAPICH: MPI over InfiniBand,10GigE/iWARP and RoCE. http://mvapich.cse.ohio-state.edu/performance/mvapich2/interNode.shtml, 2013.

[54] D. Negrut, A. Tasora, M. Anitescu, H. Mazhar, T. Heyn,and A. Pazouki. Solving large multi-body dynamics prob-lems on the GPU. GPU Gems Vol. 4, pages 269–280, 2011.

[55] D. Negrut, A. Tasora, H. Mazhar, T. Heyn, and P. Hahn.


Leveraging parallel computing in multibody dynam-ics. Multibody System Dynamics, 27:95–117, 2012.10.1007/s11044-011-9262-y.

[56] NVIDIA. CUDA Programming Guide. Available on-line at http://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html, 2014.

[57] Open Group. IEEE Std. 1003.1. Available online athttp://www.unix.org/version3/ieee std.html, 2004.

[58] Open MPI. A High Performance Message Passing Library.http://www.open-mpi.org/, 2013.

[59] OpenMP. Specification Standard 4.0. Available online athttp://openmp.org/wp/, 2013.

[60] N. Orlandea, M. A. Chace, and D. A. Calahan. A sparsity-oriented approach to the dynamic analysis and design ofmechanical systems – part I and part II. Transactions of theASME Journal of Engineering for Industry, pages 773–784,1977.

[61] J. S. Pang and D. E. Stewart. Differential variational in-equalities. Mathematical Programming, 113:1–80, 2008.

[62] M. Parks, R. Lehoucq, S. Plimpton, and S. Silling. Im-plementing peridynamics within a molecular dynamicscode. Computer Physics Communications, 179(11):777–783, 2008.

[63] David A Patterson and John L Hennessy. Computer organi-zation and design: the hardware/software interface. Mor-gan Kaufmann, 4 edition, 2011.

[64] A. Pazouki, H. Mazhar, and D. Negrut. Parallel colli-sion detection of ellipsoids with applications in large scalemultibody dynamics. Mathematics and Computers in Sim-ulation, 82(5):879 – 894, 2012.

[65] A. Pazouki and D. Negrut. Direct simulation of lateralmigration of buoyant particles in channel flow using GPUcomputing. In Proceedings of the 32nd Computers and In-formation in Engineering Conference, CIE32, August 12-15, Chicago, IL, USA. American Society of Mechanical En-gineers, 2012.

[66] S. Plimpton. Fast parallel algorithms for short-rangemolecular dynamics. Journal of Computational Physics,117(1):1–19, 1995.

[67] G. Quaranta, P. Masarati, and P. Mantegazza. Multibodyanalysis of controlled aeroelastic systems on parallel com-puters. Multibody System Dynamics, 8(1):71–102, AUG2002.

[68] G. Rodriguez. Kalman filtering, smoothing and recur-sive robot arm forward and inverse dynamics. IEEE J. ofRobotics and Automation, 3(6):624–639, 1987.

[69] G. Rodriguez, A. Jain, and K. Kreutz-Delgado. Spatial op-erator algebra for multibody system dynamics. J. Astronau-tical Sciences, 40(1):27–50, 1992.

[70] D. Rosenthal. Order n formulation for equations of motionof multibody systems. In SDIO/NASA Workshop on Multi-body Simulations, pages 1122–1150. JPL Pub. D-5190, Jet

Propulsion Lab., Pasadena, CA, 1987.[71] SBEL. Movies, Physics-Based Modeling and Simulation.

http://sbel.wisc.edu/Animations, 2013.[72] SBEL. Software Downloads, Simulation-Based Engi-

neering Laboratory, University of Wisconsin-Madison.http://sbel.wisc.edu/Software, 2013.

[73] A. Shabana. Multibody Systems. John Wiley and Sons,New York, 1989.

[74] A. A. Shabana. Dynamics of Multibody Systems. Cam-bridge University Press, third edition, 2005.

[75] David E. Stewart and Jeffrey C. Trinkle. An implicit time-stepping scheme for rigid-body dynamics with inelasticcollisions and Coulomb friction. International Journal forNumerical Methods in Engineering, 39:2673–2691, 1996.

[76] A. Tasora and M. Anitescu. A matrix-free cone com-plementarity approach for solving large-scale, nonsmooth,rigid body dynamics. Computer Methods in Applied Me-chanics and Engineering, 200(5-8):439–453, 2011.

[77] Paul B. Umbanhowar, Francisco Melo, and Harry L. Swin-ney. Localized excitations in a vertically vibrated granularlayer. Nature, 382(6594):793–796, Aug 1996.

[78] J. Wittenburg. Dynamics of Systems of Rigid Bodies. B. G.Teubner, Stuttgart, 1977.


PARALLEL COMPUTING IN MULTIBODY SYSTEM DYNAMICS:...

Documents

Transcript of PARALLEL COMPUTING IN MULTIBODY SYSTEM DYNAMICS:...