Towards the 3D multiphase flow simulation

of collapsing columns and pyroclastic flow propagation

T. Esposti Ongaro(1) , A. Neri(1), C. Cavazzoni(2), G. Erbacci(2), G. Macedonio(3) , M.V. Salvetti(4), M. de’ Michieli Vitturi(5) andF. Beux(6)

(1) Centro per la modellistica fisica e pericolosità dei processi vulcanici – INGV – Pisa - Italy(2) High Performance Computing Division - CINECA –

Casalecchio di Reno (Bologna) - Italy(3) Osservatorio Vesuviano – INGV- Napoli - Italy(4) Dip.to di Ingegneria Aerospaziale – Università degli Studi di Pisa – Italy(5) Dip.t di Matematica - Università degli Studi di Pisa – Italy(6) Scuola Normale Superiore – Pisa- Italy

The main objective of the EU-funded EXPLORIS project consists in the quantitative assessment of explosive eruption risk in densely populated EU regions through the development of volcanic risk facilities such as:

1. 3D Simulation codes (fallout and pyroclastic flows)2. Vulnerability functions and databases3. Probabilistic risk assessment protocols

Framework

Why do we need the 3D modelling?

Fundamental physical processes are intrinsically 3D Examples: Turbulence, fluid dynamic instabilities, etc.

Mt. St. Helens, 1980 (courtesy of B.Voight)

Boundary conditions are also mostly 3D Examples: Volcanic topography, wind, etc.

The Red Zone around the Somma-Vesuvio complex (from Pareschi et al, 2000)

The PDAC model (Neri et al., 2003)

Mass conservation

Gas components conservation

Momentum balance

Enthalpy balance

+ Constitutive equations (e.g. EOS, stress tensors, LES turbulence closure, etc.)

resulting in 3D in [5*(n+1) + (m-1)] independent equations(e.g., 16 equations using n=2 particulate phases and m=2 gas components )

COMPUTATIONALLY VERY DEMANDING!

Code re-engineering

• Implementation of a new modular structure in Fortran95 (easier to use and to mantain)

• Parallelization of the 3D code by adopting a Domain Decomposition approach (using “ghost cells”)

• Adoption of Message Passing Interface (MPI) (guarantees efficiency and portability)

• Improvement of code numerical accuracy by implementing Second-Order-Upwind schemes with limiters (Van Leer, minmod, Ultra Sharp)

• Implementation of “Immersed Boundary” techniques (more accurate description of the flow field over the topographic profile)

Parallelization strategy

topography proc 0

proc 1

proc 2

proc 3

proc 4

proc 5

proc 0

proc 1

proc 2

proc 3

proc 4

proc 5

proc 6

topography

proc 0 proc 2proc 3

proc 1 proc 3proc 4

proc 5

topography

The computational domain was partitioned into sub-domains with equal number of active cells (“domain decomposition” approach)

The load balancing and the scalability depend on the sub-domain topology

1) 2) 3)

Domain Decomposition into:

1) Horizontal layers

2) Vertical columns

3) 3D blocks

Benchmarks performed on a linux cluster with PIV (3.06GHz) and myrinet interconnection network

Scalability of the parallel code The total execution time (T) is the sum of the computation (Tc) and communication (Tm) times: T = Tc + Tm

The Speed-up, S(N), is the ratio between the execution time with 1 and N CPUs S(N) = T(1) / T(N). The ideal Speed-up is linear.

The saturation limit depends on the size of the computational domain and on the sub-domains topology

Estimated execution time for a 3D simulation on a volcano topography (1283 computational cells)

1 3 10 30

100

300

1000

3000

processors

seco

nd

s /

10

0 t

ime

step

s Linux (PIV 3.06GHz)SGI Altix (Itanium2 1.3GHz)IBM p690 (Power4 1.3GHz)

1 simulated second -> 200 sec / 128 CPU30 min. eruption -> 360000 sec / 128 CPU (4 days)

The code has been ported on several supercomputers and PC clusters

The performance depends strongly on the optimization of the algorithm and on the latency time of the interprocessor network

643 Cells

• Composition = Rhyolite (2% wt. H2O), Vent Diam. = 127 m, Mass flow-rate = 3.8 x 107 Kg/s (from Papale et al. 1998)

• Grain-sizes = Two classes of 30 and 500 microns in equal amount• Domain size = (1.2 x 1.2 x 2.0) Km3

• Non-uniform rectilinear mesh = 100 x 100 x 70 (cell size: 5-100 m)

Gas Temperature Isosurfaces (400, 800, 1000°K)

6s 10s 14s

3D Simulation of a axisymmetric volcanic jet

(!) circular vent is discretized by using the finest uniform cells and applying a technique for flux correction in partially filled cells

2D vs 3D axisymmetric jets

6 sec 14 sec

Isolines of the Log10 of the total particle fraction (-5:-2:-0.375) + Vector flow field of the gas velocity

• The comparison between 2D/3D is carried out on a non-uniform mesh (5-100 m) by using a First-Order-Upwind scheme• A Second-Order-Upwind scheme with minmod limiter is also shown (on a uniform 12.7 m mesh)

Simulation of a 3D volcanic jet at Vesuvius

Vent conditions for a sub-Plinian eruption from Todesco et al. (2002)

Gas temperature isosurfaces (400,800,1000°C),t= 25 s

Physical Domain: (8.0x8.0x8.0) km3

Non-uniform rectilinear mesh: 1283 cells

Horizontal resolution: 5-250 m

Simulation of a 3D dome explosion (t=20s)

UTM Georeferenced map: isolines of the gas temperature

Isosurfaces of the total particle fraction (10-3, 10-5)

Initial conditions: P=3 bar, T=1100°C, Solid vol. fract.=0.7, L=50 m

Topography from the Vesuvius DEM

Conclusions

• A first fully 3D version of the multiphase flow code PDAC has been completed and tested

• Benchmarks on the 3D parallel version of the code have shown that it is now possible to run a 3D simulation on a parallel supercomputer in a time comparable to that required for a 2D simulation on a single CPU

• Preliminary simulation results suggest that a more realistic description of the dynamics of volcanic dispersal processes can be obtained by using fully 3D models

Numerical simulation of the pyroclastic flow propagation along the Northern Sector of Vesuvius (run LL_w2_t950 of Todesco et al. 2002): isolines of the Log10 of the total particle fraction. Zoom of the gas velocity vector field near the ground

The Immersed Boundary technique• The “Immersed Boundary” technique consists in the introduction of a pseudo-force in the momentum equations to mimic the presence of a no-slip boundary

• This technique is suited to accurately model the boundary conditions on a generic solid wall even using a rectilinear mesh

• The use of the Immersed Boundary technique on a 2D numerical simulations produces an effect analogous to a decrease of the artificial roughness due to the steps

steps imm.b.

Code optimizationsExecu

tion t

ime

(rela

tive t

o F

77

code)

recover the performances of the F77 original code

code evolution

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2

F77 Serial F90 Serial 2D Parallel 3D Parallel Unrolling Memoryaccess

Parameterdependent

sub.

Slower

Faster

Numerical simulation of volcanic column formation at Vesuvius

Run L_w2_t950 in Todesco et al. 2002 (mfr = 5x107, 2% H2O)

• Domain size: (8.0x8.0x8.0) Km3

• 1283 rectilinear non-uniform mesh Horizontal resolution: 5-250 m • NO Immersed Boundaries • The internal crater topography strongly drives the decompression

Gas temperature isosurfaces (400, 800 and 1000°C) at t= 26 sec.