Hyper-resolution and Large Scale

Flood Assessments using 2D GPU

Modeling

ASFPM Conference 2017

Javier FernandezComputational Hydraulics Group

University of Zaragoza, Spain

Asier Lacasta, PhDComputational Hydraulics Group

University of Zaragoza, Spain

Mario Morales, PhDComputational Hydraulics Group

University of Zaragoza, Spain

Pilar Garcia-Navarro, PhDComputational Hydraulics Group

University of Zaragoza, Spain

Reinaldo Garcia, PhDDirector or Model Development and Applications

Hydronia, LLC

High-resolution and Large Scale Flood and

Hydrologic Simulations in 2D

Take advantage of elevation data sets

Provide adequate resolution to capture irregular terrain, and urban

areas

Long rivers reaches with wide floodplains

Large watersheds

Millions of computation cells Long runtimes

High Res Elevation Model for Managua

Terrain data: high resolution LiDAR

Terrain roughness: Aerials, Remote Sensing

Rainfall: Radar precipitation estimates (NEXRAD, etc.)

Inundation extent: Aerials, Remote Sensing

2D Model Requirements for High-Resolution and Large Scale

Simulations

Capable of resolving complex terrain and urban areas

Integration of Hydrology and Hydraulics

o Spatially and temporally variable rainfall, evaporation and

infiltration

o 2D flood routing

o [Sediment transport]

Huge memory storage capacity

o 64-bit GUI + Model Engine

Computational competence

o Reasonable Runtimes

Square-cell Grid vs Triangular-cell Mesh

400,000 Triangles 2,200,000 Squares

5.5 times more cells ==> More RAM memory, Larger runtimes

Flexible Mesh

Parallel Computation Approach

Multiple Processors (CPU or GPU)

Multiple steps at a time

Multiple Core Processors

… if the Model is not programed to run in

parallel you will not experience

any speedup

Parallelization in multiple core processors is not

enough

Number of cores is limited: 2, 4,…, 16…

Open MP parallelization often does not scale

well:

More cores ≠> better performance

Graphic Processing Units: GPUs

GPU: Graphic Processing Units

GPU is present on video cards

GPU’s often have more than 2000 processors

GTX 780GTX Titan

BlackGTX1080

Tesla c2075

Tesla k20 Tesla k40 Tesla k80

CUDA cores 2,304 2,880 2,560 448 2,496 2,880 2 x 2,496

Memory 3 Gb 6 Gb 8 Gb 6 Gb 5 Gb 12 Gb 24 Gb

~Cost US$ 500 850 700 - 2,000 3,000 4,500

RiverFlow2D 2D Flood Simulation

Model Combined Hydrologic-Hydraulic model

Flexible non-structured mesh

Dry-wet bed algorithm

Explicit Finite-Volume numerical engine

Supercritical and subcritical regimes

Hydraulic structures in 2D mesh (Bridges, etc.)

Parallelized for multiple core CPUs

High Performance GPU: up to150X faster

Add on Modules

o Erosion/Deposition

o Mud and Debris Flows

o Water Quality

SMS GUI

Autodesk Infraworks™ Flood Simulation plug-in

Tasmania State-wide Flood Model

• A preliminary investigation Into the feasibility of a

State-wide Flood Model for Tasmania

• Proof of Concept

• Prepared for the Office of Security and

Emergency Management. The Department of

Premier and Cabinet, Tasmania. May 2016

Acknowledgements

• Ted Rigby, Rienco Consulting.

• Colin Mazengarb, Mineral Resources Tasmania

• Alan Zundel and Tom Moreland, Aquaveo.

RIENCOConsulting

• 68,401 km²

• 26,410 mi2

• ~350 watersheds

315 Km

350 Km

Tasmania State-wide Flood Model

Background

The Department has been considering developing a centralised and more

consistent flood modelling for the State

Investigations until recently had suggested that construction of a single state-

wide flood model was not practical

Model Construction

RiverFlow2D &

SMS

5m and 25m

LiDAR

Ocean: 500m cells

Land: 10-250m

cells

2-5m cells on area

of interest

7.680,561 cells

Mesh View in 3D

9-hour

Storm

Comparison with Measurements

OBSERVEDWater Elevations

(m)

Coarse mesh Water Elevations

(m)Error 1

(m)

Refined Mesh Water Elevations

(m) Error 2(m)

29.28 30.25 -0.97 29.27 0.01

25.51 25.96 -0.45 25.41 0.10

20.73 22.01 -1.28 20.54 0.19

19.12 20.47 -1.35 19.11 0.01

19.07 20.48 -1.41 18.91 0.16

18.69 18.88 -0.19 18.44 0.25

16.77 16.19 0.58 16.58 0.19

15.77 15.48 0.29 15.71 0.06

13.23 13.58 -0.35 13.07 0.16

13.02 12.97 0.05 12.89 0.13

11.62 11.48 0.14 11.57 0.05

10.54 9.98 0.56 10.47 0.07

10.20 9.71 0.49 10.04 0.16

8.37 8.14 0.23 8.19 0.18

8.03 7.58 0.45 7.91 0.12

6.85 6.51 0.34 6.61 0.24

5.79 5.40 0.39 5.73 0.06

5.75 5.10 0.65 5.64 0.11

4.33 4.44 -0.11 4.12 0.21

3.96 3.90 0.06 3.85 0.11

3.04 3.20 -0.16 3.00 0.04

3.00 3.20 -0.20 2.85 0.15

1.18 1.35 -0.17 1.16 0.02

Average Error 1 0.48 Average Error 2 0.06

Tasmania State-wide RiverFlow2D Model

Runtimes for 9-hour Rainfall Storm

3 days 18 hours

4.5 hours

Conclusions

Large-scale high-resolution model is not practical using uniform fixed grid

resolution (square grid)

A single state-wide flood model for Tasmania is feasible using SMS and

RiverFlow2D GPU

Preliminary results compare well with measurements

Runtimes of few hours vs days or weeks

Use of GPU models open the door for realistic large scale 2D simulations

Red River of the North

Thanks to:

Mike DeWeese

Jonathan Thornburg

Justin Palmer

Pedro Restrepo

Mark Ziemer

NOAA – NCRFC

Chanhassen, MN

Hydronia-NOAA CRADA

Cooperative Research & Development Agreement (CRADA)

Create a large scale two-dimensional flooding simulations using advanced

GPU technology with the RiverFlow2D model

420-mile reach including the city of Fargo

Red River of the North

Inundated fields, Red River of the North (April 2013)

Hydraulic Modeling Challenges

Low accuracy of 1D approach

Complex overland flow

Excessive computational times: 1D model requires 3-9 hours!

Ideally run times should be less than 2 hours

Elevation Data Sets

+ =

In-channel surveyed cross sections

LiDAR Merged Elevations

Test B: Refined mesh 590,000 cells

Test A: Uniform mesh 1,070,000 cells

420 miles

Test C: Refined mesh 1,265,712 cells

Inflow Locations Merged Elevations

Animation created by Asier Lacasta, UZ.

Comparison with field data: Flood of 2011

20 Measurement locations

USGS Gauges and HWMs

HWM locations

RiverFlow2D model vs. HWMs 2011

1,070,000 cell mesh

R² = 0.9986

240

250

260

270

280

290

300

240 250 260 270 280 290 300

Fiel

d D

ata

Max

imu

m W

SE(m

)

RiverFlow2D Maximum WSE (m)

Data (m) Model (m)Difference

(m)

260.85 261.76 0.91

292.44 293.55 1.11

272.12 271.62 -0.51

288.90 287.78 -1.12

265.46 264.90 -0.56

284.98 285.28 0.31

295.13 294.99 -0.14

263.56 264.03 0.47

275.00 275.39 0.40

273.16 273.04 -0.11

255.71 255.59 -0.12

278.77 278.31 -0.46

241.62 241.62 0.00

243.64 243.60 -0.04

257.63 257.58 -0.05

252.96 254.03 1.07

282.71 282.59 -0.12

274.79 274.86 0.07

264.64 263.91 -0.73

247.45 246.91 -0.54

Comparison of RiverFlow2D GPU Model with Aerial

Images

2011 Flood

Comparison of RiverFlow2D GPU Model with Aerial

Images

2011 Flood

Summary of RRN Performance Tests

RiverFlow2D90-day hydrograph

GPU Time GTXTitan Black

1,070,000 cells 2.2 h

590,000 cells 36 min

HEC-RAS 1D Run Time

30-dayhydrograph

3.1 h

Performance Comparison with other 2D

Models

UK Test 5: Runtimes

57X fasterthan RAS2D

13X faster than RAS2D

99X fasterthan SRH-2D

RiverFlow2D vs FLO-2D BasicUK Test 5

1953X faster than FLO-2D

RiverFlow2D vs MIKE 21Dam-break / 200k cells / 6-hour simulation

≠

FeatureSub-Grid

Macro-cellTriangular cells

Accounts for momentum transfer inside cell No Yes

Velocity detailsNo Yes

Simplified frontal wave travel Yes No

Accurately capture complex hydrodynamics

No Yes