Possible foreseeable measures for tera-scale data handling

Kazutoshi Horiuchi*1

Keiko Takahashi*1

Hirofumi Sakuma*1

Shigemune Kitawaki*2

*1 Frontier Research System for Global Change

*2 Earth Simulator Research and Development Center

Global Change Prediction by an Integrated Three-in-one Research

Observation

Numerical Simulation

Process Study & Modeling

Accurate & spatially representative data

Optimal monitoring plan

Sophisticated high resolution model

High performance computing

Assimilation data for validation

Accurate & spatially representative data

ESRDC FRSGC

FORSGC

Frontier Research System for Global Change (Project)

Funding Bodies

• Japan Marine Science and Technology Center (JAMSTEC)

• National Space Development Agency (NASDA)

Activities

• Process Study • Model Development

(Common) Goal Global Change Prediction

On-Going Process Studies

Climate Variations ResearchHydrological Cycle ResearchGlobal Warming ResearchAtmospheric Composition ResearchEcosystem Change ResearchResearch of International Pacific Research CenterResearch of International Arctic Research Center

Current Target of Model Development Group

Coupled Model (now based on CCSR/NIES, MOM3)for Climate Change Experiment

Cloud Cluster Resolving Ultra High Resolution Model for Prediction of Typhoon/Baiu Evolution

Coupled Chemistry - Global Climate Modelfor Prediction of Atmospheric Composition Change

Next Generation Model (Cubic/Icosahedral Grid, CIP method)

4DVAR Ocean Data Assimilation Model (based on MOM3)

Current Target of Coupled Model

Development on ES

High Resolution: Atmosphere Model part: T213L50 Ocean Model part: 1/10 deg. 53 layers

High Performance

Estimation of Acceleration ratio

• Atmosphere Model: (under estimation)

• Ocean Model: 300 ~ 400 times (480PEs;60Ns) 5 days for 100 years integration

Earth Simulator Research Development Center (Project)

Funding Bodies

•Japan Atomic Energy Research Institute (JAERI)•National Space Development Agency (NASDA)•Japan Marine Science and Technology Center (JAMSTEC)

Activities

• Development of High Speed Parallel Computer• Understanding and Prediction of Global Change

(Common) Goal Global Change Prediction

Characteristics of Earth Simulator

Peak Performance: 40TFLOPSNumber of Processor Nodes: 640Number of PEs: 5120 (8PEs/Node)Interconnection Network: 16GB/s Total Memory: 10TBTotal Secondary Storage: 600-700TBTotal Mass Storage: 1PB (84Drives)

16 Nodes

Architecture of Earth Simulator

16 Nodes

Interconnection Network

TSS Cluster *1 Batch Cluster *39

…

Fiber Channel Switch

…

MM

VP

0

VP

1

VP

7

16 Nodes

…

MM

VP

0

VP

1

VP

7

…

MM

VP

0

VP

1

VP

7

…

…

Fiber Channel Switch

…

MM

VP

0

VP

1

VP

7

16 Nodes

…

MM

VP

0

VP

1

VP

7

…

MM

VP

0

VP

1

VP

7

…

Mass Storage System

WAN

WS

WS

WS

FS

…

84 Drives

…

I/O Model in Distributed Memory Parallel Computer

P P P…

communication

P P P… P P P…

(Unix) File (Data-Distributed Unix) Files

Parallel File

Parallel File System on ESTo handle distributed data as a logically single file has advantages to develop application softwares and to process post processings.

1234567

1 2 34 5 67 8 9

1 4 72 5 83 6 9

M1 M2

89

File Image

P

P

PProcessors Disks

Proc no.: Np

Dist. Size: Sd

Dist. Pattern: P= BLOCK/CYCLIC

Disk no.: Nd

Striping Size: Ss

Distribution MechanismD1

D2

D3

P1

P2

P3

1 2 3

4 5 6

7 8 9

Support for Parallel File on Several Levels

……

Unix File Parallel File

Operating System (with PFS)

FA

L

MPI-IO

F90

HPF-RTPF90-RTP

Library

Compiler

C

User Program

PFS UFS PFS UFS PFS

Hardware

HPF

Review of Model Development Flow

Improvement of Model

Execution with Model

Evaluation of Results

Input Data

Output Data

Resources for Process Study

Results of Process Study

•Analysis•Visualization

Is it Satisfactory about I/O performance?

The faster super computers are, the larger the amount of the output data generated by large-scale simulations.

The large amount of data is stored to secondary storages and/or mass storages whose devices are slower.

Is it satisfactory about I/O performance ?

Amount of Input/Output Data -Coupled Model

To answer the question, the following cases are investigated . Case I:

1000 Years Integration for the Prediction of Global Warming, Decadal Variability, etc.

Case II: 50 Years Integration for the Analysis of El nino, Dipole Mode Events, Asian Monsoon, etc.

Amount of Input/Output Data -Coupled Model

Atmosphere Model part

NOTE: The amount of output data is estimated as 2 byte integer elements

Vars Amount [GB] Times Vars Amount [GB]I 1000 T213L50 5+3 14 monthly 7+27 1,726II 50 T213L50 5+3 1 6hourly 7+27 10,498

Input OutputCase Periods [Y] Resolution

Vars Amount [GB] Times Vars Amount [GB]I 1000 0.1deg,L53 4+4 582 monthly 5+8 39,541II 50 0.1deg,L53 4+4 89 10days 5+8 6,096

Case Periods [Y] Input OutputResolution

Ocean Model part

Estimated I/O time - Coupled Model

NOTE: Time is estimated only on drive’s I/O rate. Multiple drives are assumed to be independent.

8 16 84Input 0.05 0.02 0.005 0.001Output 6.0 3.0 0.6 0.08Input 0.003 0.001 0.0003 0.00004Output 36.5 18.2 3.5 0.49Input 2.0 1.0 0.2 0.03Output 137.3 68.6 13.1 1.83Input 0.3 0.2 0.03 0.004Output 21.2 10.6 2.0 0.28

Time of ModelRun (Estimation)

Disk I/ O time(Estimation)

[unit:hour]

Atmos-phere

Model Case In/ Out I/ O time with Tape Drives:

-

-

1200

60Ocean

I

II

I

II

Summary of I/O Performance (from the viewpoint of Model Development)

Disk I/O time might be satisfactory. 0.2% of the simulation time Less than 2 hours as a total

Tape I/O time might be conspicuous. 11-35% of the simulation time for 8 tape drives 1-6 days for 8 tape drives

This inefficiency might be critical for iterative works such as model development

How to Shorten Turn Around Time of Model Development

Give up outputting numerical data.Output necessary minimum data.Output full data, with executing tape I/O

and simulations in parallel, and with tape I/O library being able to extract necessary minimum data for post processing.

Shortening of TAT by Giving up Outputting Numerical Data

－

3.0

1.9

67.5

68.1

68.0

－

0.98

0.79

Elapsed time (h)

Time increase (%)

Elapsed time (h)

Time increase (%)

CFD solver only

+ fixed camera

+ moving camera

1.6M grid (169x92x101) 6.2M grid (337x183x101)

14.5

14.7

14.9

Elapsed time for the concurrent visualization with RVSLIB in the batch processing mode on SX-4

*The number of computational time steps was 10000.*Contour and tracers were displayed at every 10 time steps and visualized animation was stored in a file.*Time integration for moving the tracers was done at every time step for greater accuracy.

* This result was provided by NEC

Shortening of TAT by Outputting Necessary Minimum Data

“Browse sets”, into which the large amount of output data is abstracted (spatially and/or temporally) within simulations, should be stored.

Specific regions of output data should be stored.

This may be Know-Hows of using ES

Shortening of TAT by Enhancement of Tape I/O Library for Full Output Data

Tape I/O should be executed with simulations in parallel.

In mass storages, output data should be re-organized, and small subsets which are needed for post processing should be able to be extracted.

This may be requirements for the improvement of ES

Conclusion

I/O performance was roughly estimated and I/O problem was apprehended.

I/O problem would be avoided with concurrent visualization and/or know-how of the usage.

However we would like to examine the efficient technique for handling the large amount of data continuously to realize comfortable environment for global change prediction.