Prof. Heon Y. Yeom Distributed Computing Systems Lab. Seoul National University FT-MPICH : Providing...

Prof. Heon Y. YeomDistributed Computing Systems Lab.Seoul National University

FT-MPICH : Providing fault

tolerance for MPI parallel applications

FT-MPICH : Providing fault

tolerance for MPI parallel applications

Heon Y. Yeom, Seoul National UniversityCondor Week 2006 [email protected]

Motivation

Condor supports Checkpoint/Restart(C/R) mechanism only in Standard Universe. Single process jobs

C/R for parallel jobs is not provided in any of current Condor universes.

We would like to make C/R available for MPI programs.


Introduction

Why Message Passing Interface (MPI)? Designing a generic FT framework is

extremely hard due to the diversity of hardware and software systems.

We have chosen MPICH series ....

MPI is the most popular programming model in cluster computing.

Providing fault-tolerance to MPI is more cost-effective than providing it to the OS or hardware…


Architecture-Concept-

Monitoring

FailureDetection

C/R Protocol

FT-MPICHFT-MPICH


Architecture-Overall System-

Ethernet

IPC

Management System

Communication

MPI Process

Communication

IPC Ethernet

MPI Process

Communication

IPCEthernet

MPI Process

Communication

IPC

Ethernet

Message Queue


Management System

ManagementSystem

Makes MPI more reliable

FailureDetection

CheckpointCoordination

Recovery

InitializationCoordination

OutputManagement

CheckpointTransfer


Manager System

MPI processLocal Manager

MPI processLocal Manager

MPI processLocal Manager Stable

Storage

Leader Manager

Initialization, CKPT cmd, CKPT transfer, Failure notification & recovery

Communication between MPI process to exchange data


Fault-tolerant MPICH_P4

FT Module Recovery Module

ConnectionRe-establishment

Checkpoint Toolkit

Atomic M

essage

Transfer

ADI(Abstract Device Interface)Ch_p4 (Ethernet)

FT-MPICH

Ethernet

Collective Operations

P2P Operations


Startup in Condor

Precondition Leader Manager already knows the machines

where MPI process is executed and the number of MPI process by user input

Binary of Local Manager and MPI process is located at the same location of each machine


Startup in Condor

Job submission description file Vanilla Universe Shell script file is used in submission

description file executable points a shell script The shell file only executes Leader Manager

Ex) Example.cmd

#!/bin/shLeader_manager …

exe.sh(shell script)

universe = Vanillaexecutable = exe.shoutput = exe.outerror = exe.errlog = exe.logqueue

Example.cmd


Startup in Condor

User submits a job using condor_submitNormal Job Startup

Condor PoolCentral Manager

Submit Machine

Submit Shadow

Schedd

Negotiator Collector

Execute Machine

Job(Leader Manager)Starter

Startd


Startup in Condor

Leader Manager executes Local Manager Local Manager executes MPI process

Condor Pool

Central Manager

Submit MachineExecute Machine

Job(Leader Manager)

Execute Machine 1 Execute Machine 2 Execute Machine 3Local Manager Local Manager Local Manager

MPI Process MPI Process MPI Process

Fork()&

Exec()


Startup in Condor

MPI processes send Communication Info and Leader Manager aggregates this info

Leader Manager broadcasts aggregated info

Condor Pool

Central Manager


Job(Leader Manager)


MPI Process MPI Process MPI ProcessMPI Process MPI Process MPI Process


Fault Tolerant MPI

To provide MPI fault-tolerance, we have adopted Coordinated checkpointing scheme

(vs. Independent scheme) The Leader Manager is the Coordinator!!

Application-level checkpointing (vs. kernel-level CKPT.)

This method does not require any efforts on the part of cluster administrators

User-transparent checkpointing scheme (vs. User-aware)

This method requires no modification of MPI source codes


Atomic Message Passing

Coordination between MPI process Assumption

Communication Channel is FIFO Lock(), Unlock()

To create atomic operation

Proc 1

Lock() Lock()

Unlock() Unlock()

AtomicRegion

CKPT SIG

CKPT SIG

CKPT SIG

Checkpoint is performed!!

Checkpoint is performed!!

Checkpoint is delayed!!

Proc 0


Atomic Message Passing(Case 1)

When MPI process receive CKPT SIG, MPI process send & receive barrier message

Proc 1

Lock() Lock()

Unlock() Unlock()

AtomicRegion

Proc 0CKPT SIGCKPT SIG

Barrier

Data

CKPT

CKPT SIGCKPT SIG CKPT


Atomic Message Passing (Case 2)

Through sending and receiving barrier message, In-transit message is pushed to the destination

Proc 1

Lock() Lock()

Unlock() Unlock()

AtomicRegion

Proc 0CKPT SIG

CKPT SIG

Barrier

Data

Delayed CKPT


Atomic Message Passing (Case 3)

The communication channel between MPI process is flushed Dependency between MPI process is removed

Proc 1

Lock() Lock()

Unlock() Unlock()

AtomicRegion

Proc 0

CKPT SIGCKPT SIG

Barrier

Data

Delayed CKPT


Checkpointing

Coordinated Checkpointing

ver 2

ver 1

LeaderManager

checkpointcommand

rank0 rank1 rank2 rank3

storage

Stack

Data

Text

Heap


Failure Recovery

MPI process recovery

Stack

Data

Text

Heap

Stack

Data

Text

Heap

CKPT Image New processRestarted Process


Failure Recovery

Connection Re-establishment Each MPI process re-opens socket and sends

IP, Port info to Local Manager This is the same with the one we did before at the initialization time.


Fault Tolerant MPI

Recovery from failure

failure detection

ver 1

LeaderManager

checkpointcommand

rank0 rank1 rank2 rank3

storage


Fault Tolerant MPI in Condor

Leader Manager controls MPI processes by issuing checkpoint command, monitoring

Condor Pool

Central Manager


Job(Leader Manager)


MPI Process MPI Process MPI Process

Condor is not aware of the failure incident


Fault-tolerant MPICH-variants(Seoul National University)

FT Module Recovery Module

ConnectionRe-establishment

Ethernet

Checkpoint Toolkit

Atomic M

essage

Transfer

ADI(Abstract Device Interface)

Globus2 (Ethernet) GM (Myrinet) MVAPICH (InfiniBand)

Collective Operations

MPICH-GF

P2P Operations

M3 SHIELD

Myrinet InfiniBand


Summary

We can provide fault-tolerance for parallel applications using MPICH on Ethernet, Myrinet, and Infiniband.

Currently, only the P4(ethernet) version works with Condor.

We look forward to working with Condor team.

Prof. Heon Y. Yeom Distributed Computing Systems Lab. Seoul National University FT-MPICH : Providing...

Documents

Transcript of Prof. Heon Y. Yeom Distributed Computing Systems Lab. Seoul National University FT-MPICH : Providing...