Stripped replication for the Grid environment as a web service

CGW 04, Stripped replication for the grid environment as a web service 1

Stripped replication for the Grid environment as a web service

Marek Ciglan, Ondrej Habala, Ladislav HluchýInstitute of informatics Slovak Academy of Sciences

CGW 04, Stripped replication for the grid environment as a web service

2

Stripped Replication for Grids

Overview

• Replication in Grid environment• Principles of stripped replication (SR)

method• Optimization of stripped replication• Prototype Implementation as a Web

Service• Experimental Results• Future Work


3


Replication in Grid environment

• Creation of multiple copies of single data source across Grid infrastructure

• Replication increases data availability• RLS - Replica Location Service• Grid monitoring services – network

monitoring


4



Storage Element 1

File 1

Storage Element 2

Storage Element 3


5



Storage Element 1

File 1

Storage Element 2

Storage Element 3

File 1


6



Storage Element 1

File 1

Storage Element 2

Storage Element 3

File 1


7



Storage Element 1

File 1

Storage Element 2

Storage Element 3

File 1

File 1


8


Stripped Replication - Principles

• Transfer from multiple Grid sites, in parallel• Transfer only a portion of file from each

Storage Element (SE)• Different file portions (stripes) are obtained

from different SEs• Parallel transfer increases replication speed• If SR is not managed properly, process

could be time consuming• Optimization of SR management is required


9


Stripped Replication - Optimization

Replicated data source

Replica 1

Replica 2 Replica 3


10



Replicated data source

Replica 1

Replica 2 Replica 3

Replica 1

Replica 2 Replica 3


11



Replica 1

Replica 2 Replica 3


12



Replica 1

Replica 2 Replica 3

Replica 1

Replica 2 Replica 3


13



Replica 1

Replica 2 Replica 3

Replica 1

Replica 2 Replica 3

Replica 1

Replica 2 Replica 3


14


SR Prototype Implementation

• Java programming language• CoG 1.2 API (GridFTP interface)• Integrated with EDG Replica Location

Service• EDG RLS API (RLS interface)• File Chunks – basic data units for transfer• Implemented as a Web Service

( motivation :OGSA, WSRF)


15


Service WorkflowStripped Replication Service

LFNGetGUID


16


Service Workflow

LFNGetGUID

ReplicaMetadataCatalog

Stripped Replication Service


17


Service Workflow

LFNGetGUID



GetPFNs


18


Service Workflow

LFNGetGUID


GetPFNs

LocalReplicaCatalog



19


Service Workflow

LFNGetGUID


GetPFNs

LocalReplicaCatalog

StrippedReplicationAlgorithm



20


Service Workflow

LFNGetGUID


GetPFNs

LocalReplicaCatalog


GridFTPSite 1

GridFTPSite N

. .

.

. .

.



21


Service Workflow

LFNGetGUID


GetPFNs

LocalReplicaCatalog


GridFTPSite 1

GridFTPSite N

. .

.

. .

.

Register Replica



22


Properties of Stripped Replication

• Parallel transfer from multiple sites increases replication process speed

• Proposed optimization does not use network monitoring services

• SR adapts to varying nature of network load

• SR optimally distributes network load


23


Experimental Results

• Motivation test case– File size 223.9Mb – Best replica transfer with standard replication tool

(EDG rm) - 713 sec– Stripped replication (2 replicas) – 405 sec (43 %)– Stripped replication (3 replicas) – 209 sec (71 %)

• Average time saving– 2 replicas – 37% time saving– 3 replica – 55% time saving


24


Future Work

• Implementation refinement– Add logging functionality– Refine error states handling

• Evaluation of SR integration in Grid projects


25


Thank you for your attention !

Stripped replication for the Grid environment as a web service

Documents

Transcript of Stripped replication for the Grid environment as a web service