IRODS performance test and SRB system at KEK Yoshimi Iida @ KEK Building data grids with iRODS 27...

iRODS performance test and SRB system at KEKYoshimi Iida @ KEK

Building data grids with iRODS

27 May 2008

27/05/2008 Building data grids with iRODS 2

Outline Performance measurement

Transfer test between CC-IN2P3 and KEK Scaling test for ICAT Concurrent test for ICAT Compare with iRODS and SRB

SRB/iRODS system at KEK


Transfer between CC-IN2P3 and KEK

1GB data transfer between CC-IN2P3 and KEK Compare with iRODS and bbcp

The network route go through USA The RTT is about 285ms

KEK, JPCC-IN2P3, FR

NY, USA


System used iRODS system at CC-IN2P3

ICAT enabled iRODS server on Solaris 10 Thumpers Sun x4500 (AMD processors) Oracle 10g is on a cluster of dedicated machines Linux file resource from local disk system iRODS 0.9

iRODS system at KEK iRODS server on RHEL3

Intel Xeon 3.0GHz ×4 Linux file resource from local disk system iRODS 0.9


From KEK to CC-IN2P3

1GB data transfer during 24 hours window size 4MB number of parallel

streams 16

bbcp often fail to connect

iput

0

10

20

30

40

50

Spe

ed [

MB/s

ec]

bbcp

0

10

20

30

40

50

Spe

ed [

MB/s

ec]


From CC-IN2P3 to KEK

iput

0

10

20

30

40

Spe

ed [

MB/s

ec] 1GB data transfer

during 12 hours window size 4MB number of paralle

l streams 16

iput is better than bbcp

bbcp

0

10

20

30

40

Spe

ed [

MB/s

ec]


Scaling test Data

Using the same directory at client machine 1000 files of 1000 bytes each

Measurement Ingesting directory and listing the collection Performance measured for every directory

operation 1 collection and 1000 files


System used ICAT based on Oracle system at CC-IN2P3

ICAT enabled iRODS server on Solaris 10 Thumpers Sun x4500 (AMD processors) Oracle 10g is on a cluster of dedicated machines

iRODS resource and client on SL4 Dual AMD Opteron Processor 848 Linux file resource from local disk system

ICAT based on PostgreSQL system at KEK ICAT enabled iRODS server on RHEL3

Dual Intel Xeon 2.8GHz PostgreSQL 8.2.5 is running on the same machine

iRODS resource and client on RHEL3 Dual Intel Xeon 2.8GHz Linux file resource from local disk system


Ingesting up to 1 million filesiput - Oracle

0

20

40

60

80

100

0 200 400 600 800 1000number of files [x 1,000]

tim

e in

sec

onds

ils - Oracle

0

1

2

3

4

5

0 200 400 600 800 1000number of files [x 1,000]

tim

e in

sec

onds

iput- PostgreSQL

0

20

40

60

80

100

0 200 400 600 800 1000number of files [x 1,000]

tme

in s

econ

ds

ils - PostgreSQL

0

1

2

3

4

5

0 200 400 600 800 1000number of files [x 1,000]

tim

e in

sec

onds

Running the other process


Nested collection test Data

Registering same directory at client machine 100 files of 100 bytes each

Measurement Ingesting directory and listing the collection Making nested collection every 10 collection Performance measured for every directory

operation 1 collection and 100 files


Nested collection testireg - PostgreSQL

010203040506070

0 50 100 150 200number of files [x 1,000]

time

in s

econ

ds

nestflat

iput - Oracle

010203040506070

0 50 100 150 200number of files [x 1,000]

time

in s

econ

ds

nestflat

ils - PostgreSQL

0

1

2

3

4

0 50 100 150 200number of files [x 1,000]

time

in s

econ

ds

nestflat

ils - Oracle

0

2

4

6

8

10

0 50 100 150 200

number of files [x 1,000]

time

in s

econ

dsnestflat

nest-121 : OCI_ERRORcoll-1220: USER_PATH_EXCEEDS_MAX


Concurrent test Data

Ingesting same directory at client machine 1000 files of 1000 bytes each

Measurement Runing multi process at the same time

read operation – ils and iget write operation – iput and ireg mixed operation – iput, ireg, ils and iget

Performance measured for every directory operation DB setting

Set the number of maximum connections to 200


Concurrent test for reading metadatails - PostgreSQL ICAT

0.1

1

10

100

1000

0 50 100 150 200 250 300

Number of clients

Tim

e in s

econds

iget - PostgreSQL ICAT

1

10

100

1000

10000

0 50 100 150 200 250 300

Number of clients

Tim

e in

sec

onds

ils - Oracle ICAT

0.1

1

10

100

1000

0 50 100 150 200 250 300

Number of clients

Tim

e in s

econds

iget - Oracle ICAT

1

10

100

1000

10000

0 50 100 150 200 250 300

Number of clients

Tim

e in

sec

onds

Error: connectToRhost failed10 clients 200 clients100 clients 300 clients×


Concurrent test for writing metadataiput - PostgreSQL ICAT

1

10

100

1000

10000

0 50 100 150 200

Number of clients

Tim

e in

se

con

ds

ireg - PostgreSQL ICAT

1

10

100

1000

10000

0 50 100 150 200

Number of clients

Tim

e in

se

con

ds

ireg - Oracle ICAT

1

10

100

1000

10000

0 50 100 150 200

Number of clients

Tim

e in

se

con

ds

iput – Oracle ICAT Because of limit of inodes, we

cannot put any more files

10 clients 100 clients 200 clients


Concurrent tests for mix - PostgreSQLiput

0.1

1

10

100

1000

10000

0 50 100 150 200

Number of clients

Tim

e in s

econds

ils

0.1

1

10

100

1000

0 50 100 150 200

Number of clients

Tim

e in

se

co

nd

s

ireg

0.1

1

10

100

1000

10000

0 50 100 150 200

Number of clients

Tim

e in

se

co

nd

s

iget

0.1

1

10

100

1000

10000

0 50 100 150 200

Number of clients

Tim

e in

se

co

nd

s



Concurrent test for mix - Oracle

ireg

1

10

100

1000

0 50 100 150 200

Number of clients

Tim

e in

se

con

ds

ils

1

10

100

1000

10000

0 50 100 150 200

Number of clients

Tim

e in

se

con

ds

iget

1

10

100

1000

10000

0 50 100 150 200

Number of clients

Tim

e in

se

con

ds

iput – Oracle ICAT Because of limit of inodes, we

cannot put any more files



Compare with iRODS and SRB For the scaling test

Data Using the same directory at client machine 1000 files of 1000 bytes each

Measurement Ingesting directory and listing the collection Performance measured for every directory operation

For the nested collection test Data

Registering same directory at client machine 100 files of 100 bytes each

Measurement Ingesting directory and listing the collection Making nested collection every 10 collection Performance measured for every directory operation


System used iRODS system at KEK

ICAT enabled iRODS server on RHEL3 Dual Intel Xeon 2.8GHz PostgreSQL 8.2.5 is running on the same machine iRODS 1.0

iRODS resource and client on RHEL3 Dual Intel Xeon 2.8GHz Linux file resource from local disk system

SRB system at KEK MCAT enabled SRB server on RHEL3

Dual Intel Xeon 2.8GHz PostgreSQL 8.2.5 is running on the same machine SRB 3.5.0

SRB resource and client on RHEL3 Dual Intel Xeon 2.8GHz Linux file resource from local disk system


Scaling test - iRODS and SRBiput

0

10

20

30

40

50

0 200 400 600 800 1000number of files [x 1,000]

tme

in s

econ

ds

Sput

0500

10001500200025003000

0 10 20 30 40 50number of files [x 1,000]

tim

e in

sec

onds

ils

0

0.1

0.2

0.3

0.4

0.5

0 200 400 600 800 1000number of files [x 1,000]

tim

e in

sec

onds

Sls

0

0.2

0.4

0.6

0.8

1

0 10 20 30 40 50number of files [x 1,000]

tim

e in

sec

onds


nested collection –iRODS and SRBireg

010203040506070

0 50 100 150 200number of files [x 1,000]

time

in s

econ

ds

nestflat

Sregister

05

1015202530

0 10 20 30 40 50number of files [x 1,000]

tim

e in

sec

onds

nestflat

ils

0

1

2

3

4

0 50 100 150 200

number of files [x 1,000]

time

in s

econ

ds

nestflat

Sls

00.10.20.30.40.50.6

0 10 20 30 40 50number of files [x 1,000]

tim

e in

sec

onds

nestflat

coll-1220: USER_PATH_EXCEEDS_MAX nest-47: Error Problem running command


SRB system for Belle at KEK

Both protocols are authorized by GSI

Setting up on GridFTP server grid-mapfile for SRB user SRB configuration file for serve

r and resource Register LCG user DN on MCAT

MelbourneMelbourneMelbourneMelbourne

NCUNCUNCUNCUASGCASGCASGCASGC

KUKUKUKU

NagoyaNagoyaNagoyaNagoya

CYFRONETCYFRONETCYFRONETCYFRONET

KEK-2KEK-2KEK-2KEK-2

KEK-1KEK-1KEK-1KEK-1

SINETSINETSINETSINET

KEK-LANKEK-LANKEK-LANKEK-LAN

GEANT2GEANT2GEANT2GEANT2

APANAPANAPANAPAN

KEK-DMZKEK-DMZKEK-DMZKEK-DMZ

SRB-DSISRB-DSISRB-DSISRB-DSI

KEK FWKEK FW

HSMHSMHSMHSMMCATMCATMCATMCAT

PluggableExtension

SRBSRB client client

SRBSRB server server

3.5PB3.5PB

GridFTPGridFTP

SRBSRB

NFSNFS

Enhanced Enhanced GridFTGridFTP serviceP service Belle NetBelle Net

dedicated insidededicated inside

Belle NetBelle Netdedicated insidededicated inside

LCG user

Computing FarmStill not integration with Grid

LSFLSF

LCG user

Belle analysis user


Plan for iRODS system at KEK Data transfer for J-PARC project

Generate huge amount of imaging data at Tokai About 1PB data in a year in total

Store the data at Tokai storage once, then copy to KEK and distribute for collaborators Storage at Tokai are recycled

Bandwidth between 2sites

will be 10Gbps

KEK

Tokai

60km


Acknowledgements Special thanks to Jean-Yves Nief from CC-IN

2P3 for his help to setup iRODS system and supporting at CC-IN2P3

Thanks to Adil Hasan from RAL for his help to these tests


Back up


Bandwidth Lyon-KEK

0

20

40

60

80

100

120

0 10 20 30 40 50 60

Time [min]

Spe

ed [

MByt

es/s

ec] Lyon to KEK

KEK to Lyon

iperf with some options; -w 4M : TCP window size [Bytes] -P 16 : the number of parallel threads -i 5 : periodic bandwidth reports [sec]


Summery of iRODS Performance Transfer from KEK to CC-IN2P3 is not stable but ip

ut is better than simple transfer software (bbcp) iRODS can manage 1 million files stability

In case of very nested collection, PostgreSQL ICAT takes long time to register data

Better performance than SRB Oracle ICAT can handle more than 300 clients at the

same time, but it takes time according to increase the number of clients

IRODS performance test and SRB system at KEK Yoshimi Iida @ KEK Building data grids with iRODS 27...

Documents

Transcript of IRODS performance test and SRB system at KEK Yoshimi Iida @ KEK Building data grids with iRODS 27...