Oct 24, 2002Michael Ernst, Fermilab

DRM for Tier1 and Tier2 centers DRM for Tier1 and Tier2 centers

Michael ErnstMichael Ernst

FermilabFermilab

February 3, 2003February 3, 2003

February 3, 2003Michael Ernst, Fermilab 2

Production Data Flow Today …Production Data Flow Today …Tier1 at FNAL

dCache/Enstore

CATALOG

Local Disk

CATALOG

Tier2Worker Nodes Tier2

Head/Storage Node

CATALOG

MOP Master

February 3, 2003Michael Ernst, Fermilab 3

Typical Tier2 Today …Typical Tier2 Today …

Gri

dFT

P, b

bcp

Head/Storage Node

Worker Nodes

February 3, 2003Michael Ernst, Fermilab 4

Tier2 w/dCacheTier2 w/dCache

dCache

Admin Node

Gri

dF

TP

February 3, 2003Michael Ernst, Fermilab 5

Tier2 w/dCacheTier2 w/dCache

dCacheG

rid

FT

P

dCache

Gri

dF

TP

dCache/Enstore

Catalog

Gri

dF

TP

GridFTP

CASTOR

FNAL

CERN

UCSD

Florida, …

Cat

alog

Cat

alo g

Catalog

February 3, 2003Michael Ernst, Fermilab 6

dCache PlacementdCache Placement

Application (e.g. dccp)

dCap library

xxxFTP GRID Access Method(s)

Applications

dCache

Enstore

OSM

HSM X

Local Disk

PNFS NamespaceManager

Tertiary StorageSystems

February 3, 2003Michael Ernst, Fermilab 7

Distributed Pool ArchitectureDistributed Pool Architecture

Topic Caches

Tertiary Storage

Enstore, OSM, …

Super Cluster Caches(Experiment)

Central Cache

HostCaches

Cluster Caches(Working Group)

externally enforced attraction

destination determined attraction

February 3, 2003Michael Ernst, Fermilab 8

What is dCache ?What is dCache ?

dCachedCache is/provides is/provides• Compact system making access to HSM systems more efficientCompact system making access to HSM systems more efficient• Generic tool for caching, storing and easily accessing peta byte scale Generic tool for caching, storing and easily accessing peta byte scale

datasets distributed among a large set of heterogeneous caching nodesdatasets distributed among a large set of heterogeneous caching nodes• Able to run with and without HSM backend (can be used as scalable file Able to run with and without HSM backend (can be used as scalable file

store)store)• Very flexible HSM BackendVery flexible HSM Backend

Supports multiple instances of same HSM or different HSMs within same dCache Supports multiple instances of same HSM or different HSMs within same dCache instanceinstance

Rate Adaption (tape Rate Adaption (tape disk) disk) Deferred Write (aggregates small files up to threshold (time, space))Deferred Write (aggregates small files up to threshold (time, space)) StagingStaging Read AheadRead Ahead dCache allows to relax performance requirements on HSM storage components (e.g. dCache allows to relax performance requirements on HSM storage components (e.g.

tape drives and robots) w/o overall performance penaltytape drives and robots) w/o overall performance penalty

• Attraction schemes used to optimize data placementAttraction schemes used to optimize data placement• Load BalancingLoad Balancing• Automated Replication (to prevent hot spots)Automated Replication (to prevent hot spots)

February 3, 2003Michael Ernst, Fermilab 9

dCache Data Access dCache Data Access • LocalLocal

dccp (req. mounted pnfs fs), kerberized and non-kerberizeddccp (req. mounted pnfs fs), kerberized and non-kerberized dcap lib (API)dcap lib (API) dcap Preload lib dcap Preload lib URL-style Addressing (e.g. dccp URL-style Addressing (e.g. dccp

dcap://door.do.name:port#/pnfs/do.name/path/to/file)dcap://door.do.name:port#/pnfs/do.name/path/to/file) GridFTP (Server embedded in dCache, client with globus-url-copy)GridFTP (Server embedded in dCache, client with globus-url-copy)

• RemoteRemote URL-style AddressingURL-style Addressing GridFTPGridFTP

February 3, 2003Michael Ernst, Fermilab 10

The dCap LibraryThe dCap Library

Posix open/read/write/close dCache System

Door Node

Mover Node

Door Node

NFS Nodepnfsnative FS

libc.o

libdcache.so

Application

Data

Namespace Operations

February 3, 2003Michael Ernst, Fermilab 11

Current StatusCurrent StatusAccess to Mass Storage Systems (dCache/Enstore) at FNALAccess to Mass Storage Systems (dCache/Enstore) at FNAL• LocalLocal

dccp / encp (req. mounted pnfs fs)dccp / encp (req. mounted pnfs fs) URL-style AddressingURL-style Addressing GridFTP (Server embedded in dCache, client with globus-url-copy)GridFTP (Server embedded in dCache, client with globus-url-copy)

• RemoteRemote URL-style AddressingURL-style Addressing GridFTPGridFTP

in Service today atin Service today at– CERN (lxcmsa)CERN (lxcmsa)– UCSD (cms-dcache-serv, t2cms0)UCSD (cms-dcache-serv, t2cms0)– Using kerberized certificates when communicating w/dCache @ FNALUsing kerberized certificates when communicating w/dCache @ FNAL

» requires installation/configuration ofrequires installation/configuration of» NMI kx509/KCA bins,libsNMI kx509/KCA bins,libs» Kerberos bins, libs, krb5.conf Kerberos bins, libs, krb5.conf

• Mass Storage elsewhere Mass Storage elsewhere dCache up&running at UCSD dCache up&running at UCSD dCache @ CERN w/interface to CASTOR will be nextdCache @ CERN w/interface to CASTOR will be next

• Still MissingStill Missing Global Dataset Catalog, Replication ManagementGlobal Dataset Catalog, Replication Management

February 3, 2003Michael Ernst, Fermilab 12

Storage ManagementStorage Management• Evaluating Catalogs and Storage ManagersEvaluating Catalogs and Storage Managers

There is no LHC Data/Storage Management System, nor a Global Dataset Catalog as of yetThere is no LHC Data/Storage Management System, nor a Global Dataset Catalog as of yet US CMS is looking into SRB (developed by SDSC)US CMS is looking into SRB (developed by SDSC)

SRB servers are now running on US CMS servers at UCSD, Caltech, Fermilab, and at CERN SRB servers are now running on US CMS servers at UCSD, Caltech, Fermilab, and at CERN soonsoon

Distributed Storage Resources(database systems, archival storage systems, file systems, (Grid)FTP, http, …)

Application

SRB Server

MCAT

HRM DB2, Oracle, Illustra, ObjectStore HPSS, ADSM, dCache

• The Storage Resource Broker is a middleware• SRB is a Distributed Filesystem• It virtualises resource access• It mediates access to distributed heterogeneous resources• It uses a MetaCATalog to facilitate the brokering• It integrates data and metadata

February 3, 2003Michael Ernst, Fermilab 13

Global Data Management w/SRBGlobal Data Management w/SRB

…GridFTP

LocalMCAT

dCache

GridFTP

LocalMCAT

dCache

GridFTP

LocalMCAT

dCache

Global Catalog

Site A Site B Site N