Post on 28-Dec-2015
SRM at Clemson
Michael Fenn
What is a Storage Element?
Provides grid-accessible storage space.
Is accessible to applications running on OSG through either GridFTP and/or SRM interface.
Has GIP setup and configured properly which publishes its information.
Has well-defined policy for cleanup and usage.
Is registered with OSG.
Architecture
Firebox: Dell PowerEdge: Head node and VPN server
Oiltank1-9: Dell PowerEdge: storage node
Birdnest: Xen VM, hosting CE and SE, VPN client
Why bother with VPN?
We have a unique situation where the grid head node is not co-located with the rest of the cluster
Distributed Filesystem
After researching multiple distributed filesystems, we settled on PVFS.
Simple configuration and easy to add extra clients.
Server runs in userspace and client requires only a kernel module.
PVFS Server Setup
./configuremake make installPVFS then provides a command to help create
your server config file/usr/bin/pvfs2-genconfig /etc/pvfs2-fs.conf
Script will prompt for your desired protocol, servers, and other configuration options.
PVFS Server Setup
This same config file can be distributed to each of the desired storage nodes at /etc/pvfs-fs.conf
To initialize the server and allow it to allocate space, run
/usr/sbin/pvfs2-server /etc/pvfs2-fs.conf -f
To start the server normally, simply omit '-f'
PVFS Client Setup
./configure –with-kernel-source=/path/to/kernelsrc
make just_kmod
insmod /usr/src/pvfs2/src/kernel/linux-2.6/pvfs2.ko
mkdir /mnt/pvfs2/
Will need to make file /etc/pvfs2tab with form:
tcp://testhost:3334/pvfs2-fs /mnt/pvfs2 pvfs2 defaults,noauto 0 0
pvfs2-client -p ./pvfs2-client-core
mount -t pvfs2 tcp://testhost:3334/pvfs2-fs /mnt/pvfs2
Grid Interfaces
GridFTP-based SE is simply a file system directory accessible via GSI-authenticated FTP.
Provides no space management functions
Limited permission management functions.
SRM/dCache - This is a full implementation of a Storage Element
Enforces stronger constraints
Can manage a space spanning more volumes.
Grid Interfaces -- BeStMan
SRM/BeStMan - BeStMan is a full implementation of SRM v2.2, developed by Lawrence Berkeley National Laboratory, for a small disk based storage and mass storage systems.
Works on top of existing disk-based UNIX file system. Works with any existing file transfer service, such as
gsiftp, http, https, bbftp and ftp. Requires minimal administrative efforts on the
deployment and updates.Source:
https://twiki.grid.iu.edu/bin/view/Documentation/AboutStorageElements
SRM/BeStMan at Clemson
We had an existing filesystem, so BeStMan best fit our situation.
Available for easy installation from the VDT
pacman -get ITB:Bestman
Configure BeStMan depending on your site setup and needs.
Our config is on the next slide
SRM/BeStMan at Clemson $ ./configure
--with-java-home=/opt/osg-1.0/jdk1.6 --with-srm-home=/opt/osg-1.0/bestman --with-srm-owner=daemon --enable-sudofsmng --with-cacert-path=/opt/osg-1.0/globus/TRUSTED_CA --with-certfile-path=/etc/grid-security/http/httpcert.pem --with-keyfile-path=/etc/grid-security/http/httpkey.pem --with-eventlog-path=/opt/osg-1.0/vdt-app-data/bestman/logs --with-cachelog-path=/opt/osg-1.0/vdt-app-data/bestman/logs --with-http-port=10080 --with-https-port=10443 --with-replica-storage-path=/mnt/pvfs2/sereplica --with-replica-storage-size=1000000 --enable-gums --with-gums-url=https://osggums.cs.clemson.edu:8443/gums/services/GUMSAuthorizationServicePort --with-gums-dn=/DC=org/DC=doegrids/OU=Services/CN=http/birdnest.cs.clemson.edu
Red options required for GUMS authentication Explanations of the options are available in the Administration guide. Source: https://twiki.grid.iu.edu/bin/view/ReleaseDocumentation/Bestman
Testing the SRM Interface
• SRM commands used to test our install• First you should create a proxy from a submit
host on which you are registered so that you can run commands on the site. – voms-proxy-init -voms Engage -
valid 72:00
• This will create a proxy cert for the Virtual Organization Engage which will be valid for 72 hours.
• https://twiki.grid.iu.edu/bin/view/Integration/ITB092/Bestman
Testing the SRM Interface
• Pinging the server:– srm-ping srm://birdnest.cs.clemson.edu:10443/srm/v2/server
• Using ls:– srm-ls srm://birdnest.cs.clemson.edu:10443/srm/v2/server\?SFN=/mnt/pvfs2/sedata/engage/test.txt
• SFN is the path to the file or directory which you have interest in.
• Remember to escape the ? in Bash!
Testing the SRM Interface
• Make a directory:– srm-mkdir srm://birdnest.cs.clemson.edu:10443/srm/v2/server\?SFN=/mnt/pvfs2/sedata/engage/testdir/
• Delete a directory:– srm-rmdir srm://birdnest.cs.clemson.edu:10443/srm/v2/server\?SFN=/mnt/pvfs2/sedata/engage/testdir/
Testing the SRM Interface• SRM commands expect full URLs, including
filenames• Will not infer that the source and destination filenames
are the same• Transfering files to the storage node– srm-copy file:///home/user/test.txt srm://birdnest.cs.clemson.edu:10443/srm/v2/server\?SFN=/mnt/pvfs2/sedata/engage/test.txt
• Transfering file from the storage node– srm-copy srm://birdnest.cs.clemson.edu:10443/srm/v2/server\?SFN=/mnt/pvfs2/sedata/engage/test.txt file:///home/user/test.txt
Registering for SRM Daily Tests• Go to http://datagrid.lbl.gov/• Select storage site registration and fill out the form. • You will need to know the service endpoint on your
server. – Ex: srm://birdnest.cs.clemson.edu:10443/srm/v2/server/
• You will also need to know the path which will be write accessible to those using your SE.– Ex: /mnt/pvfs2/sedata/
• Within a few days you will be contacted through email for verification and to make sure your site is setup correctly.
• If the test run is successful, you will be registered for daily storage test reporting.
• http://datagrid.lbl.gov/v22/srmtester-ftest.v22.php?sitename=CIRG-CU-SRM&date=03-18-2009_09_20&vo=OSG