European Middleware Initiative (EMI)gridka-school.scc.kit.edu/2011/downloads/Data... · E M I IN F...
Transcript of European Middleware Initiative (EMI)gridka-school.scc.kit.edu/2011/downloads/Data... · E M I IN F...
![Page 1: European Middleware Initiative (EMI)gridka-school.scc.kit.edu/2011/downloads/Data... · E M I IN F S O-R I-2 6 1 6 1 1 2011-09-05 GridKa 2011: Data Storage 9 (Local) File systems](https://reader035.fdocuments.us/reader035/viewer/2022071210/6021f652d35d0942de7df873/html5/thumbnails/1.jpg)
Data Storage
Paul MillardCache
![Page 2: European Middleware Initiative (EMI)gridka-school.scc.kit.edu/2011/downloads/Data... · E M I IN F S O-R I-2 6 1 6 1 1 2011-09-05 GridKa 2011: Data Storage 9 (Local) File systems](https://reader035.fdocuments.us/reader035/viewer/2022071210/6021f652d35d0942de7df873/html5/thumbnails/2.jpg)
EM
I INF
SO
-RI-
26
16
11
2011-09-05 GridKa 2011: Data Storage 2
Overview
• Introducing storage• How storage is used• Challenges and future directions
![Page 3: European Middleware Initiative (EMI)gridka-school.scc.kit.edu/2011/downloads/Data... · E M I IN F S O-R I-2 6 1 6 1 1 2011-09-05 GridKa 2011: Data Storage 9 (Local) File systems](https://reader035.fdocuments.us/reader035/viewer/2022071210/6021f652d35d0942de7df873/html5/thumbnails/3.jpg)
EM
I INF
SO
-RI-
26
16
11
2011-09-05 GridKa 2011: Data Storage 3
(Magnetic) Hard Disks
![Page 4: European Middleware Initiative (EMI)gridka-school.scc.kit.edu/2011/downloads/Data... · E M I IN F S O-R I-2 6 1 6 1 1 2011-09-05 GridKa 2011: Data Storage 9 (Local) File systems](https://reader035.fdocuments.us/reader035/viewer/2022071210/6021f652d35d0942de7df873/html5/thumbnails/4.jpg)
EM
I INF
SO
-RI-
26
16
11
2011-09-05 GridKa 2011: Data Storage 4
Tape systems
![Page 5: European Middleware Initiative (EMI)gridka-school.scc.kit.edu/2011/downloads/Data... · E M I IN F S O-R I-2 6 1 6 1 1 2011-09-05 GridKa 2011: Data Storage 9 (Local) File systems](https://reader035.fdocuments.us/reader035/viewer/2022071210/6021f652d35d0942de7df873/html5/thumbnails/5.jpg)
EM
I INF
SO
-RI-
26
16
11
2011-09-05 GridKa 2011: Data Storage 5
Disk enclosures
![Page 6: European Middleware Initiative (EMI)gridka-school.scc.kit.edu/2011/downloads/Data... · E M I IN F S O-R I-2 6 1 6 1 1 2011-09-05 GridKa 2011: Data Storage 9 (Local) File systems](https://reader035.fdocuments.us/reader035/viewer/2022071210/6021f652d35d0942de7df873/html5/thumbnails/6.jpg)
EM
I INF
SO
-RI-
26
16
11
2011-09-05 GridKa 2011: Data Storage 6
RAID systems
![Page 7: European Middleware Initiative (EMI)gridka-school.scc.kit.edu/2011/downloads/Data... · E M I IN F S O-R I-2 6 1 6 1 1 2011-09-05 GridKa 2011: Data Storage 9 (Local) File systems](https://reader035.fdocuments.us/reader035/viewer/2022071210/6021f652d35d0942de7df873/html5/thumbnails/7.jpg)
EM
I INF
SO
-RI-
26
16
11
2011-09-05 GridKa 2011: Data Storage 7
Types of RAID
![Page 8: European Middleware Initiative (EMI)gridka-school.scc.kit.edu/2011/downloads/Data... · E M I IN F S O-R I-2 6 1 6 1 1 2011-09-05 GridKa 2011: Data Storage 9 (Local) File systems](https://reader035.fdocuments.us/reader035/viewer/2022071210/6021f652d35d0942de7df873/html5/thumbnails/8.jpg)
EM
I INF
SO
-RI-
26
16
11
2011-09-05 GridKa 2011: Data Storage 8
(Local) File systems
Ext3, Ext4, XFS, ...
![Page 9: European Middleware Initiative (EMI)gridka-school.scc.kit.edu/2011/downloads/Data... · E M I IN F S O-R I-2 6 1 6 1 1 2011-09-05 GridKa 2011: Data Storage 9 (Local) File systems](https://reader035.fdocuments.us/reader035/viewer/2022071210/6021f652d35d0942de7df873/html5/thumbnails/9.jpg)
EM
I INF
SO
-RI-
26
16
11
2011-09-05 GridKa 2011: Data Storage 9
(Local) File systems
ZFS, BtrFS
![Page 10: European Middleware Initiative (EMI)gridka-school.scc.kit.edu/2011/downloads/Data... · E M I IN F S O-R I-2 6 1 6 1 1 2011-09-05 GridKa 2011: Data Storage 9 (Local) File systems](https://reader035.fdocuments.us/reader035/viewer/2022071210/6021f652d35d0942de7df873/html5/thumbnails/10.jpg)
EM
I INF
SO
-RI-
26
16
11
2011-09-05 GridKa 2011: Data Storage 10
Cluster filesystems
![Page 11: European Middleware Initiative (EMI)gridka-school.scc.kit.edu/2011/downloads/Data... · E M I IN F S O-R I-2 6 1 6 1 1 2011-09-05 GridKa 2011: Data Storage 9 (Local) File systems](https://reader035.fdocuments.us/reader035/viewer/2022071210/6021f652d35d0942de7df873/html5/thumbnails/11.jpg)
EM
I INF
SO
-RI-
26
16
11
2011-09-05 GridKa 2011: Data Storage 11
Storage Element
![Page 12: European Middleware Initiative (EMI)gridka-school.scc.kit.edu/2011/downloads/Data... · E M I IN F S O-R I-2 6 1 6 1 1 2011-09-05 GridKa 2011: Data Storage 9 (Local) File systems](https://reader035.fdocuments.us/reader035/viewer/2022071210/6021f652d35d0942de7df873/html5/thumbnails/12.jpg)
EM
I INF
SO
-RI-
26
16
11
2011-09-05 GridKa 2011: Data Storage 12
Storage Element
![Page 13: European Middleware Initiative (EMI)gridka-school.scc.kit.edu/2011/downloads/Data... · E M I IN F S O-R I-2 6 1 6 1 1 2011-09-05 GridKa 2011: Data Storage 9 (Local) File systems](https://reader035.fdocuments.us/reader035/viewer/2022071210/6021f652d35d0942de7df873/html5/thumbnails/13.jpg)
EM
I INF
SO
-RI-
26
16
11
2011-09-05 GridKa 2011: Data Storage 13
Example of redirection
![Page 14: European Middleware Initiative (EMI)gridka-school.scc.kit.edu/2011/downloads/Data... · E M I IN F S O-R I-2 6 1 6 1 1 2011-09-05 GridKa 2011: Data Storage 9 (Local) File systems](https://reader035.fdocuments.us/reader035/viewer/2022071210/6021f652d35d0942de7df873/html5/thumbnails/14.jpg)
EM
I INF
SO
-RI-
26
16
11
2011-09-05 GridKa 2011: Data Storage 14
Protocols
• Transferring data– Redirecting the client is important!– LAN access (for worker nodes):
• NFS v4.1, dcap, rfio, xrootd, (HTTP?)
– WAN access (for transferring data)• GridFTP, HTTP, WebDAV, (xrootd?)
• Management– SRM v2.2
• Standardisation:– GSI vs SSL/TLS
![Page 15: European Middleware Initiative (EMI)gridka-school.scc.kit.edu/2011/downloads/Data... · E M I IN F S O-R I-2 6 1 6 1 1 2011-09-05 GridKa 2011: Data Storage 9 (Local) File systems](https://reader035.fdocuments.us/reader035/viewer/2022071210/6021f652d35d0942de7df873/html5/thumbnails/15.jpg)
EM
I INF
SO
-RI-
26
16
11
2011-09-05 GridKa 2011: Data Storage 15
Grid storage
• Lots of sites (so, lots of SEs)• Data appears in multiple locations• Current Grid-level services:
– FTS: moving data– File Catalogues: finding the files
• Experiment provides:– File grouping (data sets)– Access framework (software)
• Unfortunately it adds layer of indirection between end-users and sites(!!)
![Page 16: European Middleware Initiative (EMI)gridka-school.scc.kit.edu/2011/downloads/Data... · E M I IN F S O-R I-2 6 1 6 1 1 2011-09-05 GridKa 2011: Data Storage 9 (Local) File systems](https://reader035.fdocuments.us/reader035/viewer/2022071210/6021f652d35d0942de7df873/html5/thumbnails/16.jpg)
EM
I INF
SO
-RI-
26
16
11
2011-09-05 GridKa 2011: Data Storage 16
Grid Storage: catalogues
Diagram from P. Fuhrmann
![Page 17: European Middleware Initiative (EMI)gridka-school.scc.kit.edu/2011/downloads/Data... · E M I IN F S O-R I-2 6 1 6 1 1 2011-09-05 GridKa 2011: Data Storage 9 (Local) File systems](https://reader035.fdocuments.us/reader035/viewer/2022071210/6021f652d35d0942de7df873/html5/thumbnails/17.jpg)
EM
I INF
SO
-RI-
26
16
11
2011-09-05 GridKa 2011: Data Storage 17
Grid problems
• Communication:– VOs have many storage provides– Sites (typically) have many VOs
• VOs have many users
• Diagnosing problems is hard– A networking problem could involve:
• end-user and VO,• src and dest storage elements (the sites),• FTS, catalogue(s), network providers, ..
• Use of non-standards doesn't help!
![Page 18: European Middleware Initiative (EMI)gridka-school.scc.kit.edu/2011/downloads/Data... · E M I IN F S O-R I-2 6 1 6 1 1 2011-09-05 GridKa 2011: Data Storage 9 (Local) File systems](https://reader035.fdocuments.us/reader035/viewer/2022071210/6021f652d35d0942de7df873/html5/thumbnails/18.jpg)
EM
I INF
SO
-RI-
26
16
11
2011-09-05 GridKa 2011: Data Storage 18
Monte Carlo
Diagram from Dr. G. Stewart
![Page 19: European Middleware Initiative (EMI)gridka-school.scc.kit.edu/2011/downloads/Data... · E M I IN F S O-R I-2 6 1 6 1 1 2011-09-05 GridKa 2011: Data Storage 9 (Local) File systems](https://reader035.fdocuments.us/reader035/viewer/2022071210/6021f652d35d0942de7df873/html5/thumbnails/19.jpg)
EM
I INF
SO
-RI-
26
16
11
2011-09-05 GridKa 2011: Data Storage 19
Data taking
Diagram from Dr. G. Stewart
![Page 20: European Middleware Initiative (EMI)gridka-school.scc.kit.edu/2011/downloads/Data... · E M I IN F S O-R I-2 6 1 6 1 1 2011-09-05 GridKa 2011: Data Storage 9 (Local) File systems](https://reader035.fdocuments.us/reader035/viewer/2022071210/6021f652d35d0942de7df873/html5/thumbnails/20.jpg)
EM
I INF
SO
-RI-
26
16
11
2011-09-05 GridKa 2011: Data Storage 20
Reconstruction
Diagram from Dr. G. Stewart
![Page 21: European Middleware Initiative (EMI)gridka-school.scc.kit.edu/2011/downloads/Data... · E M I IN F S O-R I-2 6 1 6 1 1 2011-09-05 GridKa 2011: Data Storage 9 (Local) File systems](https://reader035.fdocuments.us/reader035/viewer/2022071210/6021f652d35d0942de7df873/html5/thumbnails/21.jpg)
EM
I INF
SO
-RI-
26
16
11
2011-09-05 GridKa 2011: Data Storage 21
Chaotic analysis
Diagram from P. Fuhrmann
![Page 22: European Middleware Initiative (EMI)gridka-school.scc.kit.edu/2011/downloads/Data... · E M I IN F S O-R I-2 6 1 6 1 1 2011-09-05 GridKa 2011: Data Storage 9 (Local) File systems](https://reader035.fdocuments.us/reader035/viewer/2022071210/6021f652d35d0942de7df873/html5/thumbnails/22.jpg)
EM
I INF
SO
-RI-
26
16
11
2011-09-05 GridKa 2011: Data Storage 22
Grid storage in context
![Page 23: European Middleware Initiative (EMI)gridka-school.scc.kit.edu/2011/downloads/Data... · E M I IN F S O-R I-2 6 1 6 1 1 2011-09-05 GridKa 2011: Data Storage 9 (Local) File systems](https://reader035.fdocuments.us/reader035/viewer/2022071210/6021f652d35d0942de7df873/html5/thumbnails/23.jpg)
EM
I INF
SO
-RI-
26
16
11
2011-09-05 GridKa 2011: Data Storage 23
WLCG site storage capacity
CERN BNL FNAL IN2P3 KIT CNAF PIC0
2
4
6
8
10
12
14
16
Total
Used
Pe
tab
yte
s
![Page 24: European Middleware Initiative (EMI)gridka-school.scc.kit.edu/2011/downloads/Data... · E M I IN F S O-R I-2 6 1 6 1 1 2011-09-05 GridKa 2011: Data Storage 9 (Local) File systems](https://reader035.fdocuments.us/reader035/viewer/2022071210/6021f652d35d0942de7df873/html5/thumbnails/24.jpg)
EM
I INF
SO
-RI-
26
16
11
2011-09-05 GridKa 2011: Data Storage 24
+ some non-WLCG sites
FacebookCERN
BNLFNAL
IN2P3KIT
CNAFWayBackMachine
PICWoW
0
5
10
15
20
25
Total
Used
Pe
tab
yte
s
![Page 25: European Middleware Initiative (EMI)gridka-school.scc.kit.edu/2011/downloads/Data... · E M I IN F S O-R I-2 6 1 6 1 1 2011-09-05 GridKa 2011: Data Storage 9 (Local) File systems](https://reader035.fdocuments.us/reader035/viewer/2022071210/6021f652d35d0942de7df873/html5/thumbnails/25.jpg)
EM
I INF
SO
-RI-
26
16
11
2011-09-05 GridKa 2011: Data Storage 25
Distributed storage
WLCG Amazon S3 Google/Gmail0
20
40
60
80
100
120
140
160
180
200
Total
Used
Pe
tab
yte
s
![Page 26: European Middleware Initiative (EMI)gridka-school.scc.kit.edu/2011/downloads/Data... · E M I IN F S O-R I-2 6 1 6 1 1 2011-09-05 GridKa 2011: Data Storage 9 (Local) File systems](https://reader035.fdocuments.us/reader035/viewer/2022071210/6021f652d35d0942de7df873/html5/thumbnails/26.jpg)
EM
I INF
SO
-RI-
26
16
11
2011-09-05 GridKa 2011: Data Storage 26
Sites + IBM Almaden
IBM AlmadenFacebook
CERNBNL
FNALIN2P3
KITCNAF
WayBackMachinePIC
WoW
0
20
40
60
80
100
120
Total
Used
Pe
tab
yte
s
![Page 27: European Middleware Initiative (EMI)gridka-school.scc.kit.edu/2011/downloads/Data... · E M I IN F S O-R I-2 6 1 6 1 1 2011-09-05 GridKa 2011: Data Storage 9 (Local) File systems](https://reader035.fdocuments.us/reader035/viewer/2022071210/6021f652d35d0942de7df873/html5/thumbnails/27.jpg)
EM
I INF
SO
-RI-
26
16
11
2011-09-05 GridKa 2011: Data Storage 27
Current challenges and Future directions
![Page 28: European Middleware Initiative (EMI)gridka-school.scc.kit.edu/2011/downloads/Data... · E M I IN F S O-R I-2 6 1 6 1 1 2011-09-05 GridKa 2011: Data Storage 9 (Local) File systems](https://reader035.fdocuments.us/reader035/viewer/2022071210/6021f652d35d0942de7df873/html5/thumbnails/28.jpg)
EM
I INF
SO
-RI-
26
16
11
2011-09-05 GridKa 2011: Data Storage 28
Dynamic data placement
• Example from ATLAS– Data was copied based on what people
thought would be useful– Turns out they didn't know!
• Lots of data copied but never read.
• Try replicating based on use:– Example policy:
When a T2 pulling in a file from T1, make two additional replicas elsewhere.
– So far, working pretty well.
![Page 29: European Middleware Initiative (EMI)gridka-school.scc.kit.edu/2011/downloads/Data... · E M I IN F S O-R I-2 6 1 6 1 1 2011-09-05 GridKa 2011: Data Storage 9 (Local) File systems](https://reader035.fdocuments.us/reader035/viewer/2022071210/6021f652d35d0942de7df873/html5/thumbnails/29.jpg)
EM
I INF
SO
-RI-
26
16
11
2011-09-05 GridKa 2011: Data Storage 29
Standardisation
• HEP storage requirements aren't that enormous any more.– Others are finding solutions, don't
reinvent the wheel!
• EMI: we're switching from Grid-specific protocols to standards– GSI to SSL/TLS,– GridFTP to HTTP/WebDAV,– LAN custom protocols to NFS v4.1
![Page 30: European Middleware Initiative (EMI)gridka-school.scc.kit.edu/2011/downloads/Data... · E M I IN F S O-R I-2 6 1 6 1 1 2011-09-05 GridKa 2011: Data Storage 9 (Local) File systems](https://reader035.fdocuments.us/reader035/viewer/2022071210/6021f652d35d0942de7df873/html5/thumbnails/30.jpg)
EM
I INF
SO
-RI-
26
16
11
2011-09-05 GridKa 2011: Data Storage 30
The death of Monach
• Monach is a rigid Tier structure.– T0, T1, T2.– Rational: network will be a bottleneck
• Reality:– Prolifically of classifications:
• Non-geo. T1, “Large” T2, T3, Exp. “Clouds”
– Backbone network isn't a bottleneck
• Gradual relaxing of rules• Eventually: any file from anywhere.
![Page 31: European Middleware Initiative (EMI)gridka-school.scc.kit.edu/2011/downloads/Data... · E M I IN F S O-R I-2 6 1 6 1 1 2011-09-05 GridKa 2011: Data Storage 9 (Local) File systems](https://reader035.fdocuments.us/reader035/viewer/2022071210/6021f652d35d0942de7df873/html5/thumbnails/31.jpg)
EM
I INF
SO
-RI-
26
16
11
2011-09-05 GridKa 2011: Data Storage 31
Global namespace
![Page 32: European Middleware Initiative (EMI)gridka-school.scc.kit.edu/2011/downloads/Data... · E M I IN F S O-R I-2 6 1 6 1 1 2011-09-05 GridKa 2011: Data Storage 9 (Local) File systems](https://reader035.fdocuments.us/reader035/viewer/2022071210/6021f652d35d0942de7df873/html5/thumbnails/32.jpg)
EM
I INF
SO
-RI-
26
16
11
2011-09-05 GridKa 2011: Data Storage 32
Future of tape
• Only really HEP that uses TAPE storage in-band.– elsewhere used for archiving data.
• Still need tape for archive, but..– Data processing move to (almost)
completely on disk– Fetching from tape will be like a copy– Tape will be “write once, read never”
![Page 33: European Middleware Initiative (EMI)gridka-school.scc.kit.edu/2011/downloads/Data... · E M I IN F S O-R I-2 6 1 6 1 1 2011-09-05 GridKa 2011: Data Storage 9 (Local) File systems](https://reader035.fdocuments.us/reader035/viewer/2022071210/6021f652d35d0942de7df873/html5/thumbnails/33.jpg)
EM
I INF
SO
-RI-
26
16
11
2011-09-05 GridKa 2011: Data Storage 33
Disks: where are SSDs?
• SSDs are FLASH memory in a block-device format– Much faster than Mag. Disks for
reading (writing is slower)– Predicted introduction in data centres
hasn't happened (yet)
• Why?– Errors are sudden, unpredictable.– They're still expensive– Software support isn't here (yet)
![Page 34: European Middleware Initiative (EMI)gridka-school.scc.kit.edu/2011/downloads/Data... · E M I IN F S O-R I-2 6 1 6 1 1 2011-09-05 GridKa 2011: Data Storage 9 (Local) File systems](https://reader035.fdocuments.us/reader035/viewer/2022071210/6021f652d35d0942de7df873/html5/thumbnails/34.jpg)
EM
I INF
SO
-RI-
26
16
11
2011-09-05 GridKa 2011: Data Storage 34
Satellites
• Structure storage– SSDs for random access (analysis)– Mag. disks for “archival storage”
• Support?– In filesystems: ZFS – In cluster filesystems: GPFS– In storage systems: EOS
• and dCache (soon)
![Page 35: European Middleware Initiative (EMI)gridka-school.scc.kit.edu/2011/downloads/Data... · E M I IN F S O-R I-2 6 1 6 1 1 2011-09-05 GridKa 2011: Data Storage 9 (Local) File systems](https://reader035.fdocuments.us/reader035/viewer/2022071210/6021f652d35d0942de7df873/html5/thumbnails/35.jpg)
EM
I INF
SO
-RI-
26
16
11
2011-09-05 GridKa 2011: Data Storage 35
Disks: new technologies
Diagram from “Storage Class Memory, Technology, and Uses” David A. Pease, IBM AlmadenResearch Centre.
![Page 36: European Middleware Initiative (EMI)gridka-school.scc.kit.edu/2011/downloads/Data... · E M I IN F S O-R I-2 6 1 6 1 1 2011-09-05 GridKa 2011: Data Storage 9 (Local) File systems](https://reader035.fdocuments.us/reader035/viewer/2022071210/6021f652d35d0942de7df873/html5/thumbnails/36.jpg)
EM
I INF
SO
-RI-
26
16
11
2011-09-05 GridKa 2011: Data Storage 36
Data integrity
• More data means more likely to see corruption
• Detecting corruption:– Disk (T10 DIF)– RAID systems (scrubbing)– Filesystems (ZFS, BtrFS)– Storage Element (file-level checksums
when uploading; scrubbing)– Tape: (proposed)
![Page 37: European Middleware Initiative (EMI)gridka-school.scc.kit.edu/2011/downloads/Data... · E M I IN F S O-R I-2 6 1 6 1 1 2011-09-05 GridKa 2011: Data Storage 9 (Local) File systems](https://reader035.fdocuments.us/reader035/viewer/2022071210/6021f652d35d0942de7df873/html5/thumbnails/37.jpg)
EM
I INF
SO
-RI-
26
16
11
2011-09-05 GridKa 2011: Data Storage 37
What is EMI?
• EMI is an EU-funded project to provide Grid software– Combines four technologies (ARC,
dCache, gLite, UNICORE)– Single responsibility allowing
• Mix-n-match usage.• Consolidation.
• First major release, EMI-1, is now available
![Page 38: European Middleware Initiative (EMI)gridka-school.scc.kit.edu/2011/downloads/Data... · E M I IN F S O-R I-2 6 1 6 1 1 2011-09-05 GridKa 2011: Data Storage 9 (Local) File systems](https://reader035.fdocuments.us/reader035/viewer/2022071210/6021f652d35d0942de7df873/html5/thumbnails/38.jpg)
EMI is partially funded by the European Commission under Grant Agreement RI-261611
2011-09-05 GridKa 2011: Data Storage 38
Thank you!
![Page 39: European Middleware Initiative (EMI)gridka-school.scc.kit.edu/2011/downloads/Data... · E M I IN F S O-R I-2 6 1 6 1 1 2011-09-05 GridKa 2011: Data Storage 9 (Local) File systems](https://reader035.fdocuments.us/reader035/viewer/2022071210/6021f652d35d0942de7df873/html5/thumbnails/39.jpg)
EM
I INF
SO
-RI-
26
16
11
2011-09-05 GridKa 2011: Data Storage 39
(Magnetic) Hard disks
• Block device (addressable units of fixed size)
• Characteristics• Streaming is fast• Random access is slow• The more concurrent activity, the poorer
the overall throughput
• Failure modes are well understood– J-curve bath-tub (wrong!)– See Google Con– SMART!
• Enterprise and consumer disks• Mostly, hardware is the same• The difference comes with the firmware
![Page 40: European Middleware Initiative (EMI)gridka-school.scc.kit.edu/2011/downloads/Data... · E M I IN F S O-R I-2 6 1 6 1 1 2011-09-05 GridKa 2011: Data Storage 9 (Local) File systems](https://reader035.fdocuments.us/reader035/viewer/2022071210/6021f652d35d0942de7df873/html5/thumbnails/40.jpg)
EM
I INF
SO
-RI-
26
16
11
2011-09-05 GridKa 2011: Data Storage 40
Storage: from small to big
• Disk• RAID• Filesystems• Tape• Cluster filesystems• HSM• Storage element• Grid
![Page 41: European Middleware Initiative (EMI)gridka-school.scc.kit.edu/2011/downloads/Data... · E M I IN F S O-R I-2 6 1 6 1 1 2011-09-05 GridKa 2011: Data Storage 9 (Local) File systems](https://reader035.fdocuments.us/reader035/viewer/2022071210/6021f652d35d0942de7df873/html5/thumbnails/41.jpg)
EM
I INF
SO
-RI-
26
16
11
2011-09-05 GridKa 2011: Data Storage 41
Tape / disk separation
• Motivations:– Avoid “accidental staging”
• Want clear separation (separately addressable) between disk and tape.– Store data is either disk or tape, never
both
• Part of a move away from including tape as part of normal data-flow.
![Page 42: European Middleware Initiative (EMI)gridka-school.scc.kit.edu/2011/downloads/Data... · E M I IN F S O-R I-2 6 1 6 1 1 2011-09-05 GridKa 2011: Data Storage 9 (Local) File systems](https://reader035.fdocuments.us/reader035/viewer/2022071210/6021f652d35d0942de7df873/html5/thumbnails/42.jpg)
EM
I INF
SO
-RI-
26
16
11
2011-09-05 GridKa 2011: Data Storage 42
HSM storage
• Files migrate to slower media• Based on policies or explicit
commands• Commercially available (TSM,
SAMFS/QFS, ..)•
![Page 43: European Middleware Initiative (EMI)gridka-school.scc.kit.edu/2011/downloads/Data... · E M I IN F S O-R I-2 6 1 6 1 1 2011-09-05 GridKa 2011: Data Storage 9 (Local) File systems](https://reader035.fdocuments.us/reader035/viewer/2022071210/6021f652d35d0942de7df873/html5/thumbnails/43.jpg)
EM
I INF
SO
-RI-
26
16
11
2011-09-05 GridKa 2011: Data Storage 43
Networking
• 10G is now cheap enough for ubiq. use– Sites are (or have) rolled out 10G– Needs CAT6a (or CAT6) cabling
• 40G and 100G MUX exist– too expensive for host connect– could be used for switch interconnect