The Next Generation Root File Server
description
Transcript of The Next Generation Root File Server
![Page 1: The Next Generation Root File Server](https://reader035.fdocuments.us/reader035/viewer/2022062520/56815932550346895dc66ca7/html5/thumbnails/1.jpg)
The Next Generation Root File Server
Andrew HanushevskyStanford Linear Accelerator Center
27-September-2004
http://xrootd.slac.stanford.edu
![Page 2: The Next Generation Root File Server](https://reader035.fdocuments.us/reader035/viewer/2022062520/56815932550346895dc66ca7/html5/thumbnails/2.jpg)
27-Sep-04 2: xrootd
What is xrootd?
File Server Provides high performance file-based access Scalable, extensible, naively usable Fault tolerant
Server failures handled in a natural way Servers may be dynamically added and removed
Secure Framework allows use of almost any protocol
Rootd Compatible
![Page 3: The Next Generation Root File Server](https://reader035.fdocuments.us/reader035/viewer/2022062520/56815932550346895dc66ca7/html5/thumbnails/3.jpg)
27-Sep-04 3: xrootd
Goals II
Simplicity Can run xrootd out of the box
No config file needed for non-complicated/small installations
Generality Can configure xrootd for ultimate performance
Meant for intermediate to large-scale sites
![Page 4: The Next Generation Root File Server](https://reader035.fdocuments.us/reader035/viewer/2022062520/56815932550346895dc66ca7/html5/thumbnails/4.jpg)
27-Sep-04 4: xrootd
How Is high performance achieved?
Rich but efficient server protocol Combines file serving with P2P elements Allows client hints for improved performance
Pre-read, prepare, client access & processing hints, Multiplexed request stream
Multiple parallel requests allowed per client
An extensible base architecture Heavily multi-threaded
Clients are dedicated threads whenever possible Extensive use of OS I/O features
Async I/O, device polling, etc. Load adaptive reconfiguration.
![Page 5: The Next Generation Root File Server](https://reader035.fdocuments.us/reader035/viewer/2022062520/56815932550346895dc66ca7/html5/thumbnails/5.jpg)
27-Sep-04 5: xrootd
xrootd Server Architecture
Protocol LayerProtocol Layer
Filesystem Logical LayerFilesystem Logical Layer
Filesystem Physical LayerFilesystem Physical Layer
Filesystem ImplementationFilesystem Implementation
Protocol & Thread ManagerProtocol & Thread Manager
(included in distribution)
p2p heart
![Page 6: The Next Generation Root File Server](https://reader035.fdocuments.us/reader035/viewer/2022062520/56815932550346895dc66ca7/html5/thumbnails/6.jpg)
27-Sep-04 6: xrootd
Rootd Bilateral Compatibility
xrootdxrootd
rootdrootd
xrootdxrootdrootd compabilityrootd compabilityTNetFile
Application
TNetFile
TXNetFile
Application
rootdrootd
Client-Side Compatibility
Server-Side Compatibility
clientclient
![Page 7: The Next Generation Root File Server](https://reader035.fdocuments.us/reader035/viewer/2022062520/56815932550346895dc66ca7/html5/thumbnails/7.jpg)
27-Sep-04 7: xrootd
How performant is it?
Can deliver data at disk speeds (streaming mode) Assuming good network & proper TCP buffer size
Low CPU overhead 75% less CPU than NFS for same data load
It is memory hungry, however.
General requirements Middling speed machine
The more CPU’s the better
1-2GB of RAM
![Page 8: The Next Generation Root File Server](https://reader035.fdocuments.us/reader035/viewer/2022062520/56815932550346895dc66ca7/html5/thumbnails/8.jpg)
27-Sep-04 8: xrootd
How is scalability achieved?
Protocol allows server scalability Server directed I/O segmenting Request deferral to pace client load Unsolicited responses for ad hoc client steering P2P elements for lashing servers together
Request redirection key element Integrated with a P2P control network
olbd servers provide control information
![Page 9: The Next Generation Root File Server](https://reader035.fdocuments.us/reader035/viewer/2022062520/56815932550346895dc66ca7/html5/thumbnails/9.jpg)
27-Sep-04 9: xrootd
How does it scale?
xrootd scales in multiple dimensions Can run multiple load balanced xrootd’s
Provides single uniform name and data space Scales from 1 to over 32,000 cooperating data servers Architected as self-configuring structured peer-to-peer
(SP2) data servers Servers can be added & removed at any time
Client (TXNetFile) understands SP2 configurations xrootd informs client when running in this mode Client has more recovery options in the event of failure
![Page 10: The Next Generation Root File Server](https://reader035.fdocuments.us/reader035/viewer/2022062520/56815932550346895dc66ca7/html5/thumbnails/10.jpg)
27-Sep-04 10: xrootd
Load Balancing Implementation
Control Interface (olbd) Load balancing meta operations
Find files, change status, forwarded requests
Data Interface (xrootd) Data is provided to clients Interfaces to olbd via the ofs layer
Separation is important Allows use of any protocol Client need not know the control protocol
![Page 11: The Next Generation Root File Server](https://reader035.fdocuments.us/reader035/viewer/2022062520/56815932550346895dc66ca7/html5/thumbnails/11.jpg)
27-Sep-04 11: xrootd
Entities & Relationships
datadataxrootd
olbdxrootd
olbd
Data Clients
Redirectors
Data Servers
MM
SS
ctlctl
olbdolbd Control Network
Managers & Servers(resource info, file location)
xrootdxrootd Data Network
(redirectors steer clients to dataData servers provide data)
![Page 12: The Next Generation Root File Server](https://reader035.fdocuments.us/reader035/viewer/2022062520/56815932550346895dc66ca7/html5/thumbnails/12.jpg)
27-Sep-04 12: xrootd
Typical SP2 Configuration
Dynamic
Selection
redirector
subscribe
subscribe
subscribe
subscribe
![Page 13: The Next Generation Root File Server](https://reader035.fdocuments.us/reader035/viewer/2022062520/56815932550346895dc66ca7/html5/thumbnails/13.jpg)
27-Sep-04 13: xrootd
Example: SLAC Configuration
client machinesclient machines
kan01 kan02 kan03 kan04 kanxx
bbr-olb03 bbr-olb04 kanolb-a
![Page 14: The Next Generation Root File Server](https://reader035.fdocuments.us/reader035/viewer/2022062520/56815932550346895dc66ca7/html5/thumbnails/14.jpg)
27-Sep-04 14: xrootd
Why do this?
Can transparently & incrementally scale
Servers can come and go Load balancing effects recovery
New servers can be added at any time Servers may be brought down for maintenance Files can be moved around in real-time
Client simply adjust to the new configuration TXNetFile object handles recovery protocol
![Page 15: The Next Generation Root File Server](https://reader035.fdocuments.us/reader035/viewer/2022062520/56815932550346895dc66ca7/html5/thumbnails/15.jpg)
27-Sep-04 15: xrootd
What we have seen
For a single server: 1,000 simultaneous clients 2,200 simultaneous open files
Bottlenecks Disk I/O (memory next behind)
![Page 16: The Next Generation Root File Server](https://reader035.fdocuments.us/reader035/viewer/2022062520/56815932550346895dc66ca7/html5/thumbnails/16.jpg)
27-Sep-04 16: xrootd
What Have We Heard
The system is too stable Users run extra-long jobs (1-2 weeks) now
Error not discovered until weeks later
The system is too aggressive New servers are immediately taken over
Easy configuration but startling for administrators
![Page 17: The Next Generation Root File Server](https://reader035.fdocuments.us/reader035/viewer/2022062520/56815932550346895dc66ca7/html5/thumbnails/17.jpg)
27-Sep-04 17: xrootd
Next: Getting remote data
SLAC
IN2P3RAL
xroo
td’s
xroo
td’sRALRAL proxyproxy
IN2P3IN2P3 proxyproxy
Firewalls requireProxy servers
![Page 18: The Next Generation Root File Server](https://reader035.fdocuments.us/reader035/viewer/2022062520/56815932550346895dc66ca7/html5/thumbnails/18.jpg)
27-Sep-04 18: xrootd
Proxy Service
Attempts to address competing goals Security
Deal with firewalls Scalability
Administrative Configuration
Performance Ad hoc forwarding for near-zero wait time Intelligent caching in local domain
![Page 19: The Next Generation Root File Server](https://reader035.fdocuments.us/reader035/viewer/2022062520/56815932550346895dc66ca7/html5/thumbnails/19.jpg)
27-Sep-04 19: xrootd
Proxy Implementation
Uses capabilities of olbd and xrootd Simply an extension of local load balancing Implemented as a special file system type
Interfaces in the ofs layer Functions in the oss layer
Primary developer is Heinz Stockinger
![Page 20: The Next Generation Root File Server](https://reader035.fdocuments.us/reader035/viewer/2022062520/56815932550346895dc66ca7/html5/thumbnails/20.jpg)
27-Sep-04 20: xrootd
Proxy Interactions
client machinesclient machines
red01 data02 data03 proxy01
local olb
data01 data02 data03 data04
proxy olb
proxy olb
local olbRAL
SLAC
11
22
33
55
44
![Page 21: The Next Generation Root File Server](https://reader035.fdocuments.us/reader035/viewer/2022062520/56815932550346895dc66ca7/html5/thumbnails/21.jpg)
27-Sep-04 21: xrootd
Why This Arrangement?
Minimizes cross-domain knowledge Necessary for scalability in all areas
Security Configuration Fault tolerance & recovery
![Page 22: The Next Generation Root File Server](https://reader035.fdocuments.us/reader035/viewer/2022062520/56815932550346895dc66ca7/html5/thumbnails/22.jpg)
27-Sep-04 22: xrootd
Scalable Proxy Security
SLAC PROXY OLBD RAL PROXY OLBD
3
2 2
1
11 Authenticate & develop session key22 Distribute session key to authenticated subscribers
33 Data servers can log into each other using session key
Data ServersData Servers
![Page 23: The Next Generation Root File Server](https://reader035.fdocuments.us/reader035/viewer/2022062520/56815932550346895dc66ca7/html5/thumbnails/23.jpg)
27-Sep-04 23: xrootd
Proxy Performance
Introduces minimal latency overhead Virtually undetectably from US/Europe Negligible on faster links
2% slower on fast US/US links 10% slower on LAN
Can be further improved Parallel streams Better window size calculation Asynchronous I/O
![Page 24: The Next Generation Root File Server](https://reader035.fdocuments.us/reader035/viewer/2022062520/56815932550346895dc66ca7/html5/thumbnails/24.jpg)
27-Sep-04 24: xrootd
Conclusion
xrootd provides high performance file access Unique performance, usability, scalability,
security, compatibility, and recoverability characteristics
Should scale to tens of thousand clients Can support tens of thousand of servers
Distributed as part of the CERN root package Open software, supported by
SLAC (server) and INFN-Padova (client)